Przejdź do treści
Krok 4 Features Publiczna dokumentacja

robots.txt, llms.txt and AI crawlers

Block known search and AI bots in one click, serve an llms.txt guide for language models and opt out of AI training with the noai signal.

Treść tutorialu

Features

robots.txt

Acez SEO AI → robots.txt lists Known bots — including GPTBot, Google-Extended and ClaudeBot — that you can block with one click. Below the list you can add custom rules and see a preview; View the current live /robots.txt opens the result.

Blocking Google-Extended does not remove your site from Google Search. It only controls whether Google may use your content for its AI models.

If your site already has a physical robots.txt file, the plugin shows its content and offers Take over the content and disable the file: the file is renamed to a backup and its rules move into the editor.

llms.txt

Acez SEO AI → llms.txt serves an llms.txt file following the llmstxt.org standard — a concise guide to your most important content, written for language models. Turn on Serve the llms.txt file, then optionally set a custom title, description, introduction (Markdown) and footer. View the live /llms.txt opens the result.

Opting out of AI training

Send the noai/noimageai signal asks AI crawlers not to use your text and images for training. Not every crawler honours it; to keep a bot out entirely, block it in robots.txt.