Home / Free tools / AI crawler rules and llms.txt
AI crawler rules and llms.txt generator
Decide which AI crawlers may read your site, then copy ready-to-use robots.txt rules and an llms.txt file.
- Runs in your browser
- Nothing is sent or stored
- Copy or download
Your robots.txt
If you already have a robots.txt, merge these rules into it. Do not replace working rules for other tools.
Step 2. Describe your site for llms.txt
Your llms.txt
What each file does
robots.txt
- Tells crawlers which parts of your site they may read
- Lets you treat AI search bots and training bots differently
- Is a request that well-behaved crawlers follow, not a lock
llms.txt
- A plain-text summary of your site and its key pages for AI tools
- A community proposal that not every AI system reads yet
- Easy to add, and harmless if a tool ignores it
How to use it
- 01
Choose your rules
Decide which AI crawlers may read your site and which paths stay private.
- 02
Copy or download
Copy the robots.txt and llms.txt text, or download both files.
- 03
Upload to your site root
The files must be reachable at yourdomain.com/robots.txt and yourdomain.com/llms.txt.
Common questions
What is llms.txt?
A plain-text file at the root of your site that describes the site and lists key pages for AI tools. It is a community proposal, and not every AI system reads it yet.
Does blocking an AI crawler remove my site from Google Search?
No. Blocking user agents such as GPTBot or ClaudeBot does not affect Google Search. Google-Extended is a separate control for Gemini use and is not the same as Googlebot.
Is robots.txt a security feature?
No. It is a request. Well-behaved crawlers follow it, but it does not stop bad actors. Keep private content behind a login or a firewall.
Should I allow AI search bots?
If you want your pages to appear in AI answers, allowing the search and user-request bots is the usual choice. Training bots are a business decision: allowing them can widen reach, while blocking them keeps your content out of training data.
Where do I upload these files?
Put robots.txt and llms.txt in the root of your domain. On WordPress, Rank Math can edit robots.txt, and llms.txt can be uploaded with a file manager.
More free tools
Want a hand with the results?
Tell us about your website and we will reply with a clear plan.