AI Crawler robots.txt Policy Builder
Choose which AI crawlers to allow or block and generate a robots.txt file, with a plain-English explanation of what each rule does. Free, private and instant.
How to use the AI Crawler robots.txt Policy Builder
- Paste or type your input into the box above.
- Choose any options you need. Results update as you type.
- Copy the result with the Copy button.
About this tool
Pick a starting policy, add exceptions, and the builder writes a robots.txt covering the best-known AI and search crawlers, with a note on what each one does. It separates training crawlers, AI search crawlers and bots that fetch a page when a user asks. robots.txt is a request, not a lock: well-behaved crawlers follow it and others may not. Blocking Google-Extended does not affect Google Search, which is controlled by Googlebot.
AI Crawler robots.txt Policy Builder is part of ToolStack Hub's SEO & Marketing Tools. It runs entirely in your browser, which keeps your data private and makes results instant. There is nothing to install, no account to create and no usage limit.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT search?
No. OAI-SearchBot is a separate crawler for search results. The builder keeps them apart so you can allow search and block training.
Does Google-Extended affect my search ranking?
No. It is a control token for whether your content can be used by Google's Gemini models. Search is controlled by Googlebot.
Will every bot obey it?
Reputable operators say they do, but robots.txt is voluntary. It cannot stop a scraper that ignores it.
Is the AI Crawler robots.txt Policy Builder free?
Yes. AI Crawler robots.txt Policy Builder is completely free with no sign-up, limits or watermarks.
Is my data private when I use the AI Crawler robots.txt Policy Builder?
Yes. It runs in your browser, so your input is not uploaded to our servers.
Does it work on mobile?
Yes. ToolStack Hub tools work on phones, tablets and desktop browsers.
Keep going
Need this automated for your business?
Vantalume builds custom AI and web automation workflows so your team can stop doing repetitive work.