How In-Browser Robots.txt Generation Works
Robots.txt Generator runs entirely in your browser with JavaScript. There is no sign-up, and whatever you type or open here is processed on your device instead of being sent to a server.
Generate RFC 9309 Robots.txt Rules in Your Browser
Define crawl rules and directory permissions for Googlebot, Bingbot, and other web crawlers directly in browser memory. Built around RFC 9309, it warns when one path is in both Allow and Disallow and leaves out Allow: / next to Disallow: /, but Disallow: / still blocks your whole site, so check the rules before you publish the file.
Robots.txt Examples: What to Enter and What You Get
Each row is a common robots.txt and what to type into the builder to get it. Leave the user-agent at * (every crawler) unless you want rules for one crawler only.
| Goal | Enter in the builder | What the file then says |
|---|---|---|
| Let every crawler in | Disallowed: (empty), Allowed: / | Every crawler may fetch every page |
| Keep a staging site out | Disallowed: /, Allowed: / | No crawler may fetch anything. Allow: / is left out, because next to Disallow: / it would undo the block |
| Block one folder | Disallowed: /admin/, Allowed: / | Everything may be fetched except URLs that start with /admin/ |
| Block every PDF | Disallowed: /*.pdf$, Allowed: / | No URL ending in .pdf may be fetched: * matches any characters and $ marks the end of the URL |
| Open one folder inside a blocked one | Disallowed: /private/, Allowed: /private/press/ | /private/press/ may be fetched but the rest of /private/ may not, because the longer rule wins |
| Keep AI training crawlers out | Tick 'Also block AI training crawlers' | A second group asks the seven AI crawlers listed below not to fetch anything; search engines keep following the rules above |
Robots.txt Directives at a Glance
Every line is a field name, a colon and a value. Field names are not case-sensitive, but the paths after them are, so /Admin/ and /admin/ are different folders.
| Line | What it does | Good to know |
|---|---|---|
| User-agent | Starts a group of rules and names the crawler it is for; * means every crawler | A crawler follows the group that names it, so a crawler with its own group ignores the * group |
| Disallow | Asks crawlers not to fetch URLs that start with the path | * matches any characters and $ marks the end of the URL |
| Allow | Lets crawlers fetch a path inside a disallowed one | The longest matching rule wins; when an Allow and a Disallow match equally, Allow wins (RFC 9309) |
| Sitemap | Gives the full address of your XML sitemap | Not part of RFC 9309, but Google and Bing read it; it applies to the whole file, not to one group |
| Crawl-delay | Asks for a pause between requests | Not part of RFC 9309 and ignored by Google; Bing supports it. This builder does not write it |
How to Create and Upload Your Robots.txt File
Download the file (it is already named robots.txt and saved as plain text) and upload it to the top folder of your website, often called public_html, so that it opens at https://yourdomain.com/robots.txt. Crawlers only look for it there; a robots.txt in a subfolder is ignored. Each host needs its own file, so a subdomain such as shop.yourdomain.com needs a robots.txt of its own. After uploading, open the address in a browser to check it, then look at the robots.txt report in Google Search Console, which shows the version Google fetched. Google usually refreshes its copy within a day.
Protecting Content from AI Scrapers
Tick 'Also block AI training crawlers' to add a second rule group that asks seven crawlers which collect data for AI models not to fetch anything. Search crawlers such as Googlebot and Bingbot keep following your main rules, so your search listings are not affected. robots.txt is a request: well-behaved crawlers follow it, but it cannot force a crawler out, and it does not remove pages that were collected before. For separate rules per crawler, edit the downloaded file.
| AI crawler | Run by | Blocking it stops |
|---|---|---|
| GPTBot | OpenAI | Collecting pages to train OpenAI's models |
| ClaudeBot | Anthropic | Collecting pages to train Anthropic's models |
| CCBot | Common Crawl | Adding pages to Common Crawl's public web archive, which many AI models are trained on |
| Google-Extended | Use of your pages to train and ground Gemini models; Google Search crawling and ranking are not affected | |
| Applebot-Extended | Apple | Use of your pages to train Apple's AI models; Applebot still crawls for Siri and Spotlight |
| Bytespider | ByteDance | Crawling by ByteDance's crawler, widely reported to gather training data for its AI models |
| Meta-ExternalAgent | Meta | Collecting pages to train Meta's AI models |
Automatic Sitemap and Tie-Breaker Validation
Add an absolute sitemap reference and get a warning when the same path is listed under both Allow and Disallow, where crawlers follow Allow. Download your finalized robots.txt file with a single click.
Why processing in your browser matters
Many online tools upload your files or text to a server to process them. This one does the work on your computer or phone, so your data is not sent to us or stored anywhere else. That lowers the risk when you work with confidential material, although you should still follow your organisation's own rules. Read how this works in our Privacy Policy.
- No uploads: files are opened and saved on your device, so there is no upload or download wait.
- No account: nothing to sign up for or install.
- Free to use: no subscriptions or paywalls.