Free robots.txt generator and tester

How to use the robots.txt generator and tester

Start on Generate. Pick the platform. Add the sitemap URL. Use the AI crawler presets if you already know whether you want training bots blocked, search bots allowed, or both blocked. Copy the file when the preview looks right. Read the install note for Shopify and Next.js before you paste.

Switch to Test when you already have a file. Fetch the live robots.txt from the domain, or paste the text. Enter a path and a user-agent. The verdict is the first answer. The highlighted line is the reason. Use “test all major bots” when the real question is about AI bots, not only Googlebot.

A red confirm appears if you try to disallow the whole site for every bot. That mistake takes a site out of search. Confirm only when you mean it.

Robots.txt syntax explained

Groups start with User-agent. Rules under that group are Allow or Disallow lines, plus optional Sitemap lines at the file level. Comments start with #. Paths are case sensitive. Patterns can use * as a wildcard and $ as an end anchor.

Google’s matching rules matter more than folklore. The most specific matching user-agent group wins. Groups with the same agent are combined. Inside a group, the longest matching path wins. If two rules tie on length, Allow wins. That is why a short Disallow: / and a longer Allow: /blog/page can still let the page through.

Crawl-delay is ignored by Google. A noindex line inside robots.txt is unsupported there. Use a meta robots tag or an HTTP header when you want a page kept out of the index. Robots.txt alone will not remove a URL that is already indexed.

How Google decides which rule wins

First Google picks the group. A Googlebot group beats a * group when both exist. Then Google scores the path rules by length after pattern matching. The longest match decides. Equal length prefers Allow. Empty Disallow lines do not block. A 4xx robots.txt is treated as allow all. A 5xx or unreachable file can be treated as a temporary block. The tester says which case it saw.

Percent-encoding is normalised before the match. That stops /a%7Eb and /a~b from disagreeing. If you paste a file with Windows line endings or a UTF-8 BOM, the parser still reads the rules, and the warnings list calls out messy syntax when it can.

Should you block AI bots?

Training bots and search bots are different. GPTBot and ClaudeBot are closer to training collection. OAI-SearchBot and similar search bots fetch pages when an answer needs the live site. Google-Extended is an opt-out token for Gemini training and grounding. It does not remove you from Google Search.

The recommended preset on this page allows AI search and blocks AI training for most businesses that still want to appear in assistant answers. That is a trade-off, not a moral rule. If you do not want any AI system fetching the site, use block all AI and accept that some answers may stop citing you. If you want maximum training inclusion, allow all AI.

Robots.txt will not erase text already used in training. It will not secure private pages. Use authentication for private content. Use noindex when the problem is Google’s index, not a fetch policy.

Platform guides

WordPress usually serves robots.txt from the root. Many SEO plugins edit it. Keep one owner for the file so plugin rules and hand edits do not fight. WooCommerce adds commerce paths worth blocking from search when they create duplicate or thin URLs.

Shopify edits robots.txt through robots.txt.liquid in the theme. You do not upload a static file the way you do on a normal host. Next.js can use app/robots.ts or a static file in public. Webflow has its own hosting rules. The generator prints the install note for the preset you picked.

Common mistakes include blocking CSS or JavaScript that Google needs to render, blocking a path that still appears in the sitemap, and shipping Disallow: / for * during a staging mix-up. Test the money URLs for Googlebot after every change.

• WordPress: confirm the plugin is not appending a second policy.
• Shopify: edit robots.txt.liquid, then fetch the live file.
• Next.js: prefer one source, either robots.ts or public/robots.txt.
• After deploy: test /, a service page, and a blocked admin path.

Common mistakes and a short test plan

Staging leftovers are the classic disaster. A Disallow: / meant for a private server gets copied to production. Test the homepage for Googlebot the day you deploy. If the verdict says blocked, stop and fix the file before you do anything else.

Blocking /wp-admin/ is normal. Blocking /wp-content/uploads/ by accident is not. Blocking parameter URLs can be useful on noisy search pages and harmful when those parameters are the only way to a real product. Know which case you have.

AI bot names change. The list in the generator lives in one config file so it can be updated when vendors rename agents. If a bot you care about is missing, say so. Do not invent a User-agent line from a blog post that is three years old without checking the vendor docs.

When the tester and the live file disagree with what you think you uploaded, you are usually looking at a cache, a CDN, or a plugin that rewrites the response. Fetch with a cache buster. View the raw text. Confirm the host is the production host.

Pair this tool with the SEO checker when you want the wider crawl picture. robots.txt is one file. Canonical tags, noindex, and sitemaps decide other parts of indexation. A clean robots.txt does not fix a noindex on the money page.

After you change AI bot rules, re-test the same path for Googlebot, GPTBot, and OAI-SearchBot. People often fix one agent and forget the others. The all-bots table exists for that habit.

Keep a copy of the previous file in git or in a note. When traffic drops the day after a deploy, the diff is the first thing I ask for. A one-line accidental disallow is easier to spot when you still have yesterday’s text.

If you only came here because a plugin said “you are blocking Google”, paste the file and test / for Googlebot before you rewrite everything. Many of those warnings are about a parameter URL or a preview environment, not the homepage.

Twenty live fetches a day is plenty for checks. Paste mode does not use that limit, so you can edit and re-test the text while you learn the rules.

When you are done, leave a Sitemap line that matches a real sitemap URL. A dead sitemap hint is a small problem. A missing one is fine. A wrong one wastes time.