SEO & WebTester

Robots.txt Tester & Validator

Test and validate your robots.txt file online. Check if Googlebot, Bingbot, or AI bots can access specific URLs with live rule debugging.

Tanbir Ahamed·Founder of ToolWise · Software Engineer

Published August 2026

Presets & Templates:

robots.txt Content

Test URL Access

Crawler / User-Agent

URL Path to Test

BLOCKEDGooglebot

Path: /admin/dashboard

Blocked by line 4: "Disallow: /admin/"

Detected Sitemaps:

🔗 https://example.com/sitemap.xml

How to Use

1Paste your robots.txt file or choose a preset template.
2Select the search engine crawler or AI bot you want to test.
3Enter the URL path you want to check (e.g. /admin or /blog/post).
4Inspect the instant Allowed / Blocked result and matching rule line.
5Fix any syntax warnings and download your validated robots.txt.

Features

✓Instant rule testing with Google-compliant longest-match precedence
✓Support for wildcards (*) and end anchors ($)
✓Test 12+ search engines and AI crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot)
✓Syntax validator with warnings for invalid directives or missing User-agents
✓Ready-to-use presets (Standard, WordPress, E-commerce, Block AI, Block All)
✓Sitemap detection & extraction
✓100% private client-side evaluation

Comprehensive Guide & Reference

Complete Guide to robots.txt Testing & Debugging

Understand crawler access controls, avoid crawl-budget waste, and prevent accidental indexation blocks.

1. Why Testing robots.txt Is Critical

A single misplaced character in your robots.txt file can accidentally de-index your entire website or expose private staging directories to search engines. Testing your rules across multiple User-agents ensures search bots crawl only the pages you want them to see, preserving crawl budget and protecting private routes.

2. Rule Precedence and Evaluation Rules

User-Agent Specificity

A named bot group (like Googlebot) completely overrides the generic (*) group. The crawler will ignore all directives in the wildcard section if a specific group exists for it.

Longest Match Wins

When multiple rules match a URL path, the rule with the longest character pattern is chosen. For example, Allow: /blog/post-1/ beats Disallow: /blog/.

3. AI Bot Directives Reference

# Block OpenAI GPTBot

User-agent: GPTBot
Disallow: /

# Block Anthropic ClaudeBot

User-agent: ClaudeBot
Disallow: /

# Block Common Crawl

User-agent: CCBot
Disallow: /

4. Privacy Guarantee

🔒 100% Client-Side Evaluation

All robots.txt parsing and URL path evaluations run entirely in your local browser using JavaScript. No test paths, URLs, or file contents are transmitted to our servers.

Real example

Worked example: a blocked admin URL

Input: Rules: Disallow: /admin · Test: https://www.freetoolwise.com/admin/settings as Googlebot
Output: →DISALLOWED — matched rule: Disallow: /admin (the URL path starts with the rule's prefix)

Robots.txt matches path prefixes, not whole directories — which is exactly why the next example trips people up.

Worked example: the prefix gotcha

Input: Rules: Disallow: /admin · Test: https://www.freetoolwise.com/administrator as Googlebot
Output: →DISALLOWED — /administrator also starts with /admin, so the prefix rule matches it too

Anchor the rule when you mean it: Google and Bing support the $ wildcard, so Disallow: /admin$ blocks only the exact path /admin.

Privacy & data handling

This tool runs entirely in your browser. What you enter is processed on your own device using JavaScript and WebAssembly — it is not uploaded to our servers, and there is no account, no signup and no per-use limit. Read our privacy policy.

Frequently Asked Questions

How does robots.txt rule precedence work?

Search engines evaluate robots.txt by first finding the most specific User-agent block (e.g. Googlebot beats *). Inside that block, the longest matching rule pattern wins. If an Allow and Disallow rule have the exact same pattern length, the Allow rule takes precedence according to Google standards.

Does robots.txt prevent pages from appearing in Google Search?

Not completely. If a page is blocked in robots.txt but has external links pointing to it, Google can still index the URL and display it in search results without reading its content. To guarantee complete de-indexing, use the noindex meta tag or X-Robots-Tag HTTP header instead.

Are wildcards (*) and end-anchors ($) supported?

Yes. Major search engines including Google, Bing, and Yandex support asterisk wildcards (*) for arbitrary characters and dollar signs ($) to match the end of a URL (e.g. Disallow: /*.pdf$ to block all PDF documents).

How do I block AI web scrapers like ChatGPT (GPTBot)?

Add dedicated User-agent sections for AI crawlers like GPTBot, ClaudeBot, PerplexityBot, and CCBot, followed by Disallow: /. Standard search engines like Googlebot and Bingbot will continue indexing your site normally.

Where should the robots.txt file be located on my website?

The robots.txt file must always be placed at the exact root of your domain (e.g., https://example.com/robots.txt). Search engine crawlers will never look for it in subdirectories.