Robots.txt checker.See who may read your site.
Enter your site or paste the file. You get the mistakes in it, which search and AI bots it lets in, and a tester that tells you whether any address is blocked and by which line.
What the checker looks at
- Search bots
- Whether Googlebot, Bingbot and Applebot may crawl your site, and which line decides it.
- AI bots
- Whether the bots behind ChatGPT, Claude, Perplexity and Gemini can read your pages, split into AI search and AI training.
- Syntax
- Lines bots skip: typos, rules before a User-agent line, paths without a slash, and directives Google no longer reads such as noindex.
- Sitemaps
- Whether the file points to your sitemap with a full address.
- Blocked resources
- Rules that hide CSS or JavaScript, which stops Google from seeing your pages as visitors do.
- Size
- Google reads only the first 500 KB of robots.txt; anything after that is ignored.
How to read the result
- Start with the red dots
- A red dot means search engines or AI answers cannot reach pages you probably want found. Fix those first.
- Test the pages you care about
- Paste the address of your most important pages into the tester and check Googlebot and OAI-SearchBot. Both should say allowed.
- Decide on AI training
- Blocking GPTBot, ClaudeBot or Google-Extended keeps your text out of model training but does not hide you from AI search. It is your call.
- Check again after editing
- Paste your edited file before you upload it, so a typo never reaches your live site.
Questions
Can't find what you're looking for? Contact our support team
A plain text file at yourwebsite.com/robots.txt that tells crawlers which parts of a site they may visit. Search engines and most AI bots read it before they crawl.
No. It stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of search, allow crawling and add a noindex meta tag to the page.
Separate search from training. Bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer questions and can send you visitors, so most sites allow them. GPTBot, ClaudeBot and Google-Extended collect text for training; blocking them is a business choice and does not affect Google Search.
Crawlers treat a missing file (a 404) as permission to crawl everything. That is fine, but a short file that lists your sitemap helps search engines find new pages.
Google sets its crawl speed from how fast your server answers. Bing and Yandex still follow crawl-delay.
No. A pasted file is checked in your browser. When you enter an address, our server fetches the public file once and does not store it.
More free SEO tools
- XML sitemap checkerFind your sitemap, validate it and see whether the pages it lists actually open.
- llms.txt generatorCreate an llms.txt file from your site's pages, so AI assistants find your key pages.
- SEO checkerScore any page for Google and AI search in seconds, with the three fixes that matter most.
- Redirect checkerFollow every redirect an address goes through, and check all four versions of your domain end in one place.