Website monitoring · No. 09
robots.txt check
In short
robots.txt is a text file in the root of your website that tells search crawlers and AI bots which URLs they may crawl. One Disallow: / line blocks the whole website, which then gradually disappears from Google and AI search. This usually happens when a staging file ends up on the live website during a launch or update.
Contents
What is robots.txt?
robots.txt is a plain text file at example.com/robots.txt where the website owner writes rules for automated visitors: search engine crawlers, AI bots, and SEO tools. Here’s how Google describes it:
“A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.”
The format is standardized in RFC 9309, the document that defines the Robots Exclusion Protocol. The file must sit in the root of your website — crawlers don’t look for it in subfolders — and Google reads only the first 500 KiB (Google documentation).
| Line | What it means |
|---|---|
User-agent: * | The rules below apply to all crawlers |
User-agent: Googlebot | The rules below apply only to the Google Search crawler |
Disallow: /cart/ | Don’t crawl URLs that start with /cart/ |
Disallow: / | Don’t crawl any page on the website |
Disallow: (empty value) | No restrictions — everything can be crawled |
Allow: /cart/help/ | An exception to a broader block |
Sitemap: https://example.com/sitemap.xml | The location of your sitemap |
Keep in mind: robots.txt is a request, not a lock. RFC 9309 states plainly, “These rules are not a form of access authorization” (RFC 9309). Well-behaved crawlers follow them, but you can’t use them to protect private data.
What can a robots.txt mistake cost you?
The most dangerous mistake is a Disallow: / line in the User-agent: * group. It tells every search engine crawler not to crawl your website. Google can’t read your page content: new pages don’t make it into search, changes to existing pages aren’t picked up, and according to Google’s documentation, a blocked page’s URL can still appear in search results — just without a description. Meanwhile, the website opens normally, so unless you’re checking, you’ll only notice when traffic drops.
The second common mistake is trying to remove a page from search with robots.txt. That doesn’t work, because the file controls crawling, not indexing:
“[…] it is not a mechanism for keeping a web page out of Google. […] While Google won’t crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web.”
To keep a page out of Google, the documentation recommends blocking indexing with noindex or password-protecting the page. And don’t block a noindex page in robots.txt at the same time: the crawler won’t load the page, so it never sees the noindex.
If you need to hide a page from search urgently, Google Search Console has the Removals tool. It blocks the URL from Google Search results for about six months, but it doesn’t stop crawling or delete the page for good: after that period, the page can come back (Search Console Help). So pair a temporary removal with noindex, a password, or deleting the page itself.
An unreachable file is dangerous too. If robots.txt returns a 5xx server error, Google stops crawling your website for the first 12 hours, then uses the last cached version of the file for up to 30 days (Google documentation). That’s why Google specifically warns against returning a 503 for robots.txt, even during maintenance (Google documentation). A 404, on the other hand, means “no restrictions.”
Visibility in AI answers
How does robots.txt affect your visibility in AI answers?
Directly. Most major AI services crawl websites with their own bots and follow robots.txt rules, so the same file decides whether your website can show up in answers from ChatGPT, Claude, or Perplexity. On AI search, OpenAI puts it bluntly:
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
Note that each service runs several bots with different jobs. You can block your website from being used for model training and still stay in AI search:
| Company | Bot in robots.txt | What it’s for |
|---|---|---|
Googlebot | Google Search, including AI Overviews and AI Mode | |
Google-Extended | Training future Gemini models and grounding answers in Gemini Apps and Vertex AI; doesn’t affect Google Search | |
| OpenAI | OAI-SearchBot | Showing websites in ChatGPT search |
| OpenAI | GPTBot | Model training |
| Anthropic | Claude-SearchBot | Improving search results for Claude users |
| Anthropic | ClaudeBot | Collecting content that may be used for training |
| Perplexity | PerplexityBot | Surfacing and linking websites in Perplexity search results |
Here’s what the companies themselves say:
- Google. Google-Extended isn’t a separate crawler: regular Googlebot does the crawling, and the robots.txt token only controls how the content is used. Per the documentation, “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google). Crawl access for Search — which includes AI Overviews and AI Mode — is governed by your robots.txt rules for Googlebot, and the page itself must be indexed (Google). In other words,
Disallow: /for Googlebot removes your website from both Google Search and Google’s AI answers. - OpenAI. “Each setting is independent of the others,” and after you update robots.txt, “it can take ~24 hours” for OpenAI’s search systems to adjust. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models” (OpenAI).
- Anthropic. The company says its bots honor “industry standard directives in robots.txt.” Blocking Claude-SearchBot, it says, “may reduce your site’s visibility and accuracy in user search results” (Anthropic).
- Perplexity. PerplexityBot “is not used to crawl content for AI foundation models,” and the company says, “we recommend allowing PerplexityBot in your site’s robots.txt file” (Perplexity).
There are exceptions. Bots that open a page at a user’s direct request work differently: OpenAI says that for ChatGPT-User, “robots.txt rules may not apply,” and Perplexity says Perplexity-User “generally ignores robots.txt rules.” Anthropic, on the other hand, notes that “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query.”
Why does robots.txt break?
People rarely change the file on purpose — it usually changes along with other work on the website:
- Moving from staging. The staging copy of the website gets
Disallow: /so it won’t be indexed, and at launch the file is copied to the live website along with everything else. - CMS or hosting settings. Some CMSs and hosting panels have a “discourage search engines” option for staging copies. It gets turned on during development and never turned off after launch.
- An SEO plugin or module. A plugin update overwrites the file with its own template or adds blocks for sections that should be in search.
- An overly broad pattern.
Disallow: /cinstead ofDisallow: /cart/blocks the/catalog/section along with the cart, because the rule applies to every URL that starts with those characters. - Blocking AI bots. Someone adds a block against “all bots” and, along with the training bots, shuts out AI search crawlers or even Googlebot.
- A server error. The file is generated dynamically, and after an update,
/robots.txtreturns a 5xx error.
How do you spot a robots.txt problem?
The signs usually show up in Search Console and in search results, not on the website itself:
| What you see | Possible cause | What to do |
|---|---|---|
| Impressions and clicks in Search Console drop sharply after a launch or update | Disallow: / ended up on the live website | Open /robots.txt, remove the site-wide block, and request recrawling |
| A page shows up in search with no description | The page is blocked in robots.txt, but Google found it through links | If the page belongs in search, allow crawling; if not, allow crawling and add noindex |
| Important pages show “Blocked by robots.txt” in Search Console | A Disallow rule that’s too broad | Make the path more specific or add an Allow rule for the section you need |
| Google has stopped crawling the website entirely | robots.txt returns a 5xx error | Make sure the file is served with a 200 status code |
| Your website doesn’t appear in ChatGPT or Perplexity search | OAI-SearchBot or PerplexityBot is blocked | Allow AI search crawlers and, if you want, keep blocking only the training bots |
How to check robots.txt yourself
You’ll need a browser and Google Search Console:
- Open
https://example.com/robots.txt, replacing it with your own domain. The file should open as plain text, without an error. - Find the
User-agent: *group and any groups for specific crawlers. Make sure there’s noDisallow: /and no overly short paths likeDisallow: /c. - Check that the
Sitemap:line points to a working sitemap URL. - Find the groups for
Googlebot,OAI-SearchBot,Claude-SearchBot, andPerplexityBot, and make sure they’re allowed to crawl everything that should be in search. - In Search Console, open the robots.txt report under Settings: it shows when Google last fetched the file and whether there were any errors. For a single page, the URL Inspection tool shows whether crawling is blocked.
- Or use the FullCheckApp “robots.txt check” and get the results in a clear visual format.
RFC 9309 lets crawlers cache the file for up to 24 hours (RFC 9309), so a fix doesn’t take effect instantly.
If you chose the last option, you’ve already made the right call: the check runs automatically, and you’ll be the first to know about any warning or critical issue.
What FullCheckApp checks
Every day, at the time you choose, FullCheckApp downloads the robots.txt file of each website you monitor and shows in the report:
- Whether indexing is allowed — that the file doesn’t block your website from search engine crawlers.
- How many sitemaps are listed — the
Sitemap:lines in the file. - Whether
Disallow: /has appeared — it’s flagged as a problem, so a move from staging doesn’t go unnoticed.
Report flags: green means OK, yellow means a warning, and red means critical; you set the severity separately for each website. After each check, you get an email report, and critical issues are also sent to Telegram.
The sitemap itself — how many pages it lists and whether that number has dropped — is covered by a separate sitemap.xml check. Blocking indexing with meta robots and X-Robots-Tag is explained in the page SEO tags check article, and page status according to Google itself is covered in the Google page indexing check article.
FAQ
Does Disallow remove a page from Google?
No. robots.txt blocks crawling a page, not indexing it. Google says it can still find and index a disallowed URL if other websites link to it — and then it shows up without a description. To remove a page from search, use noindex or put it behind a password, and don’t block crawling. You can also hide unwanted pages from search results with the Removals tool in Google Search Console, but only temporarily — for about six months.
What happens if my website doesn’t have a robots.txt file?
Nothing bad, especially if you have a small business card website, a landing page, or a blog with a dozen pages and no technical sections or duplicates worth keeping crawlers out of. If the file returns a 404, Google assumes there are no crawl restrictions and crawls the whole website. It’s worse when the file returns a 5xx server error: Google stops crawling first, then uses the last cached version of the file for up to 30 days. The upside of having robots.txt is that you can list your sitemap URL so crawlers find new content faster, and control how search engine crawlers crawl your website in general.
How do I stop AI from training on my website but stay in AI search?
In robots.txt, block the training bots — for example, GPTBot, ClaudeBot, and Google-Extended — but don’t block the search crawlers: Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot. According to the companies’ documentation, these settings are independent, and Google-Extended doesn’t affect Google Search. Check each service’s documentation for current bot names.
Does robots.txt protect pages from outsiders?
No. RFC 9309 states plainly that robots.txt rules are not a form of access authorization. Well-behaved crawlers follow them, but anyone can open a disallowed URL — and since the file is public, it even hints at which sections you want to hide. For private data, use a password or other real protection.
Why doesn’t Google bring my website back right after I fix robots.txt?
Crawlers cache robots.txt: RFC 9309 allows them to keep a copy for up to 24 hours. After that, Google still has to recrawl your pages, which takes time. You can speed things up by requesting recrawling of key pages in Search Console and making sure your sitemap is available.
Sources
- Google Search Central — Introduction to robots.txtdevelopers.google.com/search/docs/crawling-indexing/robots/intro
- Google Search Central — How Google interprets the robots.txt specificationdevelopers.google.com/search/docs/crawling-indexing/robots/robots_txt
- RFC 9309 — Robots Exclusion Protocolwww.rfc-editor.org/rfc/rfc9309.html
- Google — Google’s common crawlers (Google-Extended)developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google Search Central — AI features and your websitedevelopers.google.com/search/docs/appearance/ai-features
- OpenAI — Overview of OpenAI Crawlersdevelopers.openai.com/api/docs/bots
- Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Perplexity — Perplexity Crawlersdocs.perplexity.ai/guides/bots
- Search Console Help — Removals and SafeSearch reports toolsupport.google.com/webmasters/answer/9689846?hl=en
Check your website
Add your website address — the first check runs right away, and you’ll see the status of every item in this article.
Check for free