Key Takeaways
- It uses a protocol called the Robots Exclusion Standard to tell crawlers which parts of your site they are and are not permitted to access.
- The syntax is straightforward, but precision matters.
- The safest time to test a robots.
- These are patterns Indexed encounters regularly when auditing client sites — none of them are theoretical edge cases.
- Since 2023, a growing number of AI companies have deployed their own web crawlers to train large language models and power AI-generated answers.
- Googlebot is a compliant crawler and will follow robots.
- Robots.txt errors are unusually easy to overlook and unusually damaging when they occur. Here are four specific actions
A single misplaced line in a robots.txt file can silently block Googlebot from crawling your entire site — and you might not notice for weeks. Most technical SEO audits surface at least one robots.txt error that has been quietly suppressing organic visibility. The file itself is tiny, often just a few lines, but its consequences are disproportionately large. This robots txt file seo guide explains exactly what to write, how to test it, and where teams most commonly go wrong.
If you're looking for expert help in this area, explore how Indexed's technical SEO services can drive measurable results for your business.
What Is a Robots.txt File and Why Does It Matter for SEO?
Robots.txt is a plain-text file that sits at the root of your domain — for example, https://www.example.com/robots.txt. It uses a protocol called the Robots Exclusion Standard to tell crawlers which parts of your site they are and are not permitted to access. Crucially, it is a request, not a technical barrier. A compliant crawler like Googlebot will respect it; a bad actor scraping your content will not.
From an SEO standpoint, robots.txt matters for three reasons:
- Crawl budget: Large sites have a finite crawl allocation from Google. Directing Googlebot away from low-value URLs — staging parameters, faceted navigation duplicates, internal search results — preserves that budget for pages that actually need indexing.
- Accidental blocking: A disallow directive on the wrong path can prevent Google from crawling your homepage, your CSS and JavaScript files, or your entire product catalogue.
- Indexability signals: Blocking a URL in robots.txt does not guarantee it stays out of the index. If the URL receives external links, Google may still index a shell page with no content. For true deindexing, a
noindexmeta tag or HTTP header is required — but Googlebot must be able to crawl the page to read it.
Robots.txt vs. Noindex: Understanding the Distinction
This is one of the most common points of confusion in technical SEO. Robots.txt controls crawling; noindex controls indexing. If you block a URL in robots.txt, Googlebot will not crawl it — and therefore cannot read any noindex directive on that page. The result is a URL that may still appear in Google's index (without a snippet), because Google knows the page exists from external links but cannot verify your removal instruction. If you want a page out of the index, let Googlebot crawl it and apply noindex in the meta robots tag or the HTTP response header.
How to Write a Robots.txt File Correctly
The syntax is straightforward, but precision matters. Each record in a robots.txt file consists of at least one User-agent directive followed by one or more Disallow or Allow directives.
Basic Syntax and Structure
A minimal, correctly structured file looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: Googlebot
Disallow: /internal-search/
Sitemap: https://www.example.com/sitemap.xmlKey rules to follow:
- User-agent: * applies to all crawlers. Named agents like
GooglebotorBingbotapply only to that crawler and take precedence over the wildcard rule for that specific bot. - Disallow: / blocks the entire site. A forward slash alone is the broadest possible disallow — avoid it on production environments at all costs.
- Blank Disallow: An empty disallow (
Disallow:) means the agent is allowed to crawl everything. This is the correct way to signal open access. - Sitemap declaration: Include your sitemap URL in robots.txt so crawlers can discover it without relying solely on Google Search Console submission.
- Case sensitivity: Paths in robots.txt are case-sensitive on Linux servers.
Disallow: /Blog/will not block/blog/.
What to Disallow (and What to Leave Open)
There is no universal template, but most sites benefit from restricting access to the following categories of URL:
| URL Type | Recommended Directive | Reason |
|---|---|---|
Admin panels (e.g. /wp-admin/) |
Disallow | No SEO value; unnecessary crawl spend |
Faceted navigation (?colour=, ?sort=) |
Disallow (if canonicalised) | Reduces duplicate content crawling |
Internal search results (/search?q=) |
Disallow | Thin, near-infinite URL space |
| Staging or preview paths | Disallow (on staging server's own robots.txt) | Prevents accidental indexing of dev content |
| CSS, JavaScript, image files | Allow (or no directive) | Google needs these to render pages correctly |
| Core content pages | Allow (or no directive) | These are why you have a website |
Google's own guidance is explicit that blocking CSS and JavaScript files can impair rendering, which in turn affects how Google understands and ranks your pages. This is a particularly easy mistake to make when blanket-blocking a /assets/ or /static/ directory.
Free · No obligation
Find out what your site is losing in organic revenue.
In a free Revenue Gap Analysis, we show you exactly what's holding your rankings back — and what fixing it is worth in real revenue.
How to Test Your Robots.txt File Before It Causes Damage
The safest time to test a robots.txt change is before it goes live. Google Search Console includes a robots.txt tester within the legacy tools, and while Google retired the standalone version in 2022, the underlying logic is available through the open-source google/robotstxt library.
Practical Testing Methods
- Google Search Console URL Inspection: Paste any URL into the inspection tool and look at the coverage data. If a page is blocked by robots.txt, it will show clearly under the indexing status.
- Fetch a URL with Screaming Frog: Screaming Frog respects robots.txt by default. Run a crawl in normal mode and compare it to a crawl with robots.txt ignored — pages that appear in the second crawl but not the first are blocked.
- Direct file inspection: Visit
yourdomain.com/robots.txtand read it line by line. Ask specifically: is there anyDisallow: /or any disallow that would match a URL pattern covering production content? - robots.txt validators: Tools like Merkle's robots.txt validator allow you to paste a file and test specific URLs against it without needing server access.
Post-Deployment Monitoring
After any change to robots.txt, monitor Google Search Console's Coverage report over the following two to four weeks. A sudden increase in "Excluded — Blocked by robots.txt" entries is a reliable signal that a new rule is catching unintended URLs. Set up a weekly alert or scheduled crawl so problems surface before they compound into ranking losses.
The Robots.txt Mistakes That Most Frequently Damage SEO
These are patterns Indexed encounters regularly when auditing client sites — none of them are theoretical edge cases.
Leaving a Staging Disallow on Production
Development and staging environments often include Disallow: / to prevent Google from indexing draft content. The problem occurs when a site migration or CMS switch copies the staging robots.txt file to the live domain. The site looks operational to users but is entirely invisible to search engines. This is especially common after platform migrations managed by developers rather than SEO practitioners.
Blocking JavaScript and CSS
Older SEO conventions — and some outdated security guidance — recommend blocking /wp-content/ or equivalent asset directories. Because modern websites rely on JavaScript for rendering core content, blocking these files prevents Googlebot from rendering pages accurately. Pages may be crawled but understood only partially, which typically suppresses rankings for content that relies on dynamic rendering.
Overly Broad Wildcard Patterns
Robots.txt supports the * wildcard character within paths and the $ end-of-URL marker. A directive like Disallow: /*? is intended to block all parameterised URLs but will also block URLs where a question mark appears for legitimate reasons — including some pagination structures and tracking parameters that are part of canonical URLs.
Robots.txt and AI Crawlers: A Growing Consideration
Since 2023, a growing number of AI companies have deployed their own web crawlers to train large language models and power AI-generated answers. OpenAI's GPTBot, Anthropic's ClaudeBot, and Google's Google-Extended are all identifiable by user-agent string and can be selectively blocked or permitted through robots.txt.
If your organisation has a position on AI training data usage, robots.txt is one mechanism — though not the only one — for expressing that preference. The practical reach of these blocks is limited: a block in robots.txt prevents compliant crawlers from accessing content, but enforcement against non-compliant actors relies on terms of service rather than technical controls. This topic connects directly to a broader conversation about how AI systems interact with your site's content and brand representation — something covered in more depth in the related reading below.
See the system
The Full-Stack Search Method.
Seven compounding pillars that turn search into your highest ROI channel. See exactly how we build organic growth that lasts.
FAQ
Does Google always follow robots.txt?
Googlebot is a compliant crawler and will follow robots.txt directives in normal operation. However, Google may still index a blocked URL as a shell entry if it receives external links — it will show in results without a title or snippet because Google cannot read the page content. To remove a URL from the index entirely, you need a noindex directive that Googlebot can actually access, or use the URL Removal Tool in Google Search Console as a temporary measure.
How quickly does Google pick up changes to robots.txt?
Google typically re-fetches robots.txt files frequently — often within a day or two for active sites. However, the effect on crawling may take longer to propagate through the index. If you have removed a disallow directive to re-enable crawling of previously blocked URLs, submit those URLs for indexing via Google Search Console's URL Inspection tool to accelerate the process.
Should every website have a robots.txt file?
Yes. Even if the file simply contains User-agent: * followed by a blank Disallow: line (which permits everything), having a robots.txt file confirms to crawlers that the file exists and has been intentionally configured. Including your sitemap URL in the file also aids discovery. Without a robots.txt file, crawlers will record a 404 response each time they look for it — which is not harmful, but an unnecessary signal of incomplete technical setup.
Can robots.txt help with crawl budget?
Yes, particularly for large e-commerce, news, or classified sites with tens of thousands of URLs. By directing Googlebot away from low-value URL patterns — faceted navigation, session IDs, internal search results — you concentrate crawl activity on the pages most likely to generate organic traffic. For smaller sites with fewer than a few thousand URLs, crawl budget is rarely a material constraint and should not be the primary reason for adding disallow directives.
What to Do This Week
Robots.txt errors are unusually easy to overlook and unusually damaging when they occur. Here are four specific actions to take in the next five working days:
- Fetch your live robots.txt file and read it line by line. Look for any
Disallow: /without a specific path. If it exists, check whether it is intentional. - Open Google Search Console → Settings → robots.txt and review the current file as Google sees it. Note the last fetch date — if it is old, request a fresh fetch.
- Run a Screaming Frog crawl in two modes: once respecting robots.txt, once ignoring it. Export both lists and use a VLOOKUP or similar to identify URLs present in the second but absent in the first. Prioritise reviewing any that are core commercial or content pages.
- Confirm your sitemap URL is declared in robots.txt and that the sitemap returns a 200 status code. This is a five-minute check with meaningful impact on crawl efficiency.
Related Reading
Written by
Anjan LuthraManaging Partner, Indexed
Anjan Luthra is Managing Partner at Indexed. He has spent over a decade inside high-growth companies building organic search into their primary acquisition channel, and writes about SEO strategy, AI search, and revenue a…