The robots.txt file is the first file Googlebot reads when it visits your site. With the Google Core Update of May 2026, crawl structure became a direct indicator of site quality. Sites that waste crawl budget on unimportant pages lose rankings to sites that clean their crawl paths. This guide explains how to create and optimize robots.txt step by step.
What is robots.txt and Why Does It Matter in 2026?
robots.txt is a text file in your site's root directory that tells search engine crawlers and AI tools which pages they can visit and which are off-limits. It's a voluntary protocol (RFC 9309) — well-behaved crawlers follow it, though some scraping tools ignore it.
In 2026, robots.txt is not just a technical SEO tool. It's become an editorial decision that controls access for over 30 crawlers from AI companies: Google-Extended, GPTBot, Claude-Web, PerplexityBot, and others.
File Structure and Basic Directives
| Directive | Function | Supported By |
|---|---|---|
| User-agent | Target crawler (* for all) | All |
| Disallow | Block a path | All |
| Allow | Allow a path despite Disallow | Google, Bing |
| Sitemap | Sitemap URL | Google, Bing, AI |
| Crawl-delay | Delay between requests (seconds) | Bing, Yandex (not Google) |
| # | Comment | All |
Basic safe-site example: ``` User-agent: * Disallow:
Sitemap: https://example.com/sitemap.xml ```
This allows all crawlers everything and directs them to the sitemap.
Example with admin path blocking: ``` User-agent: * Disallow: /admin/ Disallow: /search? Disallow: /*?utm_
Sitemap: https://example.com/sitemap.xml ```
When is Crawl Budget Important?
Crawl Budget is the number of pages Googlebot crawls in a given time period. Google determines it based on site authority, server performance, and content freshness. If your site has fewer than 10,000 URLs, don't worry — crawl budget isn't your problem.
But if you exceed 10,000 URLs, every incorrect Disallow wastes crawl budget. Filter, sort, and parameter-based URLs can create millions of URLs — this is the silent killer of crawl budget.
AI Crawlers: The Strategic 2026 Addition
In 2026, these options control AI access to your content:
| Crawler | Purpose | Decision |
|---|---|---|
| GPTBot | OpenAI training/search | Block training, allow search |
| Claude-Web | Anthropic training/search | Block training, allow search |
| PerplexityBot | Perplexity search | Block training, allow search |
| Google-Extended | Google AI training | Allow for search, block for training |
| Common Crawl | Public dataset | Block for AI training |
Mistakes That Close Your Entire Site
- Disallow: / This blocks all crawlers from everything. Often accidentally left from staging.
- Blocking CSS/JS files: Googlebot needs these to render pages correctly.
- Blocking the sitemap path: The sitemap must be accessible.
- Using incorrect syntax: Missing colons, wrong paths.
- Multiple User-agent blocks with conflicting rules: The most specific match wins.
Best Practices
- Always include the sitemap URL
- Use Allow and Disallow intentionally — don't block what Google needs
- Each AI crawler gets its own block if you don't want your content used for training
- Create separate staging/production robots.txt files
- Test changes using Google Search Console's robots.txt Tester
- Monitor crawl stats after changes to ensure you're not accidentally blocking important pages