Updated October 2026. I first wrote this guide in 2016. A lot has changed since then: Google dropped a few rules this post used to recommend, its testing tools were replaced, and AI crawlers from OpenAI, Anthropic, Perplexity and others now visit your site too. This version covers all of that.
A robots.txt file tells search engines and AI crawlers which parts of your site they can and can’t crawl. Get it right and bots spend their time on the pages that matter. Get it wrong and you can block your best pages from Google, ChatGPT and every other tool that might send you customers.
This guide uses WordPress examples, but the same rules work on any CMS once you account for its URL structure.
What Is a Robots.txt File?
It’s a plain text file that lives at the root of your domain (for example, kernmedia.com/robots.txt). Crawlers check it before they crawl the rest of your site.
Two things to know up front:
- Robots.txt controls crawling, not indexing. A blocked page can still show up in Google if other sites link to it. To keep a page out of search results, use a noindex tag instead (more on that below).
- It’s a request, not a lock. Reputable crawlers follow it. Bad bots ignore it. Never use robots.txt to hide private content.
How Robots.txt Rules Work
Each group of rules starts with a User-agent line that names the crawler, followed by Disallow and Allow rules. An asterisk (*) means “all crawlers.”
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
A crawler follows the most specific group that matches its name. If you add a group for Googlebot, Googlebot follows that group and ignores the * group, so repeat any rules you still want it to follow.
Disallow a Specific URL
Disallow: /specific-url-here/
Disallow a Folder
Disallow: /subfolder/
Disallow URLs by Pattern
Google and Bing support two wildcards: * matches any characters, and $ marks the end of a URL.
Disallow: /*?s= Disallow: /*.pdf$
Note: Be careful with wildcards. One broad rule can block pages you actually want crawled. Test every rule before it goes live.
Search Engine and AI Crawler User Agents
Here are the crawlers I see most often in client server logs, and what blocking each one does.
Search Engines
- Googlebot: crawls for Google Search, including AI Overviews and AI Mode.
- Bingbot: crawls for Bing, which also powers Microsoft Copilot and some other AI tools.
- Google-Extended: not a separate crawler, but a robots.txt token that controls whether Google can use your content for Gemini models. Blocking it doesn’t affect your Google Search rankings.
OpenAI (ChatGPT)
Per OpenAI’s crawler documentation:
- OAI-SearchBot: finds pages to show in ChatGPT search. Block this and your pages can’t be cited in ChatGPT search answers.
- GPTBot: collects content for training OpenAI’s models. Blocking it doesn’t affect ChatGPT search.
- ChatGPT-User: visits a page when a ChatGPT user asks it to. OpenAI says robots.txt rules may not apply here, since a person requested the visit.
Anthropic (Claude)
Per Anthropic’s crawler documentation:
- ClaudeBot: collects content for training Claude models.
- Claude-SearchBot: crawls to improve Claude’s search results. Blocking it can reduce your visibility in Claude’s answers.
- Claude-User: fetches pages when a Claude user asks a question. Blocking it can keep your pages out of those answers.
Perplexity
Per Perplexity’s crawler documentation:
- PerplexityBot: finds pages to show and link in Perplexity answers. It follows robots.txt.
- Perplexity-User: visits pages when a user asks a question. Perplexity says it generally ignores robots.txt, since a person requested it.
Should You Block AI Crawlers?
For most of my clients, no. If you sell products or services, you want AI tools to read your pages, describe you accurately and link to you. In my AI search citation study, brands whose own website was cited in an AI answer were recommended 96% of the time, compared to 18% when their site wasn’t cited. Blocking the search crawlers works against you.
If you want to stay out of model training but still show up in AI search, block the training crawlers and allow the search ones:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
That still lets OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot crawl your site. Publishers who license their content have more reason to block training bots than an eCommerce store or service business does.
Tip: Robots.txt isn’t the only thing that can block AI crawlers. Firewalls, CDN bot settings (Cloudflare has an option to block AI bots) and security plugins can block them too. If you want AI visibility, check those settings as well.
On a free 30-minute call, we’ll talk through your site and goals.
What to Disallow on a WordPress Site
Less than you might think. The old advice (including from the first version of this post) was to block /wp-includes/, /wp-content/plugins/ and /wp-content/themes/. Don’t do that. Google needs to load your CSS and JavaScript to render your pages like a visitor sees them, and those folders hold those files.
Here’s what I’d usually block:
- /wp-admin/ (but allow
/wp-admin/admin-ajax.php, which some themes and plugins need) - Internal search results (
/?s=and/search/), which can create endless low-value URLs - Private areas like a client portal or cart and account pages on an online store
- Crawl traps like filtered and sorted URLs (see below)
Here’s the robots.txt file I use on this site:
# Kern Media robots.txt # Search engines and AI crawlers are welcome. Admin and internal areas are excluded. User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /search/ Disallow: /client-portal/ Sitemap: https://www.kernmedia.com/sitemap_index.xml
It’s short on purpose. Feel free to use it as a starting point, but change the private folders and sitemap URL to match your site.
Filtered URLs and Crawl Traps
Large sites, especially online stores, often create thousands of URLs from filters, sorting and session parameters (like ?color=blue&size=m&sort=price). Crawlers can waste a lot of time on these instead of your real pages.
Crawl your site with Screaming Frog and look for URL patterns that clearly aren’t pages you want in search results. Then write wildcard rules for them:
Disallow: /*?sort= Disallow: /*&sort=
Before you block a filter, check Google Search Console. Some filtered pages (like a “red dresses” color filter) can bring in real search traffic, and you’d want to keep those crawlable.
Don’t Block Your Ad Landing Pages
Ad platforms crawl your landing pages to check them. If you use wildcard rules that could catch campaign tracking URLs (like ?utm_medium=), make sure you aren’t blocking them. Google’s ad crawler (AdsBot-Google) ignores the * group, so it only follows rules that name it directly.
Robots.txt Can’t Remove Pages from Google
This is the mistake I see most. If a page is already indexed and you block it in robots.txt, Google can’t crawl it anymore, which means it can’t see a noindex tag either. The page can stay in search results for a long time, often without a description.
To remove pages from Google:
- Make sure the pages are not blocked in robots.txt.
- Add a
noindexmeta robots tag (or an X-Robots-Tag HTTP header). - Wait until Google has dropped them from its index.
- Then block them in robots.txt if you want to save crawl time.
The first version of this post suggested using a “Noindex” rule inside robots.txt. Google stopped supporting that in 2019, so don’t use it.
Link Your XML Sitemap
Add your XML sitemap at the bottom of the file so crawlers can find it. If you have more than one, list them all.
Sitemap: https://www.example.com/sitemap_index.xml
How to Test Your Robots.txt File
- Google Search Console robots.txt report: found under Settings. It shows which robots.txt files Google found, when it last checked them and any errors. (It replaced the old Robots.txt Tester.)
- URL Inspection tool: enter any URL to see if it’s blocked by robots.txt and how Google renders it. (It replaced “Fetch as Google.”)
- Screaming Frog: lets you test a custom robots.txt file against your site before you publish it. This is the safest way to catch a rule that blocks too much.
- Server logs: show which bots are actually visiting and whether they’re getting blocked. This is the only way to confirm AI crawlers are reaching your pages.
OpenAI notes it can take about 24 hours for ChatGPT search to pick up a robots.txt change, so give it a day before you test.
Need Help?
A robots.txt review is part of every technical SEO audit I do, along with JavaScript rendering checks for AI crawlers. If you have questions, contact me or request a free consultation.
