robots.txt for AI crawlers: the rules that actually apply

The AI crawlers your robots.txt is talking to

One vendor, several bots, different jobs. That is the whole reason this got complicated: "block OpenAI" is not a single instruction, because the crawler that gathers training data and the crawler that fetches a page when a user pastes a link are separate agents with separate names.

Two of these are easy to get wrong. Google-Extended is not a crawler at all — it is a permission token that controls whether content Googlebot already fetched may be used for AI training, so disallowing it costs you nothing in search. Applebot-Extended works the same way relative to Applebot.

The list keeps growing. Locomint's access endpoint currently checks eighteen named agents: the twelve above plus Perplexity-User, Googlebot, Amazonbot, cohere-ai, Diffbot and Timpibot. Each carries a purpose of training, search, user-triggered fetch, open crawl or commercial crawl, which is the axis most site owners actually want to decide on — and the one a bare user-agent string does not tell you.

How the wildcard group and named groups interact

This is the rule that quietly breaks robots.txt files: a crawler obeys exactly one group. It uses the group that names it, and if no group names it, it falls back to the User-agent: * group. It never merges the two. The behaviour is specified in RFC 9309, which is worth ten minutes if you maintain one of these files.

So this file does not do what its author intended:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: ClaudeBot
Crawl-delay: 10

ClaudeBot is now the only agent allowed into /admin/ and /cart/, because naming it created a group with no Disallow lines and it stopped reading the wildcard group. Every named group has to repeat the site-wide rules you want it to follow.

The second half of the rule decides conflicts within a group: the longest matching path wins, and on a tie, Allow beats Disallow. That is what makes the "block everything except one section" pattern work.

Crawl-delay sits outside RFC 9309 and vendors honour it inconsistently, so read it as a request rather than a limit you can rely on. The access endpoint returns it per agent regardless, because a site asking for ten seconds between requests has told you something about how it expects to be treated, even where nothing enforces it.

Allow search but not training: a worked robots.txt

The common position for a publisher or a business site is: appear in AI answers with a link back, do not become training data. That means allowing the search and user-fetch agents and refusing the training ones, each in its own group.

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /admin/
Disallow: /checkout/

User-agent: Claude-SearchBot
Disallow: /admin/
Disallow: /checkout/

User-agent: PerplexityBot
Disallow: /admin/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Note that the search agents repeat the two site-wide disallows. Leave them out and those agents get access to /admin/, for the reason above.

If you want the opposite — nothing automated at all except the search engine you rely on — a wildcard Disallow: / plus a named group for that one crawler is the shape. A site that blocks every AI agent disappears from the answers people increasingly ask instead of searching, and there is no separate lever that grants visibility without access.

Checking any domain's AI crawler rules with one call

Reading a robots.txt by eye scales to one site. GET /v1/domains/{domain}/access reads the site's own public files — robots.txt, llms.txt, ai.txt and the homepage — and returns a verdict per agent with the line that decided it. One credit.

curl "https://api.locomint.io/v1/domains/example.com/access" \
  -H "X-API-Key: $LOCOMINT_KEY"

Each entry in agents[] looks like this, and the last two fields are the ones that matter in an argument:

{
  "agent": "GPTBot",
  "vendor": "OpenAI",
  "purpose": "training",
  "allowed": false,
  "rule": "Disallow: /",
  "matched_by": "User-agent: GPTBot",
  "explicit": true,
  "crawl_delay": null
}

rule quotes the line, and explicit says whether the site named this agent or whether the verdict came from the User-agent: * fallback — which is exactly the distinction that decides whether a site has a policy or just a default. Alongside the per-agent verdicts you get blocked_agents, blocks_any_ai_agent and ai_training_allowed as the short answer, the sitemaps declared in robots.txt, and text_without_javascript, the number of readable words in the raw HTML before any script runs.

Pass agent=ClaudeBot to check one agent, and path=/pricing to ask about a path rather than the site root — a site that disallows /blog/ and allows everything else gives different answers to those two questions. The full field list is in the access reference, and the AI Crawler Checker page covers the rest.

Every response carries checked_at. Store it. Rules change without notice, and "the site allowed this on 11 September, and here is the line" is a different kind of statement from "we thought it was fine".

Reading a verdict where explicit is false

The field that changes what you should do is explicit. Take a site whose entire robots.txt is three lines:

User-agent: *
Disallow: /wp-admin/
Sitemap: https://example.com/sitemap.xml

Ask the access endpoint about GPTBot and you get allowed: true, rule: null, matched_by: "User-agent: *", explicit: false. Every one of the eighteen agents comes back the same way, and blocks_any_ai_agent is false.

That is not the same as a site saying yes. It is a site that has never been asked the question — most likely a CMS default, written before any of these agents existed. Compare it with a site that names ClaudeBot and gives it Disallow: /: there explicit is true, and the operator made a decision you can point at.

The practical difference is what you do when the site later complains. Against an explicit allow you can quote the line and the date. Against a wildcard fallback you have a defensible position and nothing more, so weight it accordingly: fetch conservatively, honour any crawl_delay, and stop on the first refusal rather than assuming silence was consent. Two other fields belong in the same judgement. pay_per_crawl is true when the site answers 402 Payment Required, which is an explicit price on access rather than a block. And content_signals returns any content-signal line the site has published, which is a newer way of stating usage terms inside robots.txt itself.

What llms.txt and ai.txt are worth today

Neither file controls access. llms.txt is a proposed convention for a markdown file at the site root that points a model at your best pages, in the spirit of a sitemap written for a reader rather than an indexer. ai.txt is a similar idea for stating usage terms.

As of this writing, no major AI vendor has committed to reading either one, so a rule you put in llms.txt and nowhere else is a rule that nothing enforces. Access permission lives in robots.txt, and it is the file crawlers actually fetch.

That is not an argument against adding one. It is cheap, honest documentation, and if the convention wins, the site that already has one is done. Just do not use it as your only statement, and do not assume a site that lacks one is indifferent. The access endpoint reports presence, status and size for both files, which is the fastest way to see how common they really are across the domains you care about.

Rules, law, and what a crawler operator should do

robots.txt is a convention, not a statute. Whether ignoring it exposes you to liability depends on where you are, what you fetched and what you did with it, and it is being litigated in several places at once. This is not legal advice; the broader questions are covered in the article on whether web scraping is legal.

The operational position is simpler than the legal one. Read the file before you fetch, obey what it says for the agent name you send, keep the dated evidence, and stop on any refusal rather than routing around it. A 403 or a challenge page is an answer; our own crawler treats it as one and never retries through a different address. If you are also checking whether the pages you are allowed to fetch say anything useful in machine-readable form, the Schema Validator API reads the schema.org markup on a page and reports what is missing.

Where to start this week

Fetch your own robots.txt and check it against the group-matching rule: every named group must repeat any site-wide Disallow you expect it to honour. That one check fixes the most common real defect.

Then decide the training question deliberately rather than by default, and write it down: which agents may train, which may index, which may fetch on a user's behalf. Run the access endpoint against your own domain afterwards to confirm the file says what you think, and against ten sites in your sector to see where the norm sits. Ten domains is ten credits, inside the free plan, and the pricing page shows what a larger sweep costs.

Frequently asked questions

How do I block GPTBot but still allow ChatGPT to cite my pages?

Give GPTBot its own group with Disallow: / and give OAI-SearchBot and ChatGPT-User a group that allows what you want indexed or fetched. GPTBot is the training crawler, OAI-SearchBot builds the search index behind citations, and ChatGPT-User fetches a page because a person asked for it. Blocking all three removes you from the answers as well as from the training set.

Does a named user-agent group inherit rules from the wildcard group?

No, and this is the most common mistake in an AI crawler robots.txt. A crawler uses exactly one group: the one that names it, if there is one, otherwise the User-agent: * group. If you write Disallow: /admin under the wildcard and then add an empty group for ClaudeBot, ClaudeBot is allowed into /admin, because it never reads the wildcard group at all.

Is llms.txt worth adding to my site?

It costs almost nothing and no major AI vendor has committed to reading it, so treat it as optional documentation rather than a control. Access permission is decided by robots.txt; llms.txt is a proposed convention for pointing a model at your best content. The Locomint access endpoint reports whether llms.txt and ai.txt exist, so you can see how common they are on the sites you care about.