AI Crawler Checker: know what a site allows before you fetch it
Crawling rules are now written per AI agent and they change often. This endpoint reads a site's own public files and tells you, agent by agent, whether you may fetch, quoting the line that decides it. One call, one credit, and a dated record you can show to a customer or a lawyer.
GET/v1/domains/{domain}/access
{
"data": {
"domain": "example.com",
"checked_at": "2026-09-07T13:22:40Z",
"robots": {
"present": true,
"status": 200,
"bytes": 412
},
"sitemaps": [
"https://example.com/sitemap.xml"
],
"llms_txt": {
"present": false,
"status": 404
},
"pay_per_crawl": false,
"blocks_any_ai_agent": true,
"ai_training_allowed": false,
"blocked_agents": [
"GPTBot",
"CCBot"
],
"agents": [
{
"agent": "GPTBot",
"vendor": "OpenAI",
"purpose": "training",
"allowed": false,
"rule": "Disallow: /",
"matched_by": "User-agent: GPTBot",
"explicit": true
},
{
"agent": "ClaudeBot",
"vendor": "Anthropic",
"purpose": "training",
"allowed": true,
"rule": "Allow: /",
"matched_by": "User-agent: *",
"explicit": false
}
],
"text_without_javascript": 812,
"javascript_dependent": false
},
"meta": {
"credits_used": 1,
"source": "websites"
}
}
What you get
Every field in the response.
| Field | Meaning |
|---|---|
agents[] | One verdict per AI agent: allowed, the rule that decided it, whether the site named that agent, and any crawl delay |
blocked_agents, ai_training_allowed | The short answer, for a dashboard or a gate in your pipeline |
robots, llms_txt, ai_txt | Whether each file exists, with status and size |
sitemaps, content_signals | Sitemaps declared in robots.txt and any content-signal line |
pay_per_crawl | True when the site answers 402 Payment Required |
text_without_javascript | Words of readable text before any script runs, so you know if a crawl is worth it |
How it works
What happens between your request and the response.
Four small fetches
robots.txt, llms.txt, ai.txt and the homepage. Nothing else is touched.
Evidence, not opinion
Every verdict carries the line it came from and whether the site named your agent or fell back to the wildcard group.
Any agent, any path
Ask about one agent, or about the path you actually intend to fetch rather than the site root.
A refusal is the answer
If a site refuses us, that is reported. We do not retry from a different address to get a better answer.
Pricing
One credit system for every endpoint.
One credit per domain checked, from the same monthly credits as every other endpoint. A site whose robots.txt could not be read is answered with allowed: null and costs nothing.
Run it on Apify.
For bigger runs, or without writing any code, the same engine runs on Apify as the listing below. Apify bills per result, so there is no subscription and nothing to pay us, and Apify's own free plan includes $5 of usage a month with no card.
Questions
Things people ask about the AI Crawler Checker.
Who needs this?
Anyone running a crawler or ingesting the web for a model, and anyone who has to prove afterwards that they respected a site's rules. It is also the fastest way to answer why an AI assistant cannot see a customer's site.
Which agents do you check?
The ones people ask about: OpenAI, Anthropic, Google, Perplexity, Common Crawl, ByteDance, Amazon, Apple, Meta, Cohere and others, each with what it is for. You can also name any agent yourself.
Does a verdict mean I am allowed to crawl?
It means the site's own files say so on the day you asked. Terms of service, copyright and local law are separate questions, and this is not legal advice.