Crawl Access API: know what a site allows before you fetch it

Crawling rules are now written per AI agent and they change often. This endpoint reads a site's own public files and tells you, agent by agent, whether you may fetch, quoting the line that decides it. One call, one credit, and a dated record you can show to a customer or a lawyer.

GET/v1/domains/{domain}/access

Get a free API key Read the reference

200GET /v1/domains/example.com/access

{
  "data": {
    "domain": "example.com",
    "checked_at": "2026-09-07T13:22:40Z",
    "robots": {
      "present": true,
      "status": 200,
      "bytes": 412
    },
    "sitemaps": [
      "https://example.com/sitemap.xml"
    ],
    "llms_txt": {
      "present": false,
      "status": 404
    },
    "pay_per_crawl": false,
    "blocks_any_ai_agent": true,
    "ai_training_allowed": false,
    "blocked_agents": [
      "GPTBot",
      "CCBot"
    ],
    "agents": [
      {
        "agent": "GPTBot",
        "vendor": "OpenAI",
        "purpose": "training",
        "allowed": false,
        "rule": "Disallow: /",
        "matched_by": "User-agent: GPTBot",
        "explicit": true
      },
      {
        "agent": "ClaudeBot",
        "vendor": "Anthropic",
        "purpose": "training",
        "allowed": true,
        "rule": "Allow: /",
        "matched_by": "User-agent: *",
        "explicit": false
      }
    ],
    "text_without_javascript": 812,
    "javascript_dependent": false
  },
  "meta": {
    "credits_used": 1,
    "source": "websites"
  }
}

What you get

Every field in the response.

Field	Meaning
`agents[]`	One verdict per AI agent: allowed, the rule that decided it, whether the site named that agent, and any crawl delay
`blocked_agents, ai_training_allowed`	The short answer, for a dashboard or a gate in your pipeline
`robots, llms_txt, ai_txt`	Whether each file exists, with status and size
`sitemaps, content_signals`	Sitemaps declared in robots.txt and any content-signal line
`pay_per_crawl`	True when the site answers 402 Payment Required
`text_without_javascript`	Words of readable text before any script runs, so you know if a crawl is worth it

How it works

What happens between your request and the response.

Four small fetches

robots.txt, llms.txt, ai.txt and the homepage. Nothing else is touched.

Evidence, not opinion

Every verdict carries the line it came from and whether the site named your agent or fell back to the wildcard group.

Any agent, any path

Ask about one agent, or about the path you actually intend to fetch rather than the site root.

A refusal is the answer

If a site refuses us, that is reported. We do not retry from a different address to get a better answer.

Pricing

One credit system for every endpoint.

One credit per domain checked. The same plans and credits as everything else.

See all plans

Questions

Things people ask about the AI Crawler Checker.

Who needs this?

Anyone running a crawler or ingesting the web for a model, and anyone who has to prove afterwards that they respected a site's rules. It is also the fastest way to answer why an AI assistant cannot see a customer's site.

Which agents do you check?

The ones people ask about: OpenAI, Anthropic, Google, Perplexity, Common Crawl, ByteDance, Amazon, Apple, Meta, Cohere and others, each with what it is for. You can also name any agent yourself.

Does a verdict mean I am allowed to crawl?

It means the site's own files say so on the day you asked. Terms of service, copyright and local law are separate questions, and this is not legal advice.