AI Crawlers: Who Reads Your Site, and How to Let Them In

// THE SHORT ANSWER

AI crawlers are the bots that read your pages on behalf of answer engines. The ones that matter in 2026 are GPTBot and OAI-SearchBot from OpenAI, ChatGPT-User which fires live when a person asks a question, ClaudeBot, PerplexityBot, and CCBot which feeds the public Common Crawl dataset. You find them in your raw server log rather than in analytics, because bots do not run tracking scripts. Allow them explicitly, and never block the ones that fetch during live answers.

Most site owners have no idea how much of their traffic is already machine. Analytics will not tell you, because analytics counts browsers running JavaScript and these visitors run none. The only honest record is the raw access log, and the first time you read one properly is usually a surprise.

The names matter, because they do not all mean the same thing. A GPTBot visit means your pages are being studied for a model's longer term memory. A ChatGPT-User visit means a person asked something a moment ago and the model went to fetch your page to answer it. One is a deposit. The other is a live citation happening in real time.

That distinction should decide your policy. Blocking training crawlers is a real choice with real tradeoffs, and reasonable people land on both sides of it. Blocking the live fetchers is not a policy, it is a disappearance, because it removes you from answers people are actively asking for.

Access has three layers. robots.txt decides who may read you, and naming each crawler explicitly reads very differently from a silent wildcard. llms.txt describes who you are in your own words, and its value is consistency rather than citations. Schema states the same facts in a form a machine can parse without guessing.

None of it is difficult. It is roughly twenty lines of text, one small JSON block, and ten minutes with a log file. The entries below cover each layer with the exact commands and files I run.

¶ ENTRIESON THIS TOPIC

01The working notes

ENTRY 036~~PLAYBOOK~~Four OpenAI Agents. Only One Decides If You Get QuotedFour crawlers, four jobs, and two that never read your robots.txt. Blocking GPTBot costs you training, blocking OAI-SearchBot costs you today, and most sites block the wrong one.ENTRY 034~~MANIFESTO~~The Toll Booth: When Reading You Costs MoneyHTTP 402 came back from the dead so crawlers could be charged, and new domains now start closed by default. The three lanes a crawler meets at your edge, and which one you are in.ENTRY 031~~PLAYBOOK~~RSL: The Third File AI Readsrobots.txt answers yes or no. llms.txt explains you. RSL states the terms, and it is the first file that can say yes to search and no to training separately.ENTRY 026~~WAR STORY~~The Cache That Served My Old Reputation for Two HoursA robots.txt fixed at the origin in one minute and served broken from the edge for two hours. The five copies of you, how long each one can stay wrong, and the habit that catches it.ENTRY 019~~PLAYBOOK~~Static HTML Wins the AI WarMost AI crawlers never run JavaScript, and the live ones have seconds. The two number test that shows what your page ships before anything executes.ENTRY 007~~PLAYBOOK~~llms.txt: The File AI Reads Before It Trusts Youllms.txt will not get you cited. But a wrong one will get you misquoted. What belongs in it, plus a copy-paste starter.ENTRY 009~~PLAYBOOK~~How to Know If AI Is Reading Your Site TonightChatGPT reads one of my sites 13 times a day and analytics never saw it. Find the AI crawlers in your logs tonight.ENTRY 010~~PLAYBOOK~~The 2026 robots.txt: Roll Out the Red Carpet for AIBlocking AI crawlers in 2026 is deindexing yourself from the future. The VIP allow-list robots.txt, with a copy-paste starter.ENTRY 014~~PLAYBOOK~~Schema Is How You Introduce Yourself to a MachineA page without schema is a stranger. A page with schema is a source. The properties that matter, the two that hurt, and a copy-paste starter.

// QUICK ANSWERS

>Which AI crawlers should I allow?+

Allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot. Naming them explicitly in robots.txt reads as consent, where a bare wildcard reads as silence. Never block ChatGPT-User or OAI-SearchBot, because those fetch during live answers.

>How do I check if AI is crawling my site?+

Open your raw server access log, not your analytics, and search the user agent field for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and PerplexityBot. Analytics misses them entirely because bots do not run JavaScript.

>Does llms.txt help with AI crawlers?+

It helps with accuracy rather than volume. On my own measurement llms.txt earned close to zero citations directly, but a stale or wrong one gets you described incorrectly, so keeping it honest matters more than having it.

AI Crawlers and How to Let Them In

01The working notes