How do I know if AI is crawling my site?

Open your raw server access log, not your analytics. Search the user agent field for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and PerplexityBot. Analytics tools miss machines entirely, because bots do not run tracking scripts.

What is the difference between GPTBot and ChatGPT-User?

GPTBot studies your site for training, so its visits shape what the model remembers about you. ChatGPT-User fires in real time when a person asks a question and the model fetches your page to answer it. One builds memory, the other is a live citation.

Should I block AI crawlers in robots.txt?

Never block ChatGPT-User or OAI-SearchBot unless you want to disappear from AI answers. Blocking training bots like GPTBot is a real choice with real tradeoffs, and it deserves its own entry, tomorrow.

How to Know If AI Is Reading Your Site Tonight

¶ CONTEXTTHE LOYAL READER

01The reader who never misses a day

I run a small charity site. Its human traffic is modest, a few dozen visits on a good day. Last month I opened its raw server log for an unrelated reason and found a visitor with perfect attendance.

ChatGPT fetches that site 13 times a day. Every day. It reads pages no human has opened in weeks. On one of my sites, ChatGPT is a more loyal reader than any human subscriber.

Here is the part that should bother you. Your analytics never saw those visits, because analytics scripts count browsers, and machines do not run them. Google Analytics showed a quiet site. The access log showed a busy one.

Once I saw it on one domain, I checked every site I run. The machines were everywhere. Reading quietly, on schedule, every single night.

And the loyal reader pays for its subscription. When people ask ChatGPT about the cause that site serves, the model answers from those quiet nightly reads. The visitor my analytics never counted became the loudest referrer I have.

¶ DATAWHO IS AT YOUR DOOR

02Six names, three very different meanings

AI visitors introduce themselves in the user agent string of every request. Six names cover most of what you will find, and they do not mean the same thing at all.

GPTBot is OpenAI’s training crawler. A visit means your pages are being studied for the model’s long term memory. ChatGPT-User is different: it fires the moment a real person asks ChatGPT something and the model goes to fetch your page for the answer. OAI-SearchBot builds the index behind ChatGPT search. ClaudeBot gathers training material for Anthropic’s models. PerplexityBot feeds an engine that cites its sources, and it visits often. Google-Extended is the odd one out: it is not a crawler at all, just a robots.txt token that Googlebot honors for AI training permission.

A GPTBot visit means the model is studying you. A ChatGPT-User visit means a human just asked about you. One builds memory. The other is a live citation, happening right now.

The live kind adds up faster than you think. On my agency site I counted the AI visitors across 15 days: ChatGPT-User came out on top with 402 fetches, and a single article was fetched 70 times. That is 70 moments where a person asked a question and read an answer with my name inside it.

Thirteen visits a day from a machine that quotes you is better distribution than most newsletters.

EXTRACTED — THE SENTENCE THIS ENTRY EXISTS FOR

Your analytics counts humans. Your access log counts the future.

¶ PLAYBOOKTONIGHT

03Ten minutes with your access log

You do not need a tool or a subscription. You need the raw log and two commands.

[1]

Find the log. On nginx it lives at /var/log/nginx/access.log. On Apache, /var/log/apache2/access.log. On shared hosting, look for Raw Access in cPanel. If your host only offers analytics dashboards, you are blind to all of this. Ask for the raw log.

[2]

Run the two lines below. The first counts visits per AI bot. The second shows which pages ChatGPT fetched for real people, most asked first.

// paste in your terminal — who visited, and what they read

# count visits per AI bot
grep -ioE "gptbot|chatgpt-user|oai-searchbot|claudebot|perplexitybot" access.log | sort | uniq -c | sort -rn

# which pages ChatGPT fetched for real people, most first
grep -i "chatgpt-user" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

[3]

Read what they fetch, not just how often. The pages a machine returns to are the pages it trusts you for. My most fetched article told me exactly what to write next.

[4]

Watch the rhythm for a week. A steady daily visit is indexing. A burst on one page means people are asking about that topic right now.

[5]

Do not block the answer engines. Blocking ChatGPT-User or OAI-SearchBot removes you from live answers, which is the whole prize. Training bots are a genuine choice with tradeoffs, and that decision gets its own entry tomorrow.

You cannot welcome a reader you do not know you have. Open the log tonight. Tomorrow we write the robots.txt that rolls out the red carpet.