Four OpenAI Agents. Only One Decides If You Get Quoted

01Four agents, four jobs, one line in your file

Most robots.txt files contain a single rule about OpenAI, and that rule treats the company as one animal with one appetite. It is not one crawler. It is four, they are sent for four different reasons, and two of them will never read your file at all. If you have ever blocked AI and been surprised by what happened next, or by what did not, this is usually where the answer is sitting. Most robots.txt files treat OpenAI as one animal. It is four, and they want different things.

The two that ask permission

GPTBot collects pages that feed training. OAI-SearchBot builds the index that ChatGPT search consults when a person asks a question. Both are documented, both obey robots.txt, and both will do exactly what your file tells them to do. That makes them the only two you genuinely control, so every decision worth making is a decision about these two.

The two that do not

ChatGPT-User is the live fetch that happens when somebody in a conversation points at a specific page and the model goes to look at it. OAI-AdsBot validates advertiser landing pages, which is what I wrote about yesterday. Both are exempt from robots.txt for the same stated reason: the request originated with a person rather than with automated discovery. Exempt is not the same as sneaky. It means your file is not the lever, and pretending otherwise just leaves you with a policy that does not describe what is happening.

02Only one of them decides whether you get quoted

GPTBot and OAI-SearchBot look nearly identical in a log file and do close to opposite things. Whatever GPTBot takes shows up, if it ever shows up, inside a model that has not been released yet, on a timetable nobody outside the company publishes. Whatever OAI-SearchBot takes is available the moment somebody asks a relevant question. Blocking GPTBot costs you training. Blocking OAI-SearchBot costs you today.

Training is a bet on a model that does not exist yet. Search is a bet on this afternoon. Both bets are legitimate. They are simply not the same bet, and one line in a text file has been quietly making both of them on your behalf.

Why this particular mistake is so easy to make

Nobody sits down and decides to be invisible in AI search. What happens is ordinary. Somebody reads a reasonable argument about not handing their work to a training set for nothing, opens the robots file, and writes the block wide, either as a wildcard or as a list of every agent name they could find. The training half of that is a real position and I have no argument with it. The search half is self harm, and it is silent, because a refused crawler never sends a notification and nothing in any dashboard turns red. It is the same failure as not knowing which lane you were in, arrived at from the opposite direction.

Nobody blocks the search crawler on purpose. That is exactly why it stays blocked. Order matters here too, because you cannot be selected out of a pool you were never in. All the reasoning about how a model picks who to recommend runs on candidates it can already see, so a blocked search crawler does not lower your odds. It removes you from the drawing.

03Ten minutes with the file you have not opened

This is small, unglamorous work, and it is the highest return thing most sites can do this month because it costs nothing and the downside it removes is total.

Read your own file before you change a word of it

Open robots.txt and find every rule that could reach these four names, and pay closest attention to the wildcard ones, because a broad rule written for scrapers is usually the actual culprit rather than any line that mentions an agent by name. Then make the decision deliberately. GPTBot is genuinely yours to call and either answer defends itself in public. OAI-SearchBot should be allowed unless you can state the reason out loud and still believe it.

Verify by address, not by name

A user agent string is just a text field, anything on the internet can send one, and scrapers routinely claim to be crawlers they are not in order to inherit the access those crawlers are given. Each of these four publishes its own address file, so the only check that settles an argument is whether the request came from a published range, not what it announced itself as. A user agent string is a claim anyone can type. An address range is a fact you can check.

Two of the four will visit whatever you write. One is a real decision with a real cost in either direction. One decides whether you are in the answer at all. Knowing which is which takes ten minutes, and most people have never spent them.

Tomorrow: shipping a server that lets an agent use your tool instead of merely quoting it.