The Cache That Hid My Crawlers

01Seventy five percent, overnight

On 13 October I made one change to the agency site: its pages started being kept at the edge of the content delivery network for an hour, so most visitors, human or machine, are answered by the network without the request ever reaching the server. Every deploy clears that copy, and browsers are told to check back each time. It was a speed and load change. A week later I opened the server log to count AI crawlers, as I do, and they had collapsed. ChatGPT's live agent went from 19.3 successful page fetches a day to 4.8. Bing's crawler went from 44.3 to 9.6. A server log only counts the requests that reach the server. Put a cache in front of it, and your crawlers vanish from the record, not from your site.

Nothing had changed except the counting

Before reading anything into it I checked the other columns. Full page responses from the server fell from 787 a day to 43. Revalidations, the short exchanges where the network asks whether its copy is still good and the server answers yes, rose from 2 a day to 666. The server sent 1.0 MB of pages a day instead of 19.6 MB. The machines had not left. They were being served from the copy, and the copy keeps its own records, not mine.

02Every trend line crossing 13 October

Comparisons that are now false

Every crawler count in this archive was taken at the server, including the scoreboard in sixty eight entries in and the method in how to know if AI is reading your site. For that site, any comparison across 13 October now measures the cache, not the crawlers. A chart of ChatGPT fetches drawn from the log would show a 75% fall on the day the site got faster. A trend line that crosses an infrastructure change is two different measurements glued together.

It touches this batch too. The entry on the 17th counted ChatGPT fetches over eleven days, four of them after the change, so its totals are a floor. The order it reported does not depend on the cache: in the seven clean days before the change alone, the same three pages led, the homepage with 47 fetches, the algorithm record with 36 and the careers page with 25.

The crawler that did not move

One agent held perfectly still: PerplexityBot fetched 15.6 pages a day before and 15.6 after. Googlebot fell by about a third and Amazon's crawler by a seventh. I do not know why the effect differs so much by agent. It may be how often each returns to the same page, whether it arrives inside the hour a copy is kept, or how it asks. I will not dress a guess up as a finding. When a measurement changes for reasons you cannot see, report the change and the ignorance together.

03Measure where the requests land

Mark the date, then move the counter

If you count machines from your server log, write down the day you change anything in front of it: a cache rule, a firewall rule, a new network. Then take the counts from the place that sees every request, which after a change like this is the network's own analytics or its request logs, not the server. Caching changes are also a moment to remember the lesson in the cache that served my old reputation: a copy at the edge can be wrong for as long as it lives, so clear it on every deploy, as this setup does. If you move where requests are answered, move where you count them.

Was it worth it?

Yes. The server now sends about a twentieth of the page bytes it used to, and most visitors are answered from a nearby copy. The cost was a broken chart, which is cheap if you notice and expensive if you do not. The careful version of the rule is the one I should have applied on the 13th: before changing how a site is served, export the last two weeks of crawler counts, so the before is saved in the same form as the after.

Tomorrow: the careers page that kept hiring for roles we had already filled.