The Purge That Broke Half the Fetches

01Ten seconds to load, then nothing

On the morning of 21 October I shipped a routine deploy to one of my sites: 101 files, followed, as always, by a command that tells the content delivery network to forget every page it had stored. Two minutes later pages took ten to twenty seconds to load. Some never loaded and showed the network's own error page, the one that says it could not reach the server. A refresh usually worked. That detail turned out to be the whole story. A purge does not break your site. It makes every page depend on the one path you were not watching.

The server was fine

The first thing I checked was the server, because that is where a broken site usually breaks. It was idle. Load at 0.01, the web server running, between one and seven requests a minute arriving, nothing in the error log. Requests sent straight to its address, skipping the network, answered ten times out of ten in 0.6 seconds from my machine and 0.03 seconds from another host. Whatever was failing, it was failing before the request arrived.

02Every other request

Alternating, exactly

From one edge location the results came back 200, 522, 200, 522, in strict alternation. The failed requests never appeared in the server's access log. A packet capture showed the edge's connection attempts arriving at the network card during the failures, so the requests were reaching the machine and the answers were not getting back. That points at the path between the host and the network, not at either end. The host's own status page listed urgent maintenance on its core routers that week, and a connector change scheduled for that morning. It is the most likely cause. I cannot prove it, because the fault stopped before I could capture the return traffic.

Why the purge made it visible

Before the purge, almost every visitor was answered from the edge copy and never needed the path at all. Cached pages kept loading in about 0.4 seconds throughout. The purge emptied that copy, so every page had to be fetched from the origin, and half of those fetches died on the way. A cache hides a fragile network until the moment you empty it. I had described that trade in the cache that hid my crawlers from the measurement side. This was the same trade from the availability side.

03Warm it, then watch it

Refill the edge yourself

What fixed the visible problem was not a change on the server. It was fetching every address in the sitemap through the network, with retries, until each one answered 200. That is 231 pages; 54 needed a retry and none failed. Each successful fetch put a fresh copy back at the edge, so the next visitor no longer depended on the broken path. By 10:45 the path itself had recovered. Warming the cache turns a network fault into a slow script instead of a slow website. The deploy now does this every time, straight after the purge, for every page it touched.

Rules I now follow

Probe ten uncached requests after any purge, not one, because a single success proves nothing when failures alternate. Do not ship large deploys during the hours people use the site without saying so first. Keep a standby copy that can serve the site from elsewhere if a path stays broken, which this site has. And remember who notices. Crawlers that arrived during those twenty minutes met error pages too, and an assistant that fetches a page live for someone gets the same error that person would. An outage is also a crawl event, and the crawlers do not come back on your schedule. The record of how long a site has been wrong matters, the point of the cost of being wrong, and it matters most for errors you did not see. If you check one thing after reading this, check what your deploy does in the minute after it clears the cache, which is exactly the minute I had never measured. The earlier lesson about edge copies, in the cache that served my old reputation, was about stale pages. This one is about no pages.

Tomorrow: a crawler that visited every new article and stored none of them.