Why is my CDN still serving an old robots.txt after I fixed it?

Because the edge is a separate copy with its own expiry, and fixing the origin does not touch it. Purge the file explicitly after every change, and set a short cache lifetime on robots.txt, llms.txt and sitemap.xml, because those three route every other fetch on the domain. Until the edge expires or you purge it, crawlers keep reading the broken version and have no reason to come back.

How do I check what crawlers actually see instead of what I see?

Fetch the public URL from outside your own network with the cache defeated, and read the bytes that come back rather than the file you edited. Then check your server log for what was served during the window you were wrong, because analytics only counts visitors that ran JavaScript and most machines never do.

Can a wrong character set really break robots.txt?

Yes, if it puts a stray byte in front of the first directive. The parser fails to match the line, everything grouped under it is orphaned, and nothing anywhere reports an error. Serve the file as plain UTF-8 with no leading bytes, and verify by fetching the live URL rather than opening the file on the server.

The Cache That Served My Old Reputation for Two Hours

¶ WAR STORYTHE TWO HOURS

01The file was fixed and the internet disagreed

One morning I found a broken robots.txt on a site I run. The first line, the one that names the crawler, was arriving with an invisible byte in front of it, so the parser saw a directive it did not recognise and everything underneath it was hanging off nothing. I fixed it at the server in under a minute, felt fine about my morning, and then fetched the public URL to admire the work. The broken version came back.

The internet does not serve your website. It serves copies of it, and copies age.

What I had actually broken

The file itself was innocent. It was going out with the wrong character set declared, which is enough, because a robots.txt is read by a parser with no imagination. A human skims past a stray byte without noticing. A parser reads the line, fails to match anything it knows, and moves on with no error anywhere for either of us to find.

The two hours I could not see

The origin was correct almost immediately. The edge node answering my requests was not, and it went on handing out the old copy until its own clock said otherwise. Roughly two hours, as far as I could reconstruct afterwards, which is not a long time unless you think about who was reading during it. Every machine that came through that window got the broken file, believed it, and had no particular reason to come back and check.

Nothing reported this. There is no alert for serving the correct file's predecessor. The server was healthy, the response was 200, and the only symptom available to me was that I happened to look twice. Had I closed the terminal after the fix, which is what a reasonable person does, I would have gone on believing the file was fine.

I had not fixed anything. I had fixed one copy out of a number I could not count.

¶ PLAYBOOKTHE COPY MAP

02Every copy of you, and how long it can be wrong

Once you start counting the copies, the panic in the story goes away and something more useful takes its place. There are five of them, they sit at increasing distance from the file you edited, and your control over them drops away faster than most people expect.

Reading it from the top down

The first two rows are infrastructure and you own both of them. The last three are consequences, and they arrive on their own schedule whether or not you were finished. You can purge a cache. You cannot purge a memory, and you certainly cannot purge somebody else's sentence.

The uncomfortable shape of the table is that control and consequence run in opposite directions. The copy you check most often is the one that was already right. The copies that answer questions about you sit at the bottom, where you have no button at all. Two hours of a wrong instruction at the second row is not two hours of damage. It is however long the rows underneath take to notice, multiplied by how confidently they repeat what they took.

// THE COPY MAP~~HOW LONG EACH COPY CAN STAY WRONG~~

Your origincleared by youThe only copy you actually edit, and the only one that was ever correct within a minute of you fixing it. Every check you run by habit reads this one.Seconds

The CDN edgecleared by you, if you rememberNot one copy but hundreds, each on its own clock in its own city. A visitor in another country can be reading a different version of you than the one you just tested.Hours

The search indexcleared by a recrawlWhatever a crawler took the last time it came, sitting in a system you can ask but cannot instruct. This is the copy that feeds most of the answers about you.Weeks

Model memorycleared by nobodyWhat got learned from a snapshot of your site taken long before this morning. No button exists. It changes when the next snapshot is taken, or it does not.Months

Somebody else's pagecleared by nobodyYour old sentence, quoted correctly at the time, living on a domain you do not own. The copy that outlives every fix you will ever ship.Forever

EXTRACTED — THE SENTENCE THIS ENTRY EXISTS FOR

Publishing is not the moment your words become true. It is the moment the copying starts.

¶ PLAYBOOKWHAT CHANGED

03Two settings and one habit

I fixed the character set at the origin so the file could not go out mangled again, and I cut the cache lifetime on the small set of files that machines read before they read anything else. Neither change is clever. Both of them were sitting on a default nobody had ever looked at, which is where this class of problem always lives.

The files that must never be cached for long

Three of them: robots.txt, llms.txt and sitemap.xml. They are not content, they are instructions, and every other fetch on your domain is routed by whatever they last said. Minutes, not days. A stale article is one wrong answer. A stale robots.txt is every wrong answer at once. Purge them by hand after you touch them, and if you have a way of announcing the change yourself, use it in the same breath rather than waiting to be discovered.

None of which is an argument against caching. The edge is why a page from a small server in one country answers instantly in another, and shortening every lifetime on the site would trade a real benefit for an imagined one. The distinction that matters is between the files that are read and the files that are obeyed. Cache the first group hard. Keep the second on a short leash, because a wrong instruction does not sit quietly on one page, it goes and misdirects everything else.

Check the copy, not the original

This is the habit worth more than either setting. Never verify a fix by reading the thing you fixed. Fetch the public address from outside your own network, with the cache defeated, and read what came back rather than what you know is there. Then go one step further and look at what the machines were served during the window you were wrong, which is a question your server log can answer and your analytics cannot.

The origin is where you work. The edge is where you are read.

Tomorrow: why a sentence with a number in it gets quoted, and the adjective next to it gets skipped.