What is a byte order mark and why does it break API keys?

It is an invisible character, U+FEFF, stored as the bytes EF BB BF, that some tools write at the start of text. Glued to the front of an API key it makes the key wrong while it still looks perfect on screen, so authentication fails with errors that often point somewhere else. In my case one provider said the key was incorrect and another said the request had no content.

How do I stop invisible characters getting into secrets?

Set secrets with a command that writes exactly what you give it, such as printf, never with a paste or a shell redirect. Then strip a leading U+FEFF and whitespace in code at the point where the key is used, so a bad paste cannot break production. Check the variable name against the code as well, because a correct key under the wrong name fails silently.

Can an invisible character break robots.txt or llms.txt?

Yes. A stray byte before the first directive in robots.txt can stop the rules beneath it from applying, with no error and a normal status code. Check the first bytes of every file machines obey in a tool that shows raw bytes, and check the copy that is publicly served, not only the one you edited.

One Invisible Byte Killed Two Providers

¶ WAR STORYTHE OUTAGE

01Both providers, at the same moment

On 12 August the chat assistant on one of my sites stopped answering. Every visitor got the polite fallback message that is meant to appear only when something upstream is down. That should have been close to impossible. The assistant runs on two AI providers with a fallback chain between them, built on the reasonable assumption that two separate companies will not fail at the same time.

The fallback that could not fall back

They had not failed. My keys had, both of them, in the same way, which is the one scenario a fallback chain cannot survive. Redundancy protects you from independent failures. It does nothing when both copies share the same mistake. Both keys had been saved through a terminal command that quietly wrote an invisible character, a byte order mark, onto the front of each value. Three bytes, EF BB BF, sitting before the first letter of each key.

// THE HEX VIEW~~WHAT THE DASHBOARD SHOWED VS WHAT WAS SENT~~

ON SCREEN

sk-pr…

looks perfect

IN THE FILE

EFBBBF736B2D…

three extra bytes

// WHAT EACH PROVIDER BLAMED

401Incorrect API key providedPoints at the key. Right layer.

400contents is not specifiedPoints at the request body. Wrong layer.

Same byte, same cause, two providers. One told the truth and one sent me to the wrong file.

¶ WAR STORYTHE ERROR MESSAGES

02One error told the truth

The two providers reported the same problem very differently. The first returned a 401 saying the key was incorrect, which is honest and points exactly where the fault is. The second, reached through a compatibility layer, returned a 400 saying the request contained no content. It had choked on the malformed header, dropped the body, and then blamed me for sending an empty request. An error message is a claim about where the problem is, and claims can be wrong.

Where I looked first

I did what anyone would do with the second message and went looking at the request: the payload, the message format, the tool definitions. All of it was fine, because none of it was the problem. The key looked perfect in the dashboard, perfect in the settings screen, perfect when I copied it out and pasted it back. The worst bugs are the ones your editor refuses to show you. A byte order mark has no width and no glyph. Every screen designed to show you text is designed to hide it.

What finally showed it was the raw live log, where the header appeared as bytes instead of characters and the three extra ones were sitting at the front.

EXTRACTED — THE SENTENCE THIS ENTRY EXISTS FOR

When two independent things fail together, look for the one thing they share.

¶ WAR STORYTHE RULES THAT CAME OUT OF IT

03Never paste a secret again

Write it, strip it, name it

The fix took a minute. The rules took longer, and they are the part worth keeping. Secrets now go in through a command that writes exactly the characters given, with no trailing newline and no encoding marker, never through a redirect or a paste. The code strips a leading byte order mark and surrounding whitespace at the moment it uses a key, so a bad paste can no longer break anything. And the variable name gets checked against the code before a value is set, because setting the right key under the wrong name looks exactly as successful as setting it correctly. Defensive code beats discipline, because discipline fails on the one day you are tired.

The same byte, in a file machines obey

I had met this character before without recognising it. In the cache story, a robots.txt served with the wrong character set carried a stray invisible byte in front of its first directive, and every rule beneath it was silently orphaned. No error anywhere, the file still returned a perfect status code. The lesson is the same in both places. Files that machines obey, whether a crawler reading your robots.txt or a server reading a key, are read byte by byte, and a byte you cannot see still counts.

How to check your own files

Open any file a machine is supposed to obey, robots.txt, llms.txt, your sitemap, your environment files, in a tool that shows raw bytes, and look at the first three. If they are EF BB BF, remove them. Then fetch the public copy from outside, the way you would when checking whether AI can read your site, and check the bytes there too, because the file you edited and the file being served are not always the same file. If you only ever look at text through an editor, you have never seen your file. You have seen a rendering of it.

Tomorrow: a guard against my own product inventing words, and why its threshold had to be measured rather than guessed.