One Speed Test Proves Nothing

01Three runs, three verdicts

On 13 October I spent a day making the agency site faster on phones, and I measured everything with Google's PageSpeed tool through its API. Before trusting any result I ran the same unchanged page three times in the same hour. The about page scored 66, then 83, then 99. The contact page scored 67, then 97, then 66. The pricing page scored 79, 80 and 97. Nothing on those pages changed between runs. The tool simply loads them on a simulated phone over a simulated network, and the simulation moves. A single speed test is an anecdote with a decimal point.

The test that remembers

There was a second trap. Asking the tool about exactly the same address twice in a row returned exactly the same numbers, because it keeps recent results. Two identical runs look like a stable page. They were a cache. Every run after that carried a throwaway parameter in the address so each one was a real, fresh load. Two identical speed results in a row usually mean you measured the same result twice, not the same speed twice.

02The real gains, and the fake one

What a median showed

The work itself was small. The opening text now arrives by moving into place instead of fading up from invisible, which is the same mistake behind an 8.7 second lie: a browser does not count invisible text as painted. Analytics now loads after the page. A strip of partner logos shrank from 594 KB to 85 KB. Measured as a median of three fresh runs, the largest paint on most pages fell from single runs between 8.9 and 10.8 seconds to medians between 2.6 and 4.4. The contact page, which loads a form check twice, stayed slow at a median of 11.2.

The case study that got worse, or did not

One case study page measured 1.7 seconds in a single run before the work and 4.2 in a single run after, with a median of 4.2. Did it get slower, or was the first run lucky? I do not know, because I never had three runs from before. If you only measured once before a change, you cannot know what the change did. Another change was real and bad: starting the homepage animation at a different moment pushed blocking time on desktop to a median of 1.6 seconds over four runs. I reverted it the same hour. Without the median I would have shipped it or, worse, reverted a good change by mistake.

03Speed for people, measured honestly

The site had no field data at all

The same API also reports field data, timings collected from real Chrome users who visit a page. For this site it returned nothing: not for any page, not for the site as a whole, on phone or desktop. Too few visitors share their data for Chrome to publish a number. That changes what the work was for. Google describes its Core Web Vitals in terms of that real user data, and there was none to describe. When there is no field data, speed work is for the people who visit, not for a ranking you hope to move. That is still worth doing. It is just worth being honest about.

The rule I now follow

Three fresh runs before, three after, compare medians, and write down which runs were cached. Check the field data first, because if it exists, it outranks every lab number. On the day, the medians took 30 extra mobile runs across ten pages, and the API refused requests whenever I sent them too fast, so they went two at a time. That is slow and cheap. Defending a change that never happened is fast and expensive. And keep speed in proportion. A fast page that nobody is shown, the situation in seventy five impressions, is still a page nobody sees, and a fast page with no authority still sits on page four, the lesson of perfect technical SEO and still not cited.

Tomorrow: one hundred and twenty nine people use a product through an assistant, and fourteen of them ever opened its website.