The Crawler That Filed Error Reports

01Four thousand reports from a search engine

A CV product of mine uses an error tracking service, and like many sites it sends those reports through its own address, a small route called monitoring, so ad blockers do not swallow them. Every browser that loads a page and hits an error, or records a performance sample, posts a little report there. In the fourteen days to 5 November that route received 77,648 posts. Most came from ordinary browsers. Then I filtered by client. Bing's crawler had posted 4,010 times. Googlebot, 1,497. A crawler that runs your JavaScript runs all of it, including the code that reports your errors.

Nearly a third of Bing's visits

In the same fourteen days Bing's crawler made 9,739 ordinary page requests to that site and 4,017 posts, nearly all of them to the error route. So 29% of everything Bing asked of the site was a report about how its own visit had gone. OpenAI's search crawler posted 123 times, ByteDance's 71, Yandex's 79. Even the in-app browser of Gemini's iPhone app, which carries people who tap a link inside Gemini, appears with 60.

02Rendering crawlers are visitors with odd browsers

Your error dashboard is partly a crawl log

Search crawlers that render pages run a full browser, and a full browser runs every script, the useful ones and the instrumentation alike. That has two consequences. First, the error tracker's numbers include crawlers, so a spike in errors may be a crawl wave rather than users in trouble. Second, the errors crawlers hit are real information: a crawler rendering on an unusual browser version or with scripts timing out sees a different page than your users do. An error that only crawlers trigger is an error in the page they index. It is worth reading those reports separately, not deleting them.

Do not block it blindly

The obvious move is to disallow the route in robots.txt. The decision for this site was not to. With about 1,200 addresses, crawl budget is not the constraint, and blocking a script's endpoint can make a rendering crawler record a failed request where it now records a successful one. The cleaner fix is in the error tracker: tag reports from known crawler user agents and filter them out of alerts, while keeping them in a separate view. The general rule from the referrers that never clicked still holds: a name in a request is a claim, so verify crawler addresses before treating any of this as Bing's doing. Separate crawler noise from user pain before you alert on either.

03A high priority alert that was wrong

Missing from a sitemap it was in

The same week, Bing Webmaster Tools raised a high priority issue on a different product: important pages missing in sitemaps, naming a blog article. The article had been in the live sitemap since the day it launched. Fetched as Bing's crawler, it answered 200. Every one of the 139 addresses in that sitemap was clean, and none of the pages it links to was missing. The alert was wrong. Check the live file as the crawler sees it before you change anything a dashboard tells you to change. Acting on it would have meant editing a working sitemap to satisfy a report that had misread it.

The habit that saves the most time

Both findings came from the same habit: read the raw requests, not only the summaries. Dashboards aggregate, and aggregation hides the client that is doing something strange, whether that is a crawler filing error reports or a report that disagrees with the file it is reporting on. It is the same habit behind the cache that hid my crawlers and the reason I keep a table of every warning and its true cause, as described in write down every search error once. A false alarm goes in that table too, so the next person does not fix a file that was never broken.

Tomorrow: an assistant that asked for the same long job twice, and paid twice.