We spent months making this site readable by machines: llms.txt, a full-text dump, a
machine-readable regulatory clock, and a Markdown twin of nearly every page. Then we found our
own robots.txt telling ClaudeBot, GPTBot and Google-Extended to stay out.
It came from a CDN default - Cloudflare’s managed robots.txt - sitting above our own file, and
our file said the opposite two lines below it. The machine-readable rules won: user-agent-specific
Disallow beats a wildcard Allow, so the polite comment about citations being welcome was
overruled by the block above it on every crawl. Citations are the whole point here, so the block
is gone, and the crawler policy is now written down inside robots.txt as a
decision rather than inherited: search yes, AI answer-engine use yes, training-crawler access
stated explicitly instead of left to a vendor default.
This is the same failure shape as the two before it (the lead form that 503’d behind a green
build, the last time managed robots.txt overrode the committed file): the repository was right,
the build was green, and the served origin said something else. Our check:live gate exists for
exactly this class, and it is the thing that should have been run more often.
Update, 8 August 2026: it came back, and the monitor did not save us
Five days later the managed block was above our file again, disallowing ClaudeBot, GPTBot and Google-Extended exactly as before. The zone setting had been re-enabled.
The interesting part is not the recurrence. It is that the check caught it and it made no
difference. check:live runs on a schedule twice a day, it detected the drift, and it exited
non-zero on six consecutive runs. Nobody acted, and here is why:
- The monitor was already red, and had been for a week, for a completely unrelated reason: a Cloudflare KV namespace is unbound, so rate limiting is off on the lead endpoint. Red was its resting state. A job that always fails cannot tell you that something new has failed.
- The alert said nothing. It appended a fixed body — a generic list of “likely causes” — to one long-lived issue. Eleven comments accumulated, each identical apart from its timestamp, none naming which assertion had failed. The alert about the crawler block was byte-for-byte the same as the ten alerts about the KV binding.
- The check classified the block as an editorial choice. Blocking training crawlers is a
legitimate preference, so the code reported it as information rather than a failure. For a site
whose committed file says
ai-train=yesin prose and in its content signal, a block on those bots is not a preference; it contradicts the published policy. That is a bug and it now reads as one.
So the lesson is a level down from where we left it last time. We had written “the check should have been run more often”. It was being run. Detection was never the missing piece — the missing piece was a signal that could still be heard over a chronic failure.
Three changes. Training-crawler blocks now fail the check when the committed file declares
ai-train=yes, derived from that file so flipping the signal relaxes the check on its own rather
than requiring anyone to remember this script exists. Every alert now quotes the assertions that
actually failed on that run. And robots.txt policy has its own single-purpose daily job whose red
and green mean exactly one thing, deliberately not folded into the noisy monitor, because coupling
it to a permanently-failing check is the mistake being corrected. It also closes its own issue when
the drift clears — an alert nobody ever sees resolve is an alert nobody believes.
A monitor that cries wolf about something real, every twelve hours, for a week, is not a working monitor. It is a broken one that happens to be correct.
Two badges that should never have shared one
While we were in there: EAI07 and EAI10 on the Top 10 both carried the identical badge, “out of scope for a VLA-policy red-teamer”. They are not the same thing. EAI07 - firmware, comms, teleoperation - is a real attack surface we do not test in simulation; if you deploy robots, someone else must cover it. EAI10 - evaluation, observability, incident response - has no attack surface at all; a number there would be a category error, and Provael’s own scorecard is evidence for that control, not an attack on it.
The tool has always drawn that line - its catalog defines four distinct coverage states precisely so that a governance meta-risk and an untested attack surface never read as the same kind of absence. The site was not drawing it. The badges are now split, they derive from a mirrored copy of the tool’s own catalog rather than from hand-written strings, and a build check fails if the two ever flatten again.