# We were blocking the crawlers we built for - Provael

> Markdown twin of https://www.provael.com/notes/we-blocked-the-crawlers-we-built-for/ - derived mechanically from the page's own rendered content at
> build time (it can state nothing the page does not). Curated, data-driven twins exist for the
> key pages; this one is generated so the twin convention has no holes.

RESEARCH NOTE · Aug 3, 2026

# We were blocking the crawlers we built for

Our CDN's default robots.txt told ClaudeBot, GPTBot and Google-Extended to stay out, two lines above our own file welcoming citations. Fixed, written down.

We spent months making this site readable by machines: llms.txt, a full-text dump, a
machine-readable regulatory clock, and a Markdown twin of nearly every page. Then we found our
own robots.txt telling ClaudeBot, GPTBot and Google-Extended to stay out.

It came from a CDN default - Cloudflare’s managed robots.txt - sitting above our own file, and
our file said the opposite two lines below it. The machine-readable rules won: user-agent-specific
Disallow beats a wildcard Allow, so the polite comment about citations being welcome was
overruled by the block above it on every crawl. Citations are the whole point here, so the block
is gone, and the crawler policy is now [written down inside robots.txt](https://www.provael.com/robots.txt) as a
decision rather than inherited: search yes, AI answer-engine use yes, training-crawler access
stated explicitly instead of left to a vendor default.

This is the same failure shape as the two before it (the lead form that 503’d behind a green
build, the last time managed robots.txt overrode the committed file): the repository was right,
the build was green, and the served origin said something else. Our check:live gate exists for
exactly this class, and it is the thing that should have been run more often.

## Update, 8 August 2026: it came back, and the monitor did not save us

Five days later the managed block was above our file again, disallowing ClaudeBot, GPTBot and
Google-Extended exactly as before. The zone setting had been re-enabled.

The interesting part is not the recurrence. It is that the check caught it and it made no
difference. check:live runs on a schedule twice a day, it detected the drift, and it exited
non-zero on six consecutive runs. Nobody acted, and here is why:

- The monitor was already red, and had been for a week, for a completely unrelated reason: a
Cloudflare KV namespace is unbound, so rate limiting is off on the lead endpoint. Red was its
resting state. A job that always fails cannot tell you that something new has failed.

- The alert said nothing. It appended a fixed body — a generic list of “likely causes” — to one
long-lived issue. Eleven comments accumulated, each identical apart from its timestamp, none
naming which assertion had failed. The alert about the crawler block was byte-for-byte the same
as the ten alerts about the KV binding.

- The check classified the block as an editorial choice. Blocking training crawlers is a
legitimate preference, so the code reported it as information rather than a failure. For a site
whose committed file says ai-train=yes in prose and in its content signal, a block on those
bots is not a preference; it contradicts the published policy. That is a bug and it now reads
as one.

So the lesson is a level down from where we left it last time. We had written “the check should
have been run more often”. It was being run. Detection was never the missing piece — the missing
piece was a signal that could still be heard over a chronic failure.

Three changes. Training-crawler blocks now fail the check when the committed file declares
ai-train=yes, derived from that file so flipping the signal relaxes the check on its own rather
than requiring anyone to remember this script exists. Every alert now quotes the assertions that
actually failed on that run. And robots.txt policy has its own single-purpose daily job whose red
and green mean exactly one thing, deliberately not folded into the noisy monitor, because coupling
it to a permanently-failing check is the mistake being corrected. It also closes its own issue when
the drift clears — an alert nobody ever sees resolve is an alert nobody believes.

A monitor that cries wolf about something real, every twelve hours, for a week, is not a working
monitor. It is a broken one that happens to be correct.

## Two badges that should never have shared one

While we were in there: EAI07 and EAI10 on the [Top 10](https://www.provael.com/eai-top-10/) both carried the identical
badge, “out of scope for a VLA-policy red-teamer”. They are not the same thing. EAI07 - firmware,
comms, teleoperation - is a real attack surface we do not test in simulation; if you deploy
robots, someone else must cover it. EAI10 - evaluation, observability, incident response - has
no attack surface at all; a number there would be a category error, and Provael’s own
scorecard is evidence for that control, not an attack on it.

The tool has always drawn that line - its catalog defines four distinct coverage states
precisely so that a governance meta-risk and an untested attack surface never read as the same
kind of absence. The site was not drawing it. The badges are now split, they derive from a
mirrored copy of the tool’s own catalog rather than from hand-written strings, and a build check
fails if the two ever flatten again.

Written by Sattyam Jain, founder of Provael. [About the author →](https://www.provael.com/about/)

One email a month

One email a month: the results that landed, the errata, and what is next. Nothing else, no spam, unsubscribe in one click.

Subscribe

---
Canonical: https://www.provael.com/notes/we-blocked-the-crawlers-we-built-for/