Field notes on testing robot policies.
Dated articles on red-teaming robot policies and on what Provael’s numbers mean, including mistakes found in Provael’s own work. A note’s numbers are as of its date unless it says otherwise; the latest published result is on the results page.
What an 88% attack success rate means
Written about the 9 August 2026 run (44 of 50). Since 18 September the published result is 42 of 50 [62-100 %] vs 1 of 50 benign; see the update at the top.
Read note →We printed a JS object where the signing key goes
/leaderboard rendered an object where the signing key goes, in the sentence asking you not to take our word for it. The signature stayed valid; the page didn't.
Read note →We were blocking the crawlers we built for
Our CDN's default robots.txt told ClaudeBot, GPTBot and Google-Extended to stay out, two lines above our own file welcoming citations. Fixed, written down.
Read note →Two tens: the device layer and the policy layer
A December 2025 paper enumerates ten vulnerabilities in the Unitree Go2 stack. Provael's Top 10 also has ten. Not competing lists — different layers.
Read note →Why Provael publishes null results
July 2026, about a single-task n=10 run. Re-measured on ten tasks: 42 of 50 [62-100 %] vs 1 of 50 benign (published 18 September 2026); the boxes on top say so.
Read note →The embodiment gap: what text red-teaming misses
Tools like garak and PyRIT scan what a model says. A robot policy turns language into motion, and that is a failure text-only red-teaming has nothing to score.
Read note →
One email a month: the results that landed, the errata, and what is next. Nothing else, no spam, unsubscribe in one click.