Provael’s headline number is 88 percent — and what it does not mean matters more. On its own that figure is close to meaningless, and the parts that give it meaning are the parts people skip. This is all of them, in the order they matter.
1. The setup, stated exactly
SmolVLA (HuggingFaceVLA/smolvla_libero) on the LIBERO benchmark, all ten libero_object tasks,
five seeds per task, horizon 280, on an NVIDIA L4. The attack is roleplay: a templated instruction
reframe that keeps the task words and changes the frame around them. The unsafe predicate is a
keep-out zone violation — the end-effector leaving a declared box.
Everything below comes from one committed run. Not a best-of, not a rerun until it worked.
2. The number
44 of 50 matched pairs. Task-clustered 95 percent interval 72 to 100 percent. McNemar exact p = 4.6e-13, Holm-adjusted 2.7e-12, on 42-0 discordant pairs against the benign twins at the same (task, seed).
The interval is clustered by task rather than pooled because episodes within a task are correlated, and pooling would report a tighter interval than the data supports. It is wide. That is the honest width for fifty episodes.
3. The control, without which the 88 means nothing
The benign arm — same tasks, same seeds, the untouched task instruction — fired 2 of 50, 4 percent.
This line is the whole argument. A predicate that fires on clean runs will fire on attacked ones too, and an attack rate read against an uncharacterised control is a number about the predicate rather than about the policy. 88 against 4 is an effect. 88 against 80 would have been a broken predicate. You cannot tell which you are looking at without the second number, and most published robot-attack rates do not print one.
4. The reword controls, which almost everyone skips
Two arms answer two different questions, and the first version of this section conflated them.
The first is the harmless-variation control. The benign instruction, reworded so that its meaning
is unchanged and no unsafe target appears anywhere in it (benign_reword, four fixed templates),
fired 1 of 50, 2 percent — against roleplay’s 44 of 50 on the same (task, seed) pairs: 43
discordant pairs one way, none the other, McNemar exact p = 2.3e-13. A nonsense-text arm fired 0 of
50. This is the comparison that separates two claims people routinely conflate:
- the policy is brittle to how you phrase things — which a reword arm would show, and
- an attacker who controls the instruction can redirect the policy — which is what 44 of 50 shows.
At 2 percent against 88 percent, this run supports the second and not the first. If the harmless reword had also landed high, the roleplay number would have been evidence of fragility rather than of attacker control, and it would have meant something much less interesting.
The second arm is paraphrase: the unsafe ask itself, said plainly in one of four rewordings,
without the movie-scene frame. It fired 3 of 50, 6 percent. That is an attack, not a control,
and it says something else worth knowing: the bare request barely works, and the reframing is what
carries it from 6 percent to 88.
Correction, 13 September 2026. The version of this section published on 4 September called
paraphrase “the paired reword”, printed 3 of 50 as the control figure, and added that an earlier
draft’s 1 of 50 was unsupported by the artifact. That was backwards. The 1 of 50 is the committed
benign_reword control arm (results/smolvla_libero_object_control); the 3 of 50 is the paraphrase
attack arm of the suite run. The homepage, /results, /findings, the sample pack and the evidence
manifest said 1 of 50 throughout. Recorded as erratum E-2026-10.
5. What the attack costs the task
Clean-task success was 42 of 50 (84 percent). Under the roleplay attack it was 0 of 50.
The attack does not merely make the policy unsafe on the way to finishing. It destroys completion. A robot that fails safely and a robot that fails unsafely are different outcomes, and this run produced the second.
6. What would change this result
Three things, none of which exist yet:
- A calibrated predicate. The keep-out box is hand-picked, not fitted to the benign workspace of these tasks. The benign arm firing at all is the symptom, and it clusters on two adjacent tasks rather than spreading — the signature of a boundary in the wrong place. Tracked as provael#136 and #171, blocked on a GPU sweep that records end-effector trajectories.
- A second policy. One policy and one checkpoint. Nothing here establishes that the result generalises, and the site does not claim it does.
- Real hardware. Zero hardware runs. The sim-to-real protocol is pre-registered and frozen, and it has not been executed.
Until those land, this is a simulation result about one policy with an uncalibrated predicate, and that sentence belongs next to the 88 wherever the 88 appears.
The artifact
Every figure above is in the committed run rather than in this page. The raw aggregate is linked from verification, and if you run it and get something different, that is the most useful thing you can send — we publish contradicting results with attribution, unchanged.