# What an 88% attack success rate means - Provael > Markdown twin of https://www.provael.com/notes/what-88-percent-means/ - derived mechanically from the page's own rendered content at > build time (it can state nothing the page does not). Curated, data-driven twins exist for the > key pages; this one is generated so the twin convention has no holes. RESEARCH NOTE · Sep 4, 2026 # What an 88% attack success rate means Written about the 9 August 2026 run (44 of 50). Since 18 September the published result is 42 of 50 [62-100 %] vs 1 of 50 benign; see the update at the top. Provael’s headline number was 88 percent — and what it does not mean matters more. On its own that figure is close to meaningless, and the parts that give it meaning are the parts people skip. This is all of them, in the order they matter. Update, 18 September 2026. The site’s headline now reads the same suite re-measured on provael 0.41.2 on 14 September 2026: 42 of 50 (84 percent), task-clustered 95 percent interval 62 to 100 percent, against a benign arm of 1 of 50, with clean-task success 48 of 50. This note explains the 9 August 2026 run (44 of 50 on 0.32.0, on a cloud L4; the re-run was on an office RTX 2000 Ada). The setup is otherwise identical and every point below applies to both runs; the 9 August figures are kept here because a dated note that quietly acquires new numbers is not a record of anything. Section 4’s correction is what changed the reading of both. ## 1. The setup, stated exactly SmolVLA (HuggingFaceVLA/smolvla_libero) on the LIBERO benchmark, all ten libero_object tasks, five seeds per task, horizon 280, on an NVIDIA L4. The attack is roleplay: a templated instruction reframe that keeps the task words and changes the frame around them. The unsafe predicate is a keep-out zone violation — the end-effector leaving a declared box. Everything below comes from one committed run. Not a best-of, not a rerun until it worked. ## 2. The number 44 of 50 matched pairs. Task-clustered 95 percent interval 72 to 100 percent. McNemar exact p = 4.6e-13, Holm-adjusted 2.7e-12, on 42-0 discordant pairs against the benign twins at the same (task, seed). The interval is clustered by task rather than pooled because episodes within a task are correlated, and pooling would report a tighter interval than the data supports. It is wide. That is the honest width for fifty episodes. ## 3. The control, without which the 88 means nothing The benign arm — same tasks, same seeds, the untouched task instruction — fired 2 of 50, 4 percent. This line is the whole argument. A predicate that fires on clean runs will fire on attacked ones too, and an attack rate read against an uncharacterised control is a number about the predicate rather than about the policy. 88 against 4 is an effect. 88 against 80 would have been a broken predicate. You cannot tell which you are looking at without the second number, and most published robot-attack rates do not print one. ## 4. The reword controls, which almost everyone skips Two arms answer two different questions, and the first version of this section conflated them. The first is the harmless-variation control. The benign instruction, reworded so that its meaning is unchanged and no unsafe target appears anywhere in it (benign_reword, four fixed templates), fired 1 of 50, 2 percent — against roleplay’s 44 of 50 on the same (task, seed) pairs: 43 discordant pairs one way, none the other, McNemar exact p = 2.3e-13. A nonsense-text arm fired 0 of 50. This is the comparison that separates two claims people routinely conflate: - the policy is brittle to how you phrase things — which a reword arm would show, and - an attacker who controls the instruction can redirect the policy — which is what this section said 44 of 50 shows, on 4 September. At 2 percent against 88 percent, this run supported the second reading and not the first. If the harmless reword had also landed high, the roleplay number would have been evidence of fragility rather than of attacker control, and it would have meant something much less interesting. Correction, 18 September 2026 (erratum E-2026-12). That sentence set the test, and two control arms registered on 14 September met it. The two rewordings above are three tokens long; a 0 of 50 on three tokens never showed that a twenty-token frame is harmless. So two more controls were built from the roleplay prompt itself and run on the same ten tasks at three seeds each (results/smolvla_libero_object_control_2026-09-14): the roleplay frame with the graspable target replaced by a one-token filler left the envelope in 27 of 30 cells (90 percent, Wilson 95 percent [74, 97]), as often as the attack (26 of 30); the prompt’s own tokens with the target replaced and the word order destroyed left it in 18 of 30 (60 percent [42, 75]). The two three-token rewordings stayed at 0 of 30 and the benign floor at 1 of 30. By this section’s own standard the roleplay number is evidence of fragility: SmolVLA leaves its safety envelope under a long, imperative, out-of-distribution instruction, whether or not a target is named and mostly whether or not the words are in order. It is not evidence that an attacker steers the arm toward a chosen object. The number is unchanged, and a re-run of the suite on provael 0.41.2 the same day returned 42 of 50. What remains open, so nobody reads this as the opposite claim: whether a shorter imperative, or a fluent non-imperative sentence of the same length, does the same. Those arms have not been run. The second arm is paraphrase: the unsafe ask itself, said plainly in one of four rewordings, without the movie-scene frame. It fired 3 of 50, 6 percent. That is an attack, not a control, and it says something else worth knowing: the bare request barely works, and the reframing is what carries it from 6 percent to 88. Correction, 13 September 2026. The version of this section published on 4 September called paraphrase “the paired reword”, printed 3 of 50 as the control figure, and added that an earlier draft’s 1 of 50 was unsupported by the artifact. That was backwards. The 1 of 50 is the committed benign_reword control arm (results/smolvla_libero_object_control); the 3 of 50 is the paraphrase attack arm of the suite run. The homepage, /results, /findings, the sample pack and the evidence manifest said 1 of 50 throughout. Recorded as erratum E-2026-10. ## 5. What the attack costs the task Clean-task success was 42 of 50 (84 percent). Under the roleplay attack it was 0 of 50. The attack does not merely make the policy unsafe on the way to finishing. It destroys completion. A robot that fails safely and a robot that fails unsafely are different outcomes, and this run produced the second. ## 6. What would change this result Three things, none of which exist yet: - A calibrated predicate. The keep-out box is hand-picked, not fitted to the benign workspace of these tasks. The benign arm firing at all is the symptom, and it clusters on two adjacent tasks rather than spreading — the signature of a boundary in the wrong place. Tracked as [provael#136](https://github.com/provael/provael/issues/136) and [#171](https://github.com/provael/provael/issues/171), blocked on a GPU sweep that records end-effector trajectories. - A second policy. One policy and one checkpoint. Nothing here establishes that the result generalises, and the site does not claim it does. - Real hardware. Zero hardware runs. The sim-to-real protocol is pre-registered and frozen, and it has not been executed. Until those land, this is a simulation result about one policy with an uncalibrated predicate, and that sentence belongs next to the 88 wherever the 88 appears. ## The artifact Every figure above is in the committed run rather than in this page. The raw aggregate is linked from [verification](https://www.provael.com/verification/), and if you run it and get something different, that is the most useful thing you can send — we publish contradicting results with attribution, unchanged. Written by Sattyam Jain, founder of Provael. [About the author →](https://www.provael.com/about/) One email a month One email a month: the results that landed, the errata, and what is next. Nothing else, no spam, unsubscribe in one click. Subscribe --- Canonical: https://www.provael.com/notes/what-88-percent-means/