One sentence. 100% off-task.
One reframed instruction drove a real SmolVLA policy off its intended task on every trial. Here is the number, the method, and — at the same size — exactly what it does not mean. Every figure reproduces from a public notebook.
The measured result
· ASR = share of trials driven off-task (10/10)
· CI = 95% Wilson; wide at n=10, by design
· benign FPR = control fired on 0 of 10 clean trials
· attack = roleplay · family = instruction
What this is not.
- Not a real-robot result. Simulation only, one policy, one task (libero_object/0), n=10.
- Not a calibrated hazard rate. The keep-out predicate is uncalibrated — it counts trials diverted out of the benign envelope, not a certified danger threshold.
- Not a benchmark score. Small-n; the CI is wide by design and reported.
- Not a firmware claim. UniPwn-class exploits are out of scope.
- Not a safety certificate. Evidence toward assurance, not a guarantee.
The same run, framed honestly
The 100% headline is one attack (roleplay) on one task. Here is every attack in the same run — including the families that did nothing. Publishing the zeros is the point: they are what make the 100% believable.
| Family | Attack | ASR | Trials | 95% CI |
|---|---|---|---|---|
| baseline | none (benign) | 0% | 0/10 | — |
| instruction | roleplay | 100% | 10/10 | 72–100% |
| instruction | goal_substitution | 60% | 6/10 | — |
| instruction | paraphrase | 10% | 1/10 | — |
| visual | patch | 0% | 0/10 | — |
| visual | decoy_object | 0% | 0/10 | — |
| injection | scene_text | 0% | 0/10 | — |
| instruction | family total | 56.7% | 17/30 | 39–73% |
| overall | all trials | 24.3% | 17/70 | — |
Visual (0/20) and injection (0/10) transferred nothing on this policy — honest nulls, not omissions. One task, n=10, uncalibrated keep-out predicate.
How the number was produced
- Policy: SmolVLA (checkpoint HuggingFaceVLA/smolvla_libero). Simulator: LIBERO · robosuite · MuJoCo.
- Task: libero_object/0. Trials: n=10 seeds per attack, horizon 280, on an RTX 4090.
- Headline attack: roleplay (instruction family) — a reframed goal that still reads as compliant.
- Every attack is paired with a benign control; the benign false-positive rate (0/10) is reported.
- The keep-out predicate is uncalibrated: a success is a trial diverted out of the benign envelope, not a calibrated hazard rate.
- Point estimates carry a 95% Wilson score interval, wide at small n by design.
Consistent with the literature
RoboPAIR (Robey et al., UPenn) reported an algorithmic jailbreak of LLM-controlled robots, 100% across three systems, including physically harmful actions —arXiv:2410.13691. It is cited here as independent evidence that the category is real; it is a different study on different systems, kept visually distinct from Provael’s SmolVLA result.
Run it yourself
The result reproduces from a public notebook — no assessment required. The one command behind the headline:
Reproducing the SmolVLA transfer needs the optional [lerobot] extra and a GPU; the CPU-first stub path lets you exercise the same harness without one.
22 attacks across 11 families ship with the tool; this page publishes the one that transferred to a real policy, alongside the families that returned zero. The rest are stub-validated — run them to measure your own policy. Provael does not invent numbers.