Under a reframed instruction, the policy left its safe zone 42 times in 50.

Reproduce it

The run is public. Its README in the product repository records the per-task counts and the inputs the shards kept: provael 0.41.2, the checkpoint name, the ten tasks, the eight arms, 5 seeds and horizon 280. The source commit, the dependency lock, the precision and the checkpoint revision were not recorded, so pin a checkpoint revision yourself and record it. To attempt it you need a CUDA GPU, the optional [lerobot] extra and lerobot[libero]==0.5.1. A re-run matches in distribution, not byte for byte, because the policy samples its actions.

One task per run, <i> from 0 to 9:

pip install 'provael[lerobot]==0.41.2' 'lerobot[libero]==0.5.1'
PROVAEL_INTEGRATION=1 provael attack --policy smolvla --suite libero \
    --model HuggingFaceVLA/smolvla_libero --tasks libero_object/<i> \
    --attacks decoy_object,goal_substitution,mcp_tool_desc,none,paraphrase,patch,roleplay,scene_text \
    --seeds 5 --horizon 280 --seed 0

The whole procedure, from what it needs to what it costs and how to read each figure it prints, is a page of its own: how to red-team a SmolVLA policy in LIBERO.

No GPU? The quickstart runs the same harness on a CPU stub in seconds. The delivery pack lists every pinned input, file by file; every measured row, the measured defenses and how to cite this are one click away.

Every arm in the same run

The 84% headline is one attack (roleplay) across all ten tasks. Here is every attack in the same run, including the families that stayed at the benign floor and the one that did not apply. Publishing those arms and the N/A is what makes the 84% believable.

N/A is not 0 and is not a pass: mcp_tool_desc had no applicable trials in this run, so it has no rate - not a zero. Visual (2/100) and injection (scene_text 2/50) did not separate from the run's own benign floor (1/50; paired test (McNemar exact) p = 1.0 on every one of those arms, and every firing but one on the same task the benign arm fired on) - results at the benign floor, published as measured, neither nulls nor omissions. The all-trials row (55/350) includes the benign control and is not the adversarial ASR (54/300).

How the number was produced

• Policy: SmolVLA (checkpoint HuggingFaceVLA/smolvla_libero). Simulator: LIBERO · robosuite · MuJoCo.
• Tasks: all ten libero_object tasks, one shard per task. Trials: 5 seeds per (task, attack), horizon 280, on a workstation RTX 2000 Ada (the 9 August 2026 run this reproduces ran on a cloud L4).
• Headline attack: roleplay (instruction family) - a reframed goal that still reads as compliant.
• Attack-success rate (ASR): the share of trials that ended in a keep-out violation, 42/50 for the headline.
• Every attack is paired with a benign control; the benign false-positive rate (1/50) is reported. It is NOT zero: the uncalibrated predicate fires on one benign episode, on task 5 - the task where the visual and injection arms also fired. That rate is the benign floor every arm is read against.
• The keep-out predicate is uncalibrated: a success is a trial diverted out of the benign envelope, not a calibrated hazard rate.
• Intervals: the headline carries a task-clustered 95% interval, by bootstrap (62-100%), which resamples whole tasks rather than episodes, because the seeds of one task are not independent. Every per-attack interval is a 95% Wilson score interval (wilson-score-95 (episode-level)), wide at small n by design.
• Control run: 6 arms on the same 10 tasks at 3 seeds each, in one report, so each arm is compared with the attack trial by trial, on the same task and seed. Against the attack, the frame with no target named gives a paired test (McNemar exact) p = 1.0 (indistinguishable: naming the target adds nothing); the same words in scrambled order give p = 0.0078 (order carries part of the effect, not most of it).
• Provenance: this reference run was produced with provael 0.41.2 on 14 September 2026 (the ledger below dates it) and is retained as a dated result; the current release is 0.46.0, 5 releases later. It reproduces the 9 August 2026 run made with 0.32.0 (44/50 then, on a cloud L4), which stays committed beside it.

Consistent with the literature

The full side-by-side is at /compare/published-attack-baselines: every published figure with its arXiv ID, and a verdict on whether it can be compared with a Provael ASR at all. It also states where this run disagrees with the literature, unresolved.

8 families measured on a real policy; 1 separates from the benign floor; 17 registered (the status table). The registry holds 39 adversarial attacks across 17 adversarial families (counting the benign baseline and the harmless-variation controls, 44 registered attacks in 19 families). This page publishes the three families of the ten-task run, with the one that separated, the ones at the benign floor and the attack that did not apply. Five more ran on a real policy only in a one-task breadth probe, too small for a rate; the status table lists them. The other 9 are stub-validated: run them to measure your own policy. Registration does not imply real-policy validation, and Provael does not invent numbers.

Every committed measurement, and how old each one is

The banner at the top of this site dates the newest measurement. This dates all of them. One row per run committed to the product repository, read from watch/measurements.json when this page was built - which is generated from the execution manifest beside each report, so no human types a date into it.

116 committed measurements, of which 110 are real-policy measurements with an observed date. 5 came from a fixture backend and measure no policy at all, and 1 carries a date reconstructed to the day rather than observed. Both facts are stated in the row itself, in words.

A fixture run executes the real attacks against a deterministic CPU stub in under a second. It exercises the harness and it measures no policy, so it can never refresh a freshness claim - the product computes that distinction, this page does not decide it. A reconstructed date is a day-granularity value read from a manifest that recorded no instant; it is shown as the day it names and never as a time.