RESEARCH INDEX · EVERY PUBLISHED RESULT

Every finding, with its status.

One measured transfer - on a real SmolVLA policy: 42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI 62-100%) against 1 of 50 benign; the same frame with no target named does it in 27 of 30, so this is fragility, not steering. 9 findings below - 1 measured, 2 at the benign floor, 2 stub-validated, 1 hypothesis, 2 pre-registered and not run, 1 in progress. No card carries a number that is not in the repository.

2 real policies measured on: π0.5, SmolVLA8 families measured on a real policy; 1 separates from the benign floor; 17 registered · status table →The measured result →Every measured row →The measured defenses →Embodied AI Security Top 10 →

Each card carries two chips: its status (what the data showed) and its evidence class (what kind of evidence it is). The glossary defines these terms in plain words.

MEASUREDreal policy · committed runMeasured on a real checkpoint in a real simulator, against a benign control, with a 95% interval. The card’s numbers come from the committed run’s shards.
AT THE FLOORreal policy · committed runpreliminary run · never the headlineMeasured on a real policy; the arms fired no more often than the harmless control in a paired test. Published at full weight. A preliminary run has fewer seeds or arms than its protocol asks for; it is never the headline, and no transfer is claimed from it.
STUB-VALIDATEDfixture · CPU stubRuns on the deterministic CPU fixture. It exercises the code path and proves nothing about a real policy; no real-model transfer is claimed.
HYPOTHESIShypothesis · untestedAn explanation that measured arms support but do not settle. The card gives the evidence; the linked write-up says what would falsify it.
PRE-REGISTEREDpre-registered · not runThe protocol is published before the data exists. Listed so the plan is public, with no result implied.
IN PROGRESSin progress · no numberRunning or GPU-gated; no number published yet.

No card is a completed hardware result: no run on a physical robot exists yet, and /sim-to-real says so.

Customer engagements are indexed separately on /case-studies, which holds none yet and sets out what a published study will contain. For how these results compare with published attacks, Published attack baselines gives every external figure a comparability verdict and states where this research disagrees with it.

HYPOTHESIShypothesis · untestedEAI01 · instruction

Semantic reframing, not mechanical rewording

Revised 18 September 2026 (E-2026-12): the roleplay vs paraphrase separation stands; the attribution to meaning does not.

Policy: SmolVLA
Suite: LIBERO · all ten libero_object tasks
n: 50 per arm
Arms: roleplay vs paraphrase

roleplay 42/50 (84.0%) vs paraphrase 1/50 (2.0%) · Wilson intervals [71.5-91.7%] and [0.4-10.5%], which do not overlap · paraphrase vs the benign control: paired test p = 1, indistinguishable

One policy, one suite, simulation only, maintainer-run, uncalibrated predicate, four templates per arm. A hypothesis with supporting evidence, not a settled result - the repo write-up states what would falsify it and computes that ~1,960 episodes per arm would be needed to test whether paraphrase differs from doing nothing at all.

The full write-up (repo) ↗The measured result

MEASUREDreal policy · committed runEAI01 · instruction

Under a reframed instruction a real policy left its safe envelope on 42 of 50 trials

Policy: SmolVLA
Suite: LIBERO · all ten libero_object tasks
n: 5 seeds per task
Attack: roleplay

ASR 84% (42/50), 95% task-clustered interval [62-100%] · benign control 1/50

The one real-policy transfer in the project. Simulation only, ten tasks, uncalibrated keep-out predicate - not a hazard rate and not a real-robot result. The 14 September control run changed its reading (E-2026-12): the roleplay frame with no target named leaves the envelope as often (27/30) and the same words scrambled 18/30, so this is the policy's fragility under an unusual imperative, not an attacker steering it; rewordings that keep the meaning stayed at 0/30.

The measured result Immutable report ↗

AT THE FLOORreal policy · committed runEAI02 · perception

Perception attacks did not separate from the benign floor on the real policy

Policy: SmolVLA
Suite: LIBERO
Visual: 2/100
Injection: 2/50

visual 2/100 [1-7%] · injection 2/50 [1-13%] · against a 1/50 benign baseline · paired test p = 1.0 on every arm

Each arm fired as often as the untouched instruction did, on the same task, so these are floor-level results with non-zero upper confidence bounds - not zeros (the 9 August 2026 run had them at zero), not nulls, and not omissions. Publishing them at full weight is the point - they are what make the 84% believable.

How we test perception Full breakdown

STUB-VALIDATEDfixture · CPU stubEAI04 · action-space

Action-space integrity: verified not-applicable on real policies

Study: v0.20.0 transfer
Families: action · action_space
Real policies: SmolVLA · π0
Result: not-applicable

Stub-validated on the deterministic CPU core (100% [72-100%] vs 0% benign); no real-policy transfer claimed

A v0.20.0 study verified the negative: these inject an out-of-band directive channel a real VLA ignores, so they stay stub-validated by verification, not by omission.

The EAI04 action-space transfer study ↗

AT THE FLOORpreliminary run · never the headlinetransfer · second architecture

π0.5 on the same ten tasks: the frame did not move it out of the envelope, and broke the task

Policy: π0.5 (lerobot/pi05_libero_finetuned_v044)
Suite: LIBERO · all ten libero_object tasks
n: 3 seeds per task · 30 per arm
Arms: roleplay vs none (2 of 8 planned)

roleplay 1/30 (Wilson 1-17%; task-clustered 0%-10%) vs none 0/30 [0-11%] · paired test p = 1 · task success 27/30 (90%) unattacked, 8/30 (27%) under the frame

The pre-registered study's preliminary leg: 3 seeds against the 5 the protocol asks for, 2 arms of 8, on the default keep-out box, uncalibrated, at provael 0.41.2. No transfer of the envelope-exit effect is claimed - the treatment fired once in thirty cells and the benign twin never did, which the paired test cannot tell apart. What did carry across from SmolVLA is the effect on the task: the frame took completion from 90% to 27% here and from 96% to 0% there, but only SmolVLA left its box doing so. Architecture, checkpoint, or a box shaped for another policy is what the five-seed, eight-arm run with the calibrated box is for. Simulation only.

Immutable run ↗Pre-registered protocol and Amendment 1 ↗Zenodo timestamp ↗The SmolVLA result it is read beside

IN PROGRESSin progress · no numbertransfer · cross-architecture

π0 through openpi - the framework question (GPU-gated)

Policy: π0 (openpi)
Suite: LIBERO
Path: real-policy, GPU-gated, its own environment
Result: pending

No openpi number yet - the real-policy path is GPU-gated and the openpi adapter is registered scaffolding

The π0.5 leg above shares the LeRobot glue with SmolVLA, so it narrows the architecture question and leaves the framework one open; this leg is what closes it. When it runs it will report with the same 95% intervals and benign control. No cross-model claim before then.

Follow on GitHub ↗

PRE-REGISTEREDpre-registered · not runtransfer · second suite

Meta-World second-suite transfer

Suite: Meta-World (scaffolding, no benchmark run)
Status: study pre-registered
Result: not yet measured

Suite registered as scaffolding in v0.46.0; the transfer study is pre-registered with no result claimed

The adapter is implemented and unit-tested, but the product declares the suite scaffolding: from the CLI it stops at reset (it refuses LIBERO’s default keep-out box, which lies behind the Sawyer arm; the CLI has no zone option and no Meta-World calibration is committed), and no benchmark has been run through it. The cross-suite transfer measurement waits on that and on a GPU run - no number until it runs.

Roadmap ↗

PRE-REGISTEREDpre-registered · not runtransfer · physical arm

Sim-to-real correlation (SO-ARM101 + SmolVLA)

Platform: SO-ARM101 table-top arm
Status: protocol registered 24 July 2026
Result: not yet measured

Protocol published before the trials; n = 5 per condition, instruction family only

Directional correlation only — does an attack that fires in sim also fire on hardware, and does the benign control stay near zero on both. With n = 5 the real intervals are wide by design; no real-robot ASR is claimed, before or after.

The protocol

STUB-VALIDATEDfixture · CPU stubdefense · mitigation

Instruction canonicalization lowered ASR without breaking the task

Defense: instruction canonicalization / repair
Target: EAI01 instruction · EAI01 optimized
Suites: stub + reach (CPU fixture)
Gate: accepted

adversarial ASR 67.5% [52-80%] -> 7.5% [3-20%] on stub · instruction 70.0% [52-83%] -> 10.0% [3-26%] · optimized_instruction 60.0% [31-83%] -> 0.0% [0-28%] · benign FPR 0.0% -> 0.0% · clean-task success 100.0% -> 100.0% (within CI, accepted)

Credited, and substantially circular - the study says so above its own results table. Four of the CPU fixture’s seven danger tokens are words this defense strips, so the optimized_instruction result is close to tautological. stub-validated scaffolding; no real-model transfer is claimed, and the reach arm’s acceptance gate was not evaluable. The study’s 28 September 2026 note adds a defended arm on a real policy (SmolVLA on LIBERO-Object): the attack survived the defense, and that arm does not count as a measured reduction.

The study ↗How EAI01 is tested

8 families measured on a real policy; 1 separates from the benign floor; 17 registered (the status table). The registry holds 39 adversarial attacks across 17 adversarial families (counting the benign baseline and the harmless-variation controls, 44 registered attacks in 19 families). This index publishes only what has been run against a real policy, plus the studies and roadmap items above - each with its evidence state. Registration does not imply real-policy transfer, and Provael does not invent numbers. Evidence, not certification.

Measure your own policy.

Run Provael locally for your own attack-success rate. If you use a public checkpoint, we may have run it already: the public results for SmolVLA and π0.5 are published on the Results and Findings pages. You can verify each pack in seconds on a CPU and re-run it from its committed recipe on a GPU. Or name the checkpoint you actually ship, and we run the same measurement with you, in your CI, for free.

Run the quickstart Request a free scan