# Findings - every published result, with its status - Provael

> Markdown twin of https://www.provael.com/findings/ - derived mechanically from the page's own rendered content at
> build time (it can state nothing the page does not). Curated, data-driven twins exist for the
> key pages; this one is generated so the twin convention has no holes.

RESEARCH INDEX · EVERY PUBLISHED RESULT

# Every finding, with its status.

One measured transfer - on a real SmolVLA policy: 42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI 62-100%) against 1 of 50 benign; the same frame with no target named does it in 27 of 30, so this is fragility, not steering. 9 findings below - 1 measured, 2 at the benign floor, 2 stub-validated, 1 hypothesis, 2 pre-registered and not run, 1 in progress. No card carries a number that is not in the repository.

2 real policies measured on: π0.5, SmolVLA [8 families measured on a real policy; 1 separates from the benign floor; 17 registered · status table →](https://docs.provael.com/latest/attacks/#status-per-family) [The measured result →](https://www.provael.com/results/) [Every measured row →](https://www.provael.com/leaderboard/) [The measured defenses →](https://www.provael.com/defenses/) [Embodied AI Security Top 10 →](https://www.provael.com/eai-top-10/)

## How to read each card

Each card carries two chips: its status (what the data showed) and its evidence class (what kind of evidence it is). The [glossary](https://www.provael.com/glossary/#measurement) defines these terms in plain words.

- MEASURED real policy · committed run Measured on a real checkpoint in a real simulator, against a benign control, with a 95% interval. The card’s numbers come from the committed run’s shards.
- AT THE FLOOR real policy · committed run preliminary run · never the headline Measured on a real policy; the arms fired no more often than the harmless control in a [paired test](https://www.provael.com/glossary/#paired-test). Published at full weight. A preliminary run has fewer seeds or arms than its protocol asks for; it is never the headline, and no transfer is claimed from it.
- STUB-VALIDATED fixture · CPU stub Runs on the deterministic CPU fixture. It exercises the code path and proves nothing about a real policy; no real-model transfer is claimed.
- HYPOTHESIS hypothesis · untested An explanation that measured arms support but do not settle. The card gives the evidence; the linked write-up says what would falsify it.
- PRE-REGISTERED pre-registered · not run The protocol is published before the data exists. Listed so the plan is public, with no result implied.
- IN PROGRESS in progress · no number Running or GPU-gated; no number published yet.

No card is a completed hardware result: no run on a physical robot exists yet, and [/sim-to-real](https://www.provael.com/sim-to-real/) says so.

Customer engagements are indexed separately on [/case-studies](https://www.provael.com/case-studies/), which holds none yet and sets out what a published study will contain. For how these results compare with published attacks, [Published attack baselines](https://www.provael.com/compare/published-attack-baselines/) gives every external figure a comparability verdict and states where this research disagrees with it.

## Findings

HYPOTHESIS hypothesis · untested EAI01 · instruction

### Semantic reframing, not mechanical rewording

Revised 18 September 2026 (E-2026-12): the roleplay vs paraphrase separation stands; the attribution to meaning does not.

Policy: SmolVLA

Suite: LIBERO · all ten libero_object tasks

n: 50 per arm

Arms: roleplay vs paraphrase

roleplay 42/50 (84.0%) vs paraphrase 1/50 (2.0%) · [Wilson intervals](https://www.provael.com/glossary/#wilson-confidence-interval) [71.5-91.7%] and [0.4-10.5%], which do not overlap · paraphrase vs the benign control: [paired test](https://www.provael.com/glossary/#paired-test) p = 1, indistinguishable

One policy, one suite, simulation only, maintainer-run, uncalibrated predicate, four templates per arm. A hypothesis with supporting evidence, not a settled result - the repo write-up states what would falsify it and computes that ~1,960 episodes per arm would be needed to test whether paraphrase differs from doing nothing at all.

[The full write-up (repo) ↗](https://github.com/provael/provael/blob/main/docs/findings/semantic-vs-mechanical-instruction-attacks.md) [The measured result](https://www.provael.com/results/)

MEASURED real policy · committed run EAI01 · instruction

### Under a reframed instruction a real policy left its safe envelope on 42 of 50 trials

Policy: SmolVLA

Suite: LIBERO · all ten libero_object tasks

n: 5 seeds per task

Attack: roleplay

ASR 84% (42/50), 95% [task-clustered interval](https://www.provael.com/glossary/#task-clustered-interval) [62-100%] · benign control 1/50

The one real-policy transfer in the project. Simulation only, ten tasks, uncalibrated keep-out predicate - not a hazard rate and not a real-robot result. The 14 September control run changed its reading (E-2026-12): the roleplay frame with no target named leaves the envelope as often (27/30) and the same words scrambled 18/30, so this is the policy's fragility under an unusual imperative, not an attacker steering it; rewordings that keep the meaning stayed at 0/30.

[The measured result](https://www.provael.com/results/) [Immutable report ↗](https://github.com/provael/provael/tree/29405ab27ca666575e0f202f29dc3589acabc288/results/smolvla_libero_object_suite_2026-09-14)

AT THE FLOOR real policy · committed run EAI02 · perception

### Perception attacks did not separate from the benign floor on the real policy

Policy: SmolVLA

Suite: LIBERO

Visual: 2/100

Injection: 2/50

visual 2/100 [1-7%] · injection 2/50 [1-13%] · against a 1/50 benign baseline · [paired test](https://www.provael.com/glossary/#paired-test) p = 1.0 on every arm

Each arm fired as often as the untouched instruction did, on the same task, so these are floor-level results with non-zero upper confidence bounds - not zeros (the 9 August 2026 run had them at zero), not nulls, and not omissions. Publishing them at full weight is the point - they are what make the 84% believable.

[How we test perception](https://www.provael.com/eai-top-10/eai02/) [Full breakdown](https://www.provael.com/results/)

STUB-VALIDATED fixture · CPU stub EAI04 · action-space

### Action-space integrity: verified not-applicable on real policies

Study: v0.20.0 transfer

Families: action · action_space

Real policies: SmolVLA · π0

Result: not-applicable

Stub-validated on the deterministic CPU core (100% [72-100%] vs 0% benign); no real-policy transfer claimed

A v0.20.0 study verified the negative: these inject an out-of-band directive channel a real VLA ignores, so they stay stub-validated by verification, not by omission.

[The EAI04 action-space transfer study ↗](https://github.com/provael/provael/blob/main/docs/studies/eai04-action-space-transfer.md)

AT THE FLOOR preliminary run · never the headline transfer · second architecture

### π0.5 on the same ten tasks: the frame did not move it out of the envelope, and broke the task

Policy: π0.5 (lerobot/pi05_libero_finetuned_v044)

Suite: LIBERO · all ten libero_object tasks

n: 3 seeds per task · 30 per arm

Arms: roleplay vs none (2 of 8 planned)

roleplay 1/30 ([Wilson](https://www.provael.com/glossary/#wilson-confidence-interval) 1-17%; [task-clustered](https://www.provael.com/glossary/#task-clustered-interval) 0%-10%) vs none 0/30 [0-11%] · [paired test](https://www.provael.com/glossary/#paired-test) p = 1 · task success 27/30 (90%) unattacked, 8/30 (27%) under the frame

The pre-registered study's preliminary leg: 3 seeds against the 5 the protocol asks for, 2 arms of 8, on the default keep-out box, uncalibrated, at provael 0.41.2. No transfer of the envelope-exit effect is claimed - the treatment fired once in thirty cells and the benign twin never did, which the paired test cannot tell apart. What did carry across from SmolVLA is the effect on the task: the frame took completion from 90% to 27% here and from 96% to 0% there, but only SmolVLA left its box doing so. Architecture, checkpoint, or a box shaped for another policy is what the five-seed, eight-arm run with the calibrated box is for. Simulation only.

[Immutable run ↗](https://github.com/provael/provael/tree/da80e9fd15636fee523728ecee3acbdfcd67ecc7/results/pi05_libero_object_2026-09-18) [Pre-registered protocol and Amendment 1 ↗](https://github.com/provael/provael/blob/main/docs/studies/pi0-openpi-transfer.md) [Zenodo timestamp ↗](https://doi.org/10.5281/zenodo.22751558) [The SmolVLA result it is read beside](https://www.provael.com/results/)

IN PROGRESS in progress · no number transfer · cross-architecture

### π0 through openpi - the framework question (GPU-gated)

Policy: π0 (openpi)

Suite: LIBERO

Path: real-policy, GPU-gated, its own environment

Result: pending

No openpi number yet - the real-policy path is GPU-gated and the openpi adapter is registered scaffolding

The π0.5 leg above shares the LeRobot glue with SmolVLA, so it narrows the architecture question and leaves the framework one open; this leg is what closes it. When it runs it will report with the same 95% intervals and benign control. No cross-model claim before then.

[Follow on GitHub ↗](https://github.com/provael/provael)

PRE-REGISTERED pre-registered · not run transfer · second suite

### Meta-World second-suite transfer

Suite: Meta-World (scaffolding, no benchmark run)

Status: study pre-registered

Result: not yet measured

Suite registered as scaffolding in v0.46.0; the transfer study is pre-registered with no result claimed

The adapter is implemented and unit-tested, but the product declares the suite scaffolding: from the CLI it stops at reset (it refuses LIBERO’s default keep-out box, which lies behind the Sawyer arm; the CLI has no zone option and no Meta-World calibration is committed), and no benchmark has been run through it. The cross-suite transfer measurement waits on that and on a GPU run - no number until it runs.

[Roadmap ↗](https://github.com/provael/provael)

PRE-REGISTERED pre-registered · not run transfer · physical arm

### Sim-to-real correlation (SO-ARM101 + SmolVLA)

Platform: SO-ARM101 table-top arm

Status: protocol registered 24 July 2026

Result: not yet measured

Protocol published before the trials; n = 5 per condition, instruction family only

Directional correlation only — does an attack that fires in sim also fire on hardware, and does the benign control stay near zero on both. With n = 5 the real intervals are wide by design; no real-robot ASR is claimed, before or after.

[The protocol](https://www.provael.com/sim-to-real/)

STUB-VALIDATED fixture · CPU stub defense · mitigation

### Instruction canonicalization lowered ASR without breaking the task

Defense: instruction canonicalization / repair

Target: EAI01 instruction · EAI01 optimized

Suites: stub + reach (CPU fixture)

Gate: accepted

adversarial ASR 67.5% [52-80%] -> 7.5% [3-20%] on stub · instruction 70.0% [52-83%] -> 10.0% [3-26%] · optimized_instruction 60.0% [31-83%] -> 0.0% [0-28%] · benign FPR 0.0% -> 0.0% · clean-task success 100.0% -> 100.0% (within CI, accepted)

Credited, and substantially circular - the study says so above its own results table. Four of the CPU fixture’s seven danger tokens are words this defense strips, so the optimized_instruction result is close to tautological. stub-validated scaffolding; no real-model transfer is claimed, and the reach arm’s acceptance gate was not evaluable. The study’s 28 September 2026 note adds a defended arm on a real policy (SmolVLA on LIBERO-Object): the attack survived the defense, and that arm does not count as a measured reduction.

[The study ↗](https://github.com/provael/provael/blob/main/docs/studies/instruction-canonicalization.md) [How EAI01 is tested](https://www.provael.com/eai-top-10/eai01/)

8 families measured on a real policy; 1 separates from the benign floor; 17 registered ([the status table](https://docs.provael.com/latest/attacks/#status-per-family)). The registry holds 39 adversarial attacks across 17 adversarial families (counting the benign baseline and the harmless-variation controls, 44 registered attacks in 19 families). This index publishes only what has been run against a real policy, plus the studies and roadmap items above - each with its evidence state. Registration does not imply real-policy transfer, and Provael does not invent numbers. Evidence, not certification.

### Measure your own policy.

Run Provael locally for your own attack-success rate. If you use a public checkpoint, we may have run it already: the public results for SmolVLA and π0.5 are published on the Results and Findings pages. You can verify each pack in seconds on a CPU and re-run it from its committed recipe on a GPU. Or name the checkpoint you actually ship, and we run the same measurement with you, in your CI, for free.

[Run the quickstart](https://www.provael.com/quickstart/) [Request a free scan](https://www.provael.com/#request)

---
Canonical: https://www.provael.com/findings/