Provael — findings

HYPOTHESIS: What fires is a changed meaning, not a changed wording

Thu, 03 Sep 2026 00:00:00 GMT

EAI01 · instruction — HYPOTHESIS

roleplay 44/50 (88.0%) [76.2%, 94.4%] vs paraphrase 3/50 (6.0%) [2.1%, 16.2%] - intervals disjoint; paraphrase is indistinguishable from the benign control at McNemar p = 1.0

Policy: SmolVLA
Suite: LIBERO · all ten libero_object tasks
n: 50 per arm
Arms: roleplay vs paraphrase

One policy, one suite, simulation only, maintainer-run, uncalibrated predicate, four templates per arm. A hypothesis with supporting evidence, not a settled result - the repo write-up states what would falsify it and computes that ~1,960 episodes per arm would be needed to test whether paraphrase differs from doing nothing at all.

MEASURED: A reframed instruction drove a real policy off-task on 44 of 50 trials

Thu, 03 Sep 2026 00:00:00 GMT

EAI01 · instruction — MEASURED

ASR 88% (44/50), 95% task-clustered CI [72-100%] · benign control 2/50

Policy: SmolVLA
Suite: LIBERO · all ten libero_object tasks
n: 5 seeds per task
Attack: roleplay

The one real-policy transfer in the project. Sim-only, ten tasks, uncalibrated keep-out predicate - not a hazard rate and not a real-robot result. A paired benign-reword arm fired 1/50, so this is attacker control rather than brittleness to rephrasing.

HONEST NULL: Perception attacks did not transfer to the real policy

Thu, 03 Sep 2026 00:00:00 GMT

EAI02 · perception — HONEST NULL

visual 0/100 [0-4%] · injection 0/50 [0-7%] · same 0/10 benign baseline

Policy: SmolVLA
Suite: LIBERO
Visual: 0/100
Injection: 0/50

A published null with non-zero upper confidence bounds, not an omission. Publishing the zeros is the point - they are what make the 100% believable.

STUB-VALIDATED: Action-space integrity: verified not-applicable on real policies

Thu, 03 Sep 2026 00:00:00 GMT

EAI04 · action-space — STUB-VALIDATED

Stub-validated on the deterministic CPU core (100% [72-100%] vs 0% benign); no real-policy transfer claimed

Study: v0.20.0 transfer
Families: action · action_space
Real policies: SmolVLA · π0
Result: not-applicable

A v0.20.0 study verified the negative: these inject an out-of-band directive channel a real VLA ignores, so they stay stub-validated by verification, not by omission.

The EAI04 action-space transfer study ↗

IN PROGRESS: π0 cross-architecture transfer (GPU-gated)

Thu, 03 Sep 2026 00:00:00 GMT

transfer · cross-architecture — IN PROGRESS

No cross-architecture number yet - the real-policy path is GPU-gated

Policy: π0 (openpi)
Suite: LIBERO
Path: real-policy, GPU-gated
Result: pending

When it runs it will report with the same Wilson-95 + benign-FPR discipline. No cross-model claim before then.

Follow on GitHub ↗

PRE-REGISTERED: Meta-World second-suite transfer

Thu, 03 Sep 2026 00:00:00 GMT

transfer · second suite — PRE-REGISTERED

Suite ships in v0.41.1; the transfer study is pre-registered with no result claimed

Suite: Meta-World (shipped, GPU-gated)
Status: study pre-registered
Result: not yet measured

The suite is implemented and selectable today. What is outstanding is the cross-suite transfer measurement, which needs a GPU run — no number until it runs.

Roadmap ↗

PRE-REGISTERED: Sim-to-real correlation (SO-ARM101 + SmolVLA)

Thu, 03 Sep 2026 00:00:00 GMT

transfer · physical arm — PRE-REGISTERED

Protocol published before the trials; n = 5 per condition, instruction family only

Platform: SO-ARM101 table-top arm
Status: protocol registered 24 July 2026
Result: not yet measured

Directional correlation only — does an attack that fires in sim also fire on hardware, and does the benign control stay near zero on both. With n = 5 the real intervals are wide by design; no real-robot ASR is claimed, before or after.

The protocol

MEASURED: Instruction canonicalization lowered ASR without breaking the task

Thu, 03 Sep 2026 00:00:00 GMT

defense · mitigation — MEASURED

adversarial ASR 67.5% [52-80%] -> 7.5% [3-20%] on stub · instruction 70.0% [52-83%] -> 10.0% [3-26%] · optimized_instruction 60.0% [31-83%] -> 0.0% [0-28%] · benign FPR 0.0% -> 0.0% · clean-task success 100.0% -> 100.0% (within CI, accepted)

Defense: instruction canonicalization / repair
Target: EAI01 instruction · EAI01 optimized
Suites: stub + reach (CPU fixture)
Gate: accepted

Credited, and substantially circular - the study says so above its own results table. Four of the CPU fixture’s seven danger tokens are words this defense strips, so the optimized_instruction result is close to tautological. stub-validated scaffolding; no real-model transfer is claimed, and the reach arm’s acceptance gate was not evaluable.