# Reproduce the result, cheapest check first - Provael > Markdown twin of https://www.provael.com/reproduce/ - derived mechanically from the page's own rendered content at > build time (it can state nothing the page does not). Curated, data-driven twins exist for the > key pages; this one is generated so the twin convention has no holes. REPRODUCE · 0 INDEPENDENT REPRODUCTIONS AS OF 1 NOVEMBER 2026 # Reproduce the result, cheapest check first. Start with a twenty-second run on your laptop that shows the tool works. Then check the published result at one of three levels: verify the signed board, rebuild it from the committed reports, or re-run the policy on a GPU. Only the third is an independent reproduction. [Run the quickstart →](https://www.provael.com/quickstart/) What you are reproducing 42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI 62-100%) against 1 of 50 benign; the same frame with no target named does it in 27 of 30, so this is fragility, not steering. Scope: SmolVLA (HuggingFaceVLA/smolvla_libero) × libero_object (all 10 tasks), 5 seeds per task, LIBERO · robosuite · MuJoCo — simulation only, default (uncalibrated) predicate, provael 0.41.2. Twenty seconds ## First, the tool running on your laptop No GPU and no checkpoint of yours. This runs every attack family against a built-in test policy, a deterministic stub, and prints a scorecard shaped like the published one. It shows the tool works; reproducing the result needs the real policy, at level 3 below. ``` pip install provael provael attack --recipe full-sweep ``` Expect a scorecard with an attack-success rate, its interval and the benign control, for every registered family, on the stub Python 3.12+; the container path is docker run --rm ghcr.io/provael/provael:latest attack --recipe full-sweep. The [quickstart](https://www.provael.com/quickstart/) walks the same three commands with the recording. Three levels ## Verify, rebuild, re-run Cheapest first. The first two need no GPU and are real checks: a board whose signature verifies and that rebuilds to the same digest is a board nobody altered. The third re-runs the policy, and it is the reproduction that puts your name in the register. 1 No GPU · under a minute ### Check the published board has not been altered The fastest check there is. It proves the numbers you are reading are the numbers that were signed — including the success counts, which are inside the signature, not published beside it. ``` pip install "provael[attest]" curl -fsSLO https://raw.githubusercontent.com/provael/provael/main/leaderboard/results/leaderboard.json curl -fsSLO https://raw.githubusercontent.com/provael/provael/main/leaderboard/results/leaderboard.pub provael leaderboard verify --in leaderboard.json --pubkey leaderboard.pub ``` Expect leaderboard OK keyid 8d62aa33ed5162f3 2 No GPU · a few minutes ### Rebuild the board from the committed run reports Confirms the published rows are what the committed evidence actually aggregates to. It re-derives the table from the raw reports rather than trusting the table. ``` git clone https://github.com/provael/provael && cd provael git checkout 29405ab pip install -e . provael leaderboard build --real results/smolvla_libero_object_suite_2026-09-14 --out /tmp/rebuild python -c "import json; a=json.load(open('leaderboard/results/leaderboard.json'))['inputs_digest']; \ b=json.load(open('/tmp/rebuild/leaderboard.json'))['inputs_digest']; print('match:', a==b)" ``` Expect match: True 3 GPU required · hours ### Re-run the measurement against the real policy The real reproduction: it re-runs the policy in the simulator rather than re-aggregating a stored result. This is the one that would put your name in the register below. ``` pip install 'provael[lerobot]' 'lerobot[libero]==0.5.1' apt-get install -y libosmesa6 libgl1 libglx-mesa0 # headless GL export MUJOCO_GL=osmesa PYOPENGL_PLATFORM=osmesa PROVAEL_INTEGRATION=1 provael attack \ --policy smolvla --suite libero --model HuggingFaceVLA/smolvla_libero \ --attacks none,instruction,visual,injection \ --seeds 5 --horizon 280 --seed 0 \ --out runs/repro ``` Expect A report.json whose per-family rates you can compare against the published board. The packs that exist ## Public checkpoints with committed runs A pack is a run you can re-execute: the checkpoint, the recipe, the shards and the report at a pinned commit. Two exist. Every other registered backend is listed below with the status the product’s own registry gives it. ### SmolVLA × LIBERO-Object, all ten tasks measured · published The ten-task run behind the headline count above and the per-attack rows on /results, measured with provael 0.41.2. - [The delivery pack (Markdown, the real artifact) →](https://www.provael.com/evidence/provael-sample-evidence-pack.md) - [The committed run at 29405ab →](https://github.com/provael/provael/tree/29405ab27ca666575e0f202f29dc3589acabc288/results/smolvla_libero_object_suite_2026-09-14) - [The reproduction recipe →](https://github.com/provael/provael/blob/29405ab27ca666575e0f202f29dc3589acabc288/results/smolvla_libero_object_suite_2026-09-14/README.md) ### π0.5 × LIBERO-Object - the preliminary leg preliminary 1/30 under the roleplay instruction against 0/30 benign, 10 tasks, 3 seeds against the 5 the protocol asks for, 2 arms of 8; measured 2026-09-18 with provael 0.41.2. Not the headline and not a transfer claim: the opening leg of the pre-registered cross-architecture study, published so the null is a result. - [The committed run at da80e9f →](https://github.com/provael/provael/tree/da80e9fd15636fee523728ecee3acbdfcd67ecc7/results/pi05_libero_object_2026-09-18) - [The pre-registered protocol →](https://github.com/provael/provael/blob/main/docs/studies/pi0-openpi-transfer.md) - [The pre-registration (Zenodo) →](https://doi.org/10.5281/zenodo.22751558) ### Registered, no committed run - π0 (LeRobot) runnable · no committed run - π0-FAST (LeRobot) runnable · no committed run - GR00T (LeRobot) scaffolding · never run - OpenVLA (HF transformers) scaffolding · never run - openpi (websocket client to a π0 policy server) scaffolding · never run Statuses come from the product registry at v0.46.0 (29405ab): “runnable · no committed run” means the adapter installs and has never been measured here; “scaffolding” means the adapter exists and no checkpoint has been loaded through it. A pack appears above the day its run is committed, not before. Report what you found ## How to report a reproduction Use the product’s reproduction-result form. It asks for the level you ran, the commit, the tool version, the environment and what you got, whether it agrees or not. A reproduction that disagrees is the more useful one, and it is published the same way. The register on [/verification](https://www.provael.com/verification/) holds 0 as of 1 November 2026, and that number moves only when someone files one. A result about a policy you did not train gets the same 14-day right of reply Provael gives itself ([/security](https://www.provael.com/security/)). [Report a reproduction →](https://github.com/provael/provael/issues/new?template=reproduction-result.yml) [The register](https://www.provael.com/verification/) No reproduction bounty is funded. If one is, it appears here as a dated line with its amount and terms; nothing is promised before then. --- Canonical: https://www.provael.com/reproduce/