ASR LEADERBOARD · PV-LEADERBOARD

Which VLA policies hold up, and by how much.

Attack-success rate per family, each row carrying its n, its 95% Wilson interval and the benign control it must be read against. Simulation only. Every row here is reproducible from a published recipe, and rows that measured nothing are listed at their measured 0% rather than dropped.

4 rowssmolvla × liberoGenerated 2026-09-14Sim-only · evidence, not certification

Calibrated: no — a default keep-out box, not one fitted per task, so a success is divergence out of that box rather than a hazard rate
Reproducible: no — the policy sampler is not fully seeded, so these are one draw, not a constant
Scope: 1 suite (libero), 1 checkpoint (HuggingFaceVLA/smolvla_libero)
Measured: with 0.32.0, assembled 2026-09-14
Not applicable: mcp_tool_desc produced episode records but zero applicable episodes. Excluded from every denominator — not measured, which is a different claim from measured zero.

Read this before the table

This is 4 rows, not a survey. Every row is one policy on one suite — smolvla × libero — so it compares attack families against each other, not policies against each other. A leaderboard with one entrant is a starting point, and calling it more than that would be the exact overclaim this project exists to avoid.
2 of 3 adversarial families measured 0%. The injection and visual families did not transfer on this policy. They stay in the table at 0%. A leaderboard that quietly drops its failures reports a rate that is not the rate.
The payload is signed, and you should check it rather than trust this page. Ed25519 over the whole board minus the signature field, verifiable offline with no network:provael leaderboard verify --in leaderboard.json --pubkey leaderboard.pubBoth files are in the product repo. Altering a single success count returns INVALID and exits non-zero. That covers origin and integrity — it says nothing about whether four rows are enough, which is the point above.
Assembled 2026-09-14 from commit dd1b874 — but measured with 0.32.0. Those are two different dates and the distinction matters. The board is rebuilt by aggregating committed run reports, so re-running the generator refreshes the stamp without re-measuring anything. The provenance envelope is current; the numbers are the SmolVLA × LIBERO run they have always been. A stamp date read as a measurement date is exactly the overclaim a dated record must not make.
Simulation, not a hazard rate. An attack that redirects a policy in sim is a robustness signal. It is not a real-robot incident probability, and nothing here is a certification.

OLDER TOOL VERSIONMeasured with an older tool version — not a current result

The product publishes its own verdict on this board, and the verdict is that it is behind: rows measured with provael 0.32.0 are 9 minor versions behind 0.41.2 (limit 1); re-run the underlying policy before treating these numbers as current The table is shown because the numbers were true when they were taken, not because they are current. Rebuilding a board re-stamps its date and re-runs no policy, so the provenance below is newer than the measurement.

Attack-success rate by policy, suite and attack family, with 95% Wilson confidence intervals and the benign control.
Policy	Suite	Attack family	ASR	95% CI	Benign control	Successes	Attempts	Transfer	Measured with
smolvla	libero	`instruction`	41.3%	[34–49%]	4.0% (2/50) [1–13%]	62	150	real model	0.32.0
smolvla	libero	`injection`	0.0%	[0–7%]	4.0% (2/50) [1–13%]	0	50	real model	0.32.0
smolvla	libero	`visual`	0.0%	[0–4%]	4.0% (2/50) [1–13%]	0	100	real model	0.32.0
smolvla	libero	`baseline`control	4.0%	[1–13%]	4.0% (2/50) [1–13%]	2	50	real model	0.32.0

Quoting a row? How to cite this work carries the BibTeX and — separately — the release and commit these numbers were produced by, which is not the same version as the current release.

The published contract carries attempts per row, not a separate episode or seed count. There is still no PER-ROW tool version — measured_with is one value for the whole file — so the “measured with” column repeats it only while that list has a single entry, and says mixed rather than picking one when it does not. This table shows what the artifact actually contains rather than inferring the rest. Provenance for the whole file: schema v6 · generated 2026-09-14T10:04:57Z · commit dd1b874 · digest 69396ef8d2b1d93d… · signature ed25519 by key 8d62aa33ed5162f3 (3a1cJIpCYeYMTv4R…, 88 chars — full value in the artifact).

The write-up behind the headline row →Source leaderboard.json ↗Interactive Space ↗

Submissions

Get your policy on it.

Open to any policy anyone can re-run. The bar is reproducibility, not permission: we do not need your weights, we need a run someone else can repeat.

Wondering how a rate on this board compares to the published attack literature? Mostly it does not — see published attack baselines, which gives every published figure a verdict on whether it can be compared to a Provael ASR at all, and names one row it declined to print.

Run the sweep. Install provael and run the full recipe against your policy and suite. The benign control is not optional — a rate without it is not a result.
```
pip install provael
provael attack --policy <your-policy> --suite <your-suite> \
  --recipe full-sweep --out runs/submission
```
Verify it locally first. leaderboard verify checks the report is internally consistent and carries everything a row needs before you send it.
```
provael leaderboard verify --in runs/submission
```
Open a PR adding one file: leaderboard/results/<your-name>/report.json in the product repo. The Leaderboard submission workflow validates it and confirms the board still builds. In the PR description, note the checkpoint, the suite and tasks, seeds, horizon and hardware. Can’t open a PR? The repo carries a Leaderboard submission issue template that does the same job.
```
python scripts/validate_submission.py 'results/*'
```

What gets published, and what does not

Published

Your report.json, in the public repo at the path above.
The row: ASR with its 95% Wilson interval, and the benign (none) control rate beside it — never the rate alone.
The checkpoint identity, suite and tasks, seeds and horizon, so someone else can re-run it.
A real-transfer or stub-scaffolding label, so a fixture run is never quietly mixed in with a real one.

Not published, and not asked for

Your weights. The bar is that someone else can repeat the run, which needs a checkpoint they can fetch — not a model you hand over.
Any observation, image or trajectory from your run. The report carries outcomes and statistics, not the episode contents.
Anything about you beyond the name you put on the directory.

Two things are refused rather than published, and both are refusals of the method rather than of the result: a run nobody can reproduce, and an unseeded run from a stochastic policy — one draw is not a measurement, and a board of single draws cannot support the comparison it exists for.

A result that makes Provael look bad gets published unchanged. If your policy survives the sweep, if an attack family we publish a transfer for does nothing to your model, or if the benign control fires so often that our predicate is obviously drawn in the wrong place — that is the row, and it goes up as measured, under your name and ours. We do not get an edit. The only grounds for declining a submission are the two method failures above, and a decline says which one and why, in the PR. How we’d be caught if that were untrue →

What gets checked. That the run is deterministic at the stated seed, that the benign control is present and its false-positive rate is reported alongside the ASR, that every interval is computed rather than asserted, and that the checkpoint is identified precisely enough for someone else to fetch it. A submission that cannot be re-run is declined — including ours. Full method: docs.provael.com ↗

Want your own policy measured, with the scorecard and the SARIF rather than a table row?

Get your policy measured →

Or do it yourself, for nothing: /submit is the four-step path from a local run to a row on this board.

Which VLA policies hold up, and by how much.

Leaderboard

Get your policy on it.

What gets published, and what does not