STALE: OUR MEASUREMENT

Our published benchmark result was measured with v0.41.2, 5 releases ago, past this project's own 2-release window, and the newest real-model measurement, dated 2026-09-18, is past this project's own 7-day window. That is our own number, not your assessment, which runs against your policy on the current release. Why, and what unblocks it

ProvaelGitHub, star the repositoryQuickstart
LEADERBOARD · SUBMISSION PATH

Put your policy on the leaderboard yourself.

Run the same attack suite we ran, check the report with the script our CI uses, and open a pull request. Your row then sits on the public leaderboard under your name. No account, no NDA and no permission needed: the tool is Apache-2.0 and the validation runs on a CPU.

Submit a policy →The four steps
4 steps · one pull requestCPU validation · no GPU to review0 third-party submissions so far

Nobody outside this project has submitted a result yet. Every row on the board today was produced by the maintainer, and the published board file labels each one maintainer-run in its provenance field. Yours would be the first labelled third-party-submission.

The path below has been tested: it was exercised end to end on 12 August 2026 against a dummy submission, which was then deleted. Step 4 of that test correctly refused to promote a stub run to the real board.

Before you start

What do I need?

A policy you can load in Python, and nothing else. Two shapes are supported. A checkpoint, through one of the adapters runnable today: SmolVLA, pi0, pi05 and pi0fast. Or a hosted endpoint, through the openpi client, where the weights stay on your GPU server and Provael only sends observations and reads back actions; that is the path if the checkpoint cannot leave your infrastructure. GR00T, OpenVLA and openpi are registered scaffolding today: no checkpoint has been run through them here yet. If your policy is neither shape, the adapter is one class with three methods, and we would rather you opened an issue than guessed.

What exactly do you run against it?

A matrix of attacks against a simulator suite, one episode per (task, attack, seed) cell, each paired with a benign twin at the same task and seed. Nothing touches your weights and nothing is fine-tuned. The attacks change the instruction or the observation, the simulator runs the episode, and the predicate records whether the policy left its declared safe zone. You choose the suite and the attack families on the command line.

How long does it take?

The ten-task reference run of 9 August 2026 was 400 episode records (350 measured), 15.4 GPU-hours, $12.29 on a rented L4, sharded one task per container so it finished in about two hours of wall clock. Cost is roughly $0.031 per episode and that figure is measured, not projected: the estimate beforehand was $10.17, so budget with the correction rather than the estimate. A smaller screen is proportionally cheaper. The CPU validation step takes seconds.

What do I get back?

A report.json that is a pure function of your config and seed, plus an attack-success rate with its n, a 95% interval and the benign control arm beside it. Alongside it: SARIF for code scanning, OSCAL assessment results, a CycloneDX ML-BOM, and a scorecard. Null results are in there too, at the same size as the hits.

Who sees the result?

Until you open a pull request, only you. The run happens on your machine or in your CI. The free tool has six core dependencies and none of them is an HTTP client, so it has nothing to phone home with and does not. There is no account, no API key and no telemetry. Once you open the pull request the report is public, because the board's whole claim is that a row can be checked.

Can I see it before it is published?

Yes. Publication is a separate, later and entirely voluntary act. You run the suite, you read the complete result, and then you decide whether to open a pull request. If the number is bad, do not open it: nothing has been transmitted, nothing is pending, and nobody is waiting on you.

A submitted row is also withdrawable: an open pull request can be closed, and a merged row can be removed by another one. Your row carries provenance: third-party-submission with your name on it, so it reads as yours rather than ours. We will not publish a result you did not choose to publish, or keep a copy of one you decided against.

The four steps

  1. Run the suite against your policy

    Provael needs an adapter for your policy. Eight are registered: SmolVLA, pi0, pi05 and pi0fast are runnable today; GR00T, OpenVLA and openpi are registered scaffolding that has never had a checkpoint run through it here; plus a deterministic stub. If yours is not one of them, open an issue before you start rather than after. The command below runs on a laptop CPU in under a second, so you can walk the whole route before deciding whether your own policy is worth the GPU time. Swap --policy stub --suite stub for your own once it works.

    pip install 'provael[attest]'
provael attack --policy stub --suite stub \
  --attacks none,instruction --episodes 10 --seed 0 --out results/my-run

  2. Validate the report locally

    The same script CI runs. It checks that report.json parses as a RunReport and that its aggregate rates and success counts are internally consistent. It needs no GPU, so a failure here is one you fix in seconds rather than in review. Note the none arm in step 1: a result without a benign control is refused, because an attack rate with no floor cannot tell a policy the attack broke from one that was failing anyway. Our own rows meet the same bar.

    python scripts/validate_submission.py results/my-run

  3. Build the board row with your provenance

    Two flags matter. --submitted-by is you, and --provenance third-party-submission is what separates your row from ours. Every row on the board today carries provenance: maintainer-run; yours will carry third-party-submission, on the row.

    provael leaderboard build --runs results/my-run \
  --submitted-by your-name --provenance third-party-submission

  4. Open a pull request that adds results/

    CI picks up any pull request touching results/**, revalidates every report and rebuilds the board. Nothing is taken on trust: a stub run is refused from the real board by design, so a submission that never met a real model cannot become a public number.

What gets checked, and what does not

Submission checks · what CI enforces on every pull request that touches results/
CheckedNot checked
The report parses as a valid RunReportThat you ran what you say you ran. Provenance is declared, not proven
Aggregate rates agree with the underlying success countsThat your policy is safe. An ASR is a measurement, not a verdict
The leaderboard still builds with your row in itThat your result transfers to hardware. Nothing here has run on a robot
Stub runs are refused from the real boardStatistical power. Few seeds means a wide interval, and it will show

A submitted row is evidence you produced, published under your name, with its provenance on the row. It is not certification, not a conformity assessment, and not an endorsement by this project. If you would rather someone else ran it, that is what an assessment is for.

Ready to submit?

The submission guide walks the pull request. The validation script, the adapters and the recipe behind our own published row are in the same repository.

Submit a policy →Get the tool

Stuck on an adapter, or is your policy not one of the registered ones? Email hello@provael.com and say what you are running. An adapter that does not exist is a gap worth knowing about.