Get your policy measured.
Provael is a tool, not a gate. If you have a vision-language-action policy, you can run the same suite we ran, validate it with the same script CI uses, and put the row on the leaderboard yourself. No account, no NDA, no permission needed. The core is Apache-2.0 and the validation runs on a CPU.
Nobody outside this project has submitted a result yet. Every row on the board today was produced by the maintainer and is labelled maintainer-run in its own provenance column. That is said here rather than left for you to notice, because a board that cannot tell a self-reported row from an independent one is a changelog wearing the word leaderboard. Your row would be the one that changes it.
The path below is not theoretical. It was exercised end to end on 12 August 2026 against a dummy submission, which was then deleted. Step 4 of that test correctly refused to promote a stub run to the real board, which is the guard working.
Before you start, in the order people actually ask
What do I need?
What exactly do you run against it?
How long does it take?
What do I get back?
Who sees the result?
Can I see it before it is published?
Yes, and this is the part worth being explicit about. Publication is a separate, later, entirely voluntary act. You run the suite, you read the complete result, and then you decide whether to open a pull request. If the number is bad, do not open it. Nothing has been transmitted, nothing is pending, and nobody is waiting on you.
Two more things, because the honest version of this answer includes them. A submitted row is withdrawable: an open PR can be closed, and a merged row can be removed by another PR. And your row carries provenance: third-party-submission with your name on it, so it is legible as yours rather than absorbed into ours. What we will not do is publish a result you did not choose to publish, or quietly keep a copy of one you decided against.
The four steps
Run the suite against your policy
Provael needs an adapter for your policy. Seven ship today — SmolVLA, pi0, pi05, pi0fast and NVIDIA GR00T through LeRobot, OpenVLA through transformers, and openpi through its own client — plus a deterministic stub. If yours is not one of them, open an issue before you start rather than after.
provael attack --policy smolvla --suite libero \ --attacks instruction,visual,injection --episodes 5 --out results/my-runValidate the report locally
The same script CI runs. It checks that report.json parses as a RunReport and that its aggregate rates and success counts are internally consistent. It is CPU-only and needs no GPU, so a failure here is a failure you can fix in seconds instead of in review.
python scripts/validate_submission.py results/my-runBuild the board row with your provenance
Two flags matter. --submitted-by is you, and --provenance third-party-submission is what separates your row from ours. Every row currently on the board is maintainer-run and says so; yours will say something different, in a column, on the row.
provael leaderboard build --runs results/my-run \ --submitted-by your-name --provenance third-party-submissionOpen a pull request that adds results/
CI picks up any PR touching results/**, revalidates every report and rebuilds the board. Nothing is taken on trust: a stub run is refused from the real board by design, so a submission that never met a real model cannot become a public number.
What gets checked, and what does not
| Checked | Not checked |
|---|---|
| The report parses as a valid RunReport | That you ran what you say you ran. Provenance is declared, not proven |
| Aggregate rates agree with the underlying success counts | That your policy is safe. An ASR is a measurement, not a verdict |
| The leaderboard still builds with your row in it | That your result transfers to hardware. Nothing here has run on a robot |
| Stub runs are refused from the real board | Statistical power. Few seeds means a wide interval, and it will show |
A submitted row is evidence you produced, published under your name, with its provenance on the row. It is not certification, not a conformity assessment, and not an endorsement by this project. If you would rather someone else ran it, that is what an assessment is for.
Ready to submit?
Start with the repository. The validation script, the adapters and the recipe that produced our own published row are all in there.
Stuck on an adapter, or your policy is not one of the seven? Email [email protected] and say what you are running. An adapter that does not exist is a gap worth knowing about.