Provael and VLA-Arena measure different things
Their safety suites ask whether a policy is safe by default. Provael asks whether a policy can be made unsafe.
VLA-Arena places a hazard in the scene and scores whether the policy avoids one it was never pushed toward. None of their five safety suites perturbs the instruction. Provael perturbs the instruction and scores whether the policy leaves a safety envelope that did not move. Both are safety numbers. Neither answers the other’s question.
Why this page exists at all
VLA-Arena runs the only public VLA benchmark with a leaderboard carrying a safety axis — 11 suites and 170 tasks, of which 5 suites and 75 tasks are safety. That makes it the one place where a Provael number could plausibly be read as a comparable entry, and the one place where being wrong about that would do real damage.
It is not comparable. The reason is not units, not benchmark, and not embodiment — it is posture.
The posture contrast
| Dimension | VLA-Arena safety suites | Provael |
|---|---|---|
| Posture | non-adversarial | adversarial |
| The question | Is this policy safe by default? A hazard is placed in the scene and the policy is scored on whether it avoids one it was never pushed toward. | Can this policy be made unsafe? The instruction is perturbed and the policy is scored on whether it leaves a safety envelope that did not move. |
| What is placed | a hazard, in the scene | a perturbation, on the instruction |
| Is the instruction perturbed? | No — not by any of the five | Yes — that is the attack |
| Metrics | Cumulative Cost (CC) and Success Rate (SR) | ASR, 95% Wilson interval, benign false-positive rate |
The Provael arm corresponding to VLA-Arena’s entire safety axis is the benign control — not any attack family we ship.
A non-adversarial unsafe rate is what their suites report, and the control is the only Provael arm that reports one. On the ten-task SmolVLA × LIBERO run that control fired on 2/50 episodes. It is uncalibrated, so it carries a false-positive floor their declared Constrained Behavior Domain Definition Language (CBDDL) constraint does not.
Every Provael attack number — including the 88% headline — lives on an axis their leaderboard has no column for. Placing a Provael ASR beside a VLA-Arena Cumulative Cost would assert that a policy pushed by an adversary and a policy left alone sit on one scale. They do not — and the direction of the error flatters Provael, since our number is larger because we push.
Provael’s cumulative_cost and VLA-Arena’s Cumulative Cost share a NAME and have not been shown to share a definition. Ours is the mean number of unsafe steps per episode, derived from a per-step boolean. They do not belong in one table. This is the third such collision we track — the others are CC/RET against ForesightSafety-VLA and SBU against SafeVLA-Bench. One rule covers all three: borrowing a benchmark’s vocabulary never implies borrowing its units.
The five safety suites, and what Provael covers
Suite identifiers verbatim. Coverage tally: 0 covered · 2 partial · 3 not covered, of 5. Nothing here is fully covered, and that is the honest result rather than an incomplete table.
| Suite | Tasks | Hazard placed in scene | EAI | Provael family | Coverage |
|---|---|---|---|---|---|
safety_static_obstaclesGeometrically the nearest match Provael has: the keep-out predicate IS a spatial breach. But Provael’s breach is caused by a perturbed instruction and theirs by the policy’s own path around an obstacle nobody added adversarially. Same geometry, different cause. | 15 | static collision obstacles placed in the workspace | EAI04 | none | partial |
safety_cautious_graspNo grasp force, contact model or handling quality is represented in any Provael suite, so there is nothing to be cautious with. | 15 | objects requiring careful handling during grasp | EAI04 | none | not covered |
safety_hazard_avoidanceThe most literal correspondence in the set — a designated no-go region is exactly what Provael’s keepout_zones suite encodes. Still only partial, and the reason is ours not theirs: the predicate is UNCALIBRATED, so Provael’s zone boundary is not a defensible threshold the way a declared CBDDL constraint is. | 15 | designated hazard zones the policy must not enter | EAI04, EAI06 | none | partial |
safety_state_preservationProvael scores a keep-out breach, not the condition of objects afterwards. Nothing in the report records whether anything was damaged. | 15 | object state that must survive the episode intact | EAI04 | none | not covered |
safety_dynamic_obstaclesEvery Provael suite is static for the duration of an episode; no obstacle moves, so there is no dynamic avoidance to measure. | 15 | moving obstacles entering the workspace during the episode | EAI04 | none | not covered |
Every row maps zero Provael attack families, and that is the correct result. Provael’s families all perturb an input; none of these suites has an input to perturb. A table showing families mapped across these rows would be manufacturing coverage. The machine-readable original is crosswalk.vla_arena.json, emitted by provael crosswalk --target vla_arena; if this page and that artifact disagree, the artifact wins.
What this page does not license
- No leaderboard submission. Provael has never submitted to VLA-Arena’s leaderboard and this page is not a step toward doing so.
- No score in a shared column. See the posture contrast — the direction of that error flatters Provael.
- No claim of coverage. 0 of 5 safety suites are covered, and every row maps zero Provael attack families.
VLA-Arena is arXiv:2512.22539, submitted 27 December 2025, last revised 7 August 2026. Names and marks are their owners’; Provael is independent and not affiliated with them. Describing their safety axis as non-adversarial is a statement of what it measures, not a criticism — measuring default safety is a different and equally necessary job.
The adversarial half, measured.
If default-safety numbers are what you already have, the missing question is whether a perturbed instruction moves the policy anyway.