# Provael and VLA-Arena measure different things

> VLA-Arena runs the only public VLA benchmark with a leaderboard carrying a safety axis, which
> makes it the one place a Provael number could be read as a comparable entry. It is not one, and
> the reason is POSTURE - not units, not benchmark, not embodiment.

Web page: https://www.provael.com/compare/vla-arena
Source: VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models, arXiv:2512.22539 (https://arxiv.org/abs/2512.22539), submitted 27 December 2025, revised 7 August 2026
Project: https://vla-arena.github.io/
Task specification: Constrained Behavior Domain Definition Language (CBDDL)
Scale: 11 suites / 170 tasks, of which 5 suites / 75 tasks are safety

## The posture contrast

- Their posture: NON-ADVERSARIAL. Is this policy safe by default? A hazard is placed in the scene and the policy is scored on whether it avoids one it was never pushed toward.
- Our posture: ADVERSARIAL. Can this policy be made unsafe? The instruction is perturbed and the policy is scored on whether it leaves a safety envelope that did not move.
- Is the instruction perturbed by their safety suites? NO
- Their metrics: Cumulative Cost (CC) and Success Rate (SR). Ours: ASR with a 95% Wilson interval and a benign false-positive rate.

## The consequence, and it is not flattering

The Provael arm corresponding to VLA-Arena's ENTIRE safety axis is the benign control arm (--attacks none), NOT any attack family. A non-adversarial unsafe rate is what their safety suites measure, and the control is the only Provael arm that reports one.

Reference value: 2/50 episodes on SmolVLA × LIBERO libero_object, all ten tasks — Provael’s own non-adversarial unsafe rate, and an UNCALIBRATED one, so it carries a false-positive floor their declared scene constraint does not.

Every Provael ATTACK number - including the 88% headline - lives on an axis their leaderboard has
no column for. Placing a Provael ASR beside a VLA-Arena Cumulative Cost would assert that a policy pushed by an adversary and a policy left alone sit on one scale. They do not — and the direction of the error flatters Provael, since our number is larger because we push.

## Same name, unproven equivalence

Provael’s cumulative_cost and VLA-Arena’s Cumulative Cost share a NAME and have not been shown to share a definition. Ours is the mean number of unsafe steps per episode, derived from a per-step boolean. They do not belong in one table.

This is the third such collision tracked here; the others are CC/RET against ForesightSafety-VLA
and SBU against SafeVLA-Bench. One rule covers all three: borrowing a benchmark's vocabulary never
implies borrowing its units.

## The five safety suites

Coverage tally: 0 covered / 2 partial /
3 not covered, of 5. Nothing is fully
covered, and that is the honest result rather than an incomplete table.

### safety_static_obstacles

- Tasks: 15
- Hazard placed in scene: static collision obstacles placed in the workspace
- EAI: EAI04
- Provael attack family: NONE
- Coverage: partial
- Note: Geometrically the nearest match Provael has: the keep-out predicate IS a spatial breach. But Provael’s breach is caused by a perturbed instruction and theirs by the policy’s own path around an obstacle nobody added adversarially. Same geometry, different cause.

### safety_cautious_grasp

- Tasks: 15
- Hazard placed in scene: objects requiring careful handling during grasp
- EAI: EAI04
- Provael attack family: NONE
- Coverage: not covered
- Note: No grasp force, contact model or handling quality is represented in any Provael suite, so there is nothing to be cautious with.

### safety_hazard_avoidance

- Tasks: 15
- Hazard placed in scene: designated hazard zones the policy must not enter
- EAI: EAI04, EAI06
- Provael attack family: NONE
- Coverage: partial
- Note: The most literal correspondence in the set — a designated no-go region is exactly what Provael’s keepout_zones suite encodes. Still only partial, and the reason is ours not theirs: the predicate is UNCALIBRATED, so Provael’s zone boundary is not a defensible threshold the way a declared CBDDL constraint is.

### safety_state_preservation

- Tasks: 15
- Hazard placed in scene: object state that must survive the episode intact
- EAI: EAI04
- Provael attack family: NONE
- Coverage: not covered
- Note: Provael scores a keep-out breach, not the condition of objects afterwards. Nothing in the report records whether anything was damaged.

### safety_dynamic_obstacles

- Tasks: 15
- Hazard placed in scene: moving obstacles entering the workspace during the episode
- EAI: EAI04
- Provael attack family: NONE
- Coverage: not covered
- Note: Every Provael suite is static for the duration of an episode; no obstacle moves, so there is no dynamic avoidance to measure.


Every suite maps ZERO Provael attack families, and that is correct rather than incomplete:
Provael's families all perturb an input, and none of these suites has an input to perturb.

Machine-readable original: https://github.com/provael/provael/blob/main/results/crosswalk/crosswalk.vla_arena.json
Emitted by: provael crosswalk --target vla_arena
If this page and that artifact disagree, THE ARTIFACT WINS.

## What this page does not license

- No leaderboard submission. Provael has never submitted to VLA-Arena’s leaderboard and this page is not a step toward doing so.
- No score in a shared column. See the posture contrast — the direction of that error flatters Provael.
- No claim of coverage. 0 of 5 safety suites are covered, and every row maps zero Provael attack families.

Names and marks are their owners'; Provael is independent and not affiliated with them. Describing
their safety axis as non-adversarial is a statement of what it measures, not a criticism -
measuring default safety is a different and equally necessary job.

Canonical: https://www.provael.com/compare/vla-arena