Row by row, including where we lose
VLA-ArenaCapability + safety benchmark with a public leaderboard
11 suites, 170 tasks; 5 safety suites, 75 tasks · Simulation
Not directly comparable. The only public VLA leaderboard with a safety axis, which makes it the one place a Provael ASR could be mistaken for a comparable entry. It is not one, and the reason is posture: their suites ask whether a policy is safe by default, Provael asks whether it can be made unsafe. The Provael arm corresponding to their entire safety axis is the BENIGN CONTROL (2/50 on the ten-task run), not any attack family.
Where Provael is weakerA running public leaderboard with external submissions, against our zero third-party rows. A declarative constraint language (CBDDL) for defining tasks and safety constraints, against our uncalibrated keep-out predicate.
SafeVLA-BenchPost-hoc success-safety gap measurement over rollouts
LIBERO + RoboCasa-365 · Simulation
Not directly comparable. Post-hoc where Provael is pre-hoc, and nobody causes the failure: it measures the policy’s own behaviour under ordinary instructions. Safety is an STL specification over the trajectory; ours is a boolean from an uncalibrated keep-out predicate. Provael already has a field called succ_but_unsafe that names this benchmark — the shared NAME is exactly why no number is placed beside theirs.
Where Provael is weakerA declared formal specification (Signal Temporal Logic) where we have an uncalibrated threshold. And their finding bounds ours: high-SR baselines still leave 13–15% unsafe episode rates with no adversary present — a floor a Provael ASR is blind to by construction, because it measures lift over a benign control.
RedVLAPhysical red-teaming framework + a defense (SimpleVLA-Guard)
6 VLA models, LIBERO, 10 trials per configuration · Simulation
Not directly comparable. The closest published work to this project, and the cleanest incomparability on the page. RedVLA formalises red teaming as optimisation over the environment–instruction joint space (s′₀, l′), then fixes the instruction (l′ = l) and perturbs only the initial state. Provael does the converse. Same benchmark, same simulator, complementary halves of one formalism — so 95.5% and 88% are orthogonal quantities. Note what this does NOT rest on: both are simulation, so sim-versus-hardware is not the difference here.
Where Provael is weakerSix policies against our one measured. An optimisation loop over the risk-factor state against our fixed four-template banks. A shipped, evaluated defense against our two stub-validated ones. And a typed physical-safety taxonomy grounded in risk predicates, against a predicate that has never been calibrated on LIBERO.
RoboJailBench18-category harm-outcome benchmark for embodied agents, with a public leaderboard
RoboVQA, RH20T, NVIDIA PhysicalAI-AV, RJB-Instructions · Simulation
Not directly comparable. A different layer of the stack. They evaluate VLM planners — a model that reads a scene and emits a plan — so a success is a harmful sentence. Provael evaluates closed-loop low-level policies emitting motor commands, so a success is a trajectory already outside the envelope. A jailbroken planner has said something; a redirected policy has already moved.
Where Provael is weakerA public repository with external submissions, against our zero third-party rows — the axis on which this project is weakest, and the one a benchmark is built to be strong on.
AttackVLAUnified evaluation framework for adversarial and backdoor attacks on VLAs
Existing VLA attacks plus attacks adapted from vision-language models · Simulation AND real-world robotic settings
Partially. Closest in SHAPE — a unified harness running many attacks and reporting a comparable ASR — and the row that most constrains what this project may claim. Their targeted-attack predicate ("the policy performed the attacker’s specified action sequence") is strictly harder than an envelope breach, so their number is the more demanding one.
Where Provael is weakerAttackVLA already occupies the position this project is often assumed to have invented — one harness, many attacks, comparable ASR — and it does so WITH real-robot evaluation we do not have. We do not claim to have originated a unified VLA attack harness and we do not claim parity with a benchmark that has been run on hardware.
What is actually different about Provael
Not the attacks, and — since AttackVLA — not "a unified harness with a comparable ASR" either. What is left is narrower: a deterministic CPU-only, no-download core so every number is reproducible without a GPU or model weights, and an evidence layer (SARIF, OSCAL, CycloneDX ML-BOM, signed attestation, a Wilson-CI regression gate wired into CI) that a research benchmark has no reason to build.