STALE: OUR MEASUREMENT

Our published benchmark result was measured with v0.41.2, 5 releases ago, past this project's own 2-release window, and the newest real-model measurement, dated 2026-09-18, is past this project's own 7-day window. That is our own number, not your assessment, which runs against your policy on the current release. Why, and what unblocks it

QUICKSTART

From install to your first attack run, in five minutes.

Provael is free, open source (Apache-2.0) and CPU-first. Install it, run every attack family against a built-in stub policy, and read the scorecard: no GPU and no model download. A real policy on a GPU is the advanced step at the end.

Python 3.12+ · Apache-2.0No GPU to startv0.46.0
01

Install

One package, pure Python. The CPU-first core and every attack family install with it.

$ pip install provael

Or with no local Python at all (multi-arch, public, works logged out):

$ docker run --rm ghcr.io/provael/provael:latest attack --recipe full-sweep
02

Run your first attack

This runs every registered attack family against the built-in CPU stub, with the benign control and the four harmless-variation controls beside them. It finishes in seconds. A family that cannot run on the stub shows N/A, never 0. Narrow it with --attacks.

$ provael attack --recipe full-sweep
03

Read the scorecard

Each row is one attack and its EAI code: the category it tests in the Embodied AI Security Top 10 (EAI01 is an instruction jailbreak, EAI02 adversarial perception). Every run reports an attack-success rate with a 95% interval, a benign control, and a SARIF report that drops into GitHub code scanning. On the stub, the numbers test the tool, not a robot policy.

Name an acceptance protocol (--protocol) and the run also carries a release decision: PASS means the protocol's criteria held on that run - not a hazard probability, not conformity - and without a protocol the tool reports incomplete rather than deciding.

GPU

Advanced: run a real policy on a GPU

The published result comes from this path, with SmolVLA in the LIBERO simulator: 42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI 62-100%) against 1 of 50 benign; the same frame with no target named does it in 27 of 30, so this is fragility, not steering.

It needs a CUDA GPU and the optional extra, pip install 'provael[lerobot]'. This command runs SmolVLA on the ten LIBERO-Object tasks of that result, with the benign control (none) beside the attack families, at that run's settings:

$ PROVAEL_INTEGRATION=1 provael attack --policy smolvla --suite libero --tasks libero_object/0,libero_object/1,libero_object/2,libero_object/3,libero_object/4,libero_object/5,libero_object/6,libero_object/7,libero_object/8,libero_object/9 --model HuggingFaceVLA/smolvla_libero --attacks none,instruction,visual,injection --seeds 5 --horizon 280

Leave out --seeds and --horizon and the defaults apply (10 episodes of 8 steps), which test the pipeline rather than repeat the measurement. The PROVAEL_INTEGRATION=1 in front is what the published run set, and every run records it with its provenance; SmolVLA does not need it, and the CPU steps above never do. To repeat the published run exactly (its tool version, seeds and arms), use its pinned inputs: reproduce the result.

The GPU path step by step, with what it costs and how to read what it prints: how to red-team a SmolVLA policy in LIBERO.

Recorded July 2026 · CPU stub
provael attack --policy stub
Terminal recording from July 2026 of the CPU path: pip install provael, then provael attack against the built-in stub policy prints an attack-success table by attack and EAI code, then a scorecard and a SARIF file are written. The stub measures no robot policy.

The CPU path end to end, recorded in July 2026 with an earlier release: install, an attack run against the stub, its scorecard and its SARIF file. The rates in it test the tool, not a policy, and its scorecard verdict predates release decisions under a named protocol. The real-policy result, 42/50, comes from a different run, on a GPU.