LLM red-teaming tools test what a model says. Provael tests what a robot policy does.
garak, PyRIT and promptfoo scan text and tool calls. A vision-language-action policy turns an instruction into motion, and Provael scores that motion in simulation. Use both: they cover different layers of the same system.
Five comparisons, one per question
VLA safety benchmarks →
Five published VLA safety and attack benchmarks, each with a comparability verdict and where Provael is weaker.
Published attack baselines →
Published attack figures beside a Provael attack-success rate, and which of them measure the same thing.
VLA-Arena →
The only public VLA leaderboard with a safety axis. Its safety suites are non-adversarial, so the two numbers do not share a column.
The physical-AI safety stack →
NVIDIA Halos, UL 3300 and robot-CVE work: which layer each one tests, and where a policy red-team fits.
Gray Swan →
The largest AI red-teaming arena: a crowd finds breaks, Provael measures how often one recurs. Seven axes, three of them where Gray Swan is ahead.
For how Provael’s categories map onto other benchmarks and standards, each card with its own mapping status and including the ones that are explicitly not comparable, see the crosswalks.
What each tool tests
| Tool | What it tests | Success = | Taxonomy | Evidence |
|---|---|---|---|---|
| garak documentation: garak NVIDIA · docs read July 2026 | LLM text & prompt I/O | Jailbroken / toxic / leaked text | OWASP LLM Top 10 · MITRE ATLAS | Vulnerability-scan report |
| PyRIT documentation: PyRIT Microsoft · docs read July 2026 | Generative-AI & agent I/O | Harmful / policy-violating text | OWASP · MITRE ATLAS | Red-team eval logs |
| promptfoo documentation: promptfoo OpenAI acquisition announced Mar 2026 · docs read July 2026 | LLM evals & prompt red-teaming | Failed eval / jailbreak | OWASP LLM Top 10 | Eval reports |
| Provael documentation: Provael the action layer · docs read September 2026 | VLA robot policy - the commanded action | Policy driven to an unsafe action (keep-out), as an attack-success rate | Embodied AI Security Top 10 | SARIF + conformity evidence (EU AI Act / Machinery Reg / ISO) |
The difference you can check: as of the dates in the table, none of the three documents a simulator harness, a robot-policy adapter or an action-level unsafe predicate, and those three are what Provael is built around. That describes what each tool ships today, not what any of them could be extended to do.
Each row reflects that tool’s own documentation as of the date shown against its name; capabilities evolve, so check the documentation linked from each name. No numerical ranking is offered: the tools measure different layers with different success definitions, so their numbers are not comparable. Tool names are trademarks of their respective owners; Provael is independent and not affiliated with them.
What Provael does not claim
Provael does not claim new attacks: the ones it runs are the published state of the art (RoboPAIR, BadVLA, FreezeVLA, SafeVLA). Running many of them in one harness with a comparable attack-success rate is not new either: AttackVLA does it, on real hardware too. What Provael adds is narrower: a deterministic CPU-only core, and an evidence layer (SARIF, OSCAL, a CycloneDX ML-BOM, a signed attestation and a CI gate) that a research benchmark has no reason to build. The benchmark comparison sets this out row by row.
Common questions
Whether garak or PyRIT can test a robot policy, whether this replaces LLM red-teaming, what makes the number different from a jailbreak demo, whether you need a GPU, and whether any of it happens on a real robot: answered once, on the FAQ page →
Keep your LLM red-teaming. Add the action layer.
Provael measures how often an attack moves a robot policy out of its safe zone, with a benign control run beside it.