Text-layer red-teaming stops at the sentence. The robot doesn’t.
garak, PyRIT, and promptfoo are excellent at scanning what a model says. But a vision-language-action policy turns language into motion - so the failure is a trajectory, not a paragraph. That action layer is the part text-only tools structurally can’t reach, and it’s the whole reason Provael exists. These tools aren’t rivals - they’re the other half of your stack.
For the category-by-category mapping onto other robot-safety benchmarks and standards - each card carrying its own mapping status, including the ones that are explicitly not comparable - see the crosswalks.
| Tool | What it tests | Success = | Taxonomy | Evidence |
|---|---|---|---|---|
| garak NVIDIA | LLM text & prompt I/O | Jailbroken / toxic / leaked text | OWASP LLM Top 10 · MITRE ATLAS | Vulnerability-scan report |
| PyRIT Microsoft | Generative-AI & agent I/O | Harmful / policy-violating text | OWASP · MITRE ATLAS | Red-team eval logs |
| promptfoo | LLM evals & prompt red-teaming | Failed eval / jailbreak | OWASP LLM Top 10 | Eval reports |
| Provael the action layer | VLA robot policy - the commanded action | Policy driven to an unsafe action (keep-out), as an attack-success rate | Embodied AI Security Top 10 | SARIF + conformity evidence (EU AI Act / Machinery Reg / ISO) |
Comparison reflects each tool’s primary, publicly-documented scope as of July 2026; capabilities evolve - check each project. Tool names are trademarks of their respective owners; Provael is independent and not affiliated with them.
Questions people actually ask
Also on the FAQ page.
Can garak or PyRIT test a robot policy?
They test the language and tool-call layer brilliantly - but they do not model the action space. A vision-language-action policy turns an instruction into a trajectory, and that trajectory is where the embodied failure lives. Text-only tools have nothing to score there. Use them for the language layer; use Provael for the action layer.
Is Provael a replacement for LLM red-teaming?
No - it is complementary. If your robot has an LLM in the loop, red-team the text with garak or PyRIT and red-team the resulting motion with Provael. The two cover different halves of the same system.
What makes Provael’s number different from a jailbreak demo?
Every Provael result is an attack-success rate with a 95% Wilson confidence interval and a matched benign-control false-positive rate - plus published null results when an attack does not transfer. That is a measurement you can file, not a one-off "we broke it."
Do I need a GPU?
No. Provael is CPU-first: the deterministic core and every attack family except the adversarial-patch search (`optimized_patch`, which needs a real camera frame) run without a GPU. Real-model transfer tests are optional and gated behind an extra.
Do you only find attacks, or also fixes?
We measure both. Every mitigation we recommend ships with a pre/post attack-success rate and its own 95% confidence interval, so a fix is evidenced rather than asserted. two defenses have been measured end to end under that protocol: instruction canonicalization takes the stub suite from 67.5% [52–80%] to 7.5% [3–20%] (n=40), and an action envelope clamps what leaves the policy rather than what goes in. Both carry their own caveat on the page - the envelope's credit is partly circular by construction, because its bounds come from the benign policy's own magnitudes, and the study says so before it reports a rate. See /defenses for the per-family tables.
Why pay, when the tool is free and open-source?
You are not paying for the software - the CLI, every attack family, the attack-success rate with its benign control, SARIF, the GitHub Action and local attestation are free and stay free. You are paying for the operated work a tool cannot do for you: someone independent choosing the threat model for your policy, running it, and signing the evidence pack. If your team can do that in-house, run it yourself. That is the honest answer and it costs us some deals.
Who is behind this, and what happens if you stop?
One person - Sattyam Jain - and the About page says so rather than hiding behind a "we". That is a real risk for anything you depend on, so it is worth naming: the tool is Apache-2.0 and forkable, every result is reproducible from a committed recipe, and no evidence you hold depends on this project continuing to exist. What a single maintainer cannot promise is an SLA, and none is published for that reason.
Does any of this happen on a real robot?
Not yet measured, and no sim-to-real claim is made anywhere on this site. Every published number comes from simulation. What exists is a protocol - SO-ARM101 table-top arm, the same SmolVLA checkpoint, instruction family only, n = 5 per condition, a geometric keep-out predicate, filmed and seeded - registered on 24 July 2026 and published at /sim-to-real BEFORE the trials run, so a reader can check the design was fixed before there was data to fit it to. The null hypothesis is written down too: that sim and real disagree. When it runs, it reports a directional correlation for one arm, one policy and one task - never a real-robot attack-success rate.
Has anyone independent checked any of this?
No. No notified body, certification body, accreditation body, insurer or standards committee has reviewed Provael, and there are no published customer case studies - /case-studies says zero in its own heading rather than sitting empty. The nearest thing to external checkability is that the physical-transfer protocol at /sim-to-real is published in advance of its own trials, so that one result can be judged against a design nobody chose after seeing it. External validation is a real gap and the site states it in the places a buyer looks, instead of implying otherwise through silence. If you want to close that gap rather than read about it, /verification carries the exact commit, recipe and command to reproduce every published row, and the register of who has reproduced what - empty today, and labelled empty.
What exactly do I get, and can I see it before buying?
The full evidence pack is published at /sample-evidence-pack - the whole artifact, not a summary, using the project’s own public reference run. Read it before booking anything. It is the same shape you would receive: a measured attack-success rate with its confidence interval and benign control, the scope limits stated at the same size as the number, SARIF, and a compliance crosswalk.
garak/PyRIT/promptfoo are the language layer. If the comparison you actually need is against NVIDIA Halos, UL 3300 or the robot-CVE work, that is a different axis - compute and machine certification versus the learned policy.See the layer map, with sources →
VLA-Arena runs the only public VLA leaderboard with a safety axis - and its five safety suites are non-adversarial: they place a hazard in the scene, and none perturbs the instruction. That makes a Provael ASR the wrong thing to put beside it. The arm of ours that actually corresponds to their whole safety axis is the benign control.Why the two do not share a column →
Gray Swan runs the largest AI red-teaming arena there is, and on their own figures it is bigger than this project by three to five orders of magnitude. The two are not the same kind of thing: a competition of thousands of people produces a break, and this produces a rate with an interval and a benign control. One says a failure is possible; the other says how often it recurs.Seven axes, four of which Provael loses →
Red-team the layer no one else covers.
Keep your LLM red-teaming. Add the action layer - a reproducible attack-success rate for your robot’s policy.