# FAQ — red-teaming VLA robot policies | Provael

> Markdown twin of https://www.provael.com/faq/ - derived mechanically from the page's own rendered content at
> build time (it can state nothing the page does not). Curated, data-driven twins exist for the
> key pages; this one is generated so the twin convention has no holes.

FAQ

# Questions about Provael, answered plainly.

What Provael measures and what it does not, what is free, what you pay for, and who is behind it. If your question is not here, [ask it directly](https://www.provael.com/contact/).

## What Provael does

### Can garak or PyRIT test a robot policy?

They test the language and tool-call layer, and they do not model the action space. A vision-language-action policy turns an instruction into a trajectory, and that trajectory is where the embodied failure lives. Text-only tools have nothing to score there. Use them for the language layer; use Provael for the action layer.

### Is Provael a replacement for LLM red-teaming?

No - it is complementary. If your robot has an LLM in the loop, red-team the text with garak or PyRIT and red-team the resulting motion with Provael. The two cover different halves of the same system.

### What makes Provael’s number different from a jailbreak demo?

A demo shows that an attack worked once. A Provael result says how often it works: an attack-success rate over many episodes, with a 95% confidence interval and a matched benign control run beside it. When an attack does not separate from that control, the result is still published, as a floor result or a null.

### Do I need a GPU?

Not to start. Provael is CPU-first: the deterministic core and every attack family except the adversarial-patch search (optimized_patch, which needs a real camera frame) run without a GPU. Testing a real policy, including re-running the published result, needs one; those real-model runs are optional and installed as an extra.

### Do you only find attacks, or also fixes?

Both. A defense is measured the way an attack is: the attack-success rate before and after, each with its own 95% confidence interval, so a fix is evidenced rather than asserted. Two defenses have studies, and each study ran on the CPU test fixture. Instruction canonicalization takes the stub suite from 67.5% [52–80%] to 7.5% [3–20%] (n=40), a result the study itself calls substantially circular: the fixture’s danger score is lexical, and the attack’s payload is the wording the defense strips. The action envelope clamps what leaves the policy rather than what goes in, and much of its credit is circular by construction, because its bounds come from the benign policy’s own magnitudes. Each defense also has a defended arm measured on a real policy, and none of those arms counts as a measured risk reduction. See [/defenses](https://www.provael.com/defenses/) for the per-family tables.

## Limits and trust

### Does any of this happen on a real robot?

Not yet measured, and no sim-to-real claim is made anywhere on this site. Every published number comes from simulation. What exists is a protocol: an SO-ARM101 table-top arm, the same SmolVLA checkpoint, the instruction family only, n = 5 per condition, a geometric keep-out predicate, filmed and seeded. It was registered on 24 July 2026 and published at [/sim-to-real](https://www.provael.com/sim-to-real/) before the trials run, so a reader can check the design was fixed before there was data to fit it to. Its null hypothesis is written down too: that sim and real disagree. When it runs, it reports a directional correlation for one arm, one policy and one task, never a real-robot attack-success rate.

### Has anyone independent checked any of this?

No. No notified body, certification body, accreditation body, insurer or standards committee has reviewed Provael, and there are no published customer case studies. External validation is a real gap. The nearest thing to it is that the physical-transfer protocol at [/sim-to-real](https://www.provael.com/sim-to-real/) is published before its own trials, so that one result can be judged against a design nobody chose after seeing it. To close the gap yourself, [/verification](https://www.provael.com/verification/) has the exact commit, recipe and command to reproduce every published row, and the register of who has reproduced what: 0 independent reproductions as of 1 November 2026.

### Who is behind this, and what happens if you stop?

One person, Sattyam Jain, and the About page says so. That is a real risk for anything you depend on, so it is worth naming: the tool is Apache-2.0 and forkable, every published result is reproducible from a committed recipe, and no evidence you hold depends on this project continuing to exist. What a single maintainer cannot promise is an SLA, and none is published for that reason.

## Paying for it

### What is the free scan, and do you actually run it?

If you use a public checkpoint, we may have run it already: the public results for SmolVLA and π0.5 are published on the Results and Findings pages. You can verify each pack in seconds on a CPU and re-run it from its committed recipe on a GPU. Or name the checkpoint you actually ship, and we run the same measurement with you, in your CI, for free.

### Why pay, when the tool is free and open-source?

You are not paying for the software. The CLI, every attack family, the attack-success rate with its benign control, SARIF, the GitHub Action and local attestation are free and stay free. You pay for operated work on your own checkpoint: a person running the agreed tests, reviewing the findings with you and retesting after you fix them. The Checkpoint Report does that for one eligible checkpoint at a fixed price; the red-team assessment adds a threat model chosen for your stack and integration with it. [/pricing](https://www.provael.com/pricing/) lists what each includes. The signature on the pack is free either way (your own key, locally) and establishes origin and integrity, not acceptance by an insurer or assessor. If your team can do this in-house, run it yourself.

### What exactly do I get, and can I see it before buying?

The core of the evidence pack is published at [/sample-evidence-pack](https://www.provael.com/sample-evidence-pack/), in full rather than summarised, using the project’s own public reference run. Read it before booking anything. Every paid tier delivers that shape: a measured attack-success rate with its confidence interval and benign control, the scope limits stated at the same size as the number, SARIF, and a compliance crosswalk. What each tier adds to it is listed on [/pricing](https://www.provael.com/pricing/).

## Still deciding?

Talk through your own checkpoint on a short scoping call, or read the sample report first: it is the core of every paid pack.

Book a scoping call [See the sample report](https://www.provael.com/sample-evidence-pack/)

---
Canonical: https://www.provael.com/faq/