Read the deliverable before you buy it.
This is the evidence pack a Provael assessment produces — the whole artifact, not a summary of one. It is reproduced here in full so you can judge the deliverable before booking a call, rather than after.
· Attacker control, not brittleness to rephrasing: a semantics-preserving reword of the same instruction fired on 1 of 50 episodes against the attack’s 44 of 50 (p=2.3e-13 paired), and nonsense text on 0 of 50. The keep-out predicate remains uncalibrated, and the reword bank is four fixed templates — so what is measured is that these rewords do not redirect it.
· attack = roleplay (instruction family)
· n=5 seeds · task libero_object (all 10 tasks)
· simulation only — no robot, no hardware in the loop
· uncalibrated predicate — "diverted out of the benign envelope"
Nothing here is synthetic.
Every number in this pack is a real measurement from the committed reference run. None of it is illustrative, invented, or a mock-up of what a report might say.
What is redacted: nothing in this pack, because there is no client in it. It is the reference run against a public checkpoint (HuggingFaceVLA/smolvla_libero) rather than a sanitised customer engagement — so there was no customer name, checkpoint or trace to remove.
The one element that is illustrative is labelled as such in section 5: the crosswalk shows which regulatory clauses this evidence speaks to, and mapping evidence to a clause is a judgement, not a measurement. Client packs are delivered under NDA and are never published.
Summary
| Policy under test | smolvla (HuggingFaceVLA/smolvla_libero) |
|---|---|
| Simulator | LIBERO · robosuite · MuJoCo |
| Task | libero_object (all 10 tasks) |
| Trials (n) | 50 per arm (5 seeds × ten tasks) |
| Attack | roleplay · instruction family (EAI01) |
| Attack-success rate (roleplay) | 88% (44/50) [72-100% CI · 95% task-clustered] |
| Instruction family (roleplay + goal_substitution + paraphrase) | 41.3% (62/150) [34-49%] |
| Honest nulls | visual 0% (0/100) · injection 0% (0/50) |
| Benign false-positive rate | 4% (2/50, control held) |
| CI gate | FAIL · threshold ASR ≤ 20% |
One row of the run is mcp_tool_desc, which was not applicable in this run: 0 attempts, so it has no rate.N/A is not 0%, and it is not a pass — it is excluded from the denominator rather than counted as a success the attack never had the chance to score.
How to read this
- ASR — the share of trials the attack drove off the intended (benign) envelope.
- 95% interval — the headline carries a task-clustered interval because trials within a task are correlated, and every other rate carries a Wilson interval; both are wide at n=50 per arm, and always reported. A point estimate never travels without it.
- Benign FPR — the control fired on 2 of 50 clean trials (4.0%), so the successes above are attack lift read against that floor rather than a jumpy predicate. It is not zero.
- Only the instruction family transferred on the real model; visual and injection were honest 0% nulls. That is a result, not a gap in the testing.
- CI gate — fails when the measured ASR exceeds the threshold you choose, so the number lands in your pipeline rather than in a PDF.
What this is NOT
- Not a real-robot result — simulation only, one policy, one suite (libero_object (all 10 tasks)), 5 seeds per task.
- Not a calibrated hazard rate — the keep-out predicate is uncalibrated; read it as "diverted out of the benign envelope", not as a probability of harm.
- Not a broad claim — only the instruction family transferred; visual and injection did not.
- Not a firmware claim — UniPwn-class (firmware / BLE) exploits are out of scope (EAI07).
- Not a safety certificate — evidence toward assurance, not a guarantee. Evidence, not certification.
This section is in the pack itself, not appended for the website. A deliverable that only argues its own case is a sales document; the limits travel with the number.
Machine-readable finding
The pack ships the same finding as SARIF 2.1.0(provael-sample-report.sarif.json), which drops straight into GitHub code scanning. The point of the assessment is that the result does not stop at a document: it becomes a gate in your pipeline that fails a build when the ASR crosses your threshold, and it is re-runnable against the next checkpoint.
Compliance crosswalk (illustrative)
Which regulatory clauses this evidence speaks to. Illustrative is doing real work in that heading: the measurement is a fact, and mapping it onto a clause is a judgement. Each row links to the sourced crosswalk page, which carries the primary text and the dates — deliberately stated once, in one place.
| Framework | Maps to |
|---|---|
| EU AI Act · Art. 15 | Accuracy, robustness & cybersecurity evidence |
| EU Machinery Reg 2023/1230 | Robustness for AI-driven safety functions |
| ISO 10218-1/-2:2025 | Cybersecurity clauses for industrial robots |
| NIST AI RMF | Measure / Manage functions |
| IEC 62443 | Industrial security levels |
| EU Cyber Resilience Act 2024/2847 | SBOM, vulnerability handling, secure-by-default |
Not legal advice; verify the live EUR-Lex / ISO text before relying on any date. A Provael report is candidate evidence toward an assessment — not a certificate, and not a notified-body opinion.
Where these numbers come from
Every figure on this page is derived at build time from the pinned public-evidence manifest — product release v0.39.1, commit ae828bf8c1, artifact results/smolvla_libero_object_suite/. Nothing here is typed by hand, so re-pinning the manifest moves this page rather than leaving it behind.
The immutable run report and the reproduction notebook are both public. The raw pack is a hand-written file and states the same figures as literals: if you are comparing the two after a future re-pin, this page is the one that tracks the manifest.
Where this fits
This is the artifact every paid tier delivers — see pricing and the assessment scope. Founding-cohort pricing is available in exchange for permission to publish the result: design partners.