# VLA evaluation self-check: six questions | Provael > Markdown twin of https://www.provael.com/readiness/ - derived mechanically from the page's own rendered content at > build time (it can state nothing the page does not). Curated, data-driven twins exist for the > key pages; this one is generated so the twin convention has no holes. EVALUATION-PRACTICE CHECKLIST · SELF-REPORTED # Check how you test your robot policy, in two minutes. Six questions: what runs your robot, where you evaluate it, which of the [Embodied AI Security Top 10](https://www.provael.com/eai-top-10/) risks you have tested under attack, where it ships and how far along you are. You get the gaps in your own answers and a sensible next step. It is not a certification, a legal opinion, an insurance score or a safety guarantee. Take the checklist → Evidence, not certification Runs in your browser Defensive · sim-only Why this matters In our own test of a real SmolVLA policy: 42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI 62-100%) against 1 of 50 benign; the same frame with no target named does it in 27 of 30, so this is fragility, not steering. A task-success metric never sees that. The two families that stayed at the benign floor (visual 2/100, injection 2/50) are published at that size too, as counts, not as zero. [See the measured result](https://www.provael.com/results/). Scope: SmolVLA (HuggingFaceVLA/smolvla_libero) × libero_object (all 10 tasks), 5 seeds per task, LIBERO · robosuite · MuJoCo — simulation only, default (uncalibrated) predicate, provael 0.41.2. What runs the robot? Your vision-language-action policy family. SmolVLA π0 / π0.5 family OpenVLA GR00T Custom / in-house VLA Not sure yet Where do you evaluate it? The simulator or suite you run today. LIBERO Meta-World robosuite / MuJoCo Isaac / other sim Real robot only No systematic evaluation yet Have you adversarially tested the policy? Not task success. Attacks: does a reframed instruction or a perturbed observation move it out of its safe envelope? No adversarial testing Ad-hoc / manual probing Structured red-teaming, no calibration Attack-success rate with a benign control, calibrated to the policy Which risks have you actually tested? The Embodied AI Security Top 10 (the 8 a policy red-teamer can reach). Pick all you have measured. EAI01 Policy & instruction jailbreak EAI02 Adversarial perception EAI03 Poisoning, backdoors & supply chain EAI04 Action-space integrity EAI05 Embodied prompt injection EAI06 Cross-domain safety misalignment EAI08 Access & excessive autonomy EAI09 Model & data confidentiality Where does it ship, and who underwrites it? Determines which compliance clocks apply. Pick all that apply. EU (CE / Machinery route) US Insured / underwritten Internal / research only How far along is deployment? Sets the stakes: exposure rises faster than readiness. Research Pilot Limited production Scaled production Show my gaps → No account. No tracking. Evaluated in your browser; nothing is sent until you choose to below. Your evaluation practice, as you described it No adversarial evidence yet The old 0–100 heuristic, for your own reference 0/100 — 55 points for the adversarial-testing answer (none 0 · ad-hoc 18 · structured 38 · controlled with a benign control 55) plus 45 points spread equally over the 8 reachable Top-10 categories you ticked. Unvalidated: the weights are the author's judgement, nothing has calibrated them against outcomes, and the number establishes nothing about regulatory applicability, physical safety or insurability. It is kept only so a repeat visit can be compared with the previous one. Clocks to review, by the markets you named ## What is coming, and when — for your counsel to confirm Filtered by your market answers only. Whether an instrument actually applies to your product is a legal question this page cannot answer; each card links to the sourced crosswalk with its primary text and review date. Your Top-10 gap snapshot ## Where you have evidence, and where you do not The 8 categories a policy red-teamer can reach. EAI07 (CPS / firmware / comms) and EAI10 (evaluation / observability) are out of a VLA-policy red-teamer's scope by design. ### Get your scoped next step. Optional. Sending transmits exactly this: your work email, organisation and role, plus a text summary of the answers above (practice level, ticked categories, markets, stage) — nothing else, and nothing until you press the button. A real person reads it and replies with the next useful conversation, not a sales cadence. Website Your role… I sign off on safety / compliance I build / train the policy I certify or insure these systems Other Send my snapshot → Book a scoping call Not ready to send anything? One email a month: the results that landed, the errata, and what is next. Nothing else, no spam, unsubscribe in one click. Subscribe ### You sign off on safety Turn "is this safe enough?" into a measured rate with a benign control and its limits stated, for your own release decision. [Request a scoping call →](https://www.provael.com/assessment/) [See the sample report](https://www.provael.com/sample-evidence-pack/) ### You build the policy Run the red-team yourself, free and CPU-first. Every attack family is in the open-source tool. [Run the quickstart →](https://www.provael.com/quickstart/) [Get the tool](https://github.com/provael/provael) How an engagement works ## From a self-check to a measured result The tool is free and runs on your own machine. If you want Provael to run it on your policy, these are the paid options; every price is on [the pricing page](https://www.provael.com/pricing/). Open-source tool freeApache-2.0 The CLI, every attack family, an attack-success rate with a benign control, SARIF, the GitHub Action and local attestation are free and always will be. Run it yourself: pip install provael. [Get the tool →](https://github.com/provael/provael) Checkpoint Report fixed priceone checkpoint One eligible checkpoint on an agreed simulator and task set: the agreed tests that apply, matched controls, a reproducible report and an engineering review. Eligibility is confirmed by a short form and a human reply before anything is sent or paid. [Check eligibility →](https://www.provael.com/checkpoint-report/#eligibility) Red-team assessment scoped engagementby engagement A human-led red-team of your own policy in simulation, scoped to your stack: the attack-success rate with 95% confidence intervals and a benign control, narrated attack chains, and an evidence pack under a written acceptance protocol. [Request a scoping call →](https://www.provael.com/assessment/) Assurance subscription annualinvoiced quarterly The red-team re-run each quarter, CI gated on your acceptance protocol, and a rolling per-checkpoint regression baseline, so the evidence keeps up with a policy that keeps changing. [Request a scoping call →](https://www.provael.com/contact/) This checklist is a self-reported evaluation aid. Its heuristic is unvalidated, and its market and role answers are inputs for a review, not a determination of which rules apply. Provael is a defensive, simulation-only tool for authorized testing. Not affiliated with or endorsed by OWASP, MITRE, or any standards body. Verify every regulatory date against the primary source. --- Canonical: https://www.provael.com/readiness/