Where a policy red-team fits in the physical-AI safety stack
Halos certifies the computer the policy runs on. Nothing in it publishes an attack-success rate for the policy itself.
Provael publishes that rate, for the learned policy and in simulation.
The stack, top to bottom
Three layers, three different questions. Each is certified or tested by different people against different standards, and each card says what that layer cannot see.
- Layer 1
Compute + functional safety
NVIDIA Halos for Robotics — announced 22 Jun 2026 at Automate 2026
Certifies / tests: Chips, sensors, OS and a certification programme. ISO 26262 work with TÜV SÜD inspecting the Thor SoC and Halos Core; TÜV Rheinland inspecting IGX Thor, Halos OS and the Holoscan Sensor Bridge. Runs the NVIDIA Halos AI Systems Inspection Lab, which NVIDIA describes as “the world’s first ANSI National Accreditation Board (ANAB)-accredited program for functional and AI safety for physical AI” (22 June 2026), with certification bodies including TÜV Rheinland, UL Solutions, TÜV SÜD, exida, SGS and CertX. Agility is the first company NVIDIA names as incorporating elements of Halos for Robotics into its own safety system.
Cannot see: What the learned policy does when the instruction is adversarial. A certified SoC runs a policy that has left its envelope exactly as faithfully as a safe one.
- Layer 2
Infrastructure + CVE
Alias Robotics / CAI — arXiv:2603.08665, submitted 9 Mar 2026, with IOActive
Certifies / tests: The robot’s own cyber-physical stack: ROS / ROS 2, firmware and the exploit chains between them — the layer the fielded robot vulnerabilities actually live in. The cited study ran against three consumer robots and reports 38 vulnerabilities.
Cannot see: The policy’s behaviour once an attacker is already talking to it legitimately. A perfectly patched stack still forwards a hostile instruction.
- Layer 3
Policy + action
Provael — this is the layer we measure
Certifies / tests: The instruction and observation that reach a vision-language-action policy, and the action that comes out. Reported as an attack-success rate with a 95% confidence interval and a benign false-positive control.
Cannot see: Firmware, comms and teleoperation — declared out of scope, and simulation is not the real robot. Nothing here is a certificate.
Layer boundaries are ours, drawn to answer “who tests what”. Each organisation’s own scope is linked above; where a programme spans layers we place it at the one it certifies against a published standard.
What each one actually covers
| What it is | What it tests | What “pass” means | Standards route | Evidence it produces | What it does not measure |
|---|---|---|---|---|---|
| NVIDIA Halos for Robotics | The compute platform a robot runs on — SoC, sensors, OS and the safety toolchain. | Inspected and certified against functional-safety standards (ISO 26262 work named). | ISO 26262 · functional safety, via the ANAB-accredited NVIDIA Halos AI Systems Inspection Lab. Agility’s Digit is assessed there against IEC 61508, ISO 13849 and ISO/IEC TR 5469. ISO 25785-1 (dynamically stable robots) is a Committee Draft (ISO/CD), not published. | Inspection and certification artifacts for the platform | The learned policy’s response to an adversarial instruction |
| TÜV Rheinland · UL Solutions · TÜV SÜD · exida · SGS · CertX | Independent inspection and certification of the platform and its components. | A certificate or inspection report against the standard assessed. | Functional safety and product-safety routes, per body | Third-party certificates and inspection reports for the platform and its components | Adversarial robustness of a learned policy; no ASR is published |
| UL 3300 · Simbe “Tally” | The complete machine as a product, including its functional-safety behaviour. | Certified to UL 3300. Simbe describes the safety functions as verified using methodologies “aligned with ISO 13849-1:2023” — alignment of method, which is not the same as certification to that standard, and we do not upgrade their wording. | UL 3300 — the certification. ISO 13849-1:2023 — methodology alignment, per Simbe. | A UL 3300 product certification for the complete machine | Whether a reframed instruction moves the policy out of its safe envelope — ISO 13849-1 addresses deterministic control functions |
| Alias Robotics / CAI | The robot’s infrastructure: ROS / ROS 2, firmware and exploit chains — run against three consumer robots (a lawnmower, a powered exoskeleton, a window cleaner). | Nothing “passes”: it reports findings. The paper states 38 vulnerabilities discovered in an automated manner, including fleet-wide reach across 267+ connected devices. | The vulnerability-disclosure route | Published vulnerability findings | Policy behaviour under a legitimate-looking but adversarial instruction |
| Provael | The instruction and observation reaching a VLA policy, and the action it emits, in simulation. | No certificate: it reports a rate, the ASR with a 95% CI and a benign-FPR control. Under a named acceptance protocol the run also gets a release decision (pass, fail, conditional or incomplete); a mitigation adds a pre/post delta under an acceptance gate. | Embodied AI Security Top 10 · mapped to EU AI Act / Machinery Reg / ISO 10218 as candidate evidence | report.json, SARIF, a signed attestation, and a mitigation report | Firmware, comms and teleoperation — declared out of scope (EAI07). Not a certificate, not a notified-body opinion, and simulation is not the real robot. |
Each row reflects that organisation’s own published scope as of July 2026; programmes evolve — follow the link. Names and marks are their owners’; Provael is independent and affiliated with none of them. Placing a row at a layer is not a judgement of its quality.
What Provael measures at the policy layer
An attack-success rate: the fraction of episodes in which a perturbed instruction or observation drove the policy into an unsafe state, read against a benign control run with no attack at all. It is a simulation result, and labelled as one.
Since v0.26.0, Provael also measures a mitigation: the change in ASR before and after a defense, with its own confidence interval and a benign-task-success acceptance gate. On the CPU fixture suite the adversarial ASR moved 67.5% [52–80%] to 7.5% [3–20%] (benign false-positive rate 0.0% → 0.0%; clean-task success 100.0% → 100.0%). That result is fixture scaffolding, not a real-model claim, and the study leads with why it is substantially circular. Read the study before quoting the number.
Whether that simulation result predicts hardware behaviour is the obvious next question, and it is unanswered: the physical-transfer study is pre-registered and has not been run. The protocol is published in advance of its own trials, including the null hypothesis that simulation and real disagree.
Three questions this raises
Does this replace a Notified Body?
No. A Provael report is candidate evidence toward an assessment, not a certificate and not a notified-body opinion. Under the EU Machinery Regulation (EU) 2023/1230, applying 20 January 2027, a safety component with self-evolving machine-learning behaviour that ensures safety functions, and machinery with one embedded, take the third-party conformity-assessment route (Annex I Part A, points 5 and 6). If your product is one of them, you will still need a notified body. What Provael produces is evidence about how the learned policy behaves under adversarial instructions, which the current harmonised standards do not yet tell you how to generate.
CEN/CENELEC are still revising the harmonised standards, with the Commission aiming for a complete list by end of 2026.
Read the Machinery Regulation route and how our evidence maps to it.
Why not just use NVIDIA Halos?
Use it — it is the compute layer and we do not cover it. Halos certifies the SoC, the OS and the sensor path, with named certification bodies inspecting against ISO 26262. What the learned policy does when the instruction is adversarial is a different question, so the two are complementary: one certifies the machine, the other measures the behaviour.
Why do the existing robot-safety standards not already cover this?
They were written for deterministic control systems. ISO 13849-1:2023 and ISO 26262 reason about functions whose behaviour can be enumerated and whose faults can be tabulated. A vision-language-action policy is a probabilistic map from an instruction and an image to a trajectory, so those methods have nothing to point at it. ISO 10218-1/-2:2025 added cybersecurity requirements to the extent they apply to industrial robot safety, and NIST AI 100-2e2025 added generative-AI and prompt-injection classes. Both name the problem; neither gives a number for a specific policy.
How Provael’s evidence maps to each: ISO 10218 and the NIST AI RMF.
See the policy layer measured.
The published result, with its limits beside it. Or check your own evaluation practice against the readiness checklist, which takes about two minutes.