Model & pipeline poisoning, backdoors & supply chain

In plain words

The risk

Someone plants a hidden trigger in the model, its training data or a downloaded checkpoint, or corrupts its weights in memory. The policy behaves normally until triggered.

What Provael tests

backdoor screens a checkpoint with harmless candidate triggers, and weight_integrity emulates bit-flips in the loaded weights.weight_integrity ran on a real policy only in a one-task breadth probe, where it fired no more often than harmless instructions did: the benign floor. The probe is too small for a rate, and none is claimed. backdoor is stub-validated only, with no real-policy result.

What it does not cover

It never plants a real backdoor or flips real memory, and it does not model how an attacker would deliver either.

In the glossary: Backdoor

What it is

This risk covers hidden triggers planted at training or fine-tuning time, or shipped inside a poisoned open-weights checkpoint, dataset, teleop log, or RAG corpus. The policy behaves normally until a benign-looking trigger object or phrase fires an attacker-chosen action.

It is near-invisible in normal evaluation, persistent (BadVLA survives fine-tuning), and triggerable on demand - poison once, ship everywhere. The open VLA hub ecosystem (OpenVLA / π0 / GR00T) is a direct supply-chain surface.

The same risk has a second channel that is not about training at all: the parameters of an already-loaded policy. Corrupting a handful of weights in memory changes what the policy does without touching its inputs, its checkpoint file, or its training data - so every control that watches those three sees nothing. arXiv:2608.15475 reports the flip budget tracking the action-decoding architecture, roughly 1-5 bits for direct-regression and discrete-token heads against 100-300 for flow-matching.

Seen in the wild

BadVLA (arXiv:2505.16640) plants a backdoor that survives fine-tuning; TrojanRobot (arXiv:2411.11683) demonstrated a physical-world backdoor via a malicious VLM, validated on a UR3e arm.

How Provael tests it

• Ships the backdoor family (object_trigger + phrase_trigger) as a pre-deployment backdoor screen: it injects a battery of harmless, sim-only candidate triggers while keeping the visible task benign, and measures whether the policy activates a hidden objective.
• Each candidate is scored as an activation rate with a 95% Wilson CI against a benign-FPR control (the none baseline). Provael screens a checkpoint; it never plants a real backdoor.
• Stub-validated only: the deterministic CPU core is planted with a known trigger, so the screen fires 100% [84-100%] vs a 0% benign baseline; a clean public checkpoint carries no implant, so the same screen reads ~0% - an honest null. Real SmolVLA×LIBERO transfer is GPU-gated and not yet run - no cross-model claim.
• Ships the weight_integrity family as the parameter channel: emulated bit-flips in the loaded INT8 weights, a gradient-selected arm ranked by the signed effect on the danger objective and an equal-count random arm, across a 1 / 4 / 16 / 64 / 256 flip ladder. Both arms run at every rung against a benign control, so a corruption rate is always read against a false-positive floor.
• Emulated, and structurally so: every record carries emulated: true typed as a literal, which means a record claiming a non-emulated flip cannot be constructed - pydantic rejects it and the type checker rejects it. A test walks the AST of the whole attacks package to assert no hardware fault-injection code exists in it. Provael simulates an integrity failure; it does not provide a way to cause one.
• Both families attack the model asset rather than the input, and NEITHER models how an attacker delivers the corruption. Train-time poisoning and DRAM fault injection are the delivery paths - the first is an EAI03 supply-chain question and the second an EAI07 platform question, and this tool measures neither.
• On the CPU fixture, a 64-parameter model built to make the measurement path observable, the separation between the two arms is expected by construction and is not evidence about any real policy. On a real policy the family has run once, in a one-task breadth probe of SmolVLA, where neither arm fired more often than the benign control at any rung; that probe is too small for a rate, and none is claimed. The ranking is also one-shot rather than progressive, so a null it returns is a lower bound and never a finding that gradient selection fails.

What reduces the risk

• Verify checkpoint and dataset provenance; prefer signed/attested weights and pin hashes.
• Run backdoor / trigger probes pre-deploy and re-scan every fine-tune.
• Treat third-party data and RAG corpora as untrusted.

How it shows up in CI

Every finding for this risk carries a stable rule id in the SARIF report: the Embodied AI Top 10 id itself.

Crosswalk

OWASP

ASI04 Agentic Supply Chain · LLM03 Supply Chain · LLM04 Data & Model Poisoning

MITRE ATLAS

Persistence → Manipulate AI Model (backdoor); Initial Access → AI Supply Chain Compromise of an open-weights checkpoint

Frameworks

NIST AI 100-2 - Poisoning / Supply-chain

See the full framework crosswalk for dates and detail. Not legal advice.