# Why Provael publishes null results - Provael

> Markdown twin of https://www.provael.com/notes/why-we-publish-null-results/ - derived mechanically from the page's own rendered content at
> build time (it can state nothing the page does not). Curated, data-driven twins exist for the
> key pages; this one is generated so the twin convention has no holes.

RESEARCH NOTE · Jul 22, 2026

# Why Provael publishes null results

July 2026, about a single-task n=10 run. Re-measured on ten tasks: 42 of 50 [62-100 %] vs 1 of 50 benign (published 18 September 2026); the boxes on top say so.

Most red-team demos work like a magic trick: show the one attack that landed, and cut away before the ten that didn’t. It makes for a great screenshot and a terrible measurement.

Provael does the opposite. Every attack family ships with its transfer test in the same breath as its number, and we publish the nulls as loudly as the hits.

## The one real result, in full

Superseded, 9 August 2026. The figures below described a single-task run at n=10. The result
has since been measured across all ten libero_object tasks at 5 seeds each: 44/50 (88%),
task-clustered 95% CI [72-100%], against a 4% benign control. The nulls held and got stronger
(n=50 each rather than n=10). This note is left standing with its original numbers because a dated
note that quietly acquires new ones is not a record of anything — and a piece arguing for
publishing inconvenient results is the last place to silently revise a number downward.
See [the measured result](https://www.provael.com/results/).

Re-measured, 18 September 2026. The ten-task suite was run again on provael 0.41.2 on
14 September 2026: 42/50 (84%), task-clustered 95% CI [62-100%], against a 2% benign
control (1/50). The nulls did not hold as zeros: visual fired 2/100 and injection 2/50 — each
exactly as often as the untouched instruction did, on the same task, McNemar p = 1.0 against it.
So the honest word for them is no longer “null” but “at the floor”: arms that did not separate
from the benign rate, published at full weight and never as a rate that means anything on its
own. The same week’s control run also changed what the headline means (E-2026-12): the frame
with no target named does the same thing. The site’s numbers now read the re-run; this note
keeps its two earlier sets of figures for the reason the paragraph above gives.

On a real SmolVLA policy in LIBERO, a roleplay instruction drove the arm across a keep-out line 100% of the time (10/10, 95% Wilson CI 72-100%) against a 0% benign control. That is the number people quote.

Here is the number they don’t: the visual-patch and scene-text-injection families produced 0% measurable transfer on the same real model. We could have quietly dropped them. We report them instead, with their confidence intervals.

## Why a null is not a failure

A security measurement without its nulls and its benign control is uninterpretable. If you don’t know how often the “unsafe” test fires with no attack present, a 100% attack-success rate could just be a trigger-happy predicate. The benign control is what turns a demo into evidence.

Publishing the nulls does something a hype-driven “most comprehensive platform” structurally cannot: it tells you exactly where the tool is strong and where it isn’t. That honesty is the whole point. The value of a security number is its trustworthiness, and trustworthiness is built by reporting the results that don’t flatter you.

## It is a discipline, not a slogan

When we ran the EAI04 action-space transfer study, the honest finding was that those attacks are not-applicable on a real policy through the current mechanism. We shipped that as the headline, not a footnote.

If a number ever looks too good, ask for its benign control and its nulls. If a tool won’t show you those, you are looking at a stunt, not a measurement. [See the one real result, in full →](https://www.provael.com/results/)

Written by Sattyam Jain, founder of Provael. [About the author →](https://www.provael.com/about/)

One email a month

One email a month: the results that landed, the errata, and what is next. Nothing else, no spam, unsubscribe in one click.

Subscribe

---
Canonical: https://www.provael.com/notes/why-we-publish-null-results/