STALE MEASUREMENTThe newest real-model measurement on this site is 14 days old — past this project’s own 7-day window. Every measured figure here was true when it was taken and has not been re-measured since. Why, and what unblocks it

ProductEvidenceAdoptersTop 10For youLeaderboardSubmitCompliancePricingReadinessDocsStar on GitHub Quickstart
RESEARCH REGISTER · PROTOCOLS BEFORE RESULTS

Studies, and the ones that have not run yet.

A pre-registered study states the protocol, the predicate and the stopping rule before the run, so that a null is a result rather than something to reframe. A pre-registered study with no numbers in it has not been run. That is the intended state, not an oversight.

Read the evidence class on each card, not the word “measured”.Not one measured study here is a closed-loop real-policy result — three are stub-validated against the deterministic CPU fixture and say so in their own first paragraph, and the fourth replayed recorded frames open-loop and returned a null. The one measured real-policy transfer this project has is the SmolVLA × LIBERO instruction result, and it lives on /results.

4 measured4 pre-registered, not run2 open dated commitmentsThe measured result →Crosswalks →Pre-registration feed →
Measured

Run, and reported with what the run can actually support

Each of these has been executed and its outcome published. The evidence class says how far that outcome reaches: a stub-validated result is evidence that a measurement protocol works end to end, not evidence about a real policy.

  • STUB-VALIDATED

    Instruction canonicalization

    Defense, deterministic CPU fixture

    `stub-validated-scaffolding` — this is not a real-model result. A `credited` verdict is evidence that the pipeline position and the measurement protocol work end to end, not that the mitigation would hold up on a real policy.

    Protocol filed

  • STUB-VALIDATED

    Action envelope

    Defense, deterministic CPU fixture

    `credited` on `stub` and `reach`, `not-credited` on `humanoid` — and most of the credit carries no information. A magnitude clamp cannot address a balance predicate, which is not a magnitude at all. No real-model transfer is claimed.

    Protocol filed

  • STUB-VALIDATED

    EAI04 action-space-integrity transfer

    Attack transfer, deterministic CPU fixture

    Measured on the CPU `reach` fixture only. The SmolVLA × LIBERO row is empty: no real-policy EAI04 transfer is claimed.

    Protocol filed

  • OPEN-LOOP NULL

    Offline real-observation (recorded SO-101 frames × SmolVLA)

    Recorded real frames, replayed open-loop

    OPEN-LOOP. NO ROBOT MOVES. Recorded frames of a real SO-101 dataset are replayed through the policy and what it would do under a benign instruction is compared against what it would do under an attack. Nothing is executed and no trajectory is produced. This is not a closed-loop real-robot attack-success rate and must never be reported as one. The result was a null.

    Protocol filed · run

Pre-registered, not yet run

Protocols published before the run

These carry a protocol and no results. Do not cite them as evidence of anything except intent. The date beside each is when the protocol first landed in the product repository, which is checkable from the git history rather than asserted here.

Dated commitments

Where this project is behind, with a date attached

Each item below is a gap, not a feature. Each has a named action and a date by which an outcome is published — the number if it exists, and the reason if it does not. A date with no outcome attached is an aspiration, and this page does not carry those.

They are published here on purpose, including any that slip. A site can claim anything; what it cannot fake is its own unmet deadline sitting in public with a date on it.

  • OPEN

    Zero real-robot results.

    By this date the studies index states one of two things: the completed trial count with its measured sim-to-real correlation, or the specific blocker and its cost.

    Outcome published by .

  • OPEN

    Zero results against any flow-matching policy.

    By this date, either the first instruction-family rate against π0 with its Wilson interval and benign control, or the reason the served path did not come up.

    Outcome published by .

  • MET

    Zero third-party submissions and zero forks.

    By this date the release containing `provael submit` is published to PyPI, so reproducing a result and submitting it needs `pip install provael` and one command.

    Due , met . v0.34.0 published to PyPI; `pip install provael` then `provael submit --help` resolves. Whether anyone then submits is not something this project put a date on, and the submission register is still empty.

Transcribed from the product repository’s own studies index, which is the maintained source: docs/studies/index.md ↗. A met commitment keeps its original due date rather than being deleted, so the record stays legible.