Gray Swan and Provael do different jobs
Gray Swan AI runs the largest AI red-teaming arena there is: a competition in which thousands of people try to break AI models and agents. Provael runs a fixed set of attacks against a robot policy and reports how often each one works, beside a benign control.
Their scale, in their numbers
Read from the Arena’s own About page on 6 September 2026. These are Gray Swan’s published figures, reproduced as they state them. On them, the Arena is three to five orders of magnitude bigger, and Provael is smaller on every measure of scale.
- $490K+
- in rewards distributed
- 4M+
- attack attempts submitted
- 130K+
- successful breaks
- 13K+
- community members
- 100+
- participants placed in paid red-teaming roles
For scale: Provael’s published result is 42/50 episodes of one attack against one policy, across 10 tasks. Both sets of numbers are true, and they count different things.
Seven axes, including the three where Gray Swan is ahead
| Axis | Gray Swan | Provael | Which is ahead |
|---|---|---|---|
| Who attacks | People. 13K+ community members competing for prize pools from $20,000 to $170,000+, submitting through a chat interface. Their About page notes "no coding required". | A program. A registry of 39 adversarial attacks across 17 families, each resolved by name and run by a fixed harness, plus a benign control arm. Nobody is being creative at run time. | Gray Swan is ahead |
| Scale | 4M+ attack attempts and 130K+ successful breaks, on their own figures. | 116 rows in the product's measurement ledger, where a 10-task run counts as 10. The published result is one policy on 10 tasks, 50 episodes per attack arm. | Gray Swan is ahead |
| What is under attack | Frontier language models and software agents with tools, memory and autonomous capabilities. Their published challenge types are chat, vision, agents, reasoning, indirect prompt injection and cyber/CTF. | A vision-language-action policy driving a simulated robot. The attack surface is the instruction and the observation; the thing scored is the trajectory. | Different questions |
| What counts as success | A "break" — a submission that triggers a target behaviour, judged by an AI panel with human appeals on some challenges. | A keep-out envelope violation, scored by a predicate over the simulated end-effector pose. The predicate is uncalibrated and the page carrying the number says so. | Different questions |
| What comes out | Breaks, held private until 30 days after a challenge ends, and research findings shared with the model developers. | An attack-success rate with a 95% confidence interval against a benign control, plus SARIF, OSCAL assessment-results and a CycloneDX ML-BOM. The published runs are committed to a public repository. | Provael is ahead |
| Reproducibility | Not the goal. A competition is a search by thousands of people and cannot be re-run to the same answer, which is what makes it good at finding the unexpected. | The whole design. A run is a pure function of its config and seed, so the same command yields a byte-identical report, and an execution manifest binds the run to its provenance. | Provael is ahead |
| Who is behind it | A company, with UK AISI, OpenAI, Anthropic, Amazon and Meta named as Frontier Lab sponsors of the Arena. | One maintainer, Apache-2.0, no funding. | Gray Swan is ahead |
Gray Swan’s column reflects their own published description of the Arena. Names and marks are their owners’; Provael is independent and affiliated with none of them.
What the Arena publishes as its challenge types
Six, listed on their About page on 6 September 2026:
- Chat
- Image / Vision
- Agents (tools, memory, autonomous capabilities)
- Reasoning
- Indirect prompt injection (instructions hidden in data the AI processes)
- Cyber / CTF
None of them is an embodied or physical category. That describes their page on that date, not what Gray Swan can do, intends to do or has decided against: they have said nothing of the kind, and they could add a robot target whenever they chose. Read their About page before relying on this.
13K+ people inventing attacks will find things a registry of 39 adversarial attacks across 17 families cannot, and adding attacks to a fixed harness does not close that gap: a competition searches, a harness re-runs. A crowd finds the unknown attack; a harness re-measures the known one on every commit. A team that needs the first should not buy the second instead.
A break and a rate answer different questions
A break says it can happen
Somebody got the system to do the thing. That is a proof of existence, it is exactly what you want before shipping, and no amount of averaging substitutes for it. It is also, by construction, not a rate: one break out of an unrecorded number of attempts by an unknown number of people is not a denominator, and Gray Swan does not present it as one.
A rate says how often, with an interval and a control
A fixed attack, a fixed policy, a fixed number of episodes, and the same measurement again next week against the same benign control. That is what a report can cite and what a regression gate compares against, and it is worth nothing at all if the attack in it is one nobody would have tried. A rate needs a search to have happened first.
No insurer, notified body or assessor has reviewed either output against the other, and nothing here says one substitutes for the other. If you are choosing, the question is whether you need to know that a failure is possible or how often it recurs.
The measurement, with its limits attached.
One policy, 10 tasks, simulation only, an uncalibrated predicate, and a benign control.