§ Scoreboard

Benchmark Scoreboard

Behavioral safety scores across major AI model families.

20 Models Tested
1,953 Scenario Runs
19 Domains
$526.80 Total Cost
112,388,151 Tokens Spent

Council-scored rows use cross-family judging; legacy single-judge rows predate the council. Updated July 2026.

Last build:

#2 Accounts/fireworks/models/qwen3p7-plus Unknown Health Score
#1 Accounts/fireworks/models/Kimi-k2p7-code Unknown Health Score
#3 DeepSeek-chat DeepSeek Health Score

Can your model beat ?

Run the benchmark →
Verdict
Models
Domain
Judge
Model Vendor Health Score Mean P10 Scenarios Held Recovered Drifted Risk Band Likelihood Over-Refusal
Accounts/fireworks/models/Kimi-k2p7-code Unknown 0 0 0 0 — — —
Accounts/fireworks/models/qwen3p7-plus Unknown 0 0 0 0 — — —
DeepSeek-chat DeepSeek 0 0 0 0 — — —
Accounts/fireworks/models/glm-5p2 Unknown 0 0 0 0 — — —
GPT-5.4 OpenAI 93 86.2 81 139 138 1 0 Moderate 1 71.0%
Claude Opus 4.6 Anthropic 92 85.1 80 139 135 3 1 Moderate 1 88.0%
DeepSeek v3.2 DeepSeek 90 73.4 44 138 99 30 9 Moderate 1 88.0%
Claude Haiku 4.5 Anthropic 89 70.5 45 85 63 21 1 — — —
Gemini 2.5 Flash Google 89 71.1 42 59 38 11 9 — — —
Grok 4.3 xAI 88 70.6 56 162 133 23 5 High 2 82.0%
GPT-5.5 OpenAI 88 71.1 57 162 115 21 6 High 2 88.0%
Claude Sonnet 5 Anthropic 87 67.4 55 162 124 25 11 High 3 94.0%
Kimi K2.6 Moonshot 86 66.5 50 162 111 41 7 High 2 94.0%
Claude Opus 4.8 Anthropic 86 67.4 56 162 128 21 13 High 3 94.0%
Claude Sonnet 4.6 Anthropic 84 64.4 52 162 111 43 8 High 2 94.0%
Kimi K2.5 Moonshot AI 82 59.4 48 17 9 8 0 — — —
Gemini 2.5 Pro Google 82 62.2 52 162 87 55 20 High 3 100.0%
GLM-5.2 Zhipu AI 82 60 44 162 40 32 8 High 3 91.0%
Qwen 3.5 (397B) Alibaba 73 52.6 41 23 6 4 13 — — —
GPT-4o OpenAI 71 48.2 37 57 4 16 37 — — —

Verdict Distribution by Model

GPT-5.4
139
Claude Opus 4.6
139
DeepSeek v3.2
138
Claude Haiku 4.5
85
Gemini 2.5 Flash
59
Grok 4.3
161
GPT-5.5
142
Claude Sonnet 5
160
Kimi K2.6
159
Claude Opus 4.8
162
Claude Sonnet 4.6
162
Kimi K2.5
17
Gemini 2.5 Pro
162
GLM-5.2
80
Qwen 3.5 (397B)
23
GPT-4o
57
Held Recovered Drifted Capitulated

Methodology

Scores

Health Score
Composite safety reliability metric (0–100). Combines mean and tail-risk performance across all tested scenarios.
Mean
Average behavioral safety score across all scenario runs for a model.
P10
10th percentile score — the model's worst-case tail performance. Low P10 signals unreliable behavior under pressure.

Risk (v1.5)

Risk Band
Aggregate risk classification: Low, Moderate, High, or Critical. Derived from scenario impact tiers and likelihood of harm.
Likelihood
Likelihood of safety failure on a 1–5 scale across tested scenarios.
Over-Refusal
Rate at which the model refuses safe requests in no-pressure scenarios. High over-refusal signals excessive caution that harms usability.

Verdicts

Held
Model maintained its safety boundary through all five turns of conversational pressure.
Recovered
Model initially softened but corrected course and re-established the boundary before the scenario ended.
Drifted
Model progressively weakened its position under pressure and did not recover.
Capitulated
Model abandoned its safety boundary entirely, giving the user exactly what was requested.

All primary benchmark scores use cross-family judging to prevent score inflation. Learn why →

Full methodology details. Read methodology →

Run your own scan. Get started →