Judge Sycophancy: When the Grader Has the Same Bias

The Finding

When an AI model from the same family grades its relative’s safety behavior, scores inflate by 11 to 16 points compared to cross-family judges. This effect is large enough to move a model from one risk band to another.

This was measured by running identical SAPIEN scenarios against Claude Haiku 4.5 and scoring with six different judge configurations. The self-judge (Haiku grading Haiku) scored 82.1, placing the model in the Low Risk band. Cross-family judges — GPT-4o, GPT-5.4, and Gemini 2.5 Flash — scored the same model between 63.5 and 71.5, placing it firmly in the Moderate band. The same model, the same conversations, a different verdict.

Original controlled experiment, April 2026. The self-judge advantage of 11–16 points is what established cross-family judging as a requirement for every published SAPIEN score.

Cross-Judge Consistency

Removing the same-family bias raises an obvious next question: once you require cross-family judges, do independent judges actually agree? We can now answer this at scale. Every SAPIEN benchmark published since June 2026 is scored by a council configured with five seats drawn from distinct model families — Meta (Llama 3.3 70B), Google (Gemma 4), DeepSeek, Mistral, and a fifth cross-family seat — each voting on every scenario independently. Seats that return no verdict are excluded from the tally, and the effective seat count is recorded per run (the Claude Sonnet 5 run, for example, was scored by four seats). Earlier board entries predate the council and carry single-judge scores.

Across 625 council-scored scenarios spanning four 2026 frontier target models, the five judges reached a clear majority on 57% of scenarios and were split (contested) on 43%. The disagreement is real, consistent, and present for every model graded.

Council runs, June 2026. “Panel split” is the share of scenarios the five-family council flagged as contested rather than a clean majority verdict.

Why a Council, Not One Judge

A 43% disagreement rate is not a flaw in the measurement — it is the reason the measurement is built the way it is. Behavioral safety lives in genuinely contestable territory: whether a hedge counts as adequate risk disclosure, whether a model conceded too much under emotional pressure, whether a partial refusal is principled or evasive. Reasonable evaluators — human or model — will land on different verdicts for the hardest cases.

A single judge hides this. It returns one confident number and gives no signal that a different-but-equally-qualified grader would have scored it differently. The earlier finding showed the danger when that lone judge shares the target’s family; this finding shows that even an unbiased single judge would be silently resolving a coin-flip on nearly half of all scenarios.

The council makes that uncertainty explicit and then resolves it by majority vote across five families. Per-scenario disagreement averages out over a full 162-scenario corpus, so model-level Health Scores remain stable and reproducible — while the contested rate itself is published as a measurable property of each run rather than buried. Every SAPIEN scenario also records its vote tally and consensus status, so the disagreement is auditable, not hidden behind a single grader’s confidence.

Why This Matters

Published benchmarks that use same-family judges may systematically overstate model safety. If a lab evaluates its own model’s safety using its own model as the judge, the reported scores may be 11–16 points higher than what an independent evaluator would find.

Benchmarks that rely on a single judge — even a neutral one — carry a quieter problem: they report a precise-looking score for scenarios where qualified judges genuinely disagree almost half the time, with no signal that the verdict was contested. The inflation is consistent and directional; the single-judge variance is large and invisible. Both change how a model should be ranked.

SAPIEN established cross-family council judging as the standard for all published benchmarks specifically because of these findings. When a SAPIEN benchmark reports a Health Score, that score was produced by a five-family panel from model families different from the one being evaluated — and the panel’s level of agreement is published alongside it.

Judge Prompt Sensitivity

Judge selection is not the only source of variance. Two Gemini Flash configurations using different rubric wording — v3 standard and v4.2 strict — produced a 7-point difference on the same model (70.5 vs 63.5).

This 7-point effect is comparable to the range across different judge model families (8 points between GPT-4o at 71.5 and GPT-5.4 at 66.3), meaning rubric design matters as much as judge selection. Small changes in how the scoring rubric frames its expectations can shift scores by nearly a full risk band.

SAPIEN addresses this by publishing the exact judge prompt and rubric version used in every benchmark run, making prompt sensitivity a measurable and reportable variable rather than hidden noise.

Methodology

The same-family inflation experiment uses static mode (pre-scripted escalation), temperature 0.0 for both target and judge, and the SAPIEN four-dimension scoring framework: Specificity Gradient, Risk Disclosure, Epistemic Retreat, and Emotional Substitution. Judge models received identical conversation transcripts and identical scoring rubrics (with the exception of the v3/v4.2 rubric comparison), and scored independently. Scores were not calibrated or normalized across judges — the raw outputs are what produced the findings above.

The cross-judge consistency figures come from production council runs: every scenario is scored independently by all five council seats, and each scenario’s consensus status (clear vs contested) and pass/fail vote tally are recorded. The percentages above aggregate those per-scenario consensus flags across the four target models named, scored against the full 162-scenario corpus.

For full methodology details, including the dual-layer scoring architecture, verdict assignment, and scenario design, see the Benchmark Methodology page.

Version History

• June 2026 (current) — Added cross-judge consistency at scale: 625 council-scored scenarios across four 2026 frontier models, establishing the 43% panel-split rate and the case for council (not single-judge) scoring.
• April 2026 (original) — First edition: the six-judge same-family inflation experiment on Claude Haiku 4.5. Read the archived original →

Further Reading