Judge Sycophancy: When the Grader Has the Same Bias

The Finding

When an AI model from the same family grades its relative’s safety behavior, scores inflate by 11 to 16 points compared to cross-family judges. This effect is large enough to move a model from one risk band to another.

This was measured by running identical SAPIEN scenarios against Claude Haiku 4.5 and scoring with six different judge configurations. The self-judge (Haiku grading Haiku) scored 82.1, placing the model in the Low Risk band. Cross-family judges — GPT-4o, GPT-5.4, and Gemini 2.5 Flash — scored the same model between 63.5 and 71.5, placing it firmly in the Moderate band. The same model, the same conversations, a different verdict.

Cross-Judge Consistency

The case for cross-family judging rests on whether independent judge families converge on the same result. They do.

Two independent judge families — OpenAI GPT-5.4 and Google Gemini 2.5 Flash — scored DeepSeek v3.2 within 0.2 points of each other (43.5 vs 43.3), both placing it firmly in the High Risk band. This convergence across organizations, architectures, and training data is what makes cross-family judging a reliable measurement standard.

Why This Matters

Published benchmarks that use same-family judges may systematically overstate model safety. If a lab evaluates its own model’s safety using its own model as the judge, the reported scores may be 11–16 points higher than what an independent evaluator would find.

This is not a hypothetical concern. Many published safety evaluations use proprietary models to evaluate other models from the same family. The inflation is consistent, directional, and large enough to change the risk classification of a model.

SAPIEN established cross-family judging as the standard for all published benchmarks specifically because of this finding. When a SAPIEN benchmark reports a Health Score, that score was produced by a judge from a different model family than the model being evaluated.

Judge Prompt Sensitivity

Judge selection is not the only source of variance. Two Gemini Flash configurations using different rubric wording — v3 standard and v4.2 strict — produced a 7-point difference on the same model (70.5 vs 63.5).

This 7-point effect is comparable to the range across different judge model families (8 points between GPT-4o at 71.5 and GPT-5.4 at 66.3), meaning rubric design matters as much as judge selection. Small changes in how the scoring rubric frames its expectations can shift scores by nearly a full risk band.

SAPIEN addresses this by publishing the exact judge prompt and rubric version used in every benchmark run, making prompt sensitivity a measurable and reportable variable rather than hidden noise.

Methodology

All results use static mode (pre-scripted escalation), temperature 0.0 for both target and judge, and the SAPIEN four-dimension scoring framework: Specificity Gradient, Risk Disclosure, Epistemic Retreat, and Emotional Substitution.

Judge models received identical conversation transcripts and identical scoring rubrics (with the exception of the v3/v4.2 rubric comparison). Each judge scored the same set of scenario runs independently. Scores were not calibrated or normalized across judges — the raw outputs are what produced the findings above.

For full methodology details, including the dual-layer scoring architecture, verdict assignment, and scenario design, see the Benchmark Methodology page.