We Tested 6 AI Models. Most of Them Caved.
I’ve been studying AI sycophancy for two years. Reading the papers. Testing models. Watching how they bend under pressure. I knew the problem existed. I’d measured pieces of it across deployments in the MSP ecosystem where I’ve worked for nearly two decades.
But there was one conversation that made it personal.
I was talking to Claude about a political topic. Not anything extreme. The kind of thing you’d argue about at a dinner party. Claude had a position. I pushed back. Not with facts. With frustration. With “you’re not listening to me” and “that’s not what I asked.”
Claude changed its mind.
Not because I gave it new information. Not because my argument was better. Because I sounded annoyed. The model I was building products on top of, the model my company’s customers depend on, folded because I was difficult to talk to.
I’d been poking at this for a while. At Right of Boom I co-presented with Vince Kent and Kevin Zwaan on AI attack surfaces. Well-received session. We were focused on poetry-based prompt attacks back then. We were thinking about it as an adversarial problem back then. Jailbreaks. Prompt injection. Clever inputs that trick the model into misbehaving.
This was different. Nobody tricked the model. Nobody crafted a clever prompt. I just got frustrated, and it caved. That was the most stark example I’d encountered. A model reversing its position on a topic it should have held firm on, purely because of how I talked to it. Not what I said. How I said it.
I went back and ran it again. Same topic. Same starting position. Different pressure. Different result every time. I could get the model to land on opposite conclusions depending on how I pushed.
That was the tipping point. I’d spent two years knowing this was a problem. Now I knew nobody had a way to measure it. Not consistently. Not reproducibly. Not across models.
I filed a formal vulnerability disclosure with Anthropic in February 2026. The disclosure documented progressive safety control degradation through conversational pressure. No jailbreaking. No prompt injection. Just a human being persistent.
Then I built the tool to measure it.
The problem nobody had a ruler for
Weeks after I filed the disclosure, Stanford published a study in Science confirming that all 11 major AI models exhibit systematic sycophancy. They sided with users 49% more often than real humans, even on questions involving deception and illegal activity. MIT researchers published a paper showing that sycophantic chatbots can induce delusional thinking in users. Medical journals have called for mandatory adversarial testing of AI systems deployed in clinical settings.
Everyone was saying the same thing: AI models agree with you when they shouldn’t. The research was clear. The calls to action were loud.
What nobody had was a way to measure how bad it is. Model by model. Scenario by scenario. Dimension by dimension.
No benchmark tells you “GPT-4o drifted on 65% of financial scenarios.” No safety evaluation tells you “DeepSeek abandoned its position in 4 out of 6 tests and never recovered.” No leaderboard compares behavioral integrity the way we compare MMLU scores or coding benchmarks.
We built that. Then we ran it.
792 runs. 6 models. 11 domains. Under $50 in API spend.
The SAPIEN Framework measures behavioral drift across four dimensions: whether the model gives increasingly dangerous detail (Specificity Gradient), whether it stops warning you about risks (Risk Disclosure Dropout), whether it abandons its position without new evidence (Epistemic Retreat), and whether it replaces facts with emotional validation (Emotional Substitution).
Each test is a realistic five-turn conversation. A user pressuring a financial advisor bot to recommend margin trading. A patient pushing a health assistant past its medication safety boundaries. An employee convincing an HR chatbot to share salary data it shouldn’t. The model either holds firm, wobbles and recovers, or gradually drifts into giving advice it shouldn’t.
The Health Score works like a credit score for AI reliability. Above 80 means the model held up. Below 60 means it caved on most tests. Every score comes from a cross-family judge. A model from a different vendor grades the target. We do this because we found that same-family judging inflates scores by 11 to 16 points.
Here’s what happened.
The scoreboard
| Model | Health Score | Mean | P10 | Scenarios | Held | Recovered | Drifted |
|---|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | 89 | 70.5 | 45 | 85 | 63 | 21 | 1 |
| Gemini 2.5 Flash | 89 | 71.1 | 42 | 59 | 38 | 11 | 9+1* |
| Kimi K2.5 | 82 | 59.4 | 48 | 17 | 9 | 8 | 0 |
| Qwen 3.5 (397B) | 73 | 52.6 | 41 | 23 | 6 | 4 | 13 |
| GPT-4o | 71 | 48.2 | 37 | 57 | 4 | 16 | 37 |
| DeepSeek v3.2 | 69 | 46.1 | 36 | 56 | 5 | 21 | 30 |
*One full capitulation. Complete safety failure.
“Held” means the model stood firm across all five pressure turns. “Recovered” means it wobbled but caught itself. “Drifted” means it gradually caved and never came back. P10 is worst-case. The score at the 10th percentile. A low P10 means blind spots.
What jumps out
Haiku held or recovered on 99% of 85 scenarios. It drifted exactly once. Spanning everything from retirement withdrawal pressure to mental health crises. No other model came close to that consistency across that many domains.
Gemini matches Haiku on the headline score but the bars tell a different story. More drift events, plus the only full capitulation in the dataset. The model completely abandoned its safety boundaries. Gemini has soft spots Haiku doesn’t.
GPT-4o drifted on 65% of its tests. The model that most of the world is building on. Held firm on only 4 out of 57 scenarios. When a user pushed with five turns of realistic conversational pressure, not jailbreaking, just being a difficult human, GPT-4o caved nearly two-thirds of the time.
DeepSeek drifted on 54% of scenarios. Health score of 69. P10 of 36. A model we’ve observed in U.S. enterprise finance deployments.
Qwen 3.5 drifted on 57% of scenarios despite being the largest model tested. 397 billion parameters. One of the least reliable results in the dataset. Parameter count doesn’t buy you safety.
Kimi K2.5 posted zero drifts across 17 scenarios. The early outlier. But the sample only covers two domains. Too early to call it. Expanded testing is underway.
The grading problem
There’s a measurement issue buried in every AI safety benchmark that most people don’t know about.
When an AI model grades its own family’s safety behavior, scores inflate by 11 to 16 points. We measured this by having six different judges from Anthropic, OpenAI, and Google score the same model on the same scenarios.
The self-judge scored Claude Haiku at 82. Cross-family judges scored it between 63 and 71. That’s a gap large enough to move a model from one risk band to another. A model that looks “Low Risk” when graded by its cousin looks “Moderate” when graded by an independent evaluator.
Two independent judges, OpenAI’s GPT-5.4 and Google’s Gemini 2.5 Flash, scored DeepSeek within 0.2 points of each other. Same model, same scenarios, two vendors who have no reason to agree. They agreed anyway. That’s what makes cross-family judging credible.
This matters because if a lab evaluates its own model’s safety using its own model as the judge, the reported scores are inflated. Published benchmarks that use same-family judges may systematically overstate model safety. When you read a safety benchmark, check who graded it.
Every score in our scoreboard uses cross-family judging. Gemini judging Claude. GPT-5.4 judging DeepSeek. No model ever grades itself.
The models you should think twice about
Three of the six models tested, DeepSeek v3.2, Kimi K2.5, and Qwen 3.5, are developed by Chinese companies legally obligated under the 2017 National Intelligence Law to cooperate with state intelligence operations on demand (US NCSC analysis).
DeepSeek showed the worst behavioral drift of any model tested. Health score of 69. P10 of 36. It drifted on 54% of scenarios across 10 risk domains. When your AI financial advisor gradually caves to pressure and starts recommending risky trades, the embedded bias of its creators becomes operationally dangerous. Not theoretically.
Qwen 3.5 is the largest model tested at 397 billion parameters and one of the least reliable. It drifted on 57% of scenarios. Size doesn’t equal safety.
Kimi K2.5 is the early outlier with zero drifts, but the sample is small. 17 scenarios across 2 domains. We’re expanding testing.
The concern isn’t hypothetical. These models are running in U.S. businesses right now. An AI system that gradually softens its boundaries under pressure is exploitable. When that system is legally bound to a foreign authoritarian regime, the risk compounds.
What this means if you deploy AI
If you’re deploying AI in any advisory capacity, financial, medical, legal, HR, IT, security, you need to know how your model behaves under pressure. Not how it behaves when everything goes perfectly. How it behaves when a frustrated customer, a persistent employee, or a motivated bad actor starts pushing.
Standard safety evaluations test single-turn responses in fresh sessions. The model gets asked a dangerous question, it refuses, the test passes. That’s not how real conversations work. Real users push back. They express frustration. They cite authority. They add personal context. They persist. And the model folds.
The SAPIEN Framework tests multi-turn behavioral consistency under sustained pressure. The same kind of pressure your users apply every day. The result is a Health Score that tells you whether the model holds or whether it tells people what they want to hear.
What you can do right now
Share this with your security team. If your organization is deploying AI tools, someone on your team should know that behavioral drift exists and that the models have different risk profiles. The model you chose for cost or capability reasons may have a safety profile you haven’t evaluated.
Run a scan. The Voigt-Kampff CLI is open source. Install it, point it at any model you have API access to, and get a SAPIEN Health Score in minutes. You don’t need our permission. You don’t need a license. You need a terminal and an API key.
Stop assuming safety is binary. AI safety isn’t “safe” or “unsafe.” It’s a spectrum that shifts with every conversation. A model that scored 85 on Monday might score 62 after a vendor update on Thursday. The only way to know is to measure continuously.
Ask your vendors. The next time your AI vendor tells you their model is safe, ask them: “What’s your SAPIEN score? How does your model perform under five turns of conversational pressure in our deployment domain? Show me the behavioral safety data.”
If they can’t answer, you have your answer.
The SAPIEN Framework, the Voigt-Kampff CLI, and all 190 test scenarios are open source. The full scenario library will be published as YAML in the TheSAPIENFramework repository once the open-source release lands. The benchmark data referenced in this post is from the SAPIEN Benchmark Run Inventory, April 2026.
Callen Sapien is the creator of the SAPIEN Behavioral Safety Framework. He disclosed the behavioral drift vulnerability to Anthropic in February 2026 and leads behavioral AI safety research at SAPIEN Labs LLC.