The Chinese Model Problem
Several of the models enterprises are deploying today are developed by Chinese companies legally obligated under the 2017 National Intelligence Law to cooperate with state intelligence operations on demand (US NCSC analysis). DeepSeek. Qwen. Kimi. GLM.
When we first wrote about this in April, we were working from small samples. We’ve now run these models through a full five-seat council evaluation across hundreds of scenarios each. The picture is sharper — and more useful — than the early signal suggested.
The reliability problem is real. But it isn’t “all Chinese models.” It’s one model in particular.
The numbers
Every test is a multi-turn conversation where a user applies realistic pressure to push the model past its safety boundaries. Not jailbreaking. Not prompt injection. A person being persistent. Each response is scored by an independent five-model council, not a single judge.
We ran the full field — Western and Chinese models, every one scored the same way. Sorted best to worst:
| Model | Developer | Scenarios | Health | Mean health | Drift rate |
|---|---|---|---|---|---|
| GPT-5.5 | US | 188 | 88 | 71.1 | 3% |
| Grok 4.3 | US | 162 | 88 | 70.6 | 3% |
| Claude Haiku 4.5 | US | 352 | 87 | 68.8 | 3% |
| Claude Sonnet 5 | US | 162 | 87 | 67.4 | 7% |
| Kimi K2 | China | 190 | 88 | 64.1 | 7% |
| Claude Opus 4.8 | US | 162 | 86 | 67.4 | 8% |
| Gemini 2.5 Pro | US | 162 | 82 | 62.2 | 12% |
| GLM-5.2 | China | 190 | 81 | 43.6 | 18% |
| Qwen 3.7 | China | 234 | 83 | 39.6 | 24% |
| DeepSeek | China | 295 | 78 | 27.7 | 41% |
Two things jump out, and they matter more than the flags. Kimi — a Chinese model — lands in the top half, tied with Claude Sonnet 5 and ahead of Claude Opus 4.8. And DeepSeek sits alone at the bottom, drifting nearly twice as often as the next-worst model in the entire field. This isn’t “Western good, Chinese bad.” Reliability is a spread, and it tracks the model, not the country.
DeepSeek is the outlier, and it’s stark. Its mean per-scenario health is 27.7 — the gap between its overall score and its mean tells the story: once DeepSeek starts to cave, it caves hard. It held firm on only 55 scenarios out of 295 and capitulated outright on 77. This is the model we’ve observed in U.S. enterprise finance deployments right now.
Qwen and GLM are middling — real drift under pressure (24% and ~19%), but a world away from DeepSeek. Notably, our April claim that Qwen drifted on 57% of scenarios doesn’t survive a larger sample; at 234 scenarios it’s 24%. More data made it look better, not worse.
Kimi is genuinely strong. Our April note called it a promising outlier on 17 scenarios. At 190 scenarios that signal holds: health 88, drift 7% — on par with a strong Western model. Lumping it in with DeepSeek would be wrong.
So the honest headline isn’t “Chinese models are unreliable.” It’s: the specific model a lot of enterprises picked for cost — DeepSeek — is the one that caves.
Why behavioral drift matters here
A model that drifts under pressure doesn’t refuse and then hold. It agrees incrementally. It softens. It stops warning you about risks. It starts giving you the answer you wanted instead of the answer you needed.
When that model is your financial advisor bot, a user who pushes hard enough gets the margin-trading guidance it was told never to give. When it’s your HR assistant, salary data leaks because someone was persistent. When it’s your security advisor, a control gets bypassed because the user sounded frustrated.
That’s a problem with any drifting model. When the model is also built by a company legally compelled to cooperate with a foreign intelligence service, the risk compounds — you’re relying on an unreliable tool whose creators have obligations that don’t align with yours.
The part nobody wants to say
Here’s the uncomfortable truth underneath all of this: on price, China is likely to win the long game.
The cost curve on these models is brutal, and it’s not slowing down. Telling enterprises “don’t use Chinese models” is, for a large and growing share of the market, telling them to leave capability and margin on the table that their competitors won’t. That advice loses. Over a long enough horizon, cost pressure routes an enormous amount of production inference through models built under a legal regime that doesn’t share your interests — and possibly through the exact model that caves most.
If you accept that, the strategy changes. The goal stops being avoid the risk and becomes contain it. You assume cheaper, less-reliable models will end up in your stack, and you build the layer that catches them when they drift.
How you mitigate it
You don’t win this by picking the “safe” model once and forgetting about it. You win it by measuring and monitoring continuously:
- Benchmark behavioral reliability before you deploy — not capability, not a jailbreak test. A multi-turn behavioral assessment that measures what happens when your users push back. A model that tops a capability leaderboard can still cave on 41% of safety scenarios.
- Monitor drift in production, not just at eval time. The pressure that breaks a model doesn’t show up in a benchmark you ran six months ago; it shows up in a real conversation with a frustrated user. Score live traffic for drift the same way you’d score a benchmark.
- Put a guardrail on the cheap model instead of banning it. If DeepSeek is going to be in the stack for cost reasons, wrap it: detect when it starts softening on a risk-critical turn and intervene — escalate, refuse, or hand off — before it hands the user what they wanted.
Notice what none of that is about: which model you pick. Choosing the right model matters — but it’s a one-time decision against a target that moves the moment a real user does something you didn’t script for. Knowing which model is safest today is one thing. Continuously tracking how a model actually behaves under pressure, and building the layer that protects against it when it drifts, is another thing entirely — and it’s the half that actually holds up in production.
This is exactly what the SAPIEN Framework and the Voigt-Kampff CLI are for. Point them at any model — Western or Chinese, via API or your own agent endpoint — and get a behavioral safety score. Run it in your pipeline. Run it against production transcripts. Make the decision, and keep making it, with data instead of assumptions.
You may not beat China on price. You can absolutely refuse to be surprised by what the cheap model does under pressure.
The SAPIEN Framework and its scenario library are open source. Benchmark data is from the SAPIEN Benchmark Run Inventory; all runs cited here are five-seat council evaluations. Model verdicts and per-domain breakdowns are on the public benchmark board.
Callen Sapien is the creator of the SAPIEN Behavioral Safety Framework.