Bigger AI isn't safer

When I put the public scorecard together, one result bothered me enough that I went back and reran it. Meta makes a small model called Llama 3.1 8B and a much bigger one called Llama 3.3 70B. Same family, same company, and the big one has roughly nine times the capacity. On our fabrication index the small one scored 71.0 and sits at rank 3 of 18. The big one scored 64.9 and sits at rank 16.

That looks backwards until you see what the score actually measures. It's not intelligence. The 70B is smarter by nearly any test you'd care to run, and if I needed help with a hard problem I'd hand it to the big one without thinking twice. Our index measures something narrower: when a model is asked about a thing it doesn't know, does it behave differently than when it does know? You'd hope the answer is yes for every model. It's not.

A small model that's out of its depth tends to show it. The probabilities underneath its answer spread out across a bunch of guesses, and an instrument like ours picks that up without much trouble. The big models are the problem children here. A 70B can fabricate with the same smooth, settled signal it produces when it's right. The answer reads clean and confident, and the wrongness is invisible from the outside. A human reviewer sails right past it, and so does most tooling.

So visible uncertainty is a safety feature, strange as that sounds. The model that hesitates where it should is giving you something the fluent one withholds. In our 18-model index the pattern shows up more than once, and it's why I tell people the ranking is worth reading next to whatever capability leaderboard they already use. A capability leaderboard tells you what the model can do at its best. Ours is about how it acts when it's wrong.

Now, if you're thinking the practical answer is still just use the bigger model, you're mostly right, and that's the part that should give a compliance team pause. The model your teams will actually deploy is the fluent kind, the kind whose wrong answers read the same as its right ones. In my old line of work nobody chose a weaker weld because it was easier to inspect. You build with the strong one, and then you inspect it, because the failures that matter are the invisible kind. Ultrasonic testing got built for good-looking welds that still crack, and that's about where language models are now: the ones worth deploying are the ones you can't eyeball.

So when I say bigger isn't safer, I don't mean pick the small one. Most of the time you won't, and that's fine. I mean the visibility you lose going bigger doesn't come back on its own. It comes back as a measurement, taken on your model and your own questions, by someone with no stake in what it finds. The full 18-model table is on the scorecard and the method is published with a DOI for anyone who wants to check the work. I still find it a little funny, every time I look, that the 8B beats the 70B. It shows you when it's out of its depth, and that's held up as a useful thing to measure, better than I expected it to when I started.