Many people assume those who build powerful AI must grasp how it thinks, but in an important sense they cannot: even a model's own creators often cannot fully explain why it gave one answer rather than another

Here is a claim that sounds like it must be wrong: the people who build the most capable AI systems in the world often cannot give a complete, reliable explanation of why a model produced one answer instead of another.

Not “won’t tell you.” Not simply “it’s a trade secret.”

They can describe the model’s architecture, training process and mathematical operations, but they often cannot translate everything that happened inside it into a precise, human-understandable account of why it made that particular choice. 

I am not an AI engineer or interpretability researcher, and this is a piece of reading and reflection on what the people doing that work say, not technical advice. It relies heavily on an essay by Anthropic CEO Dario Amodei and on research produced by Anthropic, alongside the broader findings of the International AI Safety Report 2026. Amodei’s essay presents an informed but interested argument rather than a settled scientific consensus.

The assumption: if you built it, you understand it

With many things humans build, understanding appears to run from the maker to the thing. A bridge, a car engine or an ordinary piece of software is assembled from parts and instructions chosen for particular purposes.

Large conventional systems can still become extremely difficult to understand or debug, but their important behaviours are generally specified much more directly by human designers.

We carry that idea over to AI without thinking. Somebody built it, so somebody must know exactly why it behaves as it does.

Large language models break that intuition, however. The systems behind tools like ChatGPT, Claude, Gemini and Llama rely, as IBM puts it, on complex neural networks whose inner workings are “difficult to interpret”. Simpler rule-based systems are generally easier to trace, but they are also far less capable at the kinds of flexible language tasks these models perform. Much of the capability comes from patterns learned during training rather than instructions written out one by one.

Why the opacity is real, and grown rather than deliberately designed

The phrase “black box by design” can be misleading because it suggests that developers intentionally chose opacity. The difficulty is better understood as something that emerges from the way the systems are trained.

Anthropic’s CEO Dario Amodei, borrowing a phrase from his co-founder Chris Olah, describes generative AI systems as “grown more than they are built,” with inner workings that are “emergent” rather than directly designed.

It is a metaphor, not literal biology, but it captures an important distinction. Builders choose the architecture, training objective, data and broad learning process. The model’s detailed internal representations and computations are then learned during training rather than individually specified by programmers.

What that leaves behind is genuinely hard to interpret. On a specific output, Amodei argues, “When a generative AI system does something, like summarize a financial document, we have no idea, at a specific or precise level, why it makes the choices it does.” He goes further, calling this “essentially unprecedented in the history of technology,” which is a strong historical judgment and clearly his, not a neutral fact. 

Part of the difficulty comes from scale. The International AI Safety Report 2026, chaired by Yoshua Bengio, says researchers often cannot trace how a particular input leads to a specific output because models contain enormous numbers of internal parameters and represent information in highly distributed ways.

Scale is not the whole explanation. The problem also lies in the nonlinear interactions among components and in the way learned representations can overlap. A concept is not generally stored like a labelled entry in a database that researchers can simply open and read. 

What interpretability research has, and hasn’t, cracked

Amodei’s essay isn’t all alarm. A field called mechanistic interpretability is trying to build something like an MRI for these systems: tools that can identify readable “features,” or patterns associated with concepts the model uses, and the “circuits” through which those features interact. It is working, partly.

Anthropic researchers trained a sparse autoencoder containing roughly 34 million feature dimensions on activations taken from one middle layer of Claude 3 Sonnet. They showed that many sampled features corresponded to recognisable concepts and could sometimes help explain or influence model behaviour. Amodei summarizes this work as identifying more than 30 million ‘features’.

That is an important result, but it was not a complete, individually verified map of more than 30 million human-readable concepts across the entire model. It involved one layer, a learned interpretability model and a set of features that varied in how clearly they could be understood.

The number also sounds large until it is set against another one. Amodei estimates that even a small model may contain a billion or more concepts. That is his estimate, not a directly measured inventory. Even so, it illustrates his larger point: tens of millions of identified features could represent both a significant achievement and only a fraction of what remains.

The AI Safety Report adds a warning worth keeping: current interpretability methods depend on major simplifying assumptions and can produce misleading conclusions when used carelessly.

Beyond the lab

It would be easier to file all this under academic curiosity if these systems were staying in the lab. They aren’t. AI systems are already being integrated into consequential workflows involving healthcare, recruitment, legal work and software development. In many of these settings they assist humans rather than make the final decision, but their outputs can still shape what those humans decide.

The gap I keep coming back to is this: we are increasingly relying on systems whose behaviour their own makers cannot fully retrace in human-understandable terms, and we are deploying them faster than the tools used to inspect them are maturing.

Amodei’s position on this is blunt, and he flags it as his opinion: “I consider it basically unacceptable for humanity to be totally ignorant of how they work.” You don’t have to share his timeline or all of his assumptions to feel the force of that.

His closing line is that “powerful AI will shape humanity’s destiny, and we deserve to understand our own creations before they radically transform our economy, our lives, and our future.”

So the position isn’t that AI is permanently unknowable, or that its creators understand nothing about it. They know how the models are constructed, trained and executed, and interpretability research is steadily revealing more about what happens inside them. What remains missing is a dependable and comprehensive explanation of how their learned internal mechanisms produce many particular answers and behaviours.

The people warning loudest about the black box are often the same ones trying to pry it open. We have built machines that work, that people increasingly rely on, and that we cannot yet fully explain in the way we explain ordinary designed systems. We are deploying them at scale while the explaining is still catching up. That tension doesn’t resolve neatly, and assuming someone, somewhere, already understands everything may be the least useful response to it.