Confidence is not understanding

The Series 3 flagship makes a fairly simple point: a convincing answer is not the same thing as a defensible basis for action.

That sounds obvious until the answer is very good.

AI systems are becoming better at explaining software. They can read a repository, trace a flow, identify patterns and produce an account that feels complete enough to rely on.

The risk is not simply that the model is “too confident”.

The harder problem is that the explanation can be correct within the part of the system it has seen and still be wrong as a description of the application as a whole.

That is where confidence stops being a useful proxy for understanding.

The boundary of the answer matters

Take a permissions service.

An AI system examines the repository and concludes that all privileged actions require manager approval. It finds a policy class, a user-interface control and several tests. It explains the flow clearly and gives you references.

Then an engineer finds a scheduled import used by operations. The import applies the same underlying changes without going through the approval path.

The first answer may still be accurate about the path it inspected.

The problem is that nobody noticed the boundary.

A repository can give us strong evidence about implementation and weak evidence about whether every operational route is represented there. A test can establish that one scenario behaves a particular way. It cannot establish that no other scenario exists.

This is the point at which a good explanation can become misleading. It moves, often without saying so, from “this is what I found in the inspected path” to “this is how the application works”.

Those are not the same claim.

A confidence score does not show you what is absent

Confidence scores can be useful.

They can help rank alternatives or indicate that a model found one answer more probable than another under its own process.

But they do not tell us whether the evidence that matters was ever available.

The missing scheduled import does not become more visible because the model is less certain about the code it did inspect. The model may be highly consistent about an incomplete view.

That is why I would not design governance around one confidence number.

What I need to know is more practical. Which sources were available? Which environment did they describe? What could not be inspected? Is there runtime evidence? Does policy disagree with production? Is there an operational exception somebody performs outside the code?

Those questions describe the state of understanding.

A confidence number does not.

Understanding has to include scope

In You cannot govern what you cannot understand, I wrote that application understanding has to extend beyond the code into operations, data, dependencies, people and policy.

That does not mean we need a perfect model of an entire estate before doing useful work.

It means we need to know the boundary within which a conclusion is being made.

Suppose we establish that a service exposes no direct external API. That may be a useful finding. It does not tell us whether another system reads its database, consumes a file it produces, or depends on a report built from its data.

The narrow claim may be strong.

The broad conclusion may still be unsupported.

For me, that is part of understanding: knowing what part of the system has actually been inspected, under which operating conditions, which sources were unavailable and which parts of the conclusion are assumptions rather than observations.

A system that preserves those limits can still be extremely useful.

A system that hides them behind a polished explanation is much harder to govern.

“I do not know” can be a useful engineering result

There is a tendency to treat “I do not know” as a failure state for AI.

In consequential engineering work, I think it can be one of the better results.

An unknown tells us where the current model of the application stops. The next question is whether that gap matters to the action being considered.

If we are reading a component to understand its internal structure, an unresolved external dependency may be tolerable.

If we are about to remove that component, the same unknown may stop the change.

The useful question is not simply whether uncertainty exists. It is what would change if the assumption were wrong.

Could we inspect another source? Could we run an observation? Could we narrow the change so that the unknown no longer matters? Could we keep an old mechanism temporarily and revisit the decision later?

That is a much more useful treatment of uncertainty than converting it into a confidence score and moving on.

Humans do this as well

This problem is not unique to AI.

Experienced engineers form confident explanations from incomplete evidence all the time. We remember how the system used to work. We infer purpose from a name. We trust a diagram because it looks official. We assume an interface is unused because nobody in the room knows who owns it.

The advantage of an evidence-first system is not that machines become sceptical while people are assumed to be reliable.

It is that both machine and human reasoning have something inspectable to work from.

A reviewer should be able to see the conclusion, the sources behind it and the gaps that remain. They can add domain knowledge, challenge the interpretation or decide that the available evidence is enough for the consequence being accepted.

That is a much stronger role than asking whether a generated explanation “feels right”.

Plausible answers can hide missing context

OWASP describes LLM misinformation as false or misleading information that appears credible, and highlights the risk of users accepting generated content without adequate verification.

Software gives that problem a particular shape.

The answer can fit the local code beautifully and still miss the system.

A dependency can live outside the repository. A business control can be performed manually. Production configuration can differ from the material available to the model. An integration can run only once a quarter.

The explanation can therefore be locally persuasive and globally incomplete.

This is why asking the model to explain its answer is not enough. The explanation is still generated from the same available material and under the same missing context.

What changes the position is additional evidence or a genuinely independent check.

The better question is not “How sure are you?”

For consequential work, I would rather ask what exactly is being claimed, what supports it and where the support stops.

I would also want to know what disagrees with it, what is missing and which decision depends on the claim being true.

Sometimes the useful result will be narrower than the answer we originally wanted.

Instead of “the interface is unused”, we may get “no use has been observed in the sources and period available to us, but one quarterly process remains untested”.

That may sound less satisfying.

It is also much closer to something we can responsibly act on.

From confidence to evidence

The point is not to remove confidence from AI systems.

It is to stop asking confidence to do a job it cannot do.

Understanding requires a defensible account of the part of the world the answer is about. For complex software, that means evidence, scope, contradiction and unknowns.

The next paper, 03.2 — From signals to evidence, looks at how we build that account from the material already distributed across an application estate.

Because code, logs, documents and telemetry do not become evidence merely because an AI system can retrieve them.

They become evidence when we know what they establish, where they came from and how much of the conclusion they really support.

---

This paper is part of The Modernisation Control Papers, the first major sequence within The Harten Papers. Read the complete series on harten.io.