# AI needs evidence, not confidence.

> AI systems can produce convincing answers without establishing why those answers should be trusted. Evidence connects claims to sources, scope, uncertainty and the decisions that depend on them.

Paper: 03
Published: 2026-09-21
Canonical: https://harten.io/papers/03-ai-needs-evidence-not-confidence/

Cover illustration: [View the title image](https://harten.io/media/papers/covers/v1/03-ai-needs-evidence-not-confidence.jpg?v=22fa903987ab). Illustrative cover artwork, not an application screenshot or a record of measured results.

In [Modernisation starts by establishing what is actually true](https://harten.io/papers/02-03-modernisation-starts-by-establishing-what-is-actually-true/), I wrote about the point discovery should reach before an organisation commits to change: a clear account of what the application does, what the organisation needs and what someone has actually authorised to change.

That leaves another question.

How do we know that account is reliable enough to act on?

AI makes this harder because it is increasingly good at producing explanations that look complete. It can inspect a repository, infer relationships, describe likely behaviour and propose what should happen next. Much of that can be useful. The problem is that a fluent explanation can still be wrong in exactly the place that matters.

I do not think the answer is to ask whether the model sounds confident.

The useful question is what supports the claim we are about to act on.

## A convincing explanation can still be incomplete

Consider a customer-service application.

An AI system examines the source and concludes that only supervisors can approve refunds above £500. It finds a user-interface check, a role name and several tests. The explanation is coherent and, taken on its own terms, probably reasonable.

Then someone finds a batch interface used by finance. It can perform the same refund operation through another path and never calls the user-interface check.

The first explanation was not necessarily careless. It may have described the material available to it accurately. The mistake was broader: the conclusion reached further than the evidence did.

That is the distinction I care about.

A statement about one interface is not automatically a statement about the application. A statement about a repository is not automatically a statement about the running service. A policy describing what should happen is not evidence that production behaves that way today.

This is why [You cannot govern what you cannot understand](https://harten.io/papers/01-3-you-cannot-govern-what-you-cannot-understand/) treats application understanding as something wider than a code-summary problem.

If AI is going to contribute to consequential engineering work, we need to be able to see not only the answer but the basis on which the answer was formed.

## Confidence does not establish support

We use the language of confidence a lot around AI.

The model may express uncertainty. A system may attach a score. A reviewer may simply feel more confident because an answer is detailed, internally consistent and agrees with what they already believe.

None of those things establishes the underlying fact.

I would rather ask where the claim came from, what conditions it applies under and what would cause us to change our mind.

The W3C's work on [provenance](https://www.w3.org/TR/prov-overview/) is useful here. It describes provenance in terms of the entities, activities and people involved in producing information so that its quality, reliability or trustworthiness can be assessed.

Applied to software intelligence, I do not think the requirement needs to be complicated. For an important claim, I want to know which source supports it, which version or environment that source describes, whether the relationship was observed or inferred, what scope the source is authoritative within and whether anything else contradicts it.

That does not make the claim true by itself. It makes the claim inspectable.

The difference matters because confidence can remain high even when the underlying view is incomplete. In the refund example, the model may be extremely confident about the path it inspected and still know nothing about the finance batch interface.

That is why I would be careful about using confidence as the control mechanism.

## Sources only help if they are evidence for the thing being claimed

More material does not automatically solve this.

A code reference can establish that one component refers to another. It may tell us very little about whether that dependency is active in production. A log can establish that something happened during a particular period. It may not tell us why. An operator may know an exception that neither source contains. A policy may be authoritative about intended behaviour while saying little about whether the implementation currently complies with it.

The source and the claim need to fit.

This is also why attaching references to generated prose is not enough. Five links are not five independent confirmations if they all repeat the same assumption.

Sometimes one direct observation is more useful than a large collection of indirect material.

I would rather have a narrow claim that is well supported than a broad one dressed up with citations.

## Disagreement is often where the useful work starts

Complex estates contain disagreement.

Documentation says one thing. Production behaviour shows another. A service owner believes an interface has been retired while network traffic suggests that someone still uses it. Two experienced operators describe the same exception differently.

There is a strong temptation, especially in answer-generating systems, to turn that into one clean narrative.

That can remove the very thing we need to understand.

The disagreement may be an incomplete migration, a hidden dependency, an undocumented exception or a difference between policy and practice. It may be the reason the proposed change should be reconsidered.

I would want those competing accounts to remain visible until we know whether they describe different scopes, one of them is wrong, or the decision can be narrowed so that the disagreement no longer matters.

Sometimes further evidence resolves it. Sometimes it does not.

The point is not to eliminate uncertainty before anything can happen. It is to avoid hiding uncertainty simply because the workflow prefers a complete answer.

## Sometimes “we do not know” is the correct result

There are going to be cases where we simply do not know.

We may not know whether an interface is still used. The production configuration may not be available. A stored procedure may construct behaviour dynamically. Two sources may disagree and there may be no immediate way to establish which one describes the current service.

I do not regard that as an automatic failure.

It is a finding about the current state of understanding.

OWASP's guidance on [LLM misinformation](https://genai.owasp.org/llmrisk/llm092025-misinformation/) describes the risk of false or misleading content that appears credible, alongside overreliance on generated material without sufficient verification.

For engineering work, the practical implication is fairly simple. If an important claim has not been established, the system needs to be allowed to say so.

We can investigate the gap. We can reduce the scope of the change. We can keep an existing control for the first release. Or, where the consequence permits it, someone can explicitly accept the uncertainty.

What we should not do is manufacture certainty so that the process can continue neatly.

## Evidence changes what I want from AI

If the only objective is an answer, fluency is useful.

If the objective is to support a consequential change, I want more than that.

I want to be able to see what was found, what supports it, where that support applies, what disagrees with it and what remains unresolved. Most importantly, I want to know why any of that matters to the decision in front of us.

That changes the role of the model.

It is no longer useful because it can always produce the correct sentence. It becomes useful because it can help assemble an evidential position that someone else can inspect, challenge and build upon.

NIST's [AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) is much broader than software modernisation, but its emphasis on incorporating trustworthiness into the design, use and evaluation of AI systems is relevant here.

The Harten position is narrower. Where AI contributes to consequential software decisions, the support for its conclusions should remain available for inspection.

## The amount of evidence should follow the consequence

Evidence-first engineering does not mean proving everything before doing anything.

Reading a repository does not carry the same consequence as removing a production dependency. A reversible refactoring does not carry the same consequence as changing authentication behaviour or financial logic.

This connects back to [The human cannot be the bottleneck](https://harten.io/papers/01-2-human-cannot-be-the-bottleneck/). Human judgement should concentrate where consequence requires it, rather than sitting in front of every machine action.

Evidence is part of what makes that possible.

Where the evidence is strong, the impact is understood and the action is bounded, the system can be given more room to work.

Where the evidence is weak, contradictory or incomplete, scrutiny should increase.

That still does not make a model's confidence score the control plane. Consequence matters. Authority matters. Reversibility matters.

Evidence tells us what we have actually established about the situation in which those decisions are being made.

## Series 3 is really about the basis for action

Series 2 ended with a requirement to establish what is actually true before committing to change.

Series 3 asks what it takes to support that statement.

The three papers that follow look at the problem from different directions.

**03.1 — Confidence is not understanding** looks at why a convincing explanation can still hide the boundary of what the system has actually understood.

**03.2 — From signals to evidence** looks at how code, logs, documents, telemetry and human knowledge become useful only when we retain their provenance, scope and relationship to the claim being made.

**03.3 — Evidence changes how we work with AI** looks at what happens once that evidence becomes part of the way work is delegated, reviewed and remembered.

The point is not to make AI cautious for its own sake.

What matters is that the basis for action survives the answer.

Because once AI can produce an answer faster than we can independently reconstruct why it should be trusted, confidence becomes cheap.

Evidence becomes the scarce thing.

---

*This paper is part of The Modernisation Control Papers, the first major sequence within The Harten Papers. [Read the complete series on harten.io](https://harten.io/papers/).*