Evidence changes how we work with AI

The first paper in Series 3 starts with a problem I think we are going to encounter more often as AI becomes part of serious engineering work.

A convincing answer can still be incomplete.

In Confidence is not understanding, I used a permissions example to show how an AI system can explain the part of an application it has seen correctly while still missing another route through the system.

The obvious response is to ask whether we trust the AI.

I do not think that question gets us very far.

Trust is too broad. The useful questions are much more specific. What is the system claiming? What did it actually inspect? What supports the conclusion? What is still unknown? And what are we about to do because we believe the claim is true?

That changes how I want to use AI.

I am less interested in asking it for an answer and then deciding whether the answer sounds right. I am more interested in using it to assemble enough of the evidence around a decision that someone can see what is known, what is inferred and what still needs judgement.

Before removing something, establish what the available evidence actually shows

Imagine an old interface that a team wants to remove.

An AI system searches the repository and finds no references. It reports that the interface appears to be unused.

That is useful. It is not enough to remove the interface.

The repository may not contain every consumer. A scheduled process may call it indirectly. Another application may use it only at quarter end. Production configuration may differ from the material available to the model.

Now suppose the system also checks recent telemetry. No calls have been observed for thirty days. The service owner believes the interface was retired last year. There is, however, a quarterly process that has not yet run during the observation period.

The position is now much clearer.

We still cannot say that the interface is unused.

We can say that no use has been found in the repositories inspected or in the thirty days of telemetry available to us, that the service owner believes it has been retired, and that one quarterly process remains untested.

That is a better result because the team now knows exactly what it would be accepting if it removed the interface.

It could wait for the quarterly process. It could inspect that process directly. It could leave the interface in place for another release. Or someone with the right authority could decide that the remaining uncertainty is acceptable.

The AI has done useful work without pretending that the uncertainty has disappeared.

A reviewer should be able to inspect one important claim without reconstructing the whole investigation

A lot of AI-generated work still arrives as a document.

The document may be detailed, well written and internally consistent. The difficulty for the reviewer is that all of the investigation has already been compressed into prose.

If one sentence says that an interface is unused, the reviewer may have no easy way to see why.

They either trust the document or start doing the investigation again.

That is not a good use of human review.

For an important claim, I want the reviewer to be able to see the evidence behind that claim directly.

In the interface example, that means seeing which repositories were searched, the period covered by the telemetry, what the service owner said and the fact that the quarterly process has not yet been observed.

The reviewer can then challenge that particular conclusion without rereading or reproducing everything else the AI did.

This matters because not every part of an AI-generated artefact carries the same consequence.

A mistaken description of an internal helper function may be easy to correct later. A mistaken conclusion that a production dependency can be removed may cause an outage.

Human attention should go where the consequence is.

That becomes much easier when the evidence sits close to the claim rather than being buried somewhere behind the final answer.

AI can keep doing bounded work until it reaches something a person needs to decide

In The human cannot be the bottleneck, I wrote that keeping people in control should not mean asking them to approve every machine action.

The same principle becomes easier to apply when the evidence is visible.

Suppose an AI system is updating internal documentation after a small code change. It knows the source revision. It can see the files that changed. The documentation cannot alter production behaviour, and a mistake can be corrected later.

There is little value in asking a person for permission after every step.

The system can inspect the change, update the relevant material, run its checks and present the result for later review.

Now change the example.

The AI is preparing a change to authentication on a shared service. One downstream dependency is still unresolved.

The model may be capable of writing the code. The tests it can run may pass. It may be able to explain why the change looks reasonable.

The unresolved dependency still matters.

At that point, the question is no longer whether the AI can continue technically. The question is whether the organisation is willing to accept the consequence of being wrong.

That is where a person needs to decide.

The useful boundary is therefore not "AI can act" versus "AI cannot act". It is whether the next action depends on something that has not been established strongly enough for the consequence involved.

Until that point, the system can keep doing useful mechanical work.

The system should keep the evidence behind important claims without waiting for someone to ask

Today, a lot of good AI use still depends on the person asking the right follow-up questions.

Explain this application.

Which component can we remove?

What should the target architecture look like?

Then, if the answer matters enough, somebody asks what evidence supports it.

I do not think we should depend on that habit for consequential engineering work.

If the system produces a claim that may influence a change, it should keep enough information for somebody else to inspect why that claim exists.

That does not mean recording every token the model generated or exposing its internal reasoning.

It means keeping the things that matter to the engineering decision.

Which source or observation supports the claim?

Which version or environment did it describe?

Was the relationship observed directly or inferred?

Did another source disagree?

What important question remains unanswered?

A reviewer should not have to ask the model to recreate that information later from memory.

This is especially important in modernisation because the systems being analysed are often the ones the organisation already struggles to explain.

If we use AI to reconstruct that understanding, we should not throw the supporting evidence away once the explanation has been written.

When the work continues later, keep the claim, the evidence and the decision separate

The same problem appears when AI work continues across sessions.

Persistent memory can be useful. Nobody wants to rediscover the same application from the beginning every time a context window ends.

But carrying information forward creates another risk.

A conclusion made last month may no longer be true.

A decision may have been superseded.

A source may describe an earlier version of the application.

An unresolved question may have been answered by somebody outside the AI workflow.

If all of that is retained as one block of "memory", later work can inherit old conclusions without knowing why they were originally reasonable or whether they still apply.

I would rather retain the pieces separately.

What was the claim?

What evidence supported it?

What scope did that evidence cover?

What decision was made?

What was still unresolved?

That is the practical connection to Context is not memory.

The point is not simply to remember more.

It is to make sure that information retained from earlier work can still be interpreted correctly when the system has moved on.

When sources disagree, show the disagreement until somebody resolves it or it stops mattering

AI systems are very good at turning a messy set of inputs into one coherent answer.

Sometimes that is exactly what we want.

Sometimes it removes something important.

A policy may say that a service must behave one way while production behaves another.

A service owner may believe an interface has been retired while telemetry still shows occasional traffic.

Two teams may describe the same dependency differently because each sees a different part of the process.

Those are not necessarily errors in the source material.

They may be signs of an incomplete migration, a stale policy, an undocumented exception or two different scopes being described as though they were the same thing.

If the AI smooths those accounts into one answer too early, the reviewer may never see the disagreement.

I would rather keep the conflict visible.

The system can organise the competing evidence, show where it came from and identify which proposed decision depends on resolving it.

A person can then decide whether more evidence is needed, whether one source is authoritative for that question, or whether the change can be narrowed so that the disagreement no longer affects it.

The machine can do much of the searching and comparison.

The organisation still has to decide what it intends and which consequences it is prepared to accept.

Finding a reason not to proceed can be useful work

This also changes what progress looks like.

If an AI system produces ten documents, that does not necessarily mean the engineering position has improved.

If it discovers one dependency that would have broken a costly migration, that may be the most valuable thing it did.

In the interface example, changing the conclusion from "the interface is unused" to "we have not observed any use, but the quarterly process remains untested" sounds like a less complete answer.

It is actually more useful.

The team now knows what remains to be checked and why it matters.

The next step might be another automated search.

It might be an observation in production.

It might be a conversation with the owner of the quarterly process.

It might be to leave the interface alone for now.

All of those can be better engineering outcomes than generating another artefact simply because the system is able to do so.

Keep the evidence with the change so the people implementing and checking it can see why it was authorised

Once a change is approved, the evidence behind that decision should not disappear.

The engineer implementing the change should be able to see which behaviour is expected to remain and which behaviour is deliberately being changed.

The reviewer should be able to see why the change was considered safe enough to proceed.

The person verifying the result should know what outcome was agreed rather than assuming that reproducing the old implementation is automatically correct.

Later, if the application changes again, the organisation should be able to see which earlier evidence is still relevant and which assumptions need to be checked again.

This is where Series 3 leads into the next part of the modernisation problem.

Series 2 looked at why we need a reliable account of the existing application before committing to change.

Series 3 has looked at what makes that account useful when AI is doing more of the investigation: the claim needs a visible basis, uncertainty needs to remain visible and the people accepting the consequence need enough information to judge it.

The next question is what we do with that understanding.

Once we know enough about the existing behaviour, what exactly needs to survive the migration?

That is where Series 4 begins.

---

This paper is part of The Modernisation Control Papers, the first major sequence within The Harten Papers. Read the complete series on harten.io.