You cannot govern what you cannot understand

Updated 28 August 2026 to expand the treatment of distributed domain knowledge, human judgement and conflicting evidence.

As AI systems become capable of analysing, modifying and verifying software at machine speed, asking a person to approve every individual action is not a sustainable model of control. In The human cannot be the bottleneck, we set out a stronger model: autonomy increases as evidence strengthens, while scrutiny increases as consequence grows.

That leads to a harder question. How does a system actually know the consequence of a change?

The answer is not contained in the file being modified. It may not be obvious from the pull request, and it may not even be exposed by a passing test suite. The consequence of changing software depends on what that software is connected to, what depends on it, what assumptions have formed around it and what happens elsewhere when its behaviour changes.

Nor can we assume that presenting the change to a human resolves the problem. The person with formal authority to approve a change may not possess all the relevant domain knowledge. The people who do hold that knowledge may each understand only one part of the system, and may interpret that part differently.

Before we can govern autonomous change, we therefore need something more fundamental: an evidence-backed understanding of the system around the code, including the operational and human behaviour on which that system depends.

Code is not the system

Modern AI coding tools are becoming exceptionally good at operating on code placed in front of them. That is useful, but a software estate is not simply a collection of source files.

A thirty-line function might sit behind a critical payment process. An apparently unused database column might feed an overnight regulatory extract. A small authentication change might affect dozens of applications through an undocumented shared dependency. A configuration file might determine where customer data is stored, while a component that looks isolated in one repository may sit at the centre of a much larger runtime dependency graph.

The size of a change tells us relatively little about its consequence. What matters is the position of that change inside the wider system.

This is especially true in legacy estates. Over years or decades, systems accumulate relationships that are not represented completely in any one place. Some live in source code, databases, infrastructure, configuration, scheduled jobs or message queues. Others exist because two systems have behaved in a particular way for fifteen years and everybody around them has learned to rely on it.

Some behaviour is not encoded in technology at all. It lives in operational practice and human judgement. An underwriter may take a view based on experience, placing weight on evidence that no formal rule describes. A supervisor may apply an exception differently from a colleague, even when both believe they follow the same process. Replace or automate that human step without first discovering the variation and the system may preserve the visible workflow while silently changing the outcome.

Documentation may describe part of the architecture, the repository another part, the running system something different again, and the people operating it several overlapping versions of its behaviour. If an AI system sees only the code it has been asked to change, it may understand the implementation while still misunderstanding the system.

Change impact is a graph problem

Consider a fairly ordinary change: converting a customer identifier from an integer to a string.

Inside one application, that may look straightforward. Change the type, update the model, modify the tests and compile.

The real work starts when we look outside that application. The database may still store the identifier as an integer. Another service may expose it through an API. A nightly batch process may assume it is numeric. It may form part of a URL, a reporting key, a partitioning strategy or an external contract. Another team may have copied the schema into a system that is not visible from the repository currently being changed.

At that point, this is no longer a simple datatype conversion. It is a traversal through a dependency graph.

But that graph is rarely complete or neutral. A developer sees the call path. An operations team sees the runtime dependency. A business specialist sees the workflow. A risk or compliance function sees the obligation. Each may hold a valid part of the system model without seeing the whole, and each interprets the relationships from a different position.

The graph therefore has to represent more than components and connections. It must also preserve claims about those connections, the evidence supporting them, the scope of the person or source making the claim and the perspectives that disagree.

This is what begins to reveal the blast radius of a change. If we want an AI system to make meaningful judgements about consequence, it needs access to that wider and more qualified context.

This is why governance cannot begin only when an agent proposes an action. It has to begin earlier, during discovery.

Discovery is part of the control system

Discovery is often treated as a preparatory phase in modernisation. Understand the estate, draw some diagrams, produce a backlog and then move into the real work.

For autonomous systems, that framing is too weak.

Discovery is part of the control system because it gives an agent the context required to judge whether an action is bounded or potentially consequential. It needs to understand what calls a component, what that component calls, what data it reads and writes, what interfaces it exposes, where it executes and what boundaries it crosses. It also needs some understanding of the tests around it, the operational processes that depend on it and the confidence we should place in those conclusions.

Discovery also has to ask where human judgement forms part of the effective system. Which decisions are discretionary? Which exceptions are routine but undocumented? Which roles possess relevant knowledge? Where do practices differ despite a shared process description? These are not peripheral organisational questions. They are part of the behavioural model that a transformation may alter.

The important distinction is between understanding the code and understanding enough of the consequences of changing the system.

They are not the same thing, and governance depends on the second.

Human authority does not guarantee human understanding

Human authority must be preserved. Human understanding cannot be assumed.

In a complex estate, approval authority, technical expertise, operational knowledge and business accountability often sit with different people. Asking one person to approve an AI-generated change does not combine those perspectives. It may simply move the uncertainty to someone with a decision to make and insufficient evidence to make it well.

The objective is not to find a single domain expert who can certify the whole system. Such a person may not exist. It is to assemble a defensible view from distributed knowledge while retaining the limits of each contribution.

A claim from a database engineer, an observation from runtime telemetry and a policy owned by a compliance team do not become equally reliable because they appear on the same diagram. The system needs to retain who or what supports each claim, the scope within which that source is authoritative, when the evidence was produced and whether independent sources agree.

Human approval becomes meaningful when the decision surface contains the relevant evidence, unresolved uncertainty and competing interpretations, and when the person approving has authority over the consequence being accepted.

Not all understanding is equally reliable

Suppose an AI system concludes that Application A depends on Service B. That relationship might come from a single static code reference, or it might be supported by static analysis, infrastructure configuration, runtime telemetry, API definitions and integration tests.

The relationship looks the same in both cases, but the strength of the evidence is very different.

That matters because a useful model of a software estate should preserve more than the relationships it has discovered. It should also preserve why those relationships are believed to exist. Where did the evidence come from? Is it current? Was the dependency inferred or directly observed? Who regards it as important? Within what scope is that person or source reliable?

The goal is not merely to build a model of the estate. It is to build a model where important claims carry provenance.

Data engineering increasingly uses lineage to show where data originated and how it moved or changed. Software and system intelligence needs an equivalent discipline. A consequential claim should remain connected to the evidence from which it was derived and to the decisions that depend upon it.

That gives the system something stronger to reason over than a diagram or a dependency list. It gives it an evidential position.

Conflicting evidence is a discovery surface

Complex estates contain contradiction. Documentation describes an intended process while runtime evidence reveals another. Two experienced operators apply the same rule differently. A policy states that an interface is no longer used while network telemetry shows continuing traffic. Tests encode one interpretation of behaviour and production incidents reveal another.

The temptation is to treat these conflicts as defects in the model and collapse them into a single answer. That can destroy precisely the information experienced architects use to find hidden complexity.

Conflicting evidence may reveal an undocumented exception, divergence between policy and practice, an incomplete migration, a hidden architectural seam or behaviour that will be lost if a transformation preserves only the dominant interpretation.

Agreement increases confidence. Contradiction increases investigative value.

The system should therefore retain competing claims, show the evidence behind each and identify which proposed decisions depend on resolving the conflict. Where resolution is not possible, the disagreement remains part of the governed state rather than being converted into false consensus.

Impact must be demonstrated

Most organisations already classify systems and components in some way. Critical, important, low risk, tier one, tier three. Those classifications are useful, but they are too static to determine whether a particular autonomous action is safe.

Impact depends on what is actually being proposed.

A shared library may be safe to inspect and highly consequential to modify. A database may tolerate a low-risk index change while requiring significantly more scrutiny for a schema migration. An internal API may normally be low consequence until the proposed change alters its authentication behaviour.

The risk is not simply a permanent property of the component. It emerges from the component, the proposed action, the surrounding dependencies, the quality of the evidence and the possible outcomes.

An impact claim should therefore be supported by a traceable path through the system model. The system should be able to show what the action could affect, how far the effect could propagate, which boundaries it could cross, whether the change is reversible, how strongly the surrounding dependencies are understood and which parts of the path remain uncertain.

Those questions are a more useful basis for deciding whether the system continues autonomously, gathers more evidence, performs further verification or requires human judgement.

Missing evidence must not become false certainty

AI systems have a tendency to turn uncertainty into an answer. Complex software estates punish that behaviour.

Sometimes the correct conclusion is that we do not know.

A dependency may not be resolvable. Documentation may contradict runtime evidence. An external system may be inaccessible. A stored procedure may construct behaviour dynamically, or an important interface may have no meaningful test coverage.

Those are not necessarily failures of discovery. They are findings about the state of our understanding.

An unknown dependency should increase scrutiny. Conflicting evidence should trigger further investigation. An unresolved architectural boundary should constrain autonomy.

The system should not manufacture certainty in order to continue operating. Unknowns should remain unknown until evidence changes their state. If evidence is weak or incomplete, that should become part of the governance decision.

From repository intelligence to system intelligence

Most coding agents begin with the repository, and for many software development tasks that is a reasonable boundary.

Modernisation is different because the important relationships often extend beyond it.

Applications depend on infrastructure. Infrastructure depends on identity. Services depend on databases. Databases feed reporting systems. Scheduled jobs move data between generations of technology. Regulations and standards constrain permissible behaviour. External organisations may rely on interfaces whose original designers left years ago. Human operators make decisions that determine how the process behaves in practice.

The unit of reasoning therefore has to become larger than the repository. It has to become the system, and in sufficiently complex environments, eventually the estate.

That requires more than a conventional knowledge graph populated with apparently factual nodes and edges. System intelligence needs an evidence-backed representation of claims. A claim should remain connected to its source, scope, date, confidence, corroborating evidence, contradictions and the decisions that rely upon it. This allows a machine to reason over the claim without mistaking it for timeless truth, and allows a human to inspect why the claim exists.

Complete knowledge remains unrealistic in almost any substantial estate. The purpose of system intelligence is not to eliminate uncertainty. It is to make what is known, why it is believed, who holds relevant authority and what remains unresolved available for governed reasoning.

Where the evidence is strong, the blast radius is known, the action is reversible and the consequence is contained, the system can be given more room to act. Where dependencies are uncertain, perspectives conflict, boundaries are unclear or the action is difficult to reverse, scrutiny should increase.

That is where discovery, evidence and governance become the same problem rather than separate phases of a delivery methodology.

Where does understanding live?

There is enormous focus across the industry on making AI better at generating and transforming software. That work matters, and the capability is improving quickly.

But execution is becoming cheaper faster than understanding.

In large legacy estates, the difficult question is often no longer whether AI can rewrite a piece of code. Increasingly, it can. The harder questions are whether the code should be changed, what else may change with it, whose judgement is embedded in the current behaviour, what evidence would tell us that the transformation is safe, and what we still do not understand about the system around it.

Those questions have to be answered if autonomous transformation is going to operate safely at scale.

This is why discovery cannot remain a one-off phase at the beginning of a modernisation programme. The estate changes. Dependencies change. Regulations change. Human practice changes. Evidence changes. The system's understanding therefore has to change with them.

Discovery has to become a persistent capability. System intelligence has to become a governed organisational asset, rather than something reconstructed inside one person's head or one model's context window whenever a change is proposed.

If the knowledge required to govern change lives across code, infrastructure, documents, regulation and human practice, where does that understanding live so that both humans and machines can reason over it?

That is the next control problem.

Because if we cannot understand the environment in which an autonomous system is acting, we cannot meaningfully govern the consequences of its actions.

And we cannot safely modernise what we have not first learned to see.

Originally published by Harten Technologies on LinkedIn. This harten.io page is the canonical archive.