How to prove the new system still behaves as required

The replacement order-management application has passed its tests. The cancellation service builds, the payment integration responds, stock is released and the deployment pipeline is green. Then somebody asks what would happen if a customer tried to cancel an order after the warehouse had handed the parcel to the carrier, but before the dispatch message reached the new application.

That is the failure we explored in 04.2 — How to describe required behaviour without copying the old implementation. The customer was told the order had been cancelled, but the parcel arrived anyway and their card was charged. In that paper, we worked through what the organisation would need to guarantee rather than simply copying the controller, stored procedure and status fields used by the old system.

We now have an agreed description of the required behaviour. The question for this paper is how we demonstrate that a different implementation actually satisfies it, especially when the warehouse, payment provider and order application do not share the same state at the same moment. This is also where AI can make a practical difference. An engineering team can use agents to follow behaviour through a large codebase, prepare variations of a scenario and compare results across thousands of executions much faster than it could do all of that manually. The useful acceleration is in assembling and testing evidence, not in allowing the agent to decide what the business should have meant by cancellation.

The Cancellation Cases the First Test Misses

Start with an order whose stock has been reserved and whose payment has been authorised but not captured. The warehouse has not dispatched it, so the customer cancels. The replacement should confirm the cancellation, prevent payment capture, release the reserved stock and notify the dependent systems. We can inspect those outcomes independently of whether the implementation uses the old database fields or an entirely different arrangement. If that fails, we have found a clear problem.

But passing this test only demonstrates cancellation when all the systems agree about the order. The more revealing scenarios are the ones that interrupted the original process: delayed dispatch messages, unavailable dependencies and requests that arrive more than once.

What happens if the inventory service is unavailable when we cancel? The system should not claim that everything is complete while the stock remains reserved indefinitely. What happens if the customer retries after a timeout? We should not release the same stock twice or produce duplicate financial effects. What happens if the cancellation reaches the order service but the downstream warehouse notification fails?

What happens if the order becomes ineligible for cancellation while two events are crossing? What happens if the same message arrives twice? These are not exotic edge cases added to make the test plan look thorough. They come directly from what we said the behaviour means. The specification gives the verification work a reason.

The aim is not to accumulate tests but to find evidence that the important properties hold when the normal sequence is disturbed. An agent can help construct cases involving late messages, repeated requests, payment timeouts and inventory failures, then run them in combinations an engineer might otherwise have to explore one by one. I would want that breadth, but I would also ask where the scenarios came from. If the agent derives them only from the replacement code, it may never explore the warehouse event that the code does not know it has missed. The test set needs the operational evidence and agreed behaviour from earlier discovery, not just more variations of what the new implementation already assumes.

The Risks of Using the Old System as a Benchmark

One of the most useful migration techniques is to feed the same case into the old and new systems and compare what happens. An AI-assisted workflow can help prepare representative inputs, run the comparisons, trace divergences back to relevant source paths and group recurring differences for an engineer to investigate. That is a good use of automation because a complex application may produce more outcomes than a person can practically inspect one at a time. If both systems receive the same eligible cancellation and the old system is known to handle it correctly, a difference deserves investigation. But the efficiency of that comparison creates a trap: it is easy to treat the old result as the correct answer simply because it is the reference against which every run was measured.

The old application contains the very behaviours we spent 04 and 04.1 deciding whether to preserve. Some are deliberate requirements. Some are defects. Some are workarounds. Some are implementation constraints we explicitly chose not to carry forward.

So old and new being identical is not always success. Imagine the old system has a known defect where a retried cancellation sometimes leaves inventory reserved until an operations team clears it manually. The replacement is supposed to fix that. If the new system behaves exactly like the old one, behavioural equivalence has proved the wrong thing. The expected result comes from the agreed requirement, not from historical behaviour alone.

That means a verification run needs three accounts in view: what the old system did, what the organisation decided should happen, and what the replacement actually did. An agent can assemble that comparison and point out discrepancies, but it must not silently resolve them by choosing whichever output appears most consistent. Where the accounts differ, we need to establish whether the replacement introduced a defect, corrected one or exposed a decision that was never properly made.

A Thirty-Second Delay in Stock Release

Suppose we run the same cancellation through both systems. The old application releases the stock immediately. The replacement releases it thirty seconds later. Is that a defect? Maybe.

If another service can allocate the stock during those thirty seconds and create an oversell, the delay may violate the required behaviour. If the requirement only says that inventory must be available before the next allocation cycle and that cycle runs every five minutes, the difference may be harmless. The comparison has found something. It has not yet told us whether it matters. Now take a harder case.

The old system accepts a cancellation after the warehouse has physically dispatched the parcel because the dispatch update arrived late. The replacement rejects it once warehouse dispatch is confirmed through a newer integration. The systems disagree. Which one is correct? That depends on what the organisation authorised.

Perhaps the replacement is deliberately fixing a race condition that caused refunds after goods had already left the warehouse. Perhaps the old behaviour was actually relied on as part of a customer-service process. Perhaps the new integration changed the point at which cancellation eligibility should end. The answer is not contained in the fact that the outputs differ. We have to go back to the behavioural decision and the evidence behind it.

This is why I think verification needs traceability. When a result differs, somebody should be able to see what requirement was being checked, why that requirement exists, what changed deliberately and what evidence supports the expected outcome. An agent may suggest that a thirty-second delay is harmless, or that the old behaviour is obsolete, but those are inferences until the relevant operational constraints have been checked. Without that separation, a confident explanation can be mistaken for proof and the review becomes a contest between plausible accounts.

The Same Mistake in AI-Generated Code and Tests

There is another failure mode that is easier to miss. Imagine a team using an AI coding agent to interpret the requirements, generate the replacement and write the tests. The agent may do that work quickly and competently, and the entire suite may pass. But if it misunderstood the boundary between accepting a cancellation request and confirming that fulfilment has stopped, it can encode that misunderstanding in both the service and the tests. The two artefacts agree because they were built from the same incomplete account of the business process. This does not mean the organisation needs a completely separate verification department for every migration. It does mean I would want the expected behaviour to have some authority outside the implementation itself.

In our cancellation example, the behavioural requirement was established before the target design. The important scenarios came from the existing application, incidents, operational practice, payment behaviour, warehouse dependencies and the decisions made during preservation. Those sources give verification something independent to challenge the implementation with. For a low-consequence behaviour, normal engineering review may be enough. For something financially or operationally consequential, I would want stronger separation.

Perhaps the scenarios are reviewed by people who were not responsible for the implementation. Production examples can be replayed, and important constraints checked independently of the generated code. An independent evaluator can also challenge an agent's work, provided it receives the approved behavioural requirement, the sources behind it and the actual execution evidence rather than only the first agent's summary. A second model is not automatically independent merely because it is a different model. If both rely on the same incomplete description, both may miss the same warehouse dependency. We need a different basis for judgement, not simply another confident answer.

The exact mechanism can vary. The important point is that "the code and its own tests agree" is not the highest standard available.

The Ten-Minute Delay Nobody Can Explain

Now imagine every cancellation scenario passes except one. There is a rare order type where the old application waits ten minutes before releasing stock. In 04.2, we could not establish why. The replacement has preserved the delay temporarily, but nobody has yet proved whether it is necessary. What should the verification report say?

Not simply "cancellation behaviour passed". Most of it did. One part remains unresolved. That distinction should survive into the release decision. The organisation might decide that the unresolved case is too consequential to release.

It might decide to keep the compatibility behaviour and investigate after release. It might isolate those orders. It might accept the risk because the case is rare and the consequence is low. Those are decisions. The verification system should provide the evidence for them, not make the uncertainty disappear by collapsing everything into a green status.

This matters particularly for behaviours that are difficult to reproduce before release. A year-end process may not run during the migration window, an external partner may not provide a realistic test environment, and a rare operational recovery may depend on circumstances that are expensive to simulate. AI can propose explanations for the unexplained ten-minute stock delay and generate tests around each possibility, which can help direct an investigation. It cannot establish which explanation is true without the necessary evidence. Nor should a verification agent fill an empty evidence field with a plausible conclusion simply to complete the report. Sometimes the honest result is that part of the behaviour has not yet been demonstrated.

Then the question becomes what evidence would close the gap and how the organisation controls the risk until it has that evidence. That is very different from releasing and hoping somebody notices if it goes wrong.

Cancellations in the First Production Rollout

There are behaviours where controlled production observation is reasonable. Suppose the cancellation is sent to a warehouse partner whose test environment does not reproduce all of the production routing. Before release, we can verify the message structure, the timing requirement, the retry behaviour and our side of the contract. We may still need to observe the first real transactions after deployment to establish that the end-to-end outcome holds. I am comfortable with that if it is explicit.

We know what has already been proved. We know what has not. We know what signal we are looking for. We know what happens if the signal is wrong. And somebody has decided that the residual risk is acceptable.

That is a controlled verification gap, not a substitute for the investigation we could have done earlier. The same principle applies to gradual rollout. If the replacement handles a small proportion of cancellations first, the organisation can compare outcomes, monitor unresolved states and watch the downstream effects before increasing exposure. AI may help correlate events across the payment, order and warehouse systems and draw attention to unusual patterns, but the source observations need to remain available for inspection. A model-generated assurance summary cannot replace a missing warehouse acknowledgement or prove that the customer received the correct outcome.

The Evidence Required to Approve the Replacement

After all of this, what are we trying to say? Not that the migration has 3,000 passing tests. Not that the deployment pipeline is green. Not that old and new produced identical output in every comparison. The useful claim is narrower and stronger: the replacement has demonstrated the behaviours the organisation decided to preserve, the intended changes behave as authorised, and the remaining gaps are visible enough for someone to decide whether they are acceptable.

For our cancellation example, I would want to see the requirement that was tested, the scenarios used, the version of the replacement involved, the results, any expected differences from the old application and anything we still have not demonstrated. If a retry was tested, show the evidence that duplicate financial or inventory effects did not occur. If inventory recovery failed and then succeeded, show that the order remained unresolved until recovery completed. If the replacement deliberately rejects a cancellation the old system accepted, show the decision that authorised that change. If one warehouse path remains unverified, say so.

That gives a reviewer something they can actually inspect. It also leaves the organisation in a better position after the migration. The next team does not have to rediscover what "correct" meant by reading the implementation and guessing. They can see the behaviour that was agreed and the evidence that showed the replacement satisfied it.

That closes Series 04. We have investigated the existing behaviour, decided which outcomes still matter, described those requirements without copying the implementation, and examined how to check that a replacement actually delivers them. The point is not to demand identical code or identical outputs from every old and new system. It is to know which differences were authorised, which controls survived and which gaps still require judgement.

This is increasingly important as agents take on more implementation and verification work. They can generate alternatives, explore failure cases and assemble results at a scale that was previously expensive. They still need an independently established account of what the organisation expects, otherwise they can produce and verify the wrong behaviour very efficiently. Once that behavioural basis is available, we can ask what architecture should support it and how the old and new systems can coexist safely. That is where Series 05 — Architecture should be a consequence of evidence begins.

---

This paper is part of The Modernisation Control Papers, the first major sequence within The Harten Papers. Read the complete series on harten.io.