KPMG's Project Exponential is being read as a scoreboard. It is actually a deployment pattern. Here is how to structure a parallel run for a regulated workflow, which seams to instrument, and the governance bill that arrives with the results.
Last Updated: November 21, 2026
A parallel run is a deployment pattern, not a result. KPMG put more than 100 agents through every phase of a real FTSE 250 audit alongside the human team and did not rely on the output for sign-off — which is the point, not the caveat. The two facts worth copying into your design doc are the uncomfortable ones: a human had to step in when an agent could not match a set of regulations to company documents, and the UK audit regulator has now written down that technology which inspects whole populations rather than samples produces a volume of findings that creates its own independence and disclosure problem. Budget for the second system, instrument three specific seams, measure intervention rate per seam, and write the findings-handling policy before the first run.
What KPMG actually ran, stripped of the headline
In an experiment code-named Project Exponential, KPMG UK deployed more than 100 AI agents to review the financial statements of an unnamed FTSE 250 company, running the experiment parallel to a conventional audit by its accountants. The agents were not confined to one procedure. They worked autonomously with each other, suggested next steps, gathered and analysed data, documented it, and formed draft conclusions and observations.
Three details matter more than the "better questions" quote that drove the coverage. First, the build was fast: a team built the agents in four months, and an audit partner then "signed off" the experimental audit, which was not relied on when KPMG formally signed off the company's accounts. Second, the human surface area was deliberate: staff set the agents' objectives, supplied information about the business and previous audits, checked the work periodically and challenged conclusions before the audit opinion was finalised. Third, the failure was specific and recognisable: in one area a human had to intervene after an agent failed to match a set of regulations to the company's documents.
That last one is the most transferable finding in the whole story. Cross-referencing a normative text against a corpus of entity-specific documents is the single most common agentic task in regulated work — it is what claims adjudication, AP policy checking, sanctions review, contract compliance and clinical coding all reduce to. And it is exactly where retrieval quality, not model reasoning, decides the outcome.
Shadow mode, parallel run and limited deployment are three different things
Teams use these terms interchangeably and then argue about results that were never comparable. They differ on one axis — what the agent is allowed to touch — and that axis determines which risks you can actually observe.
| Pattern | Agent reads | Agent writes | What you learn | What you cannot learn |
|---|---|---|---|---|
| Sandbox pilot | Synthetic or frozen extract | Nothing | Whether the prompt chain works at all | Anything about real data quality, latency, volume or edge cases |
| Shadow mode | Live production data, read-only | Nothing outside its own log | Retrieval quality, output agreement with humans, cost per unit at real volume | Whether downstream humans behave differently when the agent output is visible |
| Parallel run | Live production data, read-only | A complete, reviewable second work product that a named human signs | End-to-end defensibility: evidence trail, escalation design, reviewer workload | Behaviour under production time pressure, when the human run is removed |
| Limited deployment | Live data for a subset of teams | Into the real record, with alternative procedures still performed | Reliance behaviour, automation bias, supervision cost | Fleet-level effects at full volume |
| Full production | Live data | Into the real record, alone | Everything, expensively | — |
Project Exponential sits in row three. The regulator has already described row four in writing. The FRC's Generative and Agentic AI Guidance notes that when firms release a tool to a limited number of teams before full certification, "teams often perform alternative procedures in full, in parallel to the AI-enabled procedure". In other words the doubling of work is not an extravagance invented by one firm's innovation budget; it is the expected control while the system is uncertified.
The practical rule: shadow mode answers "does the model get it right?" A parallel run answers "can a qualified human defend this output to a third party six months from now?" Only the second question is the one that blocks production in a regulated workflow. If your governance committee is going to ask the second question, do not spend a quarter answering the first.
The three seams worth instrumenting
Most parallel runs are instrumented at the wrong granularity. Teams log the final answer and compare it to the human answer, get an agreement percentage, and learn almost nothing actionable, because when the numbers disagree nobody can say where the divergence was introduced. Agentic systems fail at joints, not in the middle of steps. Instrument the joints.
Where to put the probes
Log the exact chunks, documents and clause IDs the agent had in context when it formed a view, not just the citation it chose to display. The KPMG failure — an agent unable to match regulations to company documents — is a seam A failure, and it is invisible if you only diff conclusions.
Log every point where the agent could have stopped and asked, and what it did instead. Capture confidence, the rule that fired, and whether the handoff reached a human who had authority to act. Silent non-escalation is the dominant failure in week three of a run.
Log what happened to each finding after a human saw it: accepted, rewritten, discarded, or escalated into the official file. This is the seam that produces your retention and disclosure obligations, and the one teams build last.
Diff the agent record against the human record at each seam, classify every divergence by type and owner, and hold the classification scheme fixed for the whole cycle so the trend means something.
The evidence standard KPMG described is the right target. Its system reportedly provided "full visibility and full traceability" of the agents' work, recording interactions with documents down to individual characters and timestamps. That is a deliberate engineering choice with real cost: character-level provenance means your retrieval layer has to carry offsets through chunking, embedding, re-ranking and summarisation without losing them. Most RAG stacks throw that away at the first summarisation hop. If you want it, you design for it before you pick the framework, not after. We cover how that constrains model and tool choice in our model coverage notes.
One more seam-level detail the vendor docs tend to skip: context compression. The FRC guidance explicitly warns that an agent's understanding of the goal may drift over the course of the system's activity if the original prompt is altered through incremental compression to manage space in the finite context window. In a long-running audit-style job with hundreds of documents, that is not a theoretical risk — it is the single most common cause of an agent confidently answering a slightly different question than the one it was given. Log the compressed goal statement at every iteration. It is three lines of code and it will explain a week of weird results.
Intervention rate is the metric, not accuracy
Accuracy against a human baseline is a weak readiness signal because the human baseline is itself a judgement, and because "the agent found something we missed" scores as a disagreement. The metric that actually predicts whether you can ship is intervention rate, measured per seam and tracked as a curve, not a number.
Define it as: the proportion of work units in which a qualified human had to take an action the design did not anticipate. Split it three ways.
Corrective interventions — the human fixed a wrong output. Unblocking interventions — the agent stalled or looped and a human restarted it, as in the regulations-matching case. Judgement interventions — the agent produced a defensible output but the decision was never delegable in the first place. Only the first two should trend down. The third is the permanent floor of your design, and it is the number you should be able to state to a regulator or a board: here is the work we will never hand over.
Two curve shapes tell you something real. A corrective intervention rate that falls steeply and then plateaus well above zero means you have exhausted prompt and retrieval fixes and the remaining errors are structural — usually data quality upstream. An unblocking rate that stays flat while corrective rate falls means your escalation design is wrong: the agent is right more often but still cannot recognise when it is stuck. The second pattern is more dangerous, because teams see the accuracy chart improving and ship.
Also track reviewer load per 100 findings and median time-to-human on escalations. A fleet that halves error rate while tripling reviewer load has not improved anything; it has moved cost from one column to another. That trade is the most common reason an impressive parallel run never converts, and it is why we treat reviewer workload as a first-class design constraint in any agentic build.
Treating every divergence from the human run as an agent error
It happens because the human output is the only ground truth available, so the comparison script quietly encodes "different means wrong". The KPMG result cuts directly against this: the firm reported that in several cases the agents went deeper and asked better questions than the human team. Under a naive diff, every one of those would have been logged as a false positive and prompted away.
The cost model, honestly
A parallel run costs more than double the single-system baseline, and anyone who tells you otherwise has not run one. The structure of the bill:
The human cycle, unchanged. You do not get to thin the human team during a parallel run; if you do, you have contaminated your baseline and built a worse version of the thing you were comparing against.
The agent build. KPMG's figure — four months, two partners and one director plus senior managers and junior staff — is a useful anchor for a complex, multi-phase regulated workflow at a firm with existing methodology assets. For a narrower workflow such as AP exception handling or first-line claims triage, the equivalent build is weeks, not months, but the ratio of senior-to-junior time is similar and surprises people: the expensive input is domain experts writing down judgement rules that were previously tacit.
Inference at full population. This is the line item that blows budgets. Sampling is a cost control as much as a statistical method. The moment you run every transaction instead of 40 of them, token spend scales with the population, and long-context reasoning over whole document sets is the most expensive mode you can operate in. Model and run it before you commit to the scope.
Reconciliation labour. Someone has to diff two complete work products and classify every difference. In my experience this is consistently underestimated by a factor of two or three, because it cannot be delegated to the people who are cheapest — it requires the same seniority as the original judgement.
Evidence storage and retention. Character-level provenance over a full population produces an artefact that is often larger than the source data, and it inherits the retention clock of the regulated record it supports.
Against that, the honest benefit case for a single cycle is not savings. It is a defensible answer to the certification question, a measured intervention floor, and a list of the upstream data problems that will block automation anyway. If the sponsor needs ROI inside the pilot cycle, do not run a parallel run — run shadow mode on one procedure and accept that you are answering a smaller question.
The findings problem nobody budgets for
Here is the exposure that the "AI beat the auditors" framing completely misses. A successful parallel run does not end with a verdict. It ends with a pile of findings that someone now owns, and in a regulated workflow, possession of a finding creates obligations.
The FRC published guidance on exactly this on 16 November 2026, two days before the KPMG story broke. Its framing is that the growing prevalence of AI in audit may increase the quantity of insights, because AI facilitates new forms of procedure and acts as a catalyst for greater use of more traditional data analytics. The guidance is blunt about the consequence: technology may enable a significantly higher volume of insights to be communicated than was previously possible, which may increase the risk that management act upon an insight without critically evaluating it. It also sketches a failure mode that should be familiar to anyone running continuous controls monitoring: if technology enables near-continuous monitoring and the auditor communicates every deficiency in real time, management may come to rely on that communication as part of their own control environment, giving rise to a potential management threat.
Whatever technology a firm uses, the auditor remains accountable.
— Mark Babington, Executive Director of Regulatory Standards, FRC, Financial Times via Irish Times
Generalise this past audit and it holds everywhere. If your claims agents review 100% of open claims instead of a 2% QA sample, you now hold documented knowledge of every irregularity in the book, including the ones in closed periods. If your compliance agents read every contract rather than the renewal cohort, you hold knowledge of breaches nobody asked about. The parallel run manufactured that knowledge as a by-product of testing, and "it was only a pilot" is not a disclosure defence.
The related subtlety in the FRC's agentic guidance is that full-population outputs are not simply better evidence. It observes that AI-enabled procedures may produce outputs relating to entire populations rather than samples, while the insight for each individual item may be less conclusive than in traditional substantive work — and that comparing the persuasiveness of the two may be impossible quantitatively. Breadth is not depth. A run that touches everything shallowly can generate more findings and less assurance at the same time.
Writing the findings policy after the findings exist
Teams treat findings handling as a reporting question to settle at the end. By then every option is compromised: the data exists, people have seen it, and any decision not to escalate looks like suppression in hindsight. The pressure to quietly delete the shadow logs arrives in week six, and it is the worst possible outcome.
When not to run in parallel
Three situations where the answer is to stop.
Your source data is not clean enough. KPMG's own head of audit noted that few companies currently have data clean enough for agents to work with, and that the technology is not yet available at the scale needed to run thousands of audits. A parallel run on dirty data measures your data, not your agents, at double cost. Run a two-week data readiness probe first and spend the money on remediation if it fails.
The decision was never delegable. If the output is a judgement your regulator expects a named qualified person to form — a going-concern conclusion, a clinical determination, an adverse employment decision — then an agent can assemble evidence and draft, but the parallel run should be scoped to the assembly, not the judgement. Testing autonomy on a step you will never automate is theatre.
You cannot staff the reconciliation. The reconciliation is the deliverable. If the senior people who would do it are already fully committed to the live cycle, the run will produce logs nobody reads and a conclusion drawn from vibes. Defer a quarter.
The broader signal from this month is not that agents are ready for regulated production. It is that the serious players have converged on the same intermediate step. PwC has also run several "parallel audit" projects to test AI extensively without relying on it, according to its global assurance leader. Meanwhile stakeholder confidence is genuinely split: 79% of respondents to the 2026 Audit Trust Index are satisfied with how auditors are using AI, but 23% of audit committee chairs report limited confidence and 15% of investors report limited or no confidence. A documented parallel run with a published intervention floor is the most direct answer anyone has to that 23%.
If you are scoping one for AP, claims, compliance review or any other controlled workflow, the sequencing matters more than the model choice — see our build write-ups or start a scoping conversation.
Frequently Asked Questions
What is the difference between a parallel run and shadow mode for AI agents?
Shadow mode means agents read live production data and write only to their own log, so you can compare outputs against human decisions. A parallel run goes further: the agent fleet produces a complete, reviewable second work product that a named human reviews and signs, even though the organisation does not rely on it. Shadow mode tests model quality. A parallel run tests whether the whole system — evidence trail, escalation, reviewer capacity — is defensible.
Is a parallel run worth running two systems for a full cycle?
In a regulated workflow, usually yes — but not for ROI inside the cycle. You are buying three things: a measured intervention floor you can state to a regulator or board, a defensible evidence trail, and a prioritised list of upstream data problems. If the sponsor needs payback within the pilot, scope down to shadow mode on a single procedure instead. The FRC's own guidance notes that teams often perform alternative procedures in full alongside uncertified AI tools.
What should I measure during an agent parallel run?
Intervention rate per seam, split into corrective, unblocking and judgement interventions, tracked as a curve over the cycle. Alongside it, measure reviewer load per 100 findings and median time-to-human on escalations. Classify every divergence from the human output into four buckets — agent wrong, human wrong, both defensible, agent found something new — and report them separately rather than collapsing them into a single accuracy figure.
Why did a human have to intervene in KPMG's AI audit?
According to the Financial Times, in one area of the audit an agent failed to match a set of regulations to the company's documents and a human stepped in. That is a retrieval-to-judgement failure, the most common one in regulated agentic work: the agent cannot align a normative text with entity-specific evidence. It is invisible if you only compare final conclusions, which is why that seam needs its own logging and its own escalation rule.
Do AI agents reviewing full populations create a compliance problem?
They create a findings-volume problem. The FRC's November 2026 guidance warns that technology can enable a significantly higher volume of insights to be communicated than was previously possible, raising the risk that management act on them without critical evaluation, and that real-time communication of every deficiency can make the reviewer part of the client's control environment. Outside audit, the same dynamic produces disclosure, retention and legal-hold obligations nobody planned for.
How long should a parallel run last before you decide?
One complete business cycle of the workflow, not a fixed number of weeks. For a monthly close or claims cohort that may be four to six weeks; for an annual audit it is the full engagement. The reason is seasonality of edge cases: the hard items cluster at period boundaries. Stopping early gives you a clean-looking intervention curve built entirely on routine work, which is precisely the data that misleads a go-live decision.
What evidence should an agent system retain?
At minimum: the exact retrieved context behind each judgement, the compressed goal statement at every iteration, every escalation decision and the rule that fired, and the disposition of each finding after human review. KPMG described traceability down to individual characters and timestamps. That level of provenance has to survive chunking, re-ranking and summarisation, so it constrains your architecture from the start — it cannot be bolted on after the fact.
Research digest
AI Research Briefing
Honest insights on AI agents, Small Language Models, and local RAG. No hype. Only when we have something worth sending.
- No hype, just measurable outcomes
- Read by 2,400+ engineers
- Unsubscribe anytime