Our readiness reports are not written by one model in one pass. They are assembled by roughly a dozen specialised agents — each analysing interview transcripts from a different angle, each producing structured findings that a synthesis layer must reconcile into a single, coherent, evidence-backed document. We have been running this pipeline in production for over a year. Here is what that experience actually taught us, because very little of it appears in vendor documentation.

The first failure mode is agreement, not disagreement

We expected the hard problem to be agents contradicting each other. It wasn't. The hard problem was agents plausibly agreeing — each producing findings that looked consistent but rested on slightly different readings of the same transcript. Individually defensible, collectively misleading. The fix was not a smarter model; it was a validation layer that checks every claim against the source material before synthesis, and a policy engine that decides — per report tier, per agent — whether a partial result is acceptable or the run must stop.

That phrase, "policy engine", is doing a lot of work. It took us longer to design the rules for when an agent's output is good enough than it took to build the agents themselves. If you are planning a multi-agent deployment, budget accordingly: the agents are the cheap part.

Determinism is a feature you must build, not a property you get

Run the same transcripts through the same pipeline twice and, untreated, you will get two different reports — both reasonable, both defensible, visibly different. For an internal tool that may be tolerable. For a document a client pays for and a consultant signs, it is not. We had to engineer determinism deliberately: pinned prompts, structured output schemas, controlled sampling, and an evaluation corpus of synthetic interviews with known ground truth so that any pipeline change can be tested against expected findings before it ships.

That last item — a corpus with known ground truth — is the single most valuable artefact we have built. It converts "the report seems off" from an argument into a test failure.

Schemas age faster than models

Every agent in the pipeline emits structured output against a schema. The schemas encode assumptions — minimum counts of findings, expected sections, required evidence — and those assumptions break in ways models don't. Our sharpest production incident came not from a model change but from a schema minimum that assumed multiple interviews when a lower product tier legitimately supplied one. The model behaved perfectly; the contract around it was wrong.

The general lesson: in multi-agent systems, the interfaces between agents are where reliability lives and dies. Version them, test them, and assume every hard-coded number in them is a latent bug.

What this means for your organisation

You may never build a pipeline like this — most organisations shouldn't. But if you are evaluating agentic systems, these field notes convert into due-diligence questions: How does the system validate outputs against sources? What happens when an agent's output is partially acceptable? Can the vendor re-run yesterday's job and get yesterday's answer? What is their equivalent of a ground-truth corpus?

If the answers are vague, the demo you watched is the best behaviour you will ever see from that system.