Testing an agent in shadow mode

An order-support agent recommends sending a replacement. The operator refunds the order instead. Both decisions look plausible until someone notices that the item is no longer in stock.

A test that scores only the wording of the reply could miss that difference. Before giving the agent permission to act, compare its proposed actions with the decisions people make on the same work.

In shadow mode, the existing process continues serving customers. A parallel agent receives copies of selected inputs and records what it would have done. Its proposed writes and messages do not reach production. The purpose is to expose mistakes while the established workflow remains in control.

A copy of the input is only the beginning

Model-serving platforms already support this testing pattern. Amazon SageMaker’s shadow deployments, for example, let teams validate a candidate serving component alongside a production variant. An agent needs additional boundaries because its output can be a request to change another system.

Give the shadow worker its own credentials and an execution path that cannot send, refund or modify customer records. A dry-run instruction in a prompt is insufficient: the application must intercept proposed side effects before calling the real service.

Reads need attention too. A lookup may expose private data, incur a charge or consume a limited API quota. Decide which reads the test can perform, and use captured responses or fixtures where a live lookup would be inappropriate.

Compare decisions made from the same facts

Return to the replacement example. If the agent sees yesterday’s inventory while the operator sees today’s, the disagreement does not establish whether the agent chose badly. It establishes that the comparison used different information.

Record the input timestamp, relevant account state and tool responses available to each decision. Attach a shared case identifier so the operator’s action can be matched to the shadow run later. Capture the agent’s prompt and model configuration as well; a change during the trial otherwise makes the results harder to interpret.

Do not assume the operator is always right. A disagreement may reveal an unclear policy, a stale record or inconsistent human practice. Have someone review the evidence and assign a reason before treating the operator’s action as the expected answer.

Score the action that matters

A useful record describes the proposed operation, its target and the conditions required to permit it. For a replacement, that could include the order identifier, item, shipping destination and stock check. For a reply, retain the proposed recipient and complete message.

Separate harmless variation from consequential differences. Two acceptable phrasings should not count as a failure. A wrong recipient, an unsupported promise or a refund above the permitted amount should.

Track results by request category. An overall agreement rate can conceal poor performance on cancellations or unusual shipping cases. Record cases where the agent declined to act, reached a limit or failed to produce a result; excluding them would make the trial look better than the workload actually performed.

Decide how to handle the next step

An agent’s first proposed action changes what it would see next. If a shadow agent proposes cancelling an order, you cannot execute that cancellation merely to discover the subsequent state.

For a first trial, stop at the proposed consequential action and score it. To test longer sequences, use a simulation with defined responses or replay captured interactions. Record where the simulated path diverges from reality. A simulated successful completion is evidence about that simulation, not proof that the live workflow will finish correctly.

This limitation matters when assessing recovery. A worker that handles ordinary requests well may still fail after a timeout or partial update. Add those cases to the evaluation set rather than waiting for enough of them to appear in routine traffic.

Choose the release boundary before the trial ends

Define which categories may move forward, which errors would stop the trial and who can authorise the next stage. Base those conditions on the consequences of a mistake and the cases the trial actually covers. There is no universal number of runs that establishes readiness.

A successful shadow trial can justify a limited supervised release: the agent prepares an action, a reviewer checks the exact proposal, and a separate execution path carries it out. Keep the original handling route available. Compare results again after the agent begins affecting real workflows, because the environment now responds to its actions.

The approval design is covered in human-in-the-loop agents. The trace needed to investigate disagreements is described in agent observability.