Changing AI models without breaking production

A replacement model writes better support replies in a demonstration. In production, it also interprets a cancellation request differently and chooses a refund tool the previous version rarely used. The response still looks polished. The workflow has changed.

Treat the swap as an application release. A model participates in the behaviour of the whole system: which evidence it uses, which tools it requests and when it decides to stop. A successful API call tells you little about whether those decisions remain acceptable.

Save the configuration you are replacing

Before testing a candidate, record the model identifier, instructions, generation settings, tool definitions and output schema used by the current application. Include retrieval settings and any processing that transforms the response before another system uses it.

Save a release identifier with each run. If the candidate starts producing errors, the team needs to distinguish its traffic from the previous configuration without reconstructing the deployment from memory.

Where a provider offers a fixed version identifier, check its availability and retirement policy before relying on it for rollback. Returning to an old configuration is useful only while its dependencies still work.

Compare the same work

Build a set of representative inputs from the workflow, with appropriate protection for customer data. Include previous failures and cases that should end without an action. Preserve the reference information needed to judge each case.

Run both configurations against that set. Keep the comparison narrow enough to explain: if you change the model, prompt and retrieval process together, a difference in behaviour could come from any of them. Sometimes a new model requires a prompt adjustment; record that as a separate candidate configuration rather than silently modifying the baseline.

A feature flag can control which requests use the candidate configuration, provided both routes and their fallback behaviour are tested.

Google Cloud’s evaluation documentation describes a workflow built around task-specific datasets, chosen metrics and inspection of both aggregate scores and individual responses. That separation is useful here: an overall score can identify a change, while individual cases help explain it.

Check decisions as well as answers

For a document extractor, compare required fields, missing values and whether the cited evidence supports the extraction. For a support agent, check the proposed recipient, policy reference and action. For a research assistant, inspect whether the sources support its claims.

Some checks can be deterministic. A result either matches the required schema or it does not. A requested tool either falls within the user’s permissions or it does not. Other judgements, such as whether a reply makes an unsupported promise, need an explicit review criterion and sometimes a knowledgeable person.

Review results by task category. A candidate could improve routine questions while becoming worse on refunds. Decide whether those trade-offs are acceptable before allowing the aggregate improvement to drive the release.

Count the cost of completing the task

Compare total cost, latency and success for the same unit of work. A cheaper call can become a more expensive task if the agent makes extra attempts or sends longer responses through later stages.

Keep failures and timeouts in the comparison. If you measure only successful runs, a model that abandons difficult tasks early can appear faster without delivering more useful work.

Define practical limits for each workload. A system drafting overnight reports has different latency needs from one waiting on a customer’s reply. The limits should come from the process it serves, rather than a universal target copied from another application.

Expose a small, identifiable group first

After offline checks, use shadow mode where the architecture permits it. Record candidate decisions without letting them perform consequential actions. That can reveal differences in current inputs that the saved evaluation set did not cover.

Then release to a bounded group whose traffic can be identified and stopped. Route consistently so a single workflow does not move unpredictably between configurations. Watch the checks that matter to that group, including review corrections and failed actions.

Keep permissions and approval gates in the execution layer. A candidate passing the test set does not give it authority to bypass them. The release should change the model configuration without weakening the controls around it.

Make rollback a specific operation

Write down the signal that stops the rollout, who can make that decision and how new work returns to the previous configuration. Test the switch before relying on it during an incident.

Handle in-flight tasks separately. A conversation or long-running job may already contain candidate output. Decide whether it finishes under its original configuration, pauses for review or restarts from a safe checkpoint. Switching the model does not undo an email already sent or a database record already changed.

After the release, add newly discovered failures to the evaluation set and keep the previous results. The next model comparison should include what this rollout taught you. The method for maintaining that baseline is covered in evaluating AI agents.