What changes in your agent when its model is upgraded?

Who wrote the runs

When a safety classifier flags a request, Claude Code re-runs it on a fallback model and, by default, stays there for the rest of the session. That behaviour is documented. In the version we ran, 34 of Opus 5.5's 330 runs were handed over partway, 33 to Opus 5 and 1 to Opus 4.8, and a typical handed-over run was mostly written by the other model. Claude Code's routing has changed since.

Claude Code's own transcripts recorded every handover. Harbor, the harness that ran the benchmark, recorded none in 4 of the 37 handed-over runs across both models. A before-and-after comparison that does not read the model named on every reply compares a mix. Set those runs aside, and every result here still holds.

What it means for your agent

The rates here do not transfer to your agent; the questions do. Before an upgrade, run the unchanged agent twice on the same tasks: whatever moves between those two batches moved with nothing changed, and that is your baseline. Then read three things on each side of the change: which model wrote each reply, which tasks took a different path beyond that baseline, and which tasks you rely on stopped passing every time. Here, 17 of the 20 tasks Opus 5 passed every time still did, within what running the same model twice gives.

One run each is not enough. Had each task here been checked once before and once after, about 3 in 10 of those checks would have told you something different from what five runs of each showed.

Method and limits

We ran no model ourselves. The runs are the two public jobs behind the Terminal-Bench 4.0 leaderboard on Harbor Hub: 66 coding and system tasks, five runs each per model, both at maximum effort. We wrote down every question, and the one test that judges it, in public before downloading them.

To see how much results move when nothing changes, we mixed each task's ten runs and split them into two groups of five, a thousand times. A change counts as more than that only when no more than 25 of the 1,000 splits reach it.

This is one agent on one benchmark at five runs per task. The model, the Claude Code version it needs and the longest reply allowed changed together, as they do in an upgrade, so nothing here is caused by the model alone. List prices fell 20% between the models, which explains part of the cost drop.

Reproduce it

The full note, the pre-registration, the analysis code and two commands that rebuild every number are public.

• Full note The findings in detail, with the table of what to check before you upgrade.
• Data and code Method, deviations, and the scripts that download the runs and rebuild every report.

All five trees replay the real runs in the browser:

• layout-config-recreation2, the tree above.
• ks-solver-cpp, the largest drop: 3 of 5 runs passed on Opus 5, none on Opus 5.5.
• layout-config-recreation, the largest rise: none of 5 passed on Opus 5, all 5 on Opus 5.5.
• atrx-vep-crispr and kv-live-surgery: each holds one of the two Opus 5.5 runs that ended on a safety refusal, and both were runs Claude Code had handed to another model.