The model you requested isn’t always the model that answered

Each judging call in this pilot requested a specific model through Claude Code, run unattended in non-interactive mode. If the model serving a judge changed unnoticed, a comparison between judges would be confounded.

Three of the 190 calls were answered by another model, all on the same case: 3 of that case’s 10 answers. In one, the transcript also records the requested model’s refused attempt, which produced no answer text, before the switch. No other transcript recorded before the study’s confirmatory run (666 in all, from both labs’ models) showed a served model other than the one requested. The same fallback recurred in the study’s confirmatory run: of its 802 transcripts (again from both labs’ models), 7 were answered by a model other than the one requested, all of them Claude Code calls and 6 of them on the same case as before. (Updated 14 October 2026, once that run was unsealed. This sentence first said its counts would follow.)

Why it happened

Claude Code documents this behaviour as automatic model fallback: when a supported model’s safety classifiers flag a request and automatic switching is on, Claude Code can re-run it on a fallback model. In an interactive session you see a notice. Our runs had no one watching.

Which records saw it

Each call left several records of which model ran.

Record	What it said
The session’s init record, which our harness copied as its reported model	the model we requested
Original- and fallback-model fields in the raw output	the switch
Each message’s own model field	the switch, message by message
The final result’s model-usage summary	both models, without saying which one wrote the answer

Our harness copied the init record without checking it. The other records showed the switch: the fallback record in the raw output named the retry target and said the session switched, a fallback notice appeared in the transcript, and each message’s model field named the model that wrote it. The usage summary listed both models but has no field for which one wrote the answer; we checked that against the messages. We caught it because our audit reads every message’s model field, a check we had built for an earlier study’s question: which model serves each call.

The other lab’s client showed a similar gap without any switch. In the calls we checked, each thread’s start record named the client’s default model, while each turn’s record named the model we requested; we found no reroutes. A harness reading only the start record would have reported the default rather than the model recorded for each turn.

What it means for an evaluation

Switching was triggered by a classifier flag; the study didn’t randomise it. If flagging correlates with case content or grading behaviour, pinning a judge by request alone can bias results in a direction the study can’t see. The three falling on one case is compatible with that, and also with chance or with features that case’s inputs share. The documentation says a session’s surrounding context can trigger the classifier too; every judge call started a fresh session in the same workspace, which doesn’t rule that out. These counts establish neither bias nor its direction.

Dropping the switched calls has its own cost: it removes exactly the items the classifier flagged, so the graded set isn’t the full set either.

This is one pilot’s observed proportion, not a rate to generalise. It depends on the model, the inputs and the settings.

What we got wrong

• Our harness trusted the summary. Our coordination tool copied the session’s init record and reported the requested model on all three calls. A fix is proposed.
• We reported early numbers. “1 in 113” was counted when judging halted at the first fallback, before the remaining 77 calls ran. A later count came from a scan taken before judging finished. The standing substitution scan now records when it ran, which script ran it and how many files it read in each folder.
• We called one answer mixed. We first described one call as begun by one model and finished by another. The returned text showed the other model wrote all of it; the transcript recorded both. Our scan now counts answer text by model.
• We left automatic switching on. When we froze the study design, we considered a per-call override and skipped it, going by the tool’s built-in description rather than its documentation.

What we’d set up differently

1. Turn automatic switching off for unattended evaluation runs. The documentation says that with switchModelsOnFlag set to false, a flagged request in non-interactive mode ends with a refusal instead of switching. We tried to confirm that and couldn’t. We replayed the three flagged inputs with switching off, a day later, in a fresh folder and with different local settings, and none was flagged: the requested model answered all three. So a flag isn’t a fixed property of the input, or it depends on context the replay changed. From our next study on, we’ll turn it off on the documentation’s word and record any refusal it produces as one.
2. Read the served model per message, not from the session’s start or the harness’s summary, and check which records carry the returned text. A transcript that names two models isn’t necessarily an answer written by two.
3. Treat a refusal as a refusal. In our study, a refused call, even one a fallback answered, gives no grade and is never retried. The replay shows why that matters: a retry can come back unflagged, and grading whichever attempt passed would let the classifier choose which answers get graded. Above 5% refusals per judge per run, that judge’s batch stops for review.
4. Scan before you unseal, and make partial scans visible. A served-model scan runs over every transcript before any result is opened, and its output says what it read.

The scan and its test fixtures are public at https://github.com/SuperDuperDave/which-model-answered. The fixtures are the demo: thirteen synthetic transcripts, including a two-model transcript with no refusal marker, which a marker-only check would miss.

What would change our view

• Similar fallback proportions across cases would weaken the evidence for case-level clustering, though they wouldn’t rule out bias within cases.