The Model Name Was the Same. The Judge Wasn’t.

Organizations increasingly use AI not only to generate work, but to measure work.

Models grade answers, rank candidates, evaluate safety behavior, compare vendors, score policy compliance, select training examples, and decide whether another AI system passes a gate.

That makes the evaluating model an institutional measuring instrument.

A September 2026 preregistered study raises a continuity problem for that instrument. Researchers replayed identical requests against black-box LLM judges and found that the same named model endpoint did not behave with the stability required by their preregistered measurement criteria. Same-window repeat rankings reached a Spearman correlation of only 0.400 against a 0.90 threshold, while byte-identical next-day replays reached 0.78 agreement against a required 0.99.

The model name was the same.

The measurement behavior was not.

A Model Identifier Is Not an Instrument Identity

Traditional measurement systems distinguish an instrument from its label. Laboratories track calibration, serial numbers, configurations, environmental conditions, and maintenance history because “the same kind of instrument” does not guarantee the same measurement state.

AI governance often records much less: model name, provider, prompt, and perhaps temperature.

That may be insufficient when the service behind the identifier can change through routing, serving configuration, model updates, safety layers, infrastructure, or other provider-controlled conditions.

If those changes affect the output distribution, the persistent model name can conceal a changing instrument.

The Topology Places the Judge Inside the Evidence Chain

When an LLM judge determines a ranking or pass/fail result, its output becomes Evidence for a later Decision.

The relevant path is:

Source / candidate output → Evidence / evaluation → Authority / accepted judge → Context → Decision → Action.

If the judge cannot be reconstructed with enough precision to reproduce its measurement behavior, the Evidence relationship becomes unstable.

The institution may possess a complete log of what was submitted and what score came back while still being unable to establish whether a replay would measure the same thing.

Logging Is Not Reproducibility

This distinction matters because good engineering can preserve every request perfectly.

The cited study found that request delivery and execution logging were effectively intact. The instability appeared in the measurement itself.

A fully logged decision is therefore not automatically reproducible if the instrument that generated a consequential measurement cannot be reconstructed.

Instrument Provenance Should Match Consequence

Not every AI evaluation requires laboratory-grade reproducibility. A low-stakes quality hint can tolerate variability that a safety gate cannot.

But as consequence rises, governance should preserve more about the actual evaluator state: provider, model family and version where available, endpoint, evaluation prompt, sampling controls, time, known serving changes, calibration tests, repeatability evidence, and the policy describing how unstable or conflicting measurements are handled.

The more authority the measurement carries, the stronger the provenance should be.

AI-on-AI Governance Needs Independent Evidence

The issue also connects to another GovKM principle: adding more AI layers does not automatically add independent evidence.

If one model generates an answer and another unstable black-box model judges it, the institution has created an additional inference layer. It has not necessarily created a stable verification layer.

Evaluation systems should therefore distinguish model judgment from ground-truth evidence, human authority, deterministic checks, and reproducible external measurements.

The GovKM Proposition

When AI becomes an institutional measuring instrument, continuity requires instrument provenance—not merely a persistent model name.

An organization should be able to reconstruct enough of the evaluator’s operational state to understand whether a later measurement is meaningfully comparable to the earlier one.

The identifier can remain constant while the measurement relationship changes.

Continuity must preserve the instrument, not just its name.