The Search Found the Documents. It Still Ranked the Wrong Decision First.

A GovKM and RouteMind continuity-testing case study

Imagine you work in purchasing. You ask an AI assistant which supplier is approved for a new order. Search returns an older registration and the later registration that governs today. Both documents are genuine. Both concern the same purchase. The older one appears first.

That was the distinction explored in a small synthetic test reported by RouteMind: finding the relevant records did not consistently put the governing record at the top of the results. The measurement concerned retrieval rankings; it did not establish what a model would ultimately answer.

Executive Summary

GovKM proposed a supplier-selection test to distinguish evidence that can be found from evidence that still governs. RouteMind ran the original fixture, examined why reranking succeeded, and then wrote a silent-supersession variant that removed the explicit replacement sentence.

In the silent variant, ordinary retrieval-augmented generation (RAG) and RAG with reranking each reported a retrieval-found score of 1.000 and an operative-first score of 0.333 across three questions. In everyday terms, the evidence was found for all three questions, but the governing document ranked first for only one.

These are partner-reported scores for the baseline retrieval arms, not scores for RouteMind's routing arm. They come from one synthetic fixture and one observation per question. GovKM has not independently reproduced the run.

About RouteMind: RouteMind describes itself as “A routing table for an agent, written and kept by people.”

At a Glance

Partner-reported baseline results for the silent-supersession fixture. Both RAG and RAG with reranking reported the same aggregate scores.

The Continuity Problem

An earlier approval does not become a false document when a later decision replaces it. It can remain accurate evidence of what was approved at the time. Someone investigating an old order may need that earlier registration; someone placing a new order needs the registration that applies now.

Search relevance alone does not settle that difference. A document can closely match the question while describing a state that no longer governs. Simply choosing the newest file is also insufficient when a later report describes history without changing the underlying decision.

For this case, the practical question was whether the system could distinguish a historically valid supplier registration from the operative registration for a new purchase.

How GovKM and RouteMind Developed the Test

GovKM supplied the original continuity question and supplier-selection fixture. That fixture included a sentence explicitly stating that a later approval replaced an earlier one. RouteMind reported that reranking succeeded on the original fixture and identified that sentence as a clue available directly in the document text.

RouteMind then wrote the silent variant, removing the explicit replacement sentence and retaining separate earlier and later registrations. The earlier record did not announce its own replacement. This tested a harder situation: the relationship between the records had to be established rather than read from an explicit supersession statement.

RouteMind also disclosed that it had written the variant to test its own prediction and had scored it before presenting it for review. That author-evaluator relationship matters when interpreting the result.

What the Retrieval Test Actually Showed

The three questions concerned the currently approved supplier, whether a new order could be placed with the earlier supplier, and the delivery window applicable to an order placed today. Both baseline retrieval arms found relevant evidence, yet each placed the governing document first for only one question.

In the reranked results, the governing registration appeared second for the first two questions, behind the superseded registration. For the new-order question, the results file specifically says reranking moved the governing document from sixth to second without changing the first-place outcome. The third question put the governing document first.

This is why the headline refers to ranking. A generated-answer test, identified as Q5b in the source, was not run. A model receiving both records might use the second result, rely on the first, or seek clarification. This retrieval-stage observation does not tell us which of those occurred.

Scope and Qualification

This was a synthetic, partner-run test, not evidence of a customer purchasing incident or production harm. GovKM reviewed the public results and correspondence but did not perform an independent native reproduction or independently verify the final evidence presented to an answering model.

The sample comprises three questions in one fixture, with one observation per question and no variance estimate. The fixture's author evaluated its own prediction. These limitations prevent treating the scores as a general performance estimate for retrieval systems or RouteMind.

Other routing and discovery observations in the collaboration used different configurations. They remain separate from this baseline comparison and do not establish that a shipped RouteMind product has solved the problem described here.

GovKM's Interpretation

GovKM interprets this as a useful example of the gap between locating evidence and establishing which decision governs its reuse. The earlier registration should remain available for historical explanation. The later operative registration should govern a present purchase only where the supporting authority and context justify that conclusion.

The case can be understood through GovKM's published continuity concepts:

Source → Evidence → Authority → Context → Decision → Record → Institutional Memory → Future Reuse.

This is an interpretive view of the evidence, not a claim that the test implemented or validated the entire GovKM model. The work illustrates why preserving documents and their relationships matters when an organization expects AI to use past information in present decisions.

What Follow-on Research Needs to Establish

The collaboration subsequently explored a later historical retrospective and structural discovery. Those questions extend the original problem: can an agent reach the evidence and its continuity relationships without navigation that simply supplies the answer?

Those exploratory observations require their own configuration and evidence boundaries. They cannot be combined with the three-question baseline to claim a general solution. Further evaluation would need to distinguish finding records, reaching their relationships, establishing authority, and using the supported decision in a generated answer.

Connect and Reconstruct

For GovKM, connecting records is valuable when it helps reconstruct why a decision applied, what changed, and what can safely be reused. The aim is to preserve the earlier history while making the basis for a current decision understandable.

RouteMind's contribution was to make a narrow distinction visible and inspectable. GovKM brought the continuity-testing question; RouteMind supplied an implementation, a challenging revision, and a candid account of what its measurements did and did not establish. That exchange produced a useful case study without requiring a claim of production failure or a completed solution.

Sources and Attribution

• RouteMind pull request 1: original fixture, silent variant, and collaboration discussion.
• Preserved partner results for the silent-supersession baseline.
• RouteMind publication permission, preferred branding, and factual corrections.
• RouteMind project repository.

Hero artwork is an AI-generated conceptual illustration of a reranked example, not RouteMind software, an official logo, or a captured test output.