AI governance
Your model didn't drift. Your documents did.
Model drift measures the model. A retrieval assistant's wrong answer usually comes from a superseded document instead, and no distribution test can see that.
Cognatum Team · Oct 26, 2026 · 6 min read
Model drift has a definition, a measurement and a fix. The definition is decay in a model's performance over time. The measurement is a statistical test on the data. The fix is retraining. All three are mature. None of them looks at the document your assistant just answered from.
Cognatum governs the entry
source · version · approver · permissions
That is not a complaint about the tooling. It is a point about scope. Model drift is a property of a model. The answer your staff acted on this morning is a property of a document, a version and an approval. Those are different objects. Only one of them has a monitoring discipline built around it.
What model drift actually measures
IBM defines model drift as the degradation of machine learning model performance due to changes in data, or in the relationships between input and output variables. Stanford HAI says performance degrades because the real world data the model meets has changed from the data it was trained on. Splunk calls model drift an umbrella term that covers both concept drift and data drift. Every definition names the same subject. The subject is the model.
The tests are tests on distributions
The detection methods follow from that. The Kolmogorov-Smirnov test asks whether two data sets come from the same distribution. Wasserstein distance measures how far apart two distributions sit. The population stability index tracks how far a feature has moved, and a value above 0.2 signals a real shift. These are good tools. They all answer one question. Has the shape of the data changed?
Three reasons drift never sees a stale document
The subject is wrong
A retrieval assistant reading your policies does not drift in this sense. The model is frozen. Its weights today are its weights yesterday. What moved was a document. Somebody approved version four of a procedure on Tuesday, and the assistant is still quoting version three. Domino Data Lab makes the point from the other side. Drift, it says, is a function of the real world changing, and it is not caused by a flaw in the model itself. Neither is this wrong answer.
The clock is wrong
Drift is gradual by construction. Evidently AI describes minor variations that accumulate over time, and treats distribution comparison as a proxy you watch across batches. Supersession is not gradual. It happens on one date, to one entry, by one person's decision. One superseded procedure does not move a distribution. It changes the right answer. A test built to find a shift across a window cannot find an event of size one.
The remedy is wrong
Retraining is the standard fix, and every source above recommends it. In a retrieval system there is nothing to retrain. The corpus is read at request time, not learned. Reindexing moves the new text into the index. It does not decide whether that text was approved, whether the old entry should have been retired, or which of two contradictory entries wins. Reindexing propagates a change. It does not adjudicate one.
The fields a drift metric would need
For an answer to be checkable at all, four facts have to exist and travel with it.
- The entry the answer came from.
- The version of that entry.
- The named person who approved that version.
- The date of that approval.
None of those four is a distribution. They are discrete, dated and attributable. No statistical test produces them. No monitoring tool infers them from traffic. They either sit on the entry or they do not exist.
What the rules already ask for
EU AI Act Article 15 requires high-risk systems to achieve an appropriate level of accuracy and robustness, and to perform consistently in those respects throughout their lifecycle. Paragraph 3 requires the accuracy levels and the relevant metrics to be declared in the instructions for use. Paragraph 2 then concedes the hard part. The Commission is to encourage the development of benchmarks and measurement methodologies, working with metrology and benchmarking bodies. The duty to state a metric lands before the metric exists.
What the notice asks a firm to govern
FINRA names the use case in its own words. Regulatory Notice 24-09 says Gen AI tools may allow an associated person to easily locate and query a member firm's policies and procedures or forms. It then says a firm using such tools in its supervisory system should have policies addressing technology governance, including model risk management, data privacy and integrity, reliability and accuracy of the AI model. Model risk management. The model again. Nothing about who approved the procedure it quoted.
Retrieval was already the fix that was not enough
Researchers ran the first preregistered evaluation of AI legal research tools. Tools from LexisNexis and Thomson Reuters hallucinated between 17 and 33 percent of the time, against vendor claims of eliminating hallucinations. Those products retrieve from curated case law, which is cleaner than any corporate shared drive. Retrieval was the control, and it was not sufficient. An enterprise knowledge base is messier, and nobody publishes its error rate.
Three counts a board can read
There is a measurable version of this, and it is not statistical. Score the knowledge base itself and the numbers come out plain. How many entries duplicate another entry. How many pairs of approved entries contradict each other. How many questions have no approved answer at all. Those three counts move when the library degrades, and a person can act on each one.
What that looks like in practice
This is the work Cognatum does. When a source changes materially, every entry that depends on it is flagged and routed for re-check, and the revision queues for a human. The last approved version stays live, with its approval date visible, until a person signs off on the new one. Cognatum notifies. It does not quietly rewrite, and it is never simply current.
When two entries disagree
Where two approved entries genuinely disagree, they are held apart and routed to a named person. Neither is averaged into the other. Neither wins on recency. While the flag is open, the question comes back as a conflict rather than a confident answer. AI runs seven of the eight steps in the Cognatum Knowledge Loop. Approve is the one a person holds.
A test you can run today
Pick a procedure your team superseded this year. Ask your assistant the question that procedure answers. Then ask it for the entry, the version, the approver and the date. If the answer is right, you cannot prove it. If it is wrong, no drift dashboard in your stack has an alert for it.
Your company's knowledge isn't missing. It's unusable. Cognatum changes that. The eight steps are at cognatum.ai/the-loop.
Sources
- What Is Model Drift? (ibm.com)
- What is Model Drift? (hai.stanford.edu)
- Model Drift: What It Is and How To Avoid Drift in AI/ML Models (splunk.com)
- What is data drift in ML, and how to detect and handle it (evidentlyai.com)
- Model Drift (domino.ai)
- EU AI Act Article 15: Accuracy, robustness and cybersecurity (europa.eu)
- FINRA Regulatory Notice 24-09 on Generative AI and Large Language Models (finra.org)
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arxiv.org)
- The Cognatum Knowledge Loop (cognatum.ai)
Common questions
Questions this raises.
What is model drift?
Model drift is the decay of a machine learning model's predictive performance over time, because the production data it meets diverges from the data it was trained on. It is usually split into data drift, a change in the input distributions, and concept drift, a change in the relationship between inputs and the target. It is detected with statistical tests on distributions and normally fixed by retraining.
Does model drift apply to a RAG or retrieval assistant?
Only partly. The model's own performance can still decay, and that is worth monitoring. But most wrong answers from a retrieval assistant have nothing to do with the model. They come from the material it retrieved: an entry that was superseded last week, a duplicate copy somebody saved months ago, or two approved documents that contradict each other. The weights did not change. The library did.
Can drift detection catch a superseded document?
No. Drift detection compares distributions across a window of data. A procedure being superseded is a single discrete event on a single entry, decided by a person on a date. It does not shift a distribution, so there is nothing for a Kolmogorov-Smirnov test or a population stability index to register. Approval status, version and effective date are not statistical properties, so no drift metric computes them.
What should a regulated firm measure instead?
Measure the knowledge base, not just the model. Three counts are readable by anyone: how many entries duplicate another entry, how many pairs of approved entries conflict, and how many questions have no approved answer. Then require four fields on every answer an assistant gives: the entry, the version, the named approver and the approval date. Those are the facts a reviewer asks for, and they can be produced on demand.
Does Cognatum update entries automatically when a source changes?
No, and that is deliberate. When a source changes materially, Cognatum flags every entry that depends on it and routes a revision to the Approve gate. The last approved version stays live, with its approval date visible, until a named person signs off. Cognatum notifies rather than rewrites, so it should never be described as always current. The approval records it keeps are evidence you supply toward frameworks such as the EU AI Act, ISO 30401 and NIST AI RMF. No software confers compliance with any of them.