Financial services
Complete compliance documentation, wrong answers
Every framework tells you which documents to keep and how to control each one. None of them asks whether two of those documents say the same thing. An AI assistant reading your whole set is the first thing that notices.
Cognatum Team · Oct 8, 2026 · 5 min read
A compliance team can hand a regulator every document on the list and still be wrong. The set is complete. The answer someone gave last Tuesday came out of that set. The two do not match. That is not a filing problem. It is a consistency problem. Almost nothing in a standard compliance program tests for it.
Cognatum governs the entry
source · version · approver · permissions
What the term is taken to mean
Sprinto holds the top organic result for the term. It calls the thing “the complete record of how your organization meets regulatory and security obligations”, made up of policies, controls, evidence and outcomes. Scytale’s glossary agrees. It calls it “the detailed records, policies, procedures, and evidence a business maintains” to show that a compliance program works.
Both describe a set. Both treat that set as the sum of its parts.
The lifecycle runs one file at a time
Sprinto also lays out the lifecycle: create, review and approve, distribute, maintain, version control, then retire or retain. Look at what every stage has in common. Each one acts on a single document. Approval names an owner for that document. Version control tracks changes to that document. Retention decides how long to keep that document.
Not one stage compares a document with any other document.
Completeness gets tested. Consistency does not.
The regulatory texts have the same shape. Section 404 of Sarbanes-Oxley, codified at 15 U.S.C. 7262, requires each annual report to carry an internal control report. Management states its responsibility for the control structure and assesses how well it works. The assessment is about the controls. It says nothing about whether the documents describing them agree.
FINRA Regulatory Notice 24-09 is closer to the point. It reminds member firms that FINRA’s rules are technology neutral and still apply when a firm uses generative AI. It even names our case. These tools, it says, “may allow an associated person to, for example, easily locate and query a member firm’s policies and procedures or forms.”
What the notice asks for
The notice then lists what a firm’s supervisory procedures should address. Technology governance. Model risk management. Data privacy and integrity. Reliability and accuracy of the AI model. Four requirements, all of them about the system doing the reading. None about whether the policies being read say the same thing.
An old failure in a new place
The Basel Committee met a version of this in 2013. Its Principles for Effective Risk Data Aggregation and Risk Reporting exist because the crisis that began in 2007 showed that many banks, global systemically important ones included, could not pull their risk exposures together fully, quickly or accurately. The records were there. Assembling them into one true picture was the part that broke.
Why an assistant finds the seams first
A person reading a policy reads one policy. They open the file they were pointed at and read it through. A retrieval assistant does not work that way. It takes passages from across the whole set and builds a single answer out of several sources.
That makes it the first reader to treat your documentation as one body of text. Bodies of text that large are rarely consistent.
The conflict has a name
Researchers have a taxonomy for it. A 2024 survey of knowledge conflicts in large language models sorts them into three kinds: context-memory, inter-context and intra-memory conflict. Inter-context conflict is the enterprise case. Two retrieved documents disagree, and the model has to do something about it.
What it usually does is answer anyway. It uses the same confident tone it uses when the sources agree.
Grounding does not settle it
It is tempting to assume that pointing a tool at approved internal sources fixes this. The first preregistered evaluation of retrieval-based legal research tools says otherwise. Tools from LexisNexis and Thomson Reuters each hallucinated between 17 and 33 percent of the time, against vendor claims of eliminating hallucinations. Retrieval lowered the error rate. It did not remove it.
What the regulator asks the deployer
Article 26 of the EU AI Act sets out the duties of a deployer of a high-risk system. Use it according to the instructions. Give human oversight to people with the necessary competence, training and authority. Keep logs for at least six months. On the content itself there is one duty, in Article 26(4). Where the deployer controls the input data, it has to be relevant and sufficiently representative for the intended purpose.
That is a fit test. It asks whether the data suits the job. It does not ask who approved a document, which version it is, or whether another document in the same set says the opposite.
The page nobody owns
This is where a documentation program and a knowledge base part company. Cognatum is a knowledge base built for the second job. It does not ask you to move anything. Information stays in the departmental repositories where it already lives, and Cognatum keeps a governed layer above them with a two-way flow to the systems people already work in. Every entry carries a named approver, a date and a source.
What it does with disagreement
Where two sources contradict each other, Cognatum shows both and marks where they diverge. It does not pick a winner. Conflicts, duplicates and gaps are scored as knowledge at risk, each with a confidence score and four choices: link as related, propose retirement, request revisions, or dismiss as irrelevant. Two policies for two divisions can be a real difference rather than an error, so linking is often the right call.
When a source moves, Cognatum tells you what now depends on it. It does not rewrite anything quietly and it does not update itself. The AI finds, gathers and surfaces. A named person decides what is true, and that approval is the gate.
A test you can run this week
Pick a question your documentation answers in more than one place. A retention period. A reporting threshold. An escalation route. Ask it of the assistant your staff actually use. Then ask for the approved entry behind the answer, its approver and its date.
If the assistant gives you one clean answer and your documentation gives you two, you have found the gap. Your company’s knowledge isn’t missing. It’s unusable. Cognatum changes that.
Sources
- Compliance Documentation: Types, Benefits and Challenges (sprinto.com)
- What is Compliance Documentation (scytale.ai)
- FINRA Regulatory Notice 24-09, Generative AI and Large Language Models (finra.org)
- 15 U.S. Code 7262, Management Assessment of Internal Controls (law.cornell.edu)
- Principles for Effective Risk Data Aggregation and Risk Reporting (bis.org)
- EU AI Act Article 26, Obligations of Deployers of High-Risk AI Systems (europa.eu)
- Knowledge Conflicts for LLMs: A Survey (arxiv.org)
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arxiv.org)
- The Cognatum Knowledge Loop (cognatum.ai)
Common questions
Questions this raises.
What is compliance documentation?
It is the body of records an organization keeps to show that it meets its regulatory obligations. In practice that means policies and procedures, risk assessments, control descriptions, training records, logs and audit reports. Guides to the term describe it as the complete record of how the organization meets those obligations.
Can compliance documentation be complete and still produce a wrong answer?
Yes. Completeness is judged one document at a time, against a list of what must exist. Internal agreement across the set is not usually tested at all, so two approved documents can say different things and both still count as compliant records.
Does FINRA require firms to check that their policies agree with each other?
Regulatory Notice 24-09 reminds member firms that FINRA’s rules are technology neutral and continue to apply when generative AI is used. It says supervisory procedures should address technology governance, model risk management, data privacy and integrity, and the reliability and accuracy of the model. Those obligations sit on the system rather than on the internal consistency of the documents it reads.
Does grounding an assistant in approved internal sources solve conflicting documents?
Not on its own. The first preregistered evaluation of retrieval-based legal research tools found that products from LexisNexis and Thomson Reuters each hallucinated between 17 and 33 percent of the time. Retrieval reduced errors but did not eliminate them, and research on knowledge conflicts shows models still have to resolve sources that disagree.
What does the EU AI Act ask a deployer to do about the content an AI system reads?
Article 26 sets out deployer duties including use according to instructions, human oversight by people with the necessary competence, training and authority, and log retention for at least six months. On content, Article 26(4) asks that input data under the deployer’s control be relevant and sufficiently representative for the intended purpose. That is a fit test rather than a test of approval, version or conflict.