Cognatum

Financial services

Compliance monitoring can't sample an answer

Compliance monitoring needs a population to sample and a benchmark to test against. An assistant answering from your own documents leaves no record of what it read, and no version of the procedure it read it from.

Cognatum Team · Oct 29, 2026 · 6 min read

Compliance monitoring asks one question. Did what happened match what was meant to happen? To answer it you need two things. You need a set of events you can sample. You need a benchmark to test them against. An AI assistant that answers staff questions from company files gives you neither.

Served to one approved entry
AI assistants & agents
Proposal tools
Internal search & chat
Customer portals
Compliance & audit

Cognatum governs the entry

source · version · approver · permissions

What monitoring actually tests

IBM calls compliance monitoring the act of checking, all the time, whether a firm sticks to the rules that apply to it. Those rules include its own internal policies. Skillcast calls it day to day review of how a firm performs against its key policies.

One word in both of those carries the weight. Against. Monitoring measures what people did against something written down.

Named in rule

In US broker dealers that written thing is named in rule. FINRA Rule 3110 tells each firm to keep a supervisory system and written procedures. Rule 3110(c) adds a yearly review of the businesses the firm is in. Each inspection report has to test and verify those procedures. COSO puts out separate guidance on monitoring internal controls.

Monitoring samples, by design

Sampling is not a shortcut here. It is the sanctioned method.

Rule 3110.05 says a firm need not review each transaction in detail if it runs a reasonably designed risk based review system. Rule 3110.06 extends the same risk based thinking to the review of correspondence. The firm decides what to look at, and looks at that.

A sample needs a population. That is the first place this breaks.

An answer is not in the population

Monitoring populations are concrete. Transactions. Incoming and outgoing correspondence. Internal communications. Customer complaints. Transfers of funds. Changes to customer account information. Branch inspections. Each one is a record that exists whether or not anyone looks at it.

An assistant's answer is none of those. It is not a transaction. It is not a message to a customer. It sits upstream of both.

Someone asks what the current policy is. They read the answer. Then they act, and that act is what gets sampled. The answer that shaped it was never an item.

So a monitoring universe can be complete and still not hold this. You cannot sample a population you cannot list.

The four fields a review record carries

Rule 3110.07 is specific about what evidence of review has to show. It must clearly identify the reviewer, the communication that was reviewed, the date of review, and the actions taken. The same note adds a line worth keeping. Merely opening a communication is not sufficient review.

Those are four fields. An AI answer has none of them. No reviewer. No reviewed item. No date of review. No action taken. Nobody opened it at all.

The benchmark is versioned. The answer is not.

Rule 3110(b)(7) tells a firm to promptly amend its written procedures when law, regulation or FINRA rules change. The same paragraph makes the firm responsible for promptly passing those procedures and amendments to the people they apply to.

Rule 3110.11 goes further. A firm that distributes procedures by electronic media has to retain current and prior versions, in line with the record retention rules in SEA Rule 17a-4(e)(7).

One side of the test is dated

Read that again. The rulebook already asks for point in time provenance on the benchmark. A firm can show which version of a procedure was in force in March.

What it cannot show is which version the assistant read in March. One side of the test is dated and kept. The other side is neither.

Grounding narrows the error. It does not record it.

Researchers ran the first preregistered study of retrieval based legal research tools. Products from LexisNexis and Thomson Reuters made things up between 17% and 33% of the time, against vendor claims of eliminating hallucinations.

Retrieval helps. It still leaves behind no record of what was retrieved, which version it was, or whether anyone had approved it.

What the AI rulebooks ask for

FINRA Notice 24-09 names this use case in its own words. Gen AI tools may allow an associated person to easily locate and query a firm's policies and procedures or forms.

The notice then says what a firm using such tools in its supervisory system should cover. Technology governance. Model risk management. Data privacy and integrity. Reliability and accuracy of the AI model.

Every item there is a fact about the model. None is a fact about the document the model answered from.

The AI Act logs the system, not the source

The EU AI Act goes a step further and still stops short. Article 12 requires high risk systems to allow automatic recording of events over the lifetime of the system. Those logs have to support post market monitoring.

For one narrow category, remote biometric identification, the article sets minimum log fields. One of them is the reference database the input data was checked against. Note that this article has been amended by the Digital Omnibus on AI, and the published page says the displayed wording is not yet updated.

Precedent that does not reach you

So there is precedent for logging which source was consulted. It applies to biometric matching. It does not reach the assistant answering a question about your own suitability procedure. NIST's AI Risk Management Framework sorts practice into govern, map, measure and manage. Measurement needs something measurable.

What a monitorable answer would carry

Four things have to exist before any test can run on this channel.

A population you can list, so answers are items rather than scrollback. A named approver and a date on the source, so the benchmark can be identified. The version in force at the time of the answer, not the version in force today. A way to list the answers that are now wrong because their source changed.

Where Cognatum sits

The Cognatum Knowledge Loop runs eight steps across four phases. AI runs seven of them. Approve is the one it does not. A named person signs off, and the entry then carries an approver, a date and a source.

Cognatum tells you when a source changes. It does not rewrite the answer, and it is never always current. A person still decides.

Where two documents disagree, it shows both and says where the disagreement is rather than picking a winner. Its knowledge integrity view scores entries as healthy or at risk. At risk covers duplicates, conflicting information and gaps, not only items overdue for review.

That is what gives a monitoring programme something to sample.

Monitoring is not broken. It is scoped.

It tests the records a business keeps against the procedures a business writes. Both halves are governed.

The questions your people now ask an assistant produce neither a record nor a tested answer. That channel sits outside the plan, and it is the one that is growing.

Common questions

Questions this raises.

What is compliance monitoring?

Compliance monitoring is the ongoing process of checking whether a firm follows the rules that apply to it, including its own internal policies. In practice it means sampling records of what happened and testing them against written procedures. FINRA Rule 3110 sets out the supervisory system, written procedures and annual business reviews that US broker dealers run it against.

Does compliance monitoring cover AI assistants?

Usually not, and not because anyone decided to exclude them. Monitoring samples populations of records: transactions, correspondence, internal communications, complaints, account changes. An assistant's answer is none of those. It sits upstream of the activity that gets sampled, and it leaves no item behind to sample.

Do FINRA rules apply to generative AI tools?

Yes. FINRA says its rules are technology neutral and continue to apply when firms use generative AI. Regulatory Notice 24-09 names the case of an associated person querying a firm's policies and procedures, and says a firm using such tools in its supervisory system should address technology governance, model risk management, data privacy and integrity, and the reliability and accuracy of the model. Nothing on that list covers whether the queried document was current or approved.

What would make an AI answer monitorable?

Four things. A population of answers you can list rather than scrollback. A named approver and a date on the source. The version that was in force at the moment the answer was given. And a way to find the answers that are now wrong because their source has changed.

Does using Cognatum make us compliant with the EU AI Act or NIST?

No. No software confers compliance with any framework. Cognatum aligns with the way these frameworks are written and supplies the evidence a reviewer asks for: a named approver, a date and a source on every entry, and a record of where the knowledge stood when an answer was given. The compliance judgement stays with your firm.

Knowledge governed. Intelligence everywhere.

See it on your own content, in your own environment.