AI governance
AI red teaming never tests the source
AI red teaming simulates an attacker and tests how the model behaves under pressure. The likelier enterprise failure has no attacker in it at all: a fair question, a confident answer, and a source nobody approved.
Cognatum Team · Oct 19, 2026 · 6 min read
Your security team has a plan for the attacker. Prompt injection. Jailbreaks. Data poisoning. Model extraction. Each one has a test, a finding and a fix. The exercise has a name now, and a budget.
Cognatum governs the entry
source · version · approver · permissions
The likelier failure in a regulated firm has no attacker in it at all. An employee asks a fair question. The assistant answers in full, and sounds sure. The answer is wrong, because the document behind it was withdrawn last year and nothing told the index.
What red teaming actually tests
The definitions agree with one another. IBM Research calls red teaming a way of interactively testing AI models to protect against harmful behavior. The method is to provoke the model into saying things it was trained not to say.
Palo Alto Networks sets out the same scope in a side by side table. The attack surface is AI models, APIs, training data and outputs. The testing goal is to identify unintended or unsafe AI behavior. The techniques are adversarial prompts, data poisoning, prompt injection and model extraction.
Four areas, and a fifth nobody lists
The OWASP GenAI Red Teaming Guide divides the work into four areas. Model evaluation. Implementation testing. Infrastructure assessment. Runtime behavior analysis.
Read that list once more. Four areas, and not one of them is the company content the system reads before it answers.
The ontology assumes someone is trying
Microsoft has red teamed more than 100 generative AI products and published what it learned. Its ontology models the parts of a cyberattack: the actor, the tactics and techniques, the system weakness, and the downstream impact.
One of the eight lessons is blunt. AI red teaming is not safety benchmarking. The team's own summary of the work puts the job plainly. They break the technology so that others can build it back stronger.
That is a sound discipline. It is also a discipline organized around intent. Every box in the ontology describes someone doing something to the system. None of them describes a procedure going quietly out of date.
Regulators ask for the same test
Article 55 of the EU AI Act sets out what providers of general purpose models with systemic risk must do. They have to perform model evaluation using standardised protocols, including conducting and documenting adversarial testing of the model, in order to identify and mitigate systemic risks.
Look at the object of that sentence. The model. The duty is real and worth meeting. It is also not a duty about the approved content your own people wrote.
Grounding is not a pass mark
Many teams assume retrieval closes the gap. If the assistant quotes a company document, surely the answer holds.
The best public evidence says otherwise. A preregistered study of AI legal research tools found that Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinate between 17% and 33% of the time, against vendor claims of eliminating hallucinations. These are retrieval based products working over curated legal libraries.
That study measures how the model behaves. It does not ask whether the document it pulled was the current one. Inside a company library, the second question is the one that bites.
The test nobody runs
Here is an exercise your red team is not running. Take a policy that was revised this year. Ask the assistant the question that policy answers. Then ask three more things.
Which entry did that answer come from. Who approved that entry, and on what date. Has the source changed since.
Most stacks cannot answer any of the three. There is no finding to write up, because there is no field to fail.
Four failures with no adversary
The superseded source
A procedure is revised. The old file stays on a shared drive, and the index keeps the text it took from it. The assistant answers from last year's rule, with no sign that anything moved.
The duplicate
Someone saved a copy so they would not have to hunt for it again. Now six versions exist. Retrieval picks the nearest match by similarity, and the nearest match is not the same thing as the right one.
The contradiction
Two approved documents say different things, because two divisions wrote them for different purposes. A model settles that by choosing one. A reviewer would have flagged it instead.
The gap
No approved answer exists at all. A helpful system writes something that reads well. Nobody learns that the library never covered the question.
You cannot test a field that does not exist
An adversarial test needs a property it can violate. Red teaming works because “do not output instructions for a weapon” is a rule you can state, probe and score.
“This came from an approved, current, uncontested entry” is also a rule. It is just not one most stacks can express. A retrieved passage carries text and a relevance score. It carries no approver, no effective date and no status.
The OWASP project describes its guide as a risk based method running from model level flaws to system integration pitfalls. Breadth is not the problem here. The missing field is.
Governing the knowledge, not just the model
Cognatum is a knowledge base, and it runs on its own framework, the Cognatum Knowledge Loop. Eight steps in four phases. AI runs seven of them. A person holds one, and it is the one that decides what goes live.
Step 6 is Approve. AI assembles the entry, its sources, its history and what changed since the last approved version. A named person signs off. The decision is recorded with the approver, the date and the source, and every live entry carries that record from then on.
Notify, not rewrite
Step 3 is Clean. Entries are compared against their sources and against each other. Change a source and every entry that depends on it is flagged and routed for re-check. Cognatum tells you. It does not quietly rewrite the answer, and it is not self-updating. The last approved version stays live until a person signs off on a new one.
Where two approved entries genuinely disagree, both are held and routed to a named person. Neither is served as settled. Where nothing approved covers a question, that is recorded as a gap rather than filled with something plausible.
What it is, and what it is not
Cognatum supplies evidence toward your own obligations under frameworks such as the EU AI Act and the NIST AI Risk Management Framework. It does not confer compliance with any of them, and no software does.
None of this replaces red teaming. Your assistant still needs testing against someone who means it harm. It also needs a source that holds up when nobody is attacking at all.
Where to start
Pull your last red team report. Count the findings that concern the content the system answered from. If that count is zero, the exercise came back clean and the exposure is still open.
Your company's knowledge isn't missing. It's unusable. Cognatum changes that.
Sources
- What Is AI Red Teaming? Why You Need It and How to Implement (paloaltonetworks.com)
- What is red teaming for generative AI? (research.ibm.com)
- GenAI Red Teaming Guide (genai.owasp.org)
- Announcing the OWASP Gen AI Red Teaming Guide (genai.owasp.org)
- Lessons From Red Teaming 100 Generative AI Products (arxiv.org)
- 3 takeaways from red teaming 100 generative AI products (microsoft.com)
- AI Act Article 55: Providers of general-purpose AI models with systemic risk (europa.eu)
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arxiv.org)
- The Cognatum Knowledge Loop (cognatum.ai)
Common questions
Questions this raises.
What is AI red teaming?
AI red teaming is structured adversarial testing of an AI system. Teams simulate attacks such as prompt injection, jailbreaks, data poisoning and model extraction, then record how the system behaves under each one. The aim is to find unsafe or unintended behavior before an attacker does.
Does AI red teaming check whether an answer is factually correct?
Not in the way most people assume. Red teaming probes the model and the application for weaknesses an adversary could exploit. It does not audit the company documents the system retrieves, so it will not tell you that an answer came from a procedure withdrawn last year.
Does retrieval-augmented generation remove the need for this?
No. A preregistered study of retrieval-based legal research tools measured hallucination rates between 17% and 33%, despite vendor claims that retrieval had eliminated the problem. Retrieval lowers the error rate. It does not establish that the retrieved document was approved, current or uncontested.
What would a knowledge-side test look like?
Ask the assistant a question answered by a policy you revised this year. Then ask which entry the answer came from, who approved that entry, on what date, and whether the source has changed since. A system that cannot answer those four questions has no provenance for anyone to test.
Does red teaming satisfy the EU AI Act?
Article 55 requires providers of general purpose AI models with systemic risk to conduct and document adversarial testing of the model. That obligation sits with model providers and concerns the model itself. It says nothing about whether the content a deploying organization feeds its assistant was approved by a named person.