System of record
Runbooks are tested. Answers are not.
Runbooks are the knowledge base nobody calls one. Their authority comes from an owner, a field test and a version number. An assistant answering from them delivers the steps and none of the three.
Cognatum Team · Oct 30, 2026 · 6 min read
A runbook is the document a company trusts most and reviews least. It tells somebody what to do at two in the morning, on a system they may not own, while an outage runs. It is also a knowledge base. Nobody calls it one, and nobody governs it like one.
Cognatum governs the entry
source · version · approver · permissions
What a runbook actually is
Wikipedia defines a runbook as a compilation of routine procedures and operations that the system administrator or operator carries out. It holds the steps to begin, stop, supervise and debug a system. Splunk is blunter about who it is for. A runbook lets people with limited domain expertise do the task anyway. Splunk also calls the operational runbook one of the most important knowledge resources in enterprise IT.
PagerDuty says what that means in practice. Runbooks turn knowledge that would otherwise stay in experts' heads into a shared resource. That is a knowledge base, described by an incident tooling vendor, on a page about server patching.
Read for use, not for understanding
One thing sets a runbook apart from a policy or a work instruction. A runbook is executed. The reader does not weigh it, summarise it or cite it. They do what it says, to production, now. A wrong step does not produce a wrong opinion. It restarts the wrong node, fails over the wrong region or drops the wrong table.
Where a runbook gets its authority
Not from being right. Splunk is honest about this in a way most vendors are not. The solution in a runbook may not cover every possible scenario or be the most technically accurate, and that is not the goal. The goal is one agreed step for a problem the team has seen before.
Authority comes from three other things. Somebody owns it. Somebody tested it. It has a version.
An owner
The structure PagerDuty recommends starts with a service overview naming the primary contacts, and ends with the service level agreement the service is held to. The first field is a person. Tom Limoncelli, a former Google systems administrator, set out those seven sections, and PagerDuty publishes them as the template.
A test
PagerDuty is direct about the rest. A runbook is not something you set and forget. Field test the documented process and update it as needed, because systems and applications change. The step is credible because somebody ran it on the real thing.
A version
Microsoft treats a runbook as a versioned software artifact, because in Azure Automation it is one. Runbooks are written in named runtime versions. They follow the support lifecycle of the language they are written in. Microsoft tells customers to update the outdated ones. Some runbook types can be digitally signed, and Microsoft notes exactly which ones cannot.
That is the state of the art. An owner, a test and a signature on an operational procedure.
Then somebody asks the assistant instead
Google's own site reliability engineering book sets the paging response for a user facing service at about five minutes. It then describes what that does to a person. Stress hormones impair cognitive function. The deliberate, careful way of thinking gives way to immediate, unconsidered action. The book gives an example. The same alert fires for the fourth time this week, the first three were someone else's infrastructure, and the obvious move is to assume this one is too.
Opening the runbook takes a minute. Asking an assistant takes two seconds, and it answers in the shape of an instruction. So the assistant is most attractive at the exact moment the reader has the least capacity to check it.
The law already named this
The EU AI Act names the risk directly. Article 14 requires that people overseeing a high risk system stay aware of the possible tendency to over rely on its output, which it calls automation bias, and it singles out systems used to provide information or recommendations for human decisions. The European Commission's own page for Article 14 carries a notice that the article has been amended by the Digital Omnibus on AI and that the text shown is not yet updated.
The Act governs the person reading the output. It says nothing about the version of the document the output came from.
Retrieval is not a correctness guarantee
Researchers ran the first preregistered evaluation of commercial legal research tools built on retrieval. The tools from LexisNexis and Thomson Reuters hallucinated between seventeen and thirty three per cent of the time, against vendor claims of eliminating hallucinations altogether. Pointing a model at the right library does not make the answer right.
What the answer does not carry
Ask an assistant how to fail over a payments database and you may well get a good answer. You will not get the four things the runbook would have carried.
Which runbook, and which version
An answer assembled from three runbooks, a postmortem and a two year old chat thread looks exactly like an answer from the current procedure. There is no revision number on it. There is no way to ask which one it used.
When it was last run
The field test is what made the step trustworthy. An answer inherits none of it. Nothing in the answer says the step was ever executed against the system as it stands today.
Who approved it
A runbook has an owner and, in Azure Automation, possibly a signature. An answer has an author nobody can name. It was assembled, not written.
Whether something contradicts it
Two runbooks for the same service often disagree, because one was written before a migration and the other after. A model asked a question will usually pick one and present it with the same confidence it gives everything else.
The regulated version of the same problem
Operational procedures are not ungoverned everywhere. FINRA Rule 4370 requires a US broker dealer to keep a written business continuity plan. The plan must be updated on any material change to the firm's operations, structure, business or location. It must be reviewed every year. A member of senior management, who must be a registered principal, approves it and runs that review. It must be produced promptly when FINRA asks.
Read that as a list of fields and it looks familiar. A named approver. A review date. A currency obligation tied to change in the underlying thing. An availability requirement on request.
None of it reaches the answer
The firm can show which version of the plan was in force in March. It cannot show which version the assistant read, or whether the person on the phone acted on the plan at all.
What would have to change
Your company's knowledge isn't missing. It's unusable. Cognatum changes that. The Cognatum Knowledge Loop runs eight steps across four phases, and AI runs seven of them. Approve is the one it does not. AI does the finding, gathering and surfacing, then hands the item to a named person who decides what is true and can rework it before signing.
Three things that would help
Where two sources disagree, both are surfaced with the disagreement named, rather than one being chosen quietly. When a source document changes, anything depending on it is flagged for review, which is a notification and not a silent rewrite. And every answer records where the knowledge stood when the answer was given.
The runbooks do not have to move. They stay in the repositories the teams already keep them in, and the governed layer sits above them.
A runbook is tested before anyone trusts it. That is the whole reason it works. An answer that quietly replaces it should have to clear the same bar.
Sources
- Runbook (en.wikipedia.org)
- Introduction to Runbooks (splunk.com)
- What is a Runbook? (pagerduty.com)
- Azure Automation runbook types (learn.microsoft.com)
- Site Reliability Engineering, Chapter 11: Being On-Call (sre.google)
- EU AI Act Article 14: Human oversight (ai-act-service-desk.ec.europa.eu)
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arxiv.org)
- FINRA Rule 4370: Business Continuity Plans and Emergency Contact Information (finra.org)
- The Cognatum Knowledge Loop (cognatum.ai)
Common questions
Questions this raises.
What is a runbook?
A runbook is a documented set of step by step procedures that an operator or system administrator follows to run, monitor, troubleshoot or recover a system. Typical contents include deployment steps, common fixes, alert responses and disaster recovery procedures. Its purpose is to let someone without deep expertise in that system carry out the task correctly.
What is the difference between a runbook and a playbook?
A runbook covers one specific task, in sequence, for one operator. A playbook covers the wider response to a larger event and usually works as a decision tree, often containing several runbooks inside it. Splunk and PagerDuty both draw the line that way.
Does runbook automation solve the problem of stale procedures?
No. Automation changes who executes the step, not whether the step is still right. Azure Automation runbooks are code with runtime versions and a support lifecycle, and Microsoft still tells customers to update outdated ones. An automated runbook built on a wrong assumption simply applies that assumption faster.
Should an on call engineer use an AI assistant instead of the runbook?
The honest answer is that they already do, because it is faster. The problem is not the asking, it is that the answer arrives without the things that made the runbook trustworthy. No revision number, no last tested date, no named owner and no indication that another procedure says something different.
Does using Cognatum make us compliant with an AI governance framework?
No. No software confers compliance with the EU AI Act, the NIST AI Risk Management Framework or any similar framework. What governed knowledge does is supply the evidence a reviewer asks for: who approved an entry, when, from which source, and what the state of that source was at the time the answer was given.