Evidence-Based AI Audit

The Institute already set the direction in AI Audit the Reset: the EU AI Act pushes audit back toward verified artefacts and away from a conversation about a policy pack. This article takes that reset into the engine room. If the system is software, the evidence is in the software. An auditor who cannot read that software is taking testimony.

ISO 19011 still allows interviews as one method among others. It does not make the interview the audit. ISAE-style assurance, SOC examinations, ISO/IEC 27001 and ISO/IEC 42001 all turn on sufficient appropriate evidence. For an AI system that evidence is logs, change records, test outputs, access trails, model cards that match the running image, and the code and prompts that image was built from.

The companion article Looking Backward describes why that code is mostly old, patched, and now mixed with generated functions. The September 2026 OpenAI and Anthropic disclosures, in The Role and Need of Controls, show what happens when containment and detection are assumed rather than sampled. An audit that never opens those layers cannot speak to either problem.

Why the Interview-and-Checklist Model Fails Here

A checklist asks whether a control exists. An interview asks whether someone believes it operates. Neither shows that the control ran on the system under review, at the time claimed, with the effect claimed.

Three failure modes are now common enough to name:

• The control ran and nobody kept the proof. A scan happened. The output was not retained, or was retained in a personal folder that is not the record.
• The control ran, the file exists, and it cannot be tied to a person, a system or a time. An access review marked complete, with no reviewer, no source list, no diff from the last review.
• The control was never there. The policy describes human oversight. The running agent has a tool permission that bypasses it. The interview still produces a yes.

From outside, those three look the same: a signed statement. An assessor cannot give credit for what cannot be seen. That is not a new doctrine. It is the original one. What changed is that AI systems make the gap obvious. Prompts live in repositories without change control. Weights are supplied by a vendor the organisation does not inspect. Agents write to sites the test plan did not list. A walkthrough of the architecture slide does not detect any of that.

ISACA's 2026 note on auditing agentic workflows made the same point in professional language: conventional assumptions about documented logic, sampled output, bounded action and change control break once the system can choose a path. The extra procedures it listed - versioned system prompts, boundary tests, third-party review of logs - only work if the auditor can tell a real artefact from a generated narrative about an artefact.

What Evidence Actually Looks Like

For a high-risk or otherwise material AI use, the file an auditor should be able to reconstruct is boring and specific.

• The running image or model identifier, not a marketing name.
• The prompt, tool list and permissions in the same version-control history as the application code.
• The data sources actually connected in production, matched to the RoPA or processing record if personal data is in play.
• Test results for the version that shipped, including failed tests and the decision to ship anyway.
• Logs that show what the system did, not only what the dashboard summarised, with retention and integrity that survive an argument.
• Patch and vulnerability status for the runtime and its libraries, including the deferred items and their owners.
• Incident tickets that start from a detection, not from a journalist.
• Approvals that name a person and a time, tied to a change hash, not to a meeting title.

Article 11 and Annex IV of the EU AI Act require technical documentation that a notified body can use without sitting next to the development team. Article 12 requires logs that support traceability. Those clauses are not a request for a better interview guide. They are a request for artefacts that still make sense when the people who built the system have left.

ISO/IEC 42001 asks for documented information about the AI management system. Documented information that cannot be traced to a system, a person and a time is not evidence that the system is operating. It is evidence that a document exists.

The Auditor Has to Be Able to Gather It

This is the staffing problem the reset article named as a knowledge gap and that practice has not closed.

An auditor of AI systems needs enough software experience to:

• open a repository and see whether the prompt in production matches the prompt in the file;
• read a CI log and tell a passed pipeline from a skipped test;
• ask for the image digest and check it against the registry;
• sample library versions and match them to an advisory;
• read enough of a function, a config file or a workflow YAML to see that a "human approval" step is a comment;
• know when a generated explanation of old code is plausible and when it is decoration.

That is not a demand that every auditor be a principal engineer. It is a demand that the engagement has someone who has shipped or maintained software, who has seen a bad patch, and who will not treat a dashboard screenshot as a log. Where that person is not on the team, the work is a confirmation exercise. Confirmation is not audit.

Research on AI auditability has made the same observation from the other side: competent AI audits are team efforts. They need engineers and domain people as well as audit method. A single generalist with a clause list cannot cover model behaviour, data lineage, access control and change management. Pretending otherwise is how interviews took over.

What the Auditor Does Instead of the Interview

Interviews still have a use. They point at where the artefacts should be. They do not replace the artefacts.

A workable sequence looks like this:

• Start from the running system, not from the policy library. What is in production, who owns it, what it can call.
• Pull the versioned prompt, the tool permissions, the model identifier and the last change that touched them.
• Sample logs for a defined window. Look for tool calls, outbound network, human-approval events, and silence where an event should have been.
• Match a declared patch or test control to a specific build. If the control cannot be found in that build, it did not operate on that build.
• Trace one incident or one near-miss from detection to decision. If that path is a slide, the incident control is not operating.
• Only then sit with management to explain gaps. The meeting is for residual risk and correction, not for gathering the primary evidence.

That sequence is slower than a two-day site visit with a checklist. It is also the only sequence that would have asked, of a test environment, whether agents could leave it; or of a January session, whether the review set included it. Those questions are not hostile. They are the job.

Independence Is Not a Substitute for Competence

After the September 2026 incidents, both Anthropic and OpenAI said they would give independent evaluators employee-like access. Access without the skill to use it produces a longer interview. Competence without access produces a guess. The control is both: people who can read the system, with a right to see the system that is not filtered by the team being reviewed, and a duty to keep working papers that another competent person can re-perform.

That is also why generated audit files are a hazard. A model can draft a clause-by-clause pack that looks complete. If the auditor cannot check the pack against the repository and the logs, the pack is another policy document. The reset article's point stands: the burden of proof is on the organisation that claims the control. The auditor's burden is to be able to test that claim in the place the control is supposed to live.

Where This Belongs

This is an audit-method article and a control-objectives article. Monitor, evaluate and assess only works if the assessor can see the same objects that build, run and patch. Organisations that want a defensible AI file should staff audits accordingly: software people in the team, evidence first, interviews last, checklists as a planning aid and never as the file.

This article expands the Institute's existing position on evidence-based AI audit. It is not a standard and it is not a finding on any named engagement.