AI Log Analyzer

Executive summary

Deploying AI is easy. Proving that it stayed within policy — across thousands of prompts, tool calls, and multi-turn agent sessions — is hard.

The AI Log Analyzer is a production-ready governance telemetry sink and operational control plane for AI quality and governance programmes. It ingests structured events from LLM gateways, agent harnesses, and MCP tool servers; evaluates drift, policy violations, and agentic risk signals; routes high-severity cases to a human oversight queue; and feeds alerts, post-market monitoring events, and auditor evidence packs.

One service. One ingest contract. One dashboard. Full traceability back to your AI System Register.

Why organisations choose the AI Log Analyzer

Unlike a generic log aggregator, the analyzer understands AI-specific semantics: agent identity, trajectory, action ontology, MCP tools, runtime state snapshots, and linkage to your AI system register.

Runtime ingestion and governance actions

Every agent turn or LLM call can be recorded via a single ingest endpoint. The analyzer returns a governance action the harness must honour:

• Log only — recorded; no policy breach; continue
• Escalate — drift or violation; pause and surface human approval
• Halt — critical violation under fail-closed policy; block irreversible action

Optional webhooks notify external runtimes on drift, MCP violations, and reassessment triggers.

Agentic risks — pre-execution governance

Aligned with agentic AI governance practice: shift from policy documents alone to machine-enforceable boundaries.

• Pre-execution authorize — refuse inadmissible tools or actions before they run
• Shadow-agent detection — unknown agents escalate or halt
• Lethal Trifecta — untrusted input + sensitive data + external state without mitigation can be blocked
• MCP allowlist — tools checked against registered agents
• Irreversible actions — high-tier agents require human approval
• Financial limits — configurable thresholds without approval
• Agent registry sync — mirrored from your AI system inventory

Recommended flow: authorize → execute (if allowed) → ingest for audit.

Drift, anomaly, and content detection

• Configurable threshold rules (drift, bias, out-of-range)
• Content rules for jailbreak, prompt injection, oversight evasion, and memory-poisoning patterns
• Per-system baselines: approved ontology nodes, MCP tools, prohibited topics
• Norm divergence against QMS behavioural baselines
• Multi-turn orchestration drift (“boiling frog”) heuristics
• bIOC signals across trust, propagation, context, capability, and authority axes
• Prompt-version drift when system-prompt hashes diverge from the registry

Human oversight and automation-bias metrics

• Interactive oversight queue (pending → approved / rejected / escalated)
• Review decisions persisted with justification — not fire-and-forget
• Oversight metrics: override reject rate, review latency, rubber-stamp warnings
• Feeds governance dashboards and QMS KPI views

Prompt-level monitoring

• Turn capture with prompts, responses, hashes, model id, and guardrail verdict
• Append-only event history with export for audit windows you configure
• Content-rule and baseline profile management
• Prompt versioning with drift on hash mismatch
• KPIs for evasion rate, top trajectories, and norm divergence

QMS, post-market, and evidence integration

When ingest includes AI system identifiers:

• Creates structured alerts on drift with ontology, trajectory, and MCP context
• Creates post-market monitoring events (EU AI Act Articles 72/73 style) without replacing manual monitoring
• Optional auto-spawn of change and nonconformity workflows on critical evasion

Evidence packs return a structured four-layer bundle:

1. Decision record — rules triggered, governance action, review status
2. Input / context — prompt/response (with retention and redaction), ontology, trajectory, state snapshot
3. Governance evidence — MCP policy, bIOCs, norm divergence, agentic governance detail
4. Integrity links — hashes and cross-references to watermark and QMS records

Control-mapping export supports alignment with prEN 18286 / 18282 / 18229, the EU AI Act, OWASP Agentic categories, agentic-risk frameworks, and Article 50 watermark controls.

Watermark and transparency detection

Built-in multi-modal watermark detection (image, text, PDF with OCR) for diagnostic checks — the same class of capability that supports our Watermark (Art. 50) Management System. Generation-time watermark embedding remains an upstream producer responsibility.

How it fits in the architecture

LLM Gateway / Agent Harness / MCP Server
           │
           ▼  authorize (pre-execution)
           ▼  ingest (audit)
    ┌──────────────────┐
    │  AI Log Analyzer │◄── agent-registry sync (inventory)
    │  Rules + store   │
    └────────┬─────────┘
             │
   ┌─────────┼──────────────┐
   ▼         ▼              ▼
 QMS      Webhooks     Evidence packs
 alerts   (halt /      & control mapping
 + PMS     escalate)
 events

The analyzer is intentionally decoupled from application internals — integrate via HTTP, environment configuration, and shared AI system identifiers.

Standards and frameworks supported

Who it is for

• AI platform / MLOps — one ingest contract for models and agents
• AI governance — oversight queue, metrics, evidence packs for audit
• Security — MCP policy, bIOCs, pre-execution halt, content rules
• Compliance — control-mapping export; linkage to the AI System Register
• Post-market monitoring — automated monitoring events on drift

Transparent limitations

The analyzer is a governance and diagnostic service, not a replacement for:

• Full enterprise SIEM or petabyte-scale log warehousing (retention is sized to your audit window)
• Neural-classifier guardrail models as the primary filter (rules and baselines; optional LLM-as-judge is advanced)
• Generation-time watermark embedding (detection only; marking is upstream)
• Built-in SSO UI (place an enterprise reverse proxy or API gateway in front where required)

These boundaries keep procurement and audit conversations accurate.

Getting started

1. Deploy the analyzer in your environment
2. Connect QMS credentials and optional fail-closed governance
3. Register agents in inventory; sync the registry
4. Point gateway or harness at authorize, then ingest
5. Open the oversight queue; tune rules to risk appetite
6. Export an evidence pack from the first drift alert to validate the audit trail

Pair with watermark governance

Organisations pursuing full EU AI Act transparency coverage typically deploy:

• AI Log Analyzer — runtime agent and LLM governance (this solution)
• Watermark (Art. 50) Management System — marking and detection evidence for generative content

Both share the AI System Inventory spine and integrity conventions — one QMS, one audit story.

Ready to operationalise runtime AI governance?

Contact us to discuss how the AI Log Analyzer can sit in front of your gateways and agent harnesses, and how it links to your existing risk, monitoring, and QMS stack.