Chapter 11: Audit Logging and Forensics

Why Audit Logs Are Non-Negotiable

Every earlier chapter has been about preventing bad things. This chapter assumes those defenses will sometimes fail. When they do, the difference between a five-minute investigation and a five-week nightmare comes down to whether your assistant kept an honest, complete, and trustworthy record of what it did. Audit logging is the control you build before the incident so you have something to read during it.

Reconstructing What, When, and on Whose Behalf

Because an assistant mediates all of its actions through tool calls — invocations of the capabilities you granted it — the tool-call log is the record of what the assistant actually did. Think of it as an aircraft's flight recorder: nobody straps a black box to a plane hoping to use it; its entire value is realized in the minutes after something goes wrong, when investigators need an authoritative, unalterable account.

A good log answers all three dominant questions: What did the agent do? (every tool call, in order, with inputs and outputs), When did it happen? (precise, synchronized timestamps), and On whose behalf? (the principal behind the request). That third question is what makes agent logging special: a confused-deputy attack succeeds precisely when the agent acts with the operator's authority but a stranger's intent. Only a log that records both the authority used and the true requester lets you see that divergence after the fact.

Accountability, Debugging, and Forensics

Audit logs serve three overlapping audiences. Accountability answers "who is responsible," giving non-repudiation — the record is trustworthy enough that the actor cannot credibly claim "that wasn't me." Debugging is the everyday use: replaying the exact reasoning and actions that led to a surprising result. Forensics is the incident-time use: scoping impact, finding whether a prompt injection manipulated a tool call, and locating the moment behavior went off the rails. The design consequence: log for the worst-case reader — an investigator who arrives weeks later, knows nothing about the request, and cannot ask the agent to explain itself.

Detecting Slow-Burn Abuse

Not every attack is a dramatic one-shot exfiltration. An injected instruction might quietly forward a copy of every incoming email to an attacker; a compromised skill might read one extra sensitive file per day. These patterns are invisible in the moment but obvious in aggregate. A burst of failed tool calls is the signature of probing; a steady trickle of successful calls to an unusual destination is the signature of exfiltration. Fed into monitoring or a SIEM (Security Information and Event Management platform), the log becomes a detection mechanism, not just an archive.

What and How to Log

The cardinal rule is completeness: no tool call should execute without producing a record. A partial log is worse than no log because it creates false confidence exactly where consequential behavior may have occurred.

The Three Logging Surfaces

The operational surface is non-negotiable; the cognitive surface lets an investigator find the exact moment an injected instruction hijacked the agent's plan.

Worked Example — A Tool-Call Log Record

An allowlisted user asks the assistant, over Telegram, to email a meeting summary. The resulting email.send record is a single canonical, structured entry:

{
  "entry_id": "0192f3a1-7c44-7b2e-9a01-3f6d2e8b11ce",
  "timestamp": "2026-07-08T14:03:22.481Z",
  "correlation_id": "req-8f2c9d10",
  "actor": {
    "principal": "user:telegram:441234",
    "authority": "operator-delegated",
    "capability_tier": "standard"
  },
  "source": { "channel": "telegram", "ip": "203.0.113.9" },
  "reasoning_ref": "span-cog-0192f3a1-004",
  "tool_call": {
    "name": "email.send",
    "arguments": {
      "to": "alice@example.com",
      "subject": "Meeting summary — Q3 planning",
      "body_sha256": "b1946ac9…",
      "body_length": 812
    }
  },
  "outcome": { "status": "success", "provider_message_id": "CAO=..." },
  "prev_entry_hash": "9f2b7c…",
  "entry_hash": "3a7e12…"
}

Notice what is deliberately absent. The full body is not stored inline — only its length and a SHA-256 hash (body_sha256), preserving the ability to prove later that a specific body was sent without stockpiling content. The authority field ("operator-delegated") records the power used, while principal records the true requester — exactly the pairing needed to detect a confused deputy. reasoning_ref points to the cognitive record so an investigator can jump from what to why.

Correlation IDs: Linking a Message to Its Effects

A single inbound message frequently fans out into many actions. Every downstream record must be stamped with the same identifier tying it back to the origin. The analogy is a hospital patient wristband: every lab result, medication, and X-ray gets stamped with your unique number, so anyone can pull the whole visit as one coherent story. Mature frameworks refine this into a hierarchy: a Trace ID (one per inbound request), a Span ID (one per sub-operation, nested under the trace), and a Session ID (one per ongoing conversation).

Figure 11.1: One inbound message fans out into downstream tool calls, all stamped with the same correlation ID.

Redacting While Preserving Evidence

Comprehensive logging collides with a hazard: the log itself becomes a rich target. Redaction removes or masks sensitive data while retaining forensic usefulness.

Two rules keep redaction honest: redact at capture, not after storage (never let a raw secret land in tamper-evident storage even briefly), and prefer allowlisting fields over blocklisting patterns because allowlists fail safe.

Append-Only and Tamper-Evident Logs

A log an attacker can rewrite is worthless as evidence, because the first thing a competent intruder does is erase their tracks. The defenses come in two escalating tiers: append-only storage, which physically prevents modification, and tamper-evident structure, which makes any modification mathematically detectable even if it occurs.

Append-Only, WORM, and Hash Chaining

An append-only log adds new records but never edits or removes existing ones. The strongest form is WORM (Write Once, Read Many) storage. The cloud-native realization is S3 Object Lock, which can place a retention lock so that not even an account administrator can delete or alter an object until the retention period expires.

Hash chaining adds tamper evidence: each record embeds the cryptographic hash of the previous record, forming a linear chain from a genesis entry to the tail. The entry hash is computed over all fields including the previous hash:

entry_hash = SHA256(
    entry_id || timestamp || actor_info || event_type ||
    event_data_hash || previous_entry_hash
)

This is exactly the prev_entry_hash / entry_hash pair from the worked example. Any modification to a past entry changes its hash, invalidating the previous_entry_hash stored in the next record, which changes that record's hash — cascading all the way to the tail.

Figure 11.2: Each entry embeds the previous entry's hash, so altering one record breaks every hash after it.

Hash chaining is the fastest pattern to write (the gateway keeps the tail hash in memory) but the slowest to audit, since verifying integrity means walking the entire chain. Merkle trees solve this: records are grouped into batches, pairs of hashes combine recursively into internal nodes culminating in a single root hash, and proving one record belongs to the batch requires only about log(N) hashes (a Merkle proof). The final layer is external witness anchoring: publishing the tail hash or Merkle root to an external system (a notary, blockchain, or transparency log like Sigstore Rekor), turning records "from organizational records into mathematical proofs which no one controls."

Details matter: records must use a canonical form (sorted JSON keys, UTC ISO-8601, UTF-8) so identical content hashes identically; a per-record salt prevents collisions; a per-log pepper in a key-management service blocks fabrication; and digital signatures provide strong non-repudiation.

Protecting Logs From the Agent's Own Privileges

There is a threat unique to AI-assistant logging: the thing being audited may also have the ability to tamper with the audit. A successful prompt injection could instruct the agent not only to exfiltrate data but to erase the evidence. The governing principle is separation of duties: whoever can change the system cannot alter the record of those changes. Concretely: ship logs off-box one-way (a write-only path the agent cannot read or delete), use storage-enforced immutability (S3 Object Lock compliance mode), anchor externally, and confine deletion to break-glass procedures requiring separate authorization.

Figure 11.3: The agent has only a one-way, write-only path to the log; it holds no credential to read, edit, or delete stored records.

The design goal is that the audit trail's trustworthiness does not depend on the agent behaving well. The log must survive the failure of the very component it is auditing.

Retention, Monitoring, and Alerting

Collecting logs is only half the job. Retention: keep audit logs one to three years by default, because slow-burn abuse unfolds over months and incidents are discovered late. The log must be searchable — indexed by actor, event name, IP, correlation ID, and date range. Centralize collection, normalize fields, and emit OCSF-compliant JSON so records feed a SIEM directly. Monitoring and alerting close the loop: alert on denial spikes, first-ever calls to new domains, or unusual email.send rates. Finally, the logs deserve a log of their own — record every read, search, export, and permission change against the audit logs themselves. The auditors get audited.