AI agent audit trails: what to log, and how to prove it

A log is evidence you wrote about yourself

Almost every agent platform can show you what its agent did. It ran twelve steps, called four tools, sent three emails. That record is generated by the same system that took the actions, stored where that system decides, and formatted how that system chooses. It is a self-report.

Self-reports are fine until someone doubts one. The moment a customer, an auditor or a regulator asks whether an action really happened, a log answers with the same confidence whether it is right or wrong. If the system misread a provider response and recorded a send that never left, the log says the send happened, and goes on saying it forever.

This is not a hypothetical failure mode. It is the ordinary one. A run that succeeds at sending the wrong email is a success everywhere in the stack. A step that fails and is marked skipped can leave a workflow reporting green. The record is not lying — it is faithfully reporting a belief that was wrong when it was formed.

So the useful question is not what an agent logs. It is what an agent can still prove when somebody who does not trust it asks.

An agent trail is not a model trail

Most AI observability was built for models. It captures prompts, completions, tokens, latency and versions — everything needed to debug a generation. That is genuinely useful and it is a different job.

A model produces text. An agent produces consequences. It sends the email, files the ticket, moves the money. So its trail has to carry things a model trail never needed: which external system was touched, under whose authority, with what argument, and what that system said back.

The distinction matters commercially, not just architecturally. Nobody has ever been asked to prove which prompt produced a paragraph. People are routinely asked to prove who authorised a payment.

The four questions

Strip away the vocabulary and anyone auditing an agent is asking four things in order. Each one is a separate mechanism, and a system can pass the first and fail the rest.

What did it do? Every side effect has to cross one place that records it. If actions can reach the outside world through more than one path, the ledger is a sample rather than a record, and you will not know which.

Can I find it? A row saying an email was sent is worth much less than a row carrying the provider’s own message id. The identifier turns your evidence into a pointer at their evidence — a thing in the customer’s own system, which they can open without believing anything you say.

Is it still true? This is the one almost nobody does. Going back to the provider afterwards and asking whether the artifact is still there, and still matches, is the difference between a ledger that is self-reported and one that is audited. It catches the confidently-wrong record, automatically, rather than one incident at a time.

Did I agree? An approval, a refusal, or nothing. A record that cannot distinguish "a human said yes" from "nobody was asked" is missing the fact that governance is actually about.

What sealing proves, and what it does not

Tamper-evidence is usually where audit-trail marketing arrives, and it is usually where the claim quietly outruns the mechanism. Hashing a run of records so that editing one afterwards is detectable is real and worth doing. It is also narrow.

A seal proves the ledger has not changed since it was sealed. It says nothing about whether a record was true when it was written — that is a different mechanism entirely. And it defends against accident and casual editing, not against anyone who can rewrite the seals as well, which is anyone with the same database access.

That last limit is closed by one thing only: publishing the head of the chain somewhere it cannot be rewritten, so a copy exists outside the system that produced it. Until that happens, tamper-evidence means the ledger agrees with itself.

Where Claritty stands on this today: the ledger is sealed hourly and the seals verify, but the head hash has never left our control. So it detects accident and casual editing, and nothing about someone who can rewrite the seals too. That changes when the head is published, and not before.

Coverage is the number nobody publishes

Once a system can re-check its own records, a new number appears and it is uncomfortable: how much of the ledger can be checked at all.

Most integrations have no way to read one artifact back by its id. You can send a message through them; you cannot ask afterwards whether that exact message is still there. So a re-verification pass returns three answers, not two — confirmed, not-found, and no way to check — and the third is often the largest.

The temptation is to fold the unfixable third into a rate and publish something flattering. A confirmation rate over the receipts that happened to be checkable is not a statement about the ledger, and reporting it alone is lying with arithmetic. The honest form states coverage beside the rate, always.

One more distinction that has to survive contact with a dashboard: not-found is not proof a receipt was false. A user deleting the email and a record that was never true produce exactly the same answer. It is a prompt to look, never a verdict.

Claritty re-verifies daily, on a sample, against the provider. It reaches three of the four write verbs the automation catalog actually uses, and cannot-check dominates early for the reason above. Both numbers are shown together, never folded.

How this fails in practice

The failure modes are not exotic, and they are worth naming concretely because every one of them produces a green run. We keep a register of them; these are from it.

A step fails and is recorded as skipped, so the workflow reports success. Nothing errored. The run summary is accurate about a state that was wrong.

A filter runs against a field the rows do not have. Every row scores a confident no, the filter keeps nothing, and the step downstream summarises the unfiltered set instead — a plausible report about the wrong things. Seen in production against a 900-row directory read.

A scheduled run is throttled away by an infrastructure limit and retried twice, then discarded. No error, no row, no alarm. The automation simply did not happen, and the only evidence is a metric in a different system.

A write succeeds and the provider hands back an identifier under a name the capture layer does not recognise, so the ledger records that something happened with no link to it. The row is true and useless.

A comparative check that would have caught an unusual run does not run, because the artifacts it needed died with the response on a long execution. The check exists, is correct, and never fires for the runs that most need it.

None of these is a bug in the sense of something crashing. Each is a system reporting a belief that was wrong when it formed. That is the specific thing an audit trail has to survive, and it is why re-verification against the provider matters more than any amount of internal logging.

What an auditor will accept, roughly in order

Not all evidence is equal, and the ranking is fairly stable across the people who ask for it.

Weakest: a system’s own narrative. "The agent completed 12 tasks." Unfalsifiable, and therefore worth little regardless of whether it is true.

Better: a structured record with a timestamp, an actor, and an action. Now there is something specific to disagree with. Still self-reported.

Better again: that record carrying the other system’s own identifier. The evidence now points somewhere outside you, and the person checking can open it without trusting you at all. This single step does more than everything above it.

Better still: evidence that the record has not changed since it was written. Hash-chaining does this cheaply. It is worth exactly as much as the independence of wherever the head of that chain is kept.

Strongest, and rarest: the provider confirming, after the fact, that the artifact is still there and still matches. This is the only rung where the evidence stops being yours.

Most agent platforms sit on the first or second rung and describe themselves in the language of the fifth. The gap between those is where the phrase "audit-ready" usually lives.

What to actually build, in order

One chokepoint first. Until every side effect passes through a single place, nothing downstream is a record. This is unglamorous and it is the only step that cannot be added later.

Then the provider’s own identifier on every write. It costs nothing at the time and it is what turns your ledger into a pointer at theirs. It is also easy to get silently wrong: a provider that names its identifier something unexpected will fall through a generic capture and leave rows that say an action happened with no link to it.

Then re-verification, because it is the step that changes the ledger from self-reported to audited, and the only one that catches a confidently-wrong record without a human noticing first.

Then sealing, and then publishing the head. In that order, because a seal over records nobody has checked preserves whatever was written, correct or not.

Five questions worth asking a vendor

Most of what separates real evidence from the appearance of it cannot be seen from a product page. These are the questions that surface it, and each has a wrong answer that sounds fine.

Does every side effect cross one place? The answer you want is a specific component. "It is all logged" usually means several paths log separately, which is how a record becomes a sample.

Do you store the provider’s own identifier for each write, and can I see one? A demo that shows a row saying an email was sent, with nothing that opens the email, is answering a different question.

Do you ever go back and check? Almost nobody does. A vendor that says yes should be able to tell you what fraction of their records they can check at all, and if that number sounds high, ask which verbs it covers.

What can I still prove if I stop paying you? Evidence that lives only inside a product you have churned from is not evidence. Look for an export you hold.

What does your system record when it is not sure? The best answer names a third state — dispatched but unconfirmed, or no way to check — because a system with only success and failure will file every ambiguity under one of them, and it will not be failure.