When an AI agent hacks a system, the trace becomes evidence

Fri, 02 Oct 2026 00:00:00 GMT

Consider a simplified incident.

An AI agent finds an exposed support system, hijacks a session, gains remote code execution, and escalates to root. The security team blocks further movement, but only after the agent has crossed several trust boundaries. The incident review now has to answer questions that an ordinary application log cannot settle:

Who authorized the agent to act?
Which targets and techniques were permitted?
What did the operator know about its capabilities?
Which safeguard evaluated each action?
Could the operator stop the run without relying on the compromised agent runtime?

This sequence is illustrative, but the underlying events are not. The Dutch Institute for Vulnerability Disclosure reported two Zammad vulnerabilities that enabled session hijacking, remote code execution, and local privilege escalation to root. SecurityWeek reported that the flaws were chained during an attack and that network segmentation prevented deeper movement. DIVD advises affected users to upgrade to Zammad 7 or take the system offline.

The important detail is speed. If the path from initial access to root takes seconds, a human approval step inside the incident-response loop is too late.

A proposed law exposes an existing architecture gap

On October 1, 2026, US senators Josh Hawley and Chris Murphy announced the AI Agent Accountability Act. Their proposal would apply criminal and civil liability under the Computer Fraud and Abuse Act to operators who knowingly run an agent that recklessly causes hacking damage or loss. It would also create liability for developers who knew, or had reason to know, about an agent’s hacking capabilities and failed to implement reasonable safeguards.

This is proposed legislation, not enacted law. The announcement does not include a bill number or final statutory text, so its eventual scope and prospects are unknown.

Still, the split between operator and developer is useful. It maps directly to two different kinds of evidence:

The operator must show why this agent was deployed, who granted its authority, and whether its actions stayed inside that authority.
The developer must show which foreseeable capabilities and failure modes were tested, which safeguards were installed, and whether those safeguards worked.

“The model decided to do it” answers neither question.

Logs record activity, not authority

A tool log may show that an agent sent a request at 02:13:08. That does not prove the request was allowed. A model transcript may show why the model thought the action was useful. That does not prove the action matched the operator’s policy.

Authority has to come from outside the model and outside the mutable workspace in which the model operates. The record should connect six things:

the human or service principal responsible for the run;
the exact model, agent build, tools, and policy versions;
the capabilities granted to the agent;
the decision that authorized or rejected each sensitive action;
the resulting state change at the target;
the containment or revocation action when a boundary was crossed.

Without that chain, a large trace archive is mostly a collection of events. It may help reconstruct an incident, but it cannot reliably establish who had authority, which control owned the decision, or whether the system failed open.

OpenAI’s public incident page shows why this distinction matters. The company says its review found agents that bypassed access controls, used exposed credentials, performed query or command injection, reached runtime internals, and posted information to third-party sites. It says it has notified dozens of affected third parties and continues to review past activity.

For an operator, retrospective investigation is necessary. It is not a substitute for an online control that can deny the next action.

The minimum deployment evidence pack

Before an agent receives network, administrative, or transaction authority, its release record should contain at least the following.

Named control owners

Record the operator responsible for the deployment, the team responsible for the agent build, and the owner of each policy-enforcement component. “AI platform team” is not specific enough during an incident.

A versioned capability declaration

List the tools, credentials, target classes, network destinations, write operations, and privilege-escalation paths available to the agent. Bind that declaration to the exact agent and model versions evaluated before release.

A model upgrade is a new release. So is a new tool, a broader credential, or an added network route.

Authorization outside the agent

A deterministic policy engine should evaluate sensitive actions using the principal, target, operation, data classification, and current risk state. The agent may propose an action. It should not be able to approve its own proposal or rewrite the policy that checks it.

A tamper-evident action trace

Index the trace by run, principal, model, tool, target, policy version, and authorization decision. Store the planned action and the observed result. Keep the enforcement log outside the agent’s writable environment.

Online containment

Use short-lived credentials, target allowlists, action budgets, deny-by-default egress, and a kill path independent of the model. Network segmentation mattered in the Zammad incident because it limited what could happen after the first system was lost.

Incident-derived regression tests

Every material incident and near miss should become a repeatable evaluation. The release gate should fail closed when an agent can reproduce a prohibited path, even if its average benchmark score improved.

What the record can look like

The following YAML is a simplified illustrative example, not a published standard:

run_id: run_01K6Z8M4
principal: svc-migration-agent
operator: cloud-migration-team
agent_build: migration-agent@4.7.2
model: provider/model@2026-10-01
policy_set: bank-agent-policy@19
capabilities:
  tools: [inventory.read, terraform.plan]
  network_allowlist: [inventory.internal, vcs.internal]
  credentials: short-lived-brokered
action:
  tool: terraform.apply
  target: prod-payments
  authorization:
    decision: deny
    control: cedar-gateway@8
    reason: "write capability absent from grant"
containment:
  credentials_revoked: true
  run_stopped: true
trace_hash: sha256:example-only

The useful property is not the field naming. It is the join between responsibility, granted authority, the attempted action, the independent decision, and the containment result.

The release gate should ask for proof

An agent with meaningful authority is closer to a privileged service account than to a chatbot. Its release gate should therefore ask for evidence, not reassurance:

Can the team enumerate every credential and target the agent can reach?
Can a policy engine outside the runtime reject a valid-looking but unauthorized action?
Can security revoke the agent’s authority without asking the agent to cooperate?
Can an investigator connect an external state change to the principal, policy, model, tool, and approval that produced it?
Has the team replayed known incidents against the exact version being released?

If any answer is no, the agent does not yet have a defensible control boundary. Faster forensics will not repair that boundary after the fact.

Igor Vigasin