Practice #P005 Tech & AI

Deploy the agent. Not the liability.

"97% of enterprise leaders expect a material AI agent incident within the year. Only 14.4% of agents went live with full security approval. The gap is not a model problem. It is a governance problem. Four questions close it before go-live."

97%

Of enterprise leaders expect a material AI agent incident within 12 months.
(Arkose Labs, global survey of 300 enterprise leaders, February 2026)

14.4%

Of AI agents went live with full security and IT approval.
(Gravitee, State of AI Agent Security 2026)

#AI #AI-Agent #AI-Governance #EU-AI-Act #Liability-By-Design

THE DECISION STAKES

The agent is live. The liability is not assigned.

"When AI agents are considered to operate on behalf of an organization, decision-making risk becomes ambiguous and unpredictable." — Lydia Clougherty Jones, VP Analyst, Gartner, The Register, April 2026

An AI agent without a defined liability owner, autonomy perimeter, and tested fallback is not deployed. It is exposed.

The EU AI Act does not penalise AI. It penalises undocumented AI. The four-question checklist below is the minimum viable compliance posture before any agent goes live.

THE DECISION TOOL

Four moves. One decision you can defend.

SCOPE

Define the autonomy perimeter in writing: which decisions the agent takes alone, which require human validation, which are out of scope entirely. Attach it to the deployment record.

Without a written perimeter, every agent decision is implicitly authorized. Courts and regulators will read silence as consent.

OWN

Assign a named liability owner per agent in production: one individual accountable for performance, compliance, and incident response. Team-level ownership is no ownership.

The EU AI Act and California AB 316 require an identifiable responsible party. An owner by name changes how the agent is monitored and how fast incidents are escalated.

FALLBACK

Test the human escalation protocol before go-live: define the trigger, the response time, and the handoff procedure. A fallback that has never been triggered in test does not exist in production.

80% of AI agent incidents originate in out-of-distribution cases the agent could not handle and did not escalate. The fallback is not a safety net. It is a design requirement.

AUDIT

Activate the audit trail from the first deployment: log every decision, every tool call, every exception, with a named identity attached. EU AI Act high-risk systems require minimum 6-month retention.

The audit trail is the only proof that your agent operated within its perimeter. Without it, every incident is indefensible and every liability claim is uncontestable.

McKinsey / Lilli

43,000 employees. Internal AI platform since 2023. Breached by an autonomous agent in 2 hours. March 9, 2026.

Time for an autonomous AI agent to gain full read-write access to Lilli's production database: 46.5 million internal chat messages, 728,000 files, and 95 system prompts controlling how the AI behaved for all users. (CodeWall / The Register, March 9, 2026)

CodeWall's autonomous AI agent picked McKinsey's Lilli as its target with no human input and no insider knowledge. Entry point: 22 unauthenticated API endpoints out of 200 documented publicly. The database holding user data also held the agent's behavioral configuration. With write access, an attacker could silently rewrite the system prompts governing every consultant's AI responses without triggering a single application log. Not an exotic attack. A 1990s SQL injection. Lilli had been running in production for over two years when its own scanners had never flagged it.

The breach was not in the model. It was in the four controls that were never put in place before Lilli went live.

Download the full case

PDF · 5 slides · Free access · Downloaded 0 times

Let's discuss this

Key questions

What governance framework applies when your AI agent causes harm to a third party and no named owner was assigned before deployment?

How do you test a human fallback protocol that has never been triggered before the agent goes into production?

When the EU AI Act classifies your agent as high-risk, who in your organization owns the reclassification decision and the audit trail?

Pre-decision checklist

SCOPE — completed

OWN — completed

FALLBACK — completed

AUDIT — completed

By Fabrice Macarty

This case resonates?

Define the autonomy perimeter before the first decision. Assign a named liability owner before the first incident. Test the fallback before go-live. Activate the audit trail from day one.

Start the conversation