AI agents in the enterprise: from PoC to production

An AI agent that reads documents, calls systems and handles tasks can be put together in a few days. I know because I build them — and I also work as an architect at an insurance company, so I see both sides: how fast an impressive demo comes together, and how much still stands between it and production. And most of it has little to do with technology.

Why a PoC looks finished in a week

With a proof of concept, one thing is easy to forget: it performs on scenarios its author picked. Ten showcase cases go smoothly, the audience is impressed, and everyone feels all that's left is "just deploying it".

But a PoC usually runs on curated data, with no real permissions, no audit trail, and a human sitting next to it quietly fixing the small slips. You'll have none of that in production. A PoC answers the question "can this work?" — production asks what happens when it stops working.

Klarna showed what that difference looks like in practice. In February 2024 it announced that its AI assistant had handled two-thirds of customer-support chats in its first month — "the work of 700 people". A year later the CEO admitted the AI-driven cuts had gone too far, quality had dropped, and the company was hiring people again. The press release measured how many conversations the agent handled. How well — that only came up a year later.

What the enterprise adds

In a regulated environment, requirements show up that the demo never had to deal with:

• Audit trail. Every decision the agent makes has to be traceable: what it saw, why and how it decided, what it did. "The model judged it that way" won't satisfy an auditor or a regulator.
• Accountability. When the agent makes a mistake, it has to be clear in advance who's responsible and how it gets fixed. That's a process question — and you want the answer before you let the agent into production, not after the first incident. The courts already have an opinion on this: in February 2024 a Canadian tribunal made Air Canada pay for a discount its chatbot had invented. The airline argued the chatbot was "a separate legal entity that is responsible for its own actions" — the tribunal called that a remarkable submission and upheld the claim. It was 812 Canadian dollars, but the case has been cited ever since as a precedent: the company answers for its chatbot's output.
• Data governance. In a PoC it's convenient to let the agent see everything. In production, data access has to match purpose — a claims-handling agent has no business reading payroll data, even if it would improve its answers.
• Security. An agent is a new attack surface: inputs it trusts can be forged (prompt injection is just the best-known case), and the more actions it's allowed to take, the more damage a forged input can do. For a long time this was illustrated mostly by curiosities, like the dealership chatbot promising a Chevrolet Tahoe for one dollar. Then in June 2025 came EchoLeak: the first zero-click prompt injection against a production enterprise agent (Microsoft 365 Copilot, CVSS 9.3). One ordinary-looking e-mail in the inbox was enough — the agent pulled it into context on its own, and sensitive data leaked without the user clicking anything. Luckily, security researchers found it first.

And it's not just internal caution. Under the EU AI Act, AI used for risk assessment and pricing in life and health insurance lands straight in the high-risk category, with obligations from August 2026 (Brussels is debating a delay, but the direction stands). So a good part of the bullets above will sooner or later simply be the law.

Where the agent may act on its own

The most important design decision of the whole solution: which actions the agent may take on its own, and where a human has to be in the loop. I use a simple grid for this — two axes, reversibility and impact:

• Reversible + low impact → the agent acts alone (drafting a reply, categorising, preparing materials).
• Reversible + high impact → the agent acts, humans check samples and metrics.
• Irreversible + low impact → the agent prepares, a human confirms with one click.
• Irreversible + high impact → the agent only recommends; a human decides (paying out a claim, terminating a policy).

And that boundary isn't fixed. It pays off to start conservative and give the agent more authority as the production numbers come in — that's cheaper than starting bold and locking everything down again after the first incident.

The agent as a new participant in the process

Agents often get treated as just another system to "integrate". What works better for me is the picture of a new colleague: fast, never bored, but occasionally reading the assignment their own way.

That picture leads to the practical questions you'd ask about any new colleague: who assigns their work, and in what form? Who takes their output? How do they know they're out of their depth, and who do they tell? And what happens to the process when they just "don't show up for work"? A process with no answers to these questions isn't ready for an agent — no matter how good the agent is.

What to measure so you can say "it works"

Without metrics, the debate about an agent runs on anecdotes: one person remembers a brilliant answer, another remembers a disaster. This is the minimum set I want from day one:

1. End-to-end success rate — not "the agent replied" but "the case was handled correctly", measured by the same yardstick as humans. That's exactly where the McDonald's drive-thru AI ended: two years of testing across a hundred-plus restaurants, yet order accuracy sat around 80% by franchisees' accounts, against a target of 95. The viral video with 260 nuggets in one order was just the most visible case of it.
2. Escalation rate — how often the agent hands work to a human. A low number isn't automatically good news: an agent that never escalates probably can't tell when it's out of its depth.
3. Cost per handled case — including the cost of checking and fixing the agent's work. Leave those out and the number looks better than reality.
4. Correction trend — how many of the agent's outputs people had to redo, and how that develops. This is the number that moves the autonomy boundary from the previous section.

Conclusion

Deploying an AI agent in the enterprise mostly means deciding how much trust you'll give it: what it may do on its own, who answers for it, and how you measure that it works. The technology is more ready than most organisations think. What's usually missing are the processes, responsibilities and metrics.