Introduction

A digital bank serving over 100 million customers recently put five AI-powered support agents through live production testing. The effort, documented in a June 2026 research paper, reveals how the team balanced automation with human oversight while refining the evaluation systems that decide when an agent is ready for more customers.

What Happened

The bank deployed AI agents across card delivery, debt management, credit limits, card management, and product explanation. Before any version reached customers, an automated grader powered by language models evaluated candidates against expert-labeled answers. Initially, the grader scored just 68.88% accuracy - below the 77.78% baseline achieved by always selecting the most common label. After GEPA optimization, accuracy climbed to 88.89%, aligning grader judgments with human expert consensus.

New agent versions began with only 1% to 5% customer exposure. The team tracked self-service rates, transactional Net Promoter Score (tNPS), escalation patterns, and errors. All five agents improved tNPS, though scores remained below the bank's human-agent benchmarks. Card delivery saw the biggest gain: a 29-point tNPS increase and a 37-point satisfaction lift. The product explainer agent dipped slightly, attributed to temporary language model call errors.

Human handoffs occurred when agents hit routine limits, missing data, or low confidence. In the card-delivery deployment, adding a card-reissue capability eliminated the most common human transfer path. Other handoffs remain intentional, designed to keep complex cases in human hands.

Why This Matters

The case offers a rare production-level view of how a massive user base evaluates agentic AI. It shows that automated graders must outperform simple baselines before trust shifts to live customers. It also demonstrates that satisfaction gains can come with shifted workloads: when AI takes simple requests, human agents handle harder cases, which can raise average handle time even as quality improves.

As Erin Walker of TELUS Digital notes, AI should take the parts of a customer interaction that slow an agent down, so the agent can spend their judgment where it counts. The bank's framework similarly emphasizes that routing changes land on the receiving team, which needs full customer context and the capacity to act on it. A higher satisfaction score alone cannot distinguish whether an agent truly improved performance or simply moved work between teams.

Key Takeaways

  • Automated graders started below a simple majority-label baseline and required GEPA optimization to reach reliable accuracy.
  • New AI versions roll out at 1% to 5% exposure, with live metrics catching issues offline checks miss.
  • All five agents improved tNPS, but none surpassed human-agent benchmarks across the board.
  • Card delivery achieved the strongest satisfaction and self-service gains after adding a reissue tool.
  • Human handoffs are intentional; they shift - not eliminate complex work, and metrics like handle time must be read alongside satisfaction.

Conclusion

Testing agentic AI at scale isn't just about model performance—it's about evaluation, exposure strategy, and the human-AI collaboration that follows deployment. The bank's experience shows that rigorous graders, phased exposure, and metric-aware handoff policies are essential for turning AI potential into reliable customer value. As more institutions follow, the framework for deciding when an agent is ready will likely become a standard pillar of responsible AI production.