The challenge
What was deployed
The results
Last Updated: November 9, 2026
Nubank's AI Research team built a pre-deployment simulation layer that runs candidate customer-support agents against synthetic Brazilian customers before any real person sees them. The reported effect: average time per shipped agent version fell from 21.2 days to 4.4 days, and the simulation-guided agent beat its predecessor in a live A/B test by 4.9 percentage points on self-service rate and 36.7 points on transactional NPS. A separate screen of 29 model configurations across 16,000+ simulated conversations let Nubank swap its incumbent frontier model for an open-weight Qwen configuration, adding 8.8 points of self-service and cutting p95 latency 25% with no measurable satisfaction hit.
Most published enterprise AI case studies describe the agent. This one describes the test harness. On November 5, 2026, Nubank's newsroom published results from a research paper its AI team put on arXiv in September, documenting how the Brazilian digital bank screens changes to its production chat agents against simulated customers before exposing real ones.
Nubank is not a small deployment surface. In its Q2 2026 results filed with the SEC, Nu Holdings reported 139 million customers globally and almost 118 million in Brazil. The paper, "Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale", puts the scale in its title for a reason. At that volume, a bad prompt change is not a bug report. It is a few hundred thousand degraded conversations before anyone notices.
The problem: you cannot safely learn in production at 140M scale
Nubank already had conversational AI in production. An OpenAI customer story published in 2025 describes an AI assistant resolving 55% of Tier 1 inquiries, handling over 2 million monthly chats, and cutting chat response times by 70%, alongside a call-centre copilot used by over 45% of agents. So this is not a greenfield pilot. It is the problem that arrives after the pilot works: how do you keep changing a system that millions of people depend on, without using those people as the test set?
The research paper frames the three available options bluntly. Hand-authored test cases only exercise the scenarios somebody thought to write down. Live A/B tests measure the truth but put customers in the blast radius. The third option — simulating customers on-policy at the tool boundary — is what Nubank built out.
Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust.
— Alcobaça, Rossell et al., Nubank AI Research, arXiv:2609.30137
What was actually built
Two agents sit at the centre of the work. Card Delivery handles delivery status and reissues; Card Management covers address updates and customer-requested reissues and is described in the paper as Nubank's highest-volume chat-support agent in Brazil. Card Management replaced Card Delivery and absorbed a broader slice of the card lifecycle.
The simulator is Snowglobe, a product from Guardrails AI — two of the paper's co-authors carry a Guardrails AI affiliation, which is worth noting up front. It takes an agent description and tool definitions, optionally with historical conversations, and from those builds use cases, user profiles, tool relationships and trajectory plans. An orchestrator then spreads generated conversations across use-case and interaction-style conditions.
Tool-boundary mocking
The key design choice: only the tool boundary is faked. The agent under test runs its real policy, real prompts and real reasoning loop; when it calls a backend — "look up this card's shipping status" — a mock layer, parameterised by the synthetic persona, answers instead of production. The agent then acts on that answer. Multi-step failures therefore surface in simulation, because the agent is genuinely improvising against states it has not seen, not replaying a script.
How a change gets screened before it ships
Each candidate version is tied to an explicit hypothesis and a success criterion agreed in advance — e.g. "this prompt revision reduces unnecessary transfers to humans."
Snowglobe plans coverage across use cases and interaction styles, then spawns personas carrying style, state, topics and a trajectory. Each holds multi-turn conversations with the candidate agent.
Tool calls are answered synthetically, so no production backend is invoked and no customer record is touched.
An LLM-as-a-Judge returns a binary verdict per conversation on the hypothesis under test — for Card Management, whether a transfer to a human was unnecessary. Candidates are compared against the incumbent's failure rate.
Changes that pass the screen go to production experimentation measured on self-service rate and transactional NPS.
Before trusting any of this, the team validated the simulator itself against the older Card Delivery agent. They split 8,000 real conversations into a development set for configuring the simulators and a test set for validation, then checked four properties: conversation-length distributions, transcript-embedding proximity to production, whether evaluator scores ranked the four deployed versions in the same order as production did, and whether blinded human experts could tell simulated transcripts from real ones.
The honest result on that last test is instructive. Annotators correctly identified 42 of 50 production conversations and 35 of 50 simulated ones — so 15 simulations were classified as production. Humans could still mostly tell. But on the ranking test that actually matters for decision-making, Snowglobe reproduced production's best and worst versions where the naive baseline simulator got them exactly backwards.
The measured outcome
Offline, the winning Card Management version cut the unnecessary-transfer failure rate to 16.0% from the 22.4% baseline — a 6.4-percentage-point reduction. Then it went to a live A/B test against Card Delivery.
| Live A/B test | Self-service rate | Transactional NPS | p95 latency |
|---|---|---|---|
| Card Management (simulation-guided) vs Card Delivery | +4.90 pp | +36.69 points | — |
| Qwen3.5-122B-A10B (reasoning on) vs incumbent model | +8.82 pp | −1.21 (not significant) | −25% |
Source: arXiv:2609.30137, Table 2. Following the first test, the Card Management agent was made available to Nubank's customer base in Brazil.
Every change to an AI agent is a bet on the customer experience. Simulation lets us place those bets on synthetic customers first, so only the changes that clearly win reach real people
— Rohan Ramanath, General Manager of AI, Nubank, Nubank Newsroom
The quiet second result: an open-weight model displaced the incumbent
The more commercially interesting finding is buried in section 5.3. The incumbent Card Management backbone was GPT-5.2. Nubank screened 29 configurations — across model family, reasoning effort and numerical format, including GPT-OSS-120B, Nemotron-3-Super-120B-A12B, Qwen3.5-122B-A10B and Qwen3.6-35B-A3B — over more than 16,000 simulated conversations, then selected Qwen3.5-122B-A10B with reasoning enabled and refined its prompt before live testing.
No configuration won on everything. The incumbent kept a substantial advantage on information-retrieval status (2.4% failure rate). The selected Qwen configuration won on input gathering and conversation completeness. In the live test it raised self-service by 8.8 percentage points and cut p95 response latency by 25%, with no statistically significant change in tNPS.
That is the real unlock. Evaluating 29 configurations against live customers would be commercially unthinkable; simulation made the search space affordable, and the answer it returned — an open-weight model outperforming a frontier incumbent on the metric the business cares about — is one you only find if searching is cheap.
What the public record does not tell us
The headline A/B test is not apples-to-apples. Nubank says so itself: because Card Management handles a broader set of tasks than Card Delivery, the gains reflect both the new agent and the simulation-guided process used to build it. The +36.7 tNPS figure is not a clean measurement of "what simulation bought you."
The 4.4 vs 21.2 days comparison is calendar arithmetic, not an experiment. It divides a ten-version Card Delivery cycle spanning 212 days and a five-version Card Management cycle spanning 22 days by their version counts. Different agents, different periods, different team maturity, different underlying models. Treat it as directional.
No cost figures at all. Nubank has published no inference-cost savings from the open-weight swap, no simulation compute cost, no headcount effect, and no revenue impact. A 25% p95 latency reduction and a frontier-to-open-weight migration strongly imply a unit-cost story — but the company has not published one.
Self-service rate is not resolution. Nubank flags this directly: the share of sessions completed without a request for human help does not prove every issue was actually solved. The tNPS samples backing the satisfaction claims are small (2.0K and 2.3K), and the p95 latency number is a point estimate from the serving stack with no confidence interval.
Scope is narrow. Two card-lifecycle agents, in Brazilian Portuguese, in Brazil. Nothing here establishes that the approach generalises to Nubank's Mexico or Colombia operations, to fraud or collections workflows, or to voice.
Vendor adjacency. The simulator is a commercial product whose vendor co-authored the paper, so the simulator-vs-baseline comparisons are self-reported by interested parties — though the paper does publish where Snowglobe loses.
Treating the simulator as a customer replica
Teams building synthetic-user evaluation usually set the wrong bar: make the fake customers indistinguishable from real ones. Nubank's own data shows they were not — blinded experts spotted 70% of simulated transcripts — and the project still worked, because the question being asked was never "is this realistic?" but "does this rank candidate A above candidate B the way production would?"
What an integration team should learn from this
1. The bottleneck in agent programmes is evaluation throughput, not model quality. Nubank's constraint was never "can GPT-5.2 update an address." It was "how many candidate changes can we responsibly evaluate per quarter." Most organisations hit this wall and respond by shipping more slowly rather than by building a screening layer. The automation work that pays is in the test harness.
2. Hypothesis-first beats vibes-first. Each version carried a stated hypothesis and a pre-agreed success criterion. Nubank notes a secondary benefit that regulated industries will recognise: the workflow creates a clear record of why each change was approved. That audit trail is a compliance artefact, generated for free by good engineering discipline.
3. Cheap search changes what you can procure. Once evaluation is cheap, model selection stops being a procurement decision and becomes an empirical one. A bank with 139 million customers moved its highest-volume support agent off a frontier model because 16,000 simulated conversations said it could. Teams comparing model options without a screening layer are, in practice, choosing on vendor marketing.
Nubank's OpenAI-built assistant is already resolving 55% of Tier 1 inquiries across 2M+ monthly chats, per OpenAI's published customer story.
Ten versions ship over 212 calendar days, moving from GPT-4.1 through GPT-5.1 to GPT-5.2. Four of these later serve as the ground truth for validating the simulator.
Five versions ship in 22 days under the simulation workflow. The winner cuts unnecessary transfers to 16.0% offline and wins the live A/B test.
"Screen Before You Serve" is submitted to arXiv.
Nubank publishes the results through its newsroom, with the Card Management agent live to its Brazilian customer base.
Nubank's own framing of what simulation is for is the least triumphant line in the paper: the point is to support development decisions, not to reproduce production perfectly. Teams chasing a synthetic customer who behaves exactly like a real one will build something expensive and never ship it; teams chasing a cheap, directionally-correct ranking function will ship every week.
If you are scoping similar work, the pattern to copy is the validation step, not the simulator. Start with versions you have already shipped and whose production outcomes you know, and ask whether your offline harness would have picked the winner. If it would not, nothing downstream of it is trustworthy. Our case studies and research notes cover more of this ground, and get started if you want a second opinion on an evaluation stack you already have.
Frequently Asked Questions
What did Nubank actually deploy?
Two things. First, a pre-deployment simulation layer that runs candidate customer-support agents against synthetic Brazilian customers, with backend tool calls answered by a mock layer rather than production systems. Second, the agent that layer produced: Card Management, Nubank's highest-volume chat-support agent in Brazil, covering card address updates, reissues and delivery status. It is live to Nubank's Brazilian customer base.
How much faster did development get?
Nubank reports an average of 4.4 days per agent version with simulation versus 21.2 days in an earlier workflow without it. That is calendar time divided by version count across two different agents — a ten-version cycle spanning 212 days and a five-version cycle spanning 22 days — not a controlled experiment. The direction is credible; the exact ratio should not be treated as a benchmark.
Which models does the agent use?
The Card Management agent ran on GPT-5.2 as its incumbent backbone, with earlier Card Delivery versions on GPT-4.1 and GPT-5.1. After screening 29 configurations, Nubank live-tested Qwen3.5-122B-A10B with reasoning enabled against that incumbent; it raised self-service by 8.8 percentage points and cut p95 latency by 25% with no significant change in satisfaction. The simulator is Snowglobe from Guardrails AI.
Has Nubank published cost savings?
No. Neither the newsroom release nor the research paper discloses inference cost, simulation compute cost, headcount impact or revenue effect. The published outcomes are operational and experience metrics: self-service rate, transactional NPS, unnecessary-transfer failure rate, p95 latency and development cycle time. Any ROI figure for this deployment would be an inference, not a reported number.
Does a higher self-service rate mean customers were helped?
Not on its own, and Nubank says so. Self-service rate measures the share of sessions completed without a request for human support; a customer who gives up looks the same as one who was helped. That is why the analysis pairs it with transactional NPS and with specific quality evaluations such as whether a transfer to a human was unnecessary.
Can smaller teams copy this approach?
The expensive part is not the simulator — it is the validation. You need historical conversations and shipped versions whose production outcomes you already know, so you can check that your offline harness ranks them the way production did. Nubank used 8,000 real conversations, split into development and test sets, across four previously deployed versions. Without that ground truth, a simulator is just confident noise.
Twarx analysis
Original interpretationThe binding constraint on production agent programmes is evaluation throughput, not model capability — and a simulator only needs to rank candidates the way production would, not to imitate real customers convincingly.
Read the numbers honestly
Analysis by
Rushil Shah · AI Systems Builder & Founder, Twarx