Nubank used reactive synthetic customers and simulated tool outputs to screen four deployed versions of its card-support agents before live traffic. Version-level simulation scores correlated strongly with production scores. Simulation-guided changes raised transactional net promoter score by 36.69 points in one live A/B test. A later screen of more than 16,000 conversations selected an open-weight configuration that increased self-service by 8.82 percentage points without a statistically significant tNPS decline; the agent serves a customer base reported at 140 million.
Nubank validates agent changes in simulation before serving customers
2026-09-25Frontier watchTRIAL
Nubank used reactive synthetic customers and simulated tool outputs to screen four deployed versions of its card-support agents before live traffic. Version-level simulation scores correlated strongly with production scores. Simulation-guided changes raised transactional net promoter score by 36.69 points in one live A/B test. A later screen of more than 16,000 conversations selected an open-weight configuration that increased self-service by 8.82 percentage points without a statistically significant tNPS decline; the agent serves a customer base reported at 140 million.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For production agent teams, this is unusually concrete evidence that multi-turn simulation can reduce the search space before risky online experiments. The useful pattern is not synthetic users alone: preserve tool behavior, test hypotheses against production-correlated evaluators, and require a live holdout before promotion.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Deployment | Nubank Card Delivery and Card Management agents |
|---|---|
| Scale | customer base reported at 140M |
| Validation | four deployed versions; simulation and production binary scores correlated |
| Ab Test One | +36.69 points transactional NPS |
| Ab Test Two | +8.82 percentage points self-service rate; no significant tNPS change |
| Model Screen | >16,000 simulated conversations |
Technical impact 3
Engineering value 5
Adoption signal 5
Maturity 4
Verification cost 4
Risk 2