ANZ, one of Australia's largest banks, employs over 5,000 engineers. Like every large engineering org, it needed to know whether AI coding assistants deliver measurable productivity — not vibes — before rolling one out at scale, and whether AI-written code would be maintainable enough for a regulated banking environment.
The challenge
What was deployed
ANZ engineers ran a controlled experiment: 100 engineers, split into a GitHub Copilot group and a control group, completing Python coding challenges over a four-week measurement window (mid-2023). The study was authored by ANZ's own staff and published on arXiv. Separately, GitHub published an enterprise randomised controlled trial with Accenture measuring real workflow signals (pull requests, merge rate, build success).
The results
The ANZ Copilot group completed tasks 42.36% faster than the control group, and the AI-assisted code "contained fewer code smells and bugs on average." A follow-on cohort of ~1,000 engineers showed early gains in productivity and job satisfaction. In GitHub's Accenture RCT: +8.69% pull requests per developer, +15% PR merge rate, +84% successful builds, ~30% suggestion acceptance, and 90% of developers "felt more fulfilled with their jobs."
Twarx analysis
Original interpretationThe headline is 42% faster delivery — but the more important finding is that code quality held or improved at the same time. Speed that doesn't add technical debt is the rare combination that makes AI coding assistants worth standardising.
Most coverage of the ANZ study fixates on the 42% speed-up. We think the quieter result matters more: the Copilot group's code had fewer code smells and bugs on average, so the velocity gain did not come at the cost of maintainability. In a regulated banking codebase, that distinction is everything — a faster team that ships more defects is a net negative.
The second signal worth naming is the ~30% suggestion-acceptance rate from the Accenture trial. Read pessimistically, developers reject most suggestions. Read correctly, the human stayed in the loop and curated — which is precisely why quality held. The practical takeaway: Copilot is a throughput multiplier for engineers who already review well, not a replacement for review. Where we push back on the hype is durability — ANZ's tasks were short, so we would not assume the 42% transfers cleanly to large legacy refactors.
Illustrative Twarx model
Illustrative annual engineering-hours reclaimed, by team size
50 engineers
3,600 hrs/yr
500 engineers
36,000 hrs/yr
5,000 engineers
360,000 hrs/yr
Method & assumptions: Assumes ~20% of an engineer's week on Copilot-suitable work (~360 hrs/yr: boilerplate, tests, scaffolding) and a conservative 25% speed-up on that slice — well below ANZ's 42% headline to account for review overhead and real-world dilution — giving ~72 saved hrs/engineer/yr. A Twarx model for scale intuition only; NOT measured results, and it ignores onboarding, licence cost, and quality effects.
Read the numbers honestly
Read the evidence carefully:
- Security was inconclusive. The ANZ study could not generate meaningful data on code security; it found no evidence Copilot introduced major issues, but that is not proof of safety.
- Short tasks limit generality. ANZ's challenges were short, leaving "little room for bugs" — so quality findings may not transfer to large legacy codebases.
- The Accenture numbers are a vendor (GitHub) study and omit participant count and trial length — authoritative-but-interested. The ~30% acceptance rate also implies a continuing human review burden.
- No primary dollar ROI was published by ANZ or Accenture. We deliberately do not state a payback figure; the honest cost anchor is public per-seat pricing against the documented productivity gains.
Timeline
Mid-2023
ANZ controlled experiment: 100 engineers, Python challenges
Feb 2024
ANZ publishes empirical study (42.36% faster)
May 2024
GitHub publishes Accenture enterprise RCT
Frequently asked
How much faster did developers actually work?
In ANZ's controlled experiment, the Copilot group completed coding tasks 42.36% faster than the control group. A separate GitHub/Microsoft study measured 55% faster on a single synthetic task; the Accenture enterprise trial measured workflow gains like +8.69% pull requests rather than a single speed number.
Is AI-generated code lower quality?
ANZ found Copilot code had fewer code smells and bugs on average in its tests, but the tasks were short. Security impact was inconclusive, and the ~30% suggestion-acceptance rate shows developers still review and reject most suggestions — so human review remains essential.
What does it cost and does it pay back?
GitHub Copilot is publicly priced at $19/seat/month (Business) and $39/seat/month (Enterprise). No company-audited dollar ROI or payback period has been published, so any specific payback claim should be treated with caution.
AI AgentsSoftware EngineeringDeveloper ProductivityGitHub CopilotROI
TR
Analysis by
Twarx Research Team · Applied AI Research
Twarx researches and deploys enterprise AI agents with a measurement-first method: every metric traced to a primary source, projections labelled as estimates, and limitations stated up front.
AI agentsAgentic AIDeveloper productivityEnterprise deploymentSLMs
