Sixteen programmers. 246 real tasks. A randomized controlled trial — the kind where you can't explain away the results because someone was having a bad day.
METR, an AI research nonprofit, ran the study from February to June 2025. They recruited experienced open-source developers — people with an average of five years and 1,500 commits on the repositories they were working in. Real bugs, real features, real refactors. Each task was randomly assigned: AI allowed or AI forbidden. When AI was allowed, the developers used Cursor Pro with Claude 3.5 Sonnet.
The result: developers took 19% longer to complete tasks when they used AI tools.
But that's not the finding that stopped me.
Before the study, the developers expected AI to speed them up by 24%. After experiencing the actual 19% slowdown — after living through it, task by task, for months — they still believed AI had sped them up by 20%.
A 39-point gap between what they felt and what was real.
Less than half the output survives
The METR study tracked where the time went. Developers accepted less than 44% of what the AI generated. A majority reported making major changes to clean up the code that did get accepted. Nine percent of total task time — nearly one minute in ten — was spent reviewing and fixing AI output that shouldn't have been written that way in the first place.
That's the tax nobody bills for. You prompt. You wait. You read what comes back. You fix what's wrong. You prompt again. Each cycle feels productive because something is happening on screen. But the clock is running, and the full paper measured it: the overhead from prompting, waiting, and reviewing consumed the gains and pushed the total into the red.
The METR researchers identified five factors driving the slowdown. The biggest: developers who knew their codebases deeply got slowed down more, not less. The AI didn't have access to years of implicit context — which functions call which, why that config value is hardcoded, what the last refactor was avoiding. Experienced developers already had that context loaded. The AI was an interruption, not an accelerant.
GitClear's analysis of 211 million changed lines tells the same story from a different angle. Heavy AI users generate 9x more code churn — code written and then quickly revised or reverted. Code cloning doubled from 8% to 18% of all changes. And the metric that sticks: refactored code as a share of all changes dropped from 25% in 2021 to under 10% in 2025.
Developers are writing more code. The code is worse. And the careful restructuring that keeps a codebase healthy over time has collapsed by more than 60%.
I've run 541 hours of AI coding. The data agrees.
I should be the person these numbers don't apply to.
I built nanika, a multi-agent orchestrator that runs Claude Code workers in parallel. I've spent 541 hours of AI compute time in 20 days — 34 parallel overlap events across 45 sessions. I run architecture decision records before implementation. I have 25 personas, a mission decomposer, a scheduler. I've written the guardrails, debugged the failure modes, and measured my own success rate.
It's 82%. Sounds good.
But the 82% hides 80 wrong-approach corrections across 158 sessions. It hides the mission that ran for five hours generating 1.7GB of garbage output. It hides 7 sessions that hit context exhaustion and produced nothing. And it hides the fact that "mostly achieved" in a 30-phase mission can mean phases 1 through 9 completed before the whole thing timed out and died.
Here's what bothers me: even knowing all this, even having the data in front of me, I still feel faster. When I kick off three parallel agents and they're all producing output simultaneously, it feels like I'm doing the work of four people. The feeling is strong enough that I have to actively check it against the numbers. Some days the numbers back me up. Some days they don't. But the feeling is the same either way.
That's the METR gap in first person. The sensation of speed is independent of the reality.
The speed is real. For about 20% of the work.
I don't think the METR study means AI tools are useless. I think it means the copilot model — one developer, one AI, working on the same task together in real time — has a ceiling that most people haven't hit yet because they can't feel it.
The GitClear study has a finding that doesn't get quoted as often: heavy AI users produced 4-10x more durable code during their highest-usage weeks. Not on average. At peak. When the task matched what AI is good at — boilerplate, well-scoped functions, pattern application — the speed was real and measurable.
The problem is that most coding isn't that. Most coding is the other 80% that requires the context the AI doesn't have. The function that calls three other functions you wrote last month. The config value that exists because of a production incident in November. The architectural decision that isn't documented anywhere because it lives in your head.
When you hit that 80%, the AI doesn't speed you up. It generates something plausible that you have to evaluate against knowledge you can't easily put into a prompt. That evaluation takes time. The prompt-wait-review-fix loop starts. And the clock keeps running while you do the work the AI was supposed to save you from.
The developers in the METR study weren't bad at using AI. They accepted the tool. They used it voluntarily. They had access to frontier models. They just couldn't make the tool go faster than they could go alone, because on their codebases, they were already fast.
What you can't feel
The 39-point perception gap isn't a curiosity. It's the central finding.
If you asked me a year ago whether AI made me faster, I'd have said yes without hesitating. I'd have cited the parallel sessions, the code output volume, the projects shipped. And I would have been measuring the wrong thing — output volume instead of time-to-completion, sessions launched instead of goals achieved.
The METR researchers offered a few explanations for why the gap persists. Prompting feels like doing work. The AI response feels like receiving work. The cycle has a rhythm that maps onto productivity habits — type, get result, type, get result. It looks like a fast conversation with a colleague. It doesn't look like sitting in a terminal for 19% longer than you would have otherwise.
There's also a cost that protects itself. If you've invested in learning the tool, configured your workflow around it, told your team it makes you faster — the cost of discovering it doesn't is not zero. The belief insulates.
The METR study's authors are careful to say they don't claim this generalizes to all developers or all settings. The sample was small — 16 people. The tools were early 2025, not today's. Learning effects might require several hundred hours to materialize, and their developers only had a few dozen.
All fair caveats. But the perception gap doesn't need a large sample to matter. Because the gap isn't about whether AI works. It's about whether you can tell.
If experienced developers — the people with the best chance of knowing — can't reliably detect whether they're faster or slower, then every productivity claim built on "I feel faster" is noise. Including mine.
And that's the question I keep sitting with. Not "does AI make you faster?" The data on that is mixed and will keep shifting as models improve. The question is: if you can't tell the difference between 24% faster and 19% slower, what else might you be wrong about in how you work?