ManuAIVI

AI papers, explained · Issue 1 · Published 26 October 2026

Experienced developers felt faster with AI. The clock said they were slower.

A randomised trial timed 246 real coding tasks, with and without AI. The gap between what people felt and what the clock measured is the story.

The paper

Title
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Authors
Joel Becker, Nate Rush, Elizabeth Barnes, David Rein
Research group
METR
Date
July 2025 (arXiv, revised 25 July 2025)
Type of study
Randomised controlled trial, 16 developers, 246 tasks
Open the paper

In three lines

  1. Sixteen experienced open-source developers did 246 real tasks from their own projects, and a coin decided for each task whether AI tools were allowed.
  2. They expected AI to cut their time by 24%, and afterwards believed it had cut it by 20%. Measured on the clock, tasks with AI took 19% longer.
  3. That result covers early-2025 tools and very experienced people on huge, familiar codebases. METR's own 2026 follow-up says it is now out of date.

The question

By early 2025 plenty of people said AI made them code faster, and benchmark scores for AI coding kept climbing. Very little of that had been checked on real work, done by experts, on software they know well.

METR, a research nonprofit, asked a narrow question: when experienced developers work on their own mature open-source projects, does allowing AI tools change how long a task takes?

How the study worked

  1. 01

    Pick the people

    Sixteen developers who regularly contribute to large, popular open-source projects. On average they had about five years of history on their project, and most had more than ten years of experience as engineers. The projects averaged around 23,000 GitHub stars and more than a million lines of code. Each developer was paid $150 an hour.

  2. 02

    List real work first

    Each developer listed issues they would have worked on anyway: bug fixes, features and refactors. Tasks longer than two hours were split up, and the average task took about two hours.

  3. 03

    Write down a guess

    Before anything was assigned, the developer estimated how long each task would take with AI and without it.

  4. 04

    Let chance decide

    Each task was then randomly marked AI allowed or AI not allowed. 136 tasks ended up in the first group and 110 in the second. With AI allowed, people could use any tools they liked; most used Cursor Pro with Claude 3.5 or 3.7 Sonnet, which were among the strongest models at the time. Most tasks were done in February and March 2025.

  5. 05

    Time it and record it

    Developers recorded their screens and reported how long each task took. The researchers labelled 143 hours of those recordings, about 29% of all the time spent, to see where the minutes went.

  6. 06

    Ask again afterwards

    When the study ended, developers estimated how much AI had actually sped them up. The team also collected forecasts from 34 economics experts and 54 machine learning experts.

What they found

Bars to the right mean less time with AI, bars to the left mean more time. Everyone who guessed expected a big saving. The measurement went the other way.

Time saved with AI: predicted, felt and measured
  • Economics experts' forecast (Predicted)39% less time

    34 experts, asked in advance

  • Machine learning experts' forecast (Predicted)38% less time

    54 experts, asked in advance

  • Developers' own forecast (Predicted)24% less time

    Before their tasks were assigned

  • What developers felt afterwards (Felt)20% less time

    Estimated once the study was over

  • What the clock measured (Measured)19% more time

    Tasks with AI took 19% longer. 95% interval: 2% to 39% longer

PredictedFeltMeasured95% confidence interval

  • The 19% comes from a model that adjusts for how long developers expected each task to take. Tasks that landed in the AI group happened to be forecast as slightly harder, so the raw gap (34%) overstates it. The paper says other ways of estimating gave similar results.
  • The recordings showed where the time went. With AI allowed, people spent less time writing code and searching for information, and more time writing prompts, waiting for the AI (about 4% of their time) and reviewing or cleaning up its output (about 9%). They also sat idle a bit more often.
  • Developers accepted fewer than 44% of the code suggestions Cursor generated, and most said they had to make major changes to clean up AI code.

Why it may have happened

The authors looked at many possible explanations. These are the ones they found evidence for.

  • People expected too much

    Using AI was optional on the AI-allowed tasks, so in theory nobody had to lose time to it. The authors suggest that because developers believed it was helping, before and after, they may have kept using it where it was slowing them down.

  • They already knew the code very well

    The slowdown was larger on tasks people rated themselves as expert at. Developers said their experience made it hard for the AI to add anything.

  • The projects were big and complicated

    Developers said the AI did worse in large, complex codebases. These averaged ten years old and over a million lines.

  • The output was unreliable

    With less than half of suggestions accepted, time went into reading, testing and fixing what the AI produced.

  • Much of the context was in people's heads

    Unwritten rules about how a project works were hard to hand to the tool, so its code often missed them.

How much to trust it

What makes it solid

  • Chance decided which tasks got AI, so the two groups were comparable and the comparison is fair.
  • The tasks were real work the developers listed before knowing the condition. Nobody picked tasks to make AI look good or bad.
  • Guesses were written down before the work and again after, so the gap between belief and measurement is measured directly.
  • The authors examined 21 possible explanations in the paper, including the possibility that the study setup itself caused the slowdown, and judged that unlikely to be the main cause, while saying it cannot be fully ruled out.
  • The data and methods are public, and METR published the result even though it went against what they expected.

What it can't tell you

  • Sixteen people is a small group. METR's post says 246 tasks gave just enough statistical power to tell the result apart from no effect at all.
  • These were unusually experienced developers on huge, mature projects they knew by heart. The authors say plainly that this does not show AI slows down most developers, or people in other kinds of work.
  • Only 44% had used Cursor before the study. Gains that come after more than about 50 hours with the tool cannot be ruled out.
  • Developers who agreed to join may not be typical; some experienced people may have said no.
  • The tools were from early 2025. Models and coding agents have changed a lot since, which is exactly what the 2026 follow-up below tries to measure.

Our rating

  • How well the study was run

    A real randomised trial on real work, with recordings. Small, but carefully done.

  • How much it says about today's tools

    Early-2025 tools. METR itself now calls these results out of date.

  • How much it says about you

    Unless you are a veteran on a giant codebase you know by heart, your numbers could look quite different.

  • How solid the lesson about guessing is

    Forecasts before, estimates after and the clock all point the same way: people misjudge their own speed.

Bottom line

Trust the method and the gap between feeling and measuring. Treat the 19% as a snapshot of one group in early 2025, and keep your own clock running.

What happened next: the 2026 follow-up

In August 2025 METR started a larger round: 10 developers from the first study plus 47 new ones from a wider mix of projects, including smaller and newer ones. That came to 57 developers, 143 repositories and more than 800 tasks. Pay dropped from $150 to $50 an hour.

In February 2026 METR said this round gives an unreliable signal, and that they are changing the design. A growing share of developers did not want to work without AI at all, so they declined to join. When surveyed, 30% to 50% of developers said they held back some tasks because they did not want to do them without AI. People running several AI agents at once found it hard to report how long a task really took.

Because of this, METR thinks the new estimates are likely too low. From talking to participants, they believe developers are probably more sped up in early 2026 than in early 2025, while saying their own data is only very weak evidence of how much.

Later estimates, with METR's own warning that they are probably too low
  • Returning developers, late 2025 (Measured)18% less time

    About 18% less time. 95% interval: 38% less to 9% more

  • Newly recruited developers, late 2025 (Measured)4% less time

    About 4% less time. 95% interval: 15% less to 9% more

Measured95% confidence interval

Try it on your own work

Tick them off as you go. Ticks stay in this browser.

Time it yourself

Write down your guess, flip a coin for ten tasks, and let the clock tell you whether AI saves you time. Nothing leaves your browser.

Open the calculator

Sources

  1. 1.Becker, Rush, Barnes, Rein (2025), arXiv:2507.09089The paper. Methods, appendices and the full list of factors.
  2. 2.METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (10 July 2025)METR's own summary, its list of what the study does not show, and an FAQ.
  3. 3.METR: We are Changing our Developer Productivity Experiment Design (24 February 2026)The follow-up numbers, the confidence intervals and why the design is changing.