AI tools do not make experienced developers more productive. Contrary to what many developers expect (and experience), the use of AI slows down development work by 19 percent in practice.
This is according to the study Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, conducted by the research institute METR.
METR followed sixteen developers with an average of five years of experience who contribute to huge codebases while working on programming tasks from their repositories. The tasks ranged from bug fixes to new features and took an average of about two hours per task.
A randomized controlled trial divided the participants into two groups. One group had access to AI tools, including the programming assistant Cursor Pro and advanced models such as Claude 3.5 and 3.7, and one group did not have such assistance.
Unjustified trust in AI
The results show that developers using AI took an average of 19 percent longer to complete their tasks than those working without AI. Remarkably, both the participating developers and external experts predicted in advance that AI would lead to time savings of up to 24 percent. Afterward, many participants continued to believe that AI helped them, despite the measured delay.
According to the researchers, there are several possible explanations for this unexpected result. For example, the AI tools often made irrelevant or incorrect code suggestions, which took time to evaluate and correct. The tools also struggled to understand complex project structures, and developers lost time due to context switches and waiting for AI suggestions. Other potential explanations, such as lack of experience with AI tools or distraction by new functionalities, did not prove decisive in this study.
AI impressive for novice programmers
The study raises important questions about the current enthusiasm for AI in software development. Although AI shows impressive results on benchmarks and with novice programmers, this does not automatically mean that experienced developers will benefit from it in complex, realistic work environments. The researchers emphasize that it remains useful to test such claims empirically before companies or teams switch to AI-assisted programming on a large scale.