Measure AI code tools on real work, where developers finished 26% more tasks

This image shows a Cedar Waxwing from John James Audubon's The Birds of America (1827-1838).

Real work showed a smaller gain than lab tests. But the gain was still large.

The study. Economists from MIT, Princeton, Microsoft and other institutions ran random tests with 4,867 developers at 3 companies. Some developers got GitHub Copilot earlier than others.

The finding. Developers with the AI tool finished 26.08% more tasks. The researchers counted tasks as pull requests (requests to add new code to a project). Less experienced developers used the tool more and got more benefit.

Why it matters. Lab tests showed very large speed gains on simple tasks. This study measures real work in real companies. The gain is smaller, but it is still large for an expensive team.

The takeaway. Measure AI tools with real output, such as finished tasks. Compare teams with the tool and teams without it.

Want to read more?

Subscribe to plaintalkbusiness.com to keep reading this exclusive post.

Subscribe Now