Measuring GitHub Copilot's Impact on Productivity – Communications of the ACM
Note
it's been challenging to follow these studies, many of them entangled with the companies themselves desperate for marketing terms to boost sales.
it's fascinating to see the strongest measures of performance tends to come down to how users "feel". it's not a bad thing, and emotions and motivations are important when working with tools.
unfortunately these LLMs are quick to push tight integrations to support indefinite "vibe coding sessions"
when in reality (like with any tool) learning when and how to use these tools makes the difference between actually being productive vs being less productive because you are fighting the tool.
scaffolding unit tests, auto completing basic routines or even prompting a developer with new ideas that can quickly be dismissed feel like the sweet spot for many co-pilots.
certainly, there is room for improvement both in the technology and the UX, but until then, these studies make it hard to put a high price tag on these integrations for most teams.
Published
Summary
Image

