In a mission to measure AI-assisted developer productivity, GitHub researchers recently conducted an experiment comparing the coding speed of a group that used the Copilot code completion tool to a group that relied solely on human ability. I did.
GitHub Copilot is an AI pair programming service that opened to the public earlier this year for $10 per user per month or $100 per user per year. Since launch, researchers have wanted to know if these AI tools really lead to increased developer productivity. The problem is that it’s not easy to identify good metrics to measure changes in performance.
Copilot is used as an extension for code editors such as Microsoft’s VS Code. Generate code suggestions in multiple programming languages that users can accept, reject, or edit. Suggestions are provided by OpenAI’s Codex, a system that converts natural language into code. OpenAI’s GPT-3 language model.
look: What is coding and what is it used for? A beginner’s guide
Google Research and the Google Brain Team concluded in July after studying the impact of AI code suggestions on the productivity of more than 10,000 in-house developers. The relative performance speed debate remains an “open question”This is because using a combination of a traditional rule-based semantic engine and a large language model such as Codex/Copilot “can significantly improve developer productivity with better code completion.” Despite the conclusion
But how do you measure productivity? Other researchers earlier this year used a small sample of 24 developers to found Copilot didn’t necessarily improve task completion times or success rates. However, I found that Copilot saved developers the effort of searching for code snippets online to solve a particular problem. This is a key indicator of how much AI tools like Copilot can reduce context switches when developers jump out of the editor to solve problems.
GitHub too Survey of 2,600+ developers, “Do people feel more productive with GitHub Copilot?” The researcher also has unique access to large-scale telemetry data and Research published in JuneAmong other things, researchers found that 60% to 75% of users feel more satisfied with their work when using Copilot, feel less frustrated when coding, and can focus on more satisfying tasks. .
“Our research shows that GitHub Copilot helps reduce completion times, saves developers mental energy, helps them focus on more satisfying work, and ultimately helps them get more done with the coding they do. We know you find the fun,” GitHub said.
GitHub researcher Dr. Eirini Kalliamvakou explains this approach: What do you infer from telemetry? (b) Is qualitative feedback generalized to a large user base?”
Kalliamvakou, who worked on the original research, builds on this research by conducting an experiment with 95 developers that focused on coding speed issues with and without Copilot.
The study found that a group of 45 developers using Copilot completed the task in an average of 1 hour and 11 minutes. The non-Copilot group (50 developers) completed in an average of 2 hours and 41 minutes. So the group with Copilot was 55% faster than the group without.
Kalliamvakou also found that the group using Copilot completed tasks at a higher rate. He was 78% in the Copilot group versus 70% in the non-Copilot group.
This study is limited in nature as it only compares developer speed when coding web servers in JavaScript and not other tasks involving other languages such as Python or Java. Also, I didn’t rate the code quality.
Also, this experiment did not examine factors that contribute to productivity, such as context switching. But a previous GitHub survey found that 73% of developers reported that Copilot helped them stay in flow.
In an email, Kalliamvakou explained to ZDNET what the numbers mean in terms of context switching and developer productivity.
“Reporting ‘staying in the flow’ certainly means less context switching, and we have additional evidence. 77% of those surveyed said they spend less time searching when using GitHub Copilot reported,” she wrote.
“This statement measures known context switches for developers, such as searching documentation or visiting Q&A sites like Stack Overflow to find answers or ask questions. Developers frequently use IDEs to bring them into the editor,” she explained.
But measuring productivity gains from AI code suggestions using context switching alone doesn’t tell the whole story. Also, measuring the impact of context switching is difficult because there are “good” and “bad” context switches.
look: Data Scientist vs Data Engineer: Changing Demand for These Roles
During a typical task, developers frequently switch between different activities, tools and information sources, Kalliamvakou explains.
she pointed Research published in 2014 We found that developers spend an average of 1.6 minutes in an activity before switching, or switch an average of 47 times an hour.
“Because of the nature of their work and the multitude of tools they use, it’s considered ‘good’ context switching. In contrast, there is ‘bad’ context switching due to delays and interruptions,” she said.
“We found in our previous research Not only does this greatly reduce productivity, but it also affects the developer’s own sense of progress. Context switching is difficult to measure because there is no good way to automatically distinguish between “good” and “bad” instances. Alternatively, switching may be part of completing a task and causing a break in developer flow and productivity. But there are ways to measure context switching through self-reports and observations that we use in our research. ”
Regarding Copilot’s performance in other languages, Kalliamvakou says he’s interested in experimenting with it in the future.
“It was certainly a fun experiment. These controlled experiments are going to take a lot longer as we try to be bigger and more comprehensive, but I’d definitely consider testing other languages in the future.” ‘ she said.
Kalliamvakou posted other important findings from a large GitHub survey. in a blog post It details our quest to find the best metric to measure developer productivity.
