Is Our Organization Actually Measuring Productivity? How Contrasting Organizational and Individual Measures of Engineering Success is an Opportunity to Drive Engineering Transformation

Background: Previous research highlights that common misconceptions about developer productivity lead to harmful and inaccurate evaluations of software work, pointing to the need for organizations to differentiate between measures of production, productivity, and performance as an important step that helps to suggest improvements to how we measure the success of engineering teams. Methodology: Using a card sort, we explored how a Three Layer Productivity Framework was used by 16 software engineers at a Software Engineering focused conference to rank measures of success, first in the current practice of their organization and second in their individual beliefs about the best ways to measure engineering success. Results and discussion: Overall, participants preferred organizations to 1) continue their prioritized focus on performance layer metrics, 2) increase the focus on productivity metrics, and 3) decrease their focus on production metrics. When asked about the current metrics of their organizations, while all roles reported a current focus on performance metrics, only ICs reported a strong focus on production metrics. When asked about metrics they would prefer, all roles preferred more performance metrics but only leaders and ICs also wanted productivity metrics. While all participants were aligned on performance metrics being a top preference, there was misalignment on which specific metrics are used. Our findings show that when measuring developer success, organizations should continue measurement using performance metrics, consider an increased focus on productivity metrics, and consider a decreased focus on production metrics.