Measuring Developer Productivity—Will We Ever Get It Right?

We’re lucky to have an active and engaged community of CPOs and CTOs in the Edison portfolio who have much more than a passing interest in developer productivity. Our CEO community is also interested in this topic for obvious reasons; given that Software Engineering headcount is either the largest or second largest operating expense in most of our portfolio companies, developer/development productivity is key.  Not to mention, every company in the portfolio is either a software company or a software-enabled company, so productivity in the context of delivering value on a timescale of relevance is critical.   

Okay, so everyone is aligned around the importance of developer/development productivity.  That must mean we’re making great progress in agreeing on a framework and a set of metrics for making individuals and teams as productive as possible, right? Wrong! In my view, we’re still struggling to come up with the right framework and set of metrics—and for those of us in my age group (I got started in 1982 in tech), this condition has existed for a long, long time.  That said, I think the SPACE framework comes the closest to nailing it, if there is such a thing on this topic—more on the framework in a minute.  So, I asked our community of CEOs to read Nicole’s well researched article, “The SPACE of Developer Productivity” and come prepared to discuss a series of questions (below, including some representative responses). I would ask you to do the same, but I provide a teaser on the framework (directly out of the article) below: 

Here's a summary of the SPACE framework.  It really doesn’t do the white paper justice but should provide enough of a teaser/context to get you interested in reading it.  That said, it will get you thinking and will also provide some context for our CTO responses below.  

SPACE = Satisfaction/Well-being, Performance, Activity, Communication/Collaboration, Efficiency/Flow 

• Satisfaction and well-being—in short, satisfaction is how fulfilled developers feel with their work, team, tools and culture. Well-being is how happy and healthy they are.  Satisfaction and well-being appear to be correlated to productivity and vice versa. 
• Performance—a controversial measure because of the nature of software development as a team sport; that said, evaluating outcomes as opposed to outputs appears to be the best way to measure performance. Software quality and impact are examples of outcomes. 
• Activity—relevant, but to be used with caution given the nature of software development (developers helping others, attending team meetings, brainstorming, etc.). Activity measures for design and coding (# of pull requests, commits, code reviews etc.), continuous integration and deployment (count of build, test, deployment/release), and operational activity (on-call participation, incident mitigation, etc.) are examples. 
• Communication and collaboration—how people and teams work together. This is obviously important to anyone familiar with software development.  Metrics that can be used as proxies include discoverability of documentation and expertise, onboarding time for and experience of new members, pace at which work is integrated, quality of reviews of work contributed by team members, and network metrics showing who is connected to whom and how. 
• Efficiency and flow—efficiency impacts all the SPACE dimensions. Efficiency at the individual, system, and team level are correlated with developer satisfaction. Metrics include number of handoffs in a process, number of handoffs across different teams, ability to stay in flow and to complete work without context switching, interruptions (quantity, timing, impact), time measures through a system (total time, value-added time, wait time).

Here are the questions we teed up during out Roundtable for discussion.  I have included some representative responses from our CTOs. Only a few of the folks were familiar with the framework and white paper.  

• Do you buy what they’re selling?  Does the framework make sense to you?  Where do you agree/disagree with their findings? 
• “The human tendency is to measure stuff that is easy to measure and treat it as the one thing that is of supreme importance.

 This article does a great job of dispelling that notion and urging us to focus on the softer aspects such as wellbeing, that may be hard to measure, but are nevertheless hugely important in the long run.”  Subra Viswanathan, CTO at Solutions By Text. 

• “I generally agree with them.  However, I think there are even bigger issues that could be impacting productivity - such as organizational alignment, work in process, viewing development activities as projects vs. managing a product.  Engineering leadership is also critical.  Every engineering team, every company, every leadership team is different, and it is critical to understand what you have and where you are coming from.  For example, code reviews are tricky and often not effective.  I've found pair programming much more effective because you have two experts looking at the code and working together.” Rich Howarth CTO at Terminus.   
• What challenges do you see with implementing this framework?  
• “Changing mind set, getting folks to understand and accept that the intangibles are just as important as the tangibles, this has to start top down.” Helen Johnson, CTO at Comply.
• “The challenge will be to identify and measure satisfaction, collaboration and efficiency.  Current tools (Jira, for example) do not provide an easy way to measure things like hand-offs within and between teams.  So, much of this will need to be built from scratch, and in organizations that are already stretched, investing in-house efforts to identify, design and implement these measurements will be challenging.  On the other hand, just the knowledge that Activity and Performance metrics alone are not the be-all and end-all can help to look around metrics trends and implement creative corrections.”  Subra Viswanathan, CTO at Solutions by Text. 
  
• How do you think your CEO would react to it? 
• “Favorably.  Maybe not favorable enough to set aside capacity and investment to implement all of them.  But I think the most impactful at the organization level will be ‘S’ part of these metrics. Slicing bi-annual survey metrics across organization dimensions and maybe even at team levels will provide valuable insights that will help sidestep potential longer-term risks that can be very hard to solve once they become issues.” Subra Viswanathan, CTO at Solutions By Text 
• How do you measure individual, system, team productivity today? 
• “Sprint spillover, Story points, Deployment frequency, Cycle time – all at the team and higher levels” Subra Viswanathan, CTO at Solutions By Text. 
• “By delivery of customer value - the only thing that really matters.” Rich Howarth, CTO at Terminus. 
• “Velocity, Predictability, New dev vs. maintenance vs. production issues” Helen Johnson, CTO at Comply. 

Overall, our community thought the article and framework was thought provoking at worst and timely/on point at best. The white paper’s focus was on how to fairly measure individual developer productivity. That said, the framework makes it clear that you almost can’t consider individual productivity without looking at the Individual, Team, and System. Taken together, we get a clearer picture of both individual and team/system productivity and effectiveness. Folks agreed that putting real focus and value on Satisfaction and Well-being as well as Communication and Collaboration provides a more complete picture of what impacts productivity. Performance, Activity, and Efficiency are each valuable; they just need to be used appropriately and in context. Like so many other management tools, SPACE is not meant to be used by buffoons. It expects us to use our leadership skills, experience, and general business acumen to leverage it properly.