Developer Productivity Survey

Example survey, Example analysis

While you should rely on your organizational metrics to measure developer productivity, quantitative measurement will sometimes miss important context. For example, you might be proud of how the backend developers are having a great time with their CI/CD, only to realize that the iOS engineers hate their release process that isn’t instrumented in any of your dashboards. A Developer Productivity Survey is an effective tool to bring qualitative feedback into your planning process and reduce your risk of meeting your metrics while missing your goal.

Most organizations run these surveys twice a year, with a focus on answering two questions:

• Where should you prioritize your efforts next?
• What unexpected problems or opportunties were identified?

For modest sized companies (e.g. less than thousands of engineers), surveys are often not the most effective way to measure your organization’s productivity, but a phenomenal way to better understand the developers you support.

Example analysis of Developer Productivity Survey

There are a wide range of topics you can choose to cover, and you’re going to want to balance focusing on areas you know are important and creating space to learn things you don’t already know. Some of the topics that these surveys often explore are: codebase quality, migration impact, code review experience, testing experience, CI/CD experience, and documentation.

You’ll have to figure out the right questions for your current organization, but as you design the survey remember the golden rule of survey writing: only collect information whose value you can show to the folks sharing their time to answer it.

Before You Start

Developer Productivity Surveys are a useful tool for determining what to focus on next. However, keep in mind that each time you send a survey, you’re entering into an implicit contract with the folks that you send it to: you will do something to address their concerns.

No one expects you to address all their concerns, but you should not send a survey if your schedule is already committed with other, unmovable work. If you do, you’ll find that you get fewer and fewer valuable responses over time, and that your survey becomes less valuble over time.

How to Use

1. Explore broadly for topics to consider focusing on for this survey. Remember that you’re trying to identify the areas to improve and check if your previous work has improved areas as intended. You are not trying to measure productivity, you have metrics for that. For inspiration look at others’ surveys (ex: one, two, three), or recent findings on developer productivity like State of the Octoverse or Accelerate State of DevOps Reports

2. Review your previous surveys, if any, and identify any questions where it would be valuable to have an ongoing dataset. You will need to make tradeoffs between the value of ongoing datasets (created by asking the same question in multiple surveys over time) and the desire to tweak a question

3. Determine the segments you’ll want to capture to understand the data. For example, web, mobile and backend engineers almost always have different toolchains and release processes. If you treat them as a uniform group, your analysis is likely to lead you down a confusing path. Some segments to consider are: primary toolchain, time at company. time in industry before joining this company, current role (engineer, engineering manager, etc), and current level (engineer, sr engineer, etc)

4. Combine the questions you’ve identified into your survey. Use the tool of your choice, Google Forms and Airtable are both good options. Review this example survey for suggestions. (You’ll need to create your own as Google Forms are not easily cloned.)

5. Review the proposed form with three or four engineers on your team. What is confusing? What questions were they unable to answer? Revise your questions based on their feedback

6. Announce your survey across communication mediums (e.g. Slack, email and your Engineering All Hands), and over the duration of your survey (e.g. when it launches, a week later, and a few days before it closes)

7. Once the survey closes, analyze the responses. Take inspiration from this example analysis. Don’t spend a single second worrying about whether you agree with the feedback, instead focus on identifying what’s new. Prefer looking at segmented data, particulary segmented by primary toolchain, rather than the entire corpus. Segmented data gives a much clearer view into the reality underlying the data

8. Within your team, select 2-3 areas that you’re going to prioritize improving before the next survey runs

9. Communicate the selected investment areas, how you’ll measure improvements, and your target improvement and share this out to the folks who you asked to take the survey. Showing impact is the best way to incentize future participation

10. Now you’re done! Sort of, anyway. You need to actually make those improvements you’ve committed to, and you’ll be running another survey soon!

Tips

• Many organizations set goals related to Developer Productiviy Survey results. This is a great practice when you connect it to the specific areas you’re going to invest in your current quarter or half, e.g. “0% Java developers no longer rank build times as their biggest blocker (down from 40% in Q1)”. On the other hand, it’s generally an antipattern to maintain goals on sentiment, e.g. “100% of developers feel highly productive”. The later goal is too broad to usefully measure the quality of your current work: even if you hit your target, it’s likely due to confounding factors
• Metrics are usually more effective than surveys once you’ve fully instrumented your various processes and pipelines. However, there are many cases where your process and tooling isn’t going to be instrumented anytime soon, in which case surveys are certainly better measures than having no measures at all
• It’s easy to get carried away asking too many questions in a given survey. As a rule, you should start small and slowly expand your survey over time until you see it impacting response rates. If response rates dip, pull back on size. There will be pressure to use this survey to solve every problem, but don’t fall into that trap
• Timing your survey is important. Try to run it at least a month before your planning process so that you have time to run the survey and analyze the results before you submitting plans
• Some companies attempt to also include employee engagement in this survey, but I’d recommend using a different tool dedicated for employee engagement, probably Culture Amp, so that you can see how behavior varies across teams and organizations.