When your technical audience determines your product’s success, it’s important to provide documentation and other tools to engage them. To that end, you’ll find Developer Experience (DX) teams at many top dev-focused companies. Even those without DX in their titles are likely to put an emphasis on developer success. DX connects your product’s capabilities and its real-world usability, yet many of these experiences are still sub-par. Understandably, there are many attempts to measure Developer Experience—and we’ll share a few in this post.

But measuring DX isn’t a straightforward task. You want to quantify those intangible aspects of ease, satisfaction, and enjoyment that developers get from using your product. However, there are several methods you can use to understand your DX at different levels, which will enable you to make better decisions about how you serve this audience.

In this blog post, we’ll look at different approaches to DX measurement and how you might implement each within your organization. But you can’t even get started until you understand what outcome you actually want to measure.

Know What You Want to Measure

I define Developer Experience as “every interaction a developer has with your company, service, and product.” You can’t focus on everything everywhere all at once. The Developer Experience changes with the journey, so you need to decide what part of the experience you will measure today. In time, you may have multiple measurements.

In general, we see one or more of the following problems when we talk to teams exploring Developer Experience:

• You don’t know where you stand with developers
• You can’t get developers to try your product
• You can’t get developers to use your product
• You can’t keep developers interested beyond Hello World

From these issues, you might already have some metrics in mind, such as Time to Hello World (TTHW) or percentage of developers who reach some level of success. You can quantify these and look for ways to improve them.

On the other hand, you might not know where to start. You could attempt a success measurement, but that could be premature optimization. If you can’t get a developer to try your product, for example, should you really be focused on getting them started faster?

We recommend everyone—even those who think they know the problem they’re solving—to first take a high-level view of their Developer Experience. The elements of great DX are surprisingly connected, so take stock before diving into the details. The next section covers how you can use the same framework we do in our projects at EveryDeveloper.

Look at the Components of Great DX

Before diving into detailed measurements, you should get an idea of how a developer sees your overall Developer Experience. There are small signals—some of which may be out of your control—that can impact a developer’s first impressions. It’s important to know how your company or product fares in these areas, so you can make adjustments, advocate for change, or mitigate their potential effects.

Since 2016, EveryDeveloper has used a Developer Experience Index based on a 1-10 rating to see how closely a product matches the developer ideal. These 13 weighted criteria include elements that speak to every stage of the journey.

The components of a great Developer Experience include:

1. Support for popular languages or frameworks
2. In-depth getting started guides
3. A self-serve option
4. A clear pricing page
5. A free tier or trial
6. An obvious place for questions
7. System status information
8. Accurate reference documentation
9. Sample applications for download
10. Recent technical blog posts
11. Interactive documentation
12. Prominently featured documentation
13. Example command line or dev tool usage

You can use this as a checklist for what you already include, as well as the start of your roadmap to improve Developer Experience.

Not all of these components are equally important to developers, nor must they all be part of every product. Even Stripe and Twilio score less-than-perfect on the ideal. But it can give you an idea of where you fall short, so you can improve areas that will make a difference.

For example, if you have no sample applications or technical blog posts, developers might not know when they would use your product. Or, if you do not have language-specific support, that may stop some developers from attempting your getting started guide. Finally, if there’s no opportunity for self-service, no amount of documentation is likely to overcome the uncertainty of what’s behind the “book a demo” link.

Encourage your team to discuss the 13 components together, or take our free DX quiz to walk through your Developer Experience and receive a self-assessed DX Index rating. With a better idea of where to focus, you can dive deeper into specific problems you want to solve for your audience.

Watch Real Developers Use Your Product

You may have some initial quick wins based on the components of great DX, but you’ll soon identify blockers for your audience. It’s in these circumstances that it can be invaluable to watch actual developers attempt to complete a task. You can grab someone at an event and look over their shoulder, or create a more formal usability study. It takes surprisingly few developers to identify good feedback to improve the Developer Experience.

By now, you should have a good idea of what you want to measure. It’s valuable to focus on one step of the journey at a time—don’t attempt to get all the feedback at once.

Here are some example tasks at each stage of the developer journey:

• First impression: from the home page, answer whether the product can retrieve specific data
• First experience: from the documentation, implement an API call in your preferred programming language
• First success: from the documentation, customize a sample app in your staging environment to share with a teammate
• Last interaction: from a migration document, upgrade an application to use the latest product version

Some of these tasks may require context about your product, but most can be completed by anyone with the technical skills. In fact, sometimes you’ll want someone without much understanding of your product. Your own engineers, for example, won’t provide an objective measurement.

As with getting live developer feedback, the right developers and scenario ensure you’re getting useful data to help you improve their Developer Experience.

After you watch a handful of developers attempt your task, you’ll know how many succeeded and how long they took. Even better, you’ll probably know why they struggled and where to focus your efforts.

Deploy your improved Developer Experience and compare the success of new developers. Or, build these measurements into your product and documentation with data.

Review Your Analytics Data for Patterns

The more certain you are about the problems you want to solve, the more often you’ll be able to measure your Developer Experience. When you’re focused on a specific task in the developer journey, you can look for data to guide your way. In some cases, you’ll already have what you need in existing analytics. More often, you’ll need to identify and capture the signals that lead developers toward success with your product.

When I was part of Zapier’s platform team, we launched a new, developer-friendly command-line interface. We knew it was a more robust way to build resilient integrations. But it was difficult to move partners from the legacy version of the platform. That became a problem we solved with simple analytics.

We could measure:

• The number of people who clicked the old version
• The number of people who clicked the new version

Since these were linked from the same place, we experimented with how those were displayed until we determined how the new product needed to change.

Here are some potential questions you can answer with analytics:

• What are the top clicked links from the documentation home page?
• How many visitors to the getting started page view it for at least 10 minutes?
• What percentage of new users take any action (such as an API call or new project)?
• What are the most common internal referrals to the documentation pages?

There are Developer Experience stories in your data if you know where to look. Once you’ve found something to investigate, you can measure the improvements over time. As with our experience at Zapier, it might test your assumptions, but in the end, you’ll use the data to create a better Developer Experience.

Measure Something, So You Can Improve It

While there’s not a single way to measure Developer Experience, you may use all of these methods over time. There are so many ways a developer interacts with your product, that measurement may feel elusive. What’s important is not to write it off as impossible.

First, identify the outcomes you want, and the roadblocks developers face on that journey. Then, use these methods to measure something—that gives you a benchmark to beat.

Developer Experience is all connected. Remember to look back to the components of great DX to see what else you can use to improve developer trust and enable more of them to have success with your product.