how do you think about measuring business impact when your products aren't directly monetized?

I’ve worked with developer focused tooling for almost 7 years so I know exactly what you mean here. On almost every new feature or product our teams put into motion, we have a huge list of factors to consider such as security, legal, or billing. For developer tooling, there’s usually no change to pricing or billing, no new SKU. Does that mean these types of products don't provide value? No way! Libraries, SDKs, APIs, CLIs, and other developer focused tools are a huge part of the overall product. They can open your product up in a way to create value you never imagined.

When it comes to making a business case for investing in a product feature or measuring value created by these efforts, I look at other ways the business will be impacted. Will this feature help reduce friction in the sales cycle, or is this library a market differentiator amongst similar products? Will change this improve customer sentiment if we roll it out? Will it help accelerate time to value for our customers? Will it help our customers operationalize including our product in their stack and ultimately make our product more sticky within an organization? You can put KPIs, metrics, and even dollar values around so many of these items.

When working with products that are not monetized, it is actually more critical to measure business impact to show the value of your work. Why? Often products that are not monetized, are the first ones to get resources removed when the organizaiton finds itself in a bind. This can be a very big mistake for the organization and why it's critical to get this right.

In this scenario, I'd go back to "what value does your product to your business?". And I would then tie KPIs to that value. Let's say your product is an internal tools that helps payroll to be processed faster. Then likely before your product there were X number of people spent on payroll or Y dollars spent on a service. I would focus business impact on those dollars saved by having your product as a reminder of the monetary value your payroll product brought. Then I'd focus on other items like how you've improved processing speed by delivery these X features in the business. 

Always go with value and then focus on KPIs that showcase that value to your stakeholders.

Products must have some connection back to profitability, helping to either increase income or reduce costs. You otherwise wouldn't want to make an investment unless you're choosing to make a donation to the greater good (e.g. open source). It's OK if that connection is indirect, and in some cases, even difficult to measure. The latter requires leaders to agree that the approach to measurement is inline with the values and product principles of the company. It's easiest to use examples, and I'll go to the extreme to make my point.

Every piece of software has a substrate and lattice work of capabilities upon which all the direct value driving features are built. Take your administrative dashboards, navigation UI, settings pages, notification systems, design systems, and authentication and security features. In the modern web and mobile landscape, it's dubious to think investment in any of these areas can be causal to growth and differentiation. But not meeting the Kano "threshold attribute" means that your product will feel janky and poor quality, which can lead to poor adoption or retention (and good luck with attribution there). Therefore, you need continuous investment just to meet the bar of expectation and that means time away from other KPI driving initiatives. There is no way to get there without the product principles that make space for this type of investment and improvement. Principles have to be paired with health metrics and trip wires that help diagnose the lack of investment (e.g. task completion time, dead clicks, clicks to navigate to common actions, duplicate code, account takeovers due to lack of 2FA, etc.)

I learned the phrase "anything worth doing is worth doing well" from Tobias Lütke. At Shopify, we've created a culture where improvements in many of these examples I shared are celebrated and seen as table stakes. The same is true with things like API performance, UI latency, and UX consistency. All of this takes time and investment, and we uphold it as part of the "definition of done" for most projects. We were a much smaller company at Wave, but still made some investments in our substrate to maintain our perception as the easiest to use financial management software for small businesses.

Let's circle back to products that are not directly monetized, but also not part of the substrate of software. The technique to measuring impact is about identifying the input metrics that ladder up to higher level KPIs that do ladder up to revenue. For example, the ability to do per-customer pricing is a feature expected of business-to-business (B2B) commerce systems, but not direct-to-consumer (D2C) ones. But no merchant adopts a B2B system for that single feature alone, and to some, that feature may not even matter. So while we measure win/loss reasons from the sales team along with churn reasons, we also measure usage rates of the feature and impact of per-customer pricing on average Gross Merchandise Volume (GMV) per merchant. Put another way, we're looking at the relationship of leading metrics and the KPIs that ladder up to, thus telling us how we should invest further in per-customer pricing.

There may be a number of reasons why your product does not involve monetization. Whatever the reason maybe, it is still important to have a definition for ‘success’. For example: 

• It may be an essential platform feature that is table-stakes for your customers, so your strategy is to not charge for the feature. In this case, you may use daily/weekly/monthly usage of the feature to validate customer engagement. 

• It may be a free-product that is meant to drive brand awareness and not direct revenue. Getting to a certain market share with free subscribers as a goal could be used to measure brand recall health.  

• You could be creating products for internal teams to drive operational efficiency. Man hours saved or product quality metrics like average time to solve could be used to track success. 
  

If you want to create an outcome-focused, data-driven product culture, then go beyond just monetized products and have clear success goal posts for every product.

This depends on where the product is in its lifecycle, but in general, product usage (growth of number of users, breadth of features used, and frequency of usage) are great indicators of product success. There are also other metrics you can measure to determine if a customer is getting value from your product - one example is if they are sharing results from your product or inviting others to the product. This is indicative of value and an interesting metric.