How do you know when a new feature or an update has been successful?

If you are an outcome driven product team (which I hope you are), I think this should be defined as the change in behavior you expect to see/create for users of your new feature or product. Outputs are easy, outcomes are hard. Before you ever start working on the output (e.g. building) you should clearly be able to articulate:

• What is the problem we are trying to solve

• What will be the change in user behavior if we solve the problem

• How will we know we created that change

• What are the measures we need in place to ensure we can track the change in behavior

I tend to focus on two groups of indicators.

First, did we deliver a delightful user experience that added value to the customer? 

My go-to for answering this is Google’s HEART framework: Happiness, Engagement, Adoption, Retention, and Task Success.

• For Happiness, I look at metrics like user CSAT or NPS.

• For Engagement, I consider metrics such as average session length.

• For Adoption, I focus on key activation metrics (e.g., inviting a friend to join, publishing a first post, etc.).

• For Retention, I monitor our churn rate.

• For Task Success, I evaluate key activities we want users to complete and monitor their completion rates.

Note that it’s not necessary to measure every single metric in every category for each project.

Secondly, did we add value to the business? 

Ideally, your product benefits not only your customers but also your organization. The specific metrics to consider will depend on your goals, but here are some examples:

• Did we help the business acquire new customers?

• Did we increase conversion from a free version of the product to the paid version?

• Did we enhance overall 'stickiness', leading to lower churn and increased customer lifetime value?

• Did we drive more cross-sell or up-sell, resulting in higher revenue per customer?"

First, any new feature or update should have pre-defined success criteria/metrics that are set before launch. That helps in objectively assessing the success or failure of the feature/update. The bigger/more requested a feature, the more important it becomes to validate that it was successful with metrics. As a general rule, I don't ascribe to the "launch first and we'll figure out what to measure after" approach as that can lead to measuring vanity metrics instead of user or business outcomes.

Generally speaking, the top metrics to measure for any new feature/update are:

• adoption (of new feature)

• retention (users continue to use that feature after n weeks/months)

• engagement (did users start using the product more because of this feature)

There could be other metrics in addition to the above depending on what the feature is and what additional outcomes are expected.

Key Performance Indicators (KPIs) should be identified for every feature and those should then be monitored to determine the effect of the change. This can be the number of trouble tickets filed if fixing a problem, to the average response time (non-functional change), to the use of a particular feature. Depending on the product, blue/green deployments could be used to determine if user behavior changes such as when adding a new feature to the solution.

This is not unique to a mature product per se, however a mature product normally has an established customer base which can be surveyed, but I would argue that even then, identifying a proper KPI and using that would yield more accurate results since surveys are costly, error prone and then to suffer low response rates.

The best PMs define success before building a feature. To define success, teams should think through:

1. The customer problem they're solving, what does it look like when solved (for the customer and the business)? 
2. How impactful is it if solved (time-saving, cost-saving, revenue-generating)?
3. How does solving this problem help the company achieve its longer-term vision/strategy? Is there a KPI or measure for the feature that helps the team know they're moving toward that strategy?

Once the feature is launched, there are a few different ways to determine whether a new feature or update has been successful:

• Track usage metrics: such as the number of users using the feature, the frequency with which they are using it, and any changes in retention or engagement.
• Gather customer feedback: This can be done through surveys, user interviews, or tracking social media mentions of the product.
• Monitor for changes in key performance indicators (KPIs): Depending on the goals of the feature or update, you can also monitor for changes in relevant KPIs, such as revenue, conversion rate, or customer satisfaction.
• A/B test: If you're unsure how a new feature or update will be received, you can run an A/B test to compare the performance of the new feature or update against a control group.

Lastly, a helpful question that Sean Ellis has popularized is the PMF test: ask users, “how would you feel if you could no longer use the product?” and measure the percentage who answer “very disappointed.” After benchmarking nearly a hundred startups with his customer development survey, Ellis found that the magic number was 40%. This can be a helpful tool to identify users to interview further and segment your customer base around those customers that find a lot of value in your product that you should be targeting.