AMA: Twilio Segment Staff Product Manager, SDKs and Libraries, Tasha Alfano on Product Management KPIs

The framework I use for setting KPIs for a new product or area typically looks something like this: 1. Framing: I think big picture about the product, what is the purpose, and how can I measure whether or not we are tracking towards that ultimate purpose. This can even be thought of in terms of a user story, “As an [x] user, I want to be able to [y], so that [z].” 2. Brainstorming: I like to start with many different categories that we can consider for KPIs such as usage, adoption, customer sentiment (NPS), sales friction, impacted ARR, tangentially related products, direct ties to business value, the list goes on. During this time I recommend not curating the list. 3. Crowdsourcing: At this point, enlist your team to keep building off this list. I can’t say this enough - Product Management is not a one person team. I just went through some pretty big launches and the core working team of PMM, engineering, product operations, customer success, and PM all collaborated on the many facets of the launch. Work is way more fun this way and you get the benefit of an entire teams collective experience. 4. Prioritizing: This is where the focus comes in. I recommend whittling down your target KPIs list to a few items that you can talk about and share often to really focus on. I'm definitely guilty of having a ton of different items on my dashboard, and those extra things can definitely be useful to understand what is happening with your product in the market. When it comes time to share cross functionally or with a leadership team, I find that fewer, focused targets can be useful. 5. Measuring: Whichever framework you like to work with, OKRs, KPIs, BPMs, make sure you are able to measure your targets. 6. Iterating: Start tracking, making reports or dashboards, sharing, and analyzing. Without the analysis we just have data points on a page. Ask for feedback and adjust when needed. When in doubt, share again! 

Developer Audience - now we are talking! As someone who worked as an engineer using SDKs, then building SDKs, and then moved into Product Management, I have a lot of opinions on this! We can definitely think about a ‘developer persona’ the way we all have different personas for our respective products, but all developers are definitely not the same. Building for different development platforms can add another layer of differentiation and complexity as well. I have a secret though, when you work on developer tools, you already have an amazing pilot group - the engineering team building the product! The teams I work with come to customer meetings, drive discussions with our users, write documents, and pitch new ideas. They are your biggest untapped resource if you’re working on developer focused tooling. If I'm struggling with a decision or just need a pulse check on something, the team is the perfect group of people to ask! As I mentioned before, I'm always tagging Engineering, PMM, and other team members early and often in documents or analysis as soon as I start on something new. My second biggest recommendation for working for a developer audience is to to think about all of the tangential items related to the actual product; documentation, examples, interactive apps, or code comments. In my experience, developers love an easy to use library, and they also value transparency and good documentation. The other thing I cannot reiterate enough for developer tooling is the change management aspect. When developers depend on your libraries or APIs, making a change has to be thoughtful. Whether it is promoting a library from Beta to GA, or versioning your library, you have to maintain transparency as well as versioning expectations. Breaking someone else's usage of your tools is no fun at all. This is especially important as you deprecate or end of life a tool, the notice you need to give a developer is much different than making a change to a view in a UI.

I’ve worked with developer focused tooling for almost 7 years so I know exactly what you mean here. On almost every new feature or product our teams put into motion, we have a huge list of factors to consider such as security, legal, or billing. For developer tooling, there’s usually no change to pricing or billing, no new SKU. Does that mean these types of products don't provide value? No way! Libraries, SDKs, APIs, CLIs, and other developer focused tools are a huge part of the overall product. They can open your product up in a way to create value you never imagined. When it comes to making a business case for investing in a product feature or measuring value created by these efforts, I look at other ways the business will be impacted. Will this feature help reduce friction in the sales cycle, or is this library a market differentiator amongst similar products? Will change this improve customer sentiment if we roll it out? Will it help accelerate time to value for our customers? Will it help our customers operationalize including our product in their stack and ultimately make our product more sticky within an organization? You can put KPIs, metrics, and even dollar values around so many of these items.

In the past I've worked on new library launches, major upgrades, and I'm actually in the process of an end of life (EOL) effort right now. While these types of efforts have their own nuances, the core principles remain the same. 1. Transparency - When working on developer tools you have to maintain the highest level of transparency possible. Systems, developer workflows, and customer data can all be dependent on these tools and their interfaces. 2. Communication - The way and the cadence that you distribute information about changes to libraries or tools goes hand in hand with transparency. You need to ensure that you are giving your end users enough time to adjust/migrate, test, and be confident in using the next version or tool in a production environment. 3. Cross Functional Teamwork - Assembling a group of team members from across the organization will ensure that you are considering the customer/developer experience as well as any edge cases. Working with a team like this enables you to be really thoughtful about what to communicate (transparency), and when and where to share it (communication). I like to keep a group like this going even after a launch or major change for a little while so that you can hear feedback and areas to improve.

The partnership with Product Marketing is one of the most important functions when it comes to rolling out a successful product. Don’t read too much into the last part of that statement though, a Product Marketing Manager (PMM) is a crucial teammate to include from the start, not just at launch time. If I know who I will be working with in advance, I tag them in Product Requirements Documents or other important materials. Sharing context from the beginning is so important! As far as the KPIs go, we are really talking about how you measure the success of a product, and Product Management and Product Marketing should be aligned on this. If you're the Product Manager, you’re ideally setting the big picture targets and KPIs for the initiative, and working with a cross functional team, such as PMM, Engineering, and Product Ops, to make sure these are shared targets. Part of the product rollout might include working with PMM on a webinar to drive awareness, and the PMM will likely have their own targets for things like webinar attendance, and all of these items build on each other. Ideally the webinar drives awareness, which in turn helps drives adoption.

The worst KPIs to commit to are the ones you can’t commit to at all. We can set targets and metrics and make dashboards, but that’s exactly what they are - targets. I recommend looking at past performance and trends within the data and setting a realistic yet aspirational target to work towards. After that, begin iterating on your target. Revisit the KPI, analyze, adjust, and communicate your findings.

Congratulations! I’ll answer your question with a few questions of my own. 1. Purpose: What do you want to drive? What does your organization expect you to own? In Product Management there is a lot of autonomy, but it can also feel tempting to take on everything that needs an owner. 2. Outcomes: If you don’t own a specific KPI, what will happen? In a year, will this KPI matter? What about in 3 years? Does the KPI support the goals of the business in a real way? I’d recommend starting with 2-3 target KPIs that tie directly to your purpose. From there, make the targets visible through dashboards or reports and constantly revisit them. Ask your colleagues for feedback, and never be afraid to make adjustments! Also, try to be aware of the impact these measurements are creating. Do you find yourself referring to these targets often, do they impact decision making? One of my colleagues likes to say the best way to know if a report is useful is to stop sending it and see what happens.