Mike Arcuri

This really depends on the maturity stage for your product. For 0-1 work in startups (new products, not-yet-released), I recommend the models explained by Steve Blank (e.g. customer development process, business model canvas). There are so many unknowns and intertwined hypotheses for brand new product ideas that all need to be validated, iterated on, and improved until the whole product concept and its acquisition and retention funnels are working. The business model canvas helps clarify the scope of all this and the risk in each area. You generally want to report on new information as soon as it's available, but frequency will vary (e.g. maybe daily for marketing funnel performance, and once a week for learnings from sales calls). For live products that have active users, I'd recommend: 1. focusing on engaging and retaining users first, 2. tracking these metrics daily, and 3. evaluating your product improvement hypotheses based on whether they improve engagement and retention. When retention is in a good place, then shift to growth - first by finding more users like your current users, and then by expanding to adjacent and new audiences. Continue monitoring and reporting on engagement and retention, and now also track daily growth metrics. Your high level product strategy should make it clear which maturity phase you're in and what markers you're aiming for in order to "graduate" to your next phase. E.g. "Stage 1 out of 5 for our product strategy is building consumer retention. When we achieve XYZ in terms of retention, we'll be ready to transition to Stage 2/growth." You likely also have a strategy for succeeding at the current phase (e.g. "here's how we will out-compete product alternatives to earn people's engagement and retention in Stage 1").

When you're launching a new feature in an existing app, you clarify your goals for the launch. Some example goals could be: 1. To validate that the new feature fully solves the "user problem" it was designed to solve 2. To learn about demand for the new feature 3. To test relevance of the feature to particular sub-audiences 4. To improve your product's key metrics overall (increase engagement, improve retention). 5. To resurrect churned users 6. To convert more users from free to paid 7. To grow your product's users to appeal to a new segment Launch goals #1-3 are "learning" goals that will lead to feature improvements and future updates. These types of launches should be done quietly, without any marketing to your existing audience, in order to get the cleanest data on user behavior and usability. This is often called a "soft launch." Launch goals #4-7 are more about fully serving your customers or changing the trajectory of your business with your new feature. In these cases, you've hopefully already learned quite a bit from qualitative research, betas, or prior "soft launches," and so you have confidence that your feature really meets the user need. These types of feature launches are going to be similar to full product launches in that you need to drive awareness with the target audience, and potentially operate thoughtfully designed "consideration," "installation," and "purchase" funnels in order to achieve success. The good news is by deploying your new feature into an existing product you already have an audience and established UI channels that can be used for these funnels. So the launch should be simpler and the learning faster than it would be for a new product.

At Meta/Facebook we're lucky enough to have really good internal infrastructure for setting up experiments and evaluating results. So in recent years I've come to rely on these internal tools. At other companies, I've used analytics and optimization services like: 1. Google Analytics/Firebase 2. Optimizely 3. Mixpanel 4. ApSalar (now Singular) 5. I would also expect great things from Statsig, but haven't used it yet myself. A few other points about planning experiments and your iteration strategy: 1. Learn about statistical significance and educate your stakeholders on it. 2. On early stage products or those with smaller active user bases, you may need to drive more people through your funnels specifically to reach stat sig results. 3. You can run a/b tests without code. I've used survey tools like Survey Monkey, Qualtrics, and Mturk (now deprecated) to test variations of design mocks before deciding which one to build and test in production. 4. I've seen startups falter when people in positions of power required they approve every experiment. To get good results, define your goals and metrics, then allow team members the freedom to test their ideas. 5. A/B tests and analytics will tell you what people are doing, but not why they're doing it. You should also use qualitative research techniques to form hypotheses about pain points and identify opportunities to improve. 6. Evolutionary experiments can lead you to a "local maxima." They won't reveal whether a much higher ceiling for performance exists with a boldly different approach. So talk to customers, do contextual inquiry, listen in on support calls, and build deep customer empathy within your team.

Each person in a leadership position has a unique personality and skillset. It's important they find ways to continue to offer their strengths to their team and product while also doing the job of a leader: providing direction and clarity, supporting team members of various skill levels and seniority, coaching team members, and giving them opportunities to learn and grow. In my case, I love thinking through potential visions and strategies, breaking them down into tactical plans, and actually getting into the detailed user problems and potential solutions to solve them. I love whiteboarding with engineers, participating in design sprints, and learning from data and user research. And I especially love using my products again and again to build empathy for users and form visceral product opinions on quality and what's working vs. not. I'm not shy about giving feedback in any of these kinds of activities. I can be tough in terms of holding high standards, asking hard questions, and giving detailed feedback on documents to try to drive clarity. Given my personality and skills, I do need to take steps to be sure I'm appropriately delegating and coaching. I try to regularly match important opportunities to people's skills and seniority, and I tune my level of support to each person and project's need. In many cases, PMs, engineering managers, tech leads, or designers have the skills and experience to lead projects to success with very little guidance. In these cases, all I need to do is help align folks on goals, clarify a decision process, or ask people to think through a potential future concern. Whenever this is the case, it's a win! It means I can put my own energy into solving some problem for the team that only I can solve. In other situations when hands-on help is needed, I try to check in with folks regularly to be sure I'm helping in a way that they find helpful and not disempowering.

Three ways: 1. Writing strategy docs 2. Collecting strategy feedback from other people 3. Monitoring your product metrics to track if your strategy is working Writing will help you clarify your thinking and give you a draft to "test" mentally against different frameworks and criteria (see my other answer about what makes for a good/strong product strategy). Feedback: Organizational and company leaders can give you helpful feedback (formally in group reviews, or informally on your drafts). Coworkers with specific functional roles (designers, finance folks, engineers, marketing) can give you valuable perspectives on your proposed strategy. Outside advisors (investors, board members, successful folks and subject-matter experts from your professional network), service providers (like advertising and social media experts), and at times even your best b2b customers can also provide useful perspectives and feedback. Metrics: Once you're executing on your strategy with product updates in the market, you should start to see behavioral and operational metrics "proof" that your strategy is working. A lack of metrics proof might indicate either the strategy or the execution needs to be changed/improved.

Ultimately, our job in product management is to make products that succeed in the market. So to me, that's always been the high order bit in prioritization and planning. If you're not delivering value to your customers, or the business model isn't sustainable, your product and team will fail. But the teams we work with also depend on having exec support, buy-in, and sponsorship. So you can't ignore this. You have to ensure that your exec stakeholders understand and agree with the team's strategy, priorities, and timeline. And sometimes you have "pull" important context from these executive stakeholders. E.g. "What counts as success for this organization/team/initiative in 2024?" It should be really rare that executive guidance is in opposition to delivering customer value or making a product that will succeed. But if this kind of disconnect ever comes up, start by being curious and asking questions about why the executive is giving that guidance. Likely this will expose important context (about the company, the market, or your success criteria) that you can take into account to improve your plan. Then you can continue to receive the organizational support needed in order to ship and support the product in market and achieve the ultimate goal of increased market success.

Most situations don't have "one right" product strategy. Some market/product situations have several strong/good strategies. In other situations there may be no winning strategies for your company/team. But "how should we validate that our product strategy is good?" Let's look at 3 examples: (1) validating a new strategy you're considering, (2) validating a strategy the team is executing against but without market feedback yet, and (3) validating a strategy for which you can gather real customer data. Evaluating new strategy proposals: Good strategies lay out a credible path to the right destination. 1. Does the strategy lead the product/team/company to a desirable place? Happy customers? A healthy business? 2. Is the strategy credible? Is your company/team likely able to execute on it to arrive at the desired destination? 3. Is there data to support the strategy? E.g. latent user behavior in which people are working around a missing product/solution? Market research on underserved and/or unmet needs? Early b2b sales conversations leading to strong interest? 4. Does your strategy play to your company/team strengths? Your assets and position in the market? 5. Are your strengths/advantages sustainable? (sometimes called a "moat") 6. How will competitors and new entrants likely react? Does your strategy still "win" after these reactions? 7. Does the strategy make it clear what you will NOT do? A good strategy clarifies which "good" projects/investments the team will not invest in. Evaluating a strategy your team is executing against (to build product): A good strategy aligns, stabilizes, and empowers teams. 1. Has the strategy aligned people and reduced debates (especially repeated debates)? 2. Is the strategy helping people and teams make decisions locally in a way that's consistent with the direction and destination you need to reach together? 3. Is the strategy leading to stability in product/project execution? E.g. consistent priorities for multiple quarters? Less fire fighting? Less churn? Evaluating a strategy with market feedback: Your product metrics should be telling a story consistent with your strategy. 1. If your strategy has you repositioning your product as a solution to a new set of problems, how are the market surveys changing every month/quarter? 2. If your strategy has you winning customers over from a competitor, how's that going every week/month? 3. If your strategy has you building more engagement with your users, how is engagement improving? If the product data isn't matching the path laid out in your strategy - it's time to investigate why. Maybe the execution needs to be better. Maybe the strategy needs to be changed.

When it comes to customer behavior, you usually detect that a problem exists before you figure out why that problem exists. * "What people are doing" shows up every day in your metrics. * "Why are people doing what they're doing?" is a harder question that often requires research techniques to nail down. In a small scrappy company, these techniques can involve click recording and watching replays, getting out of the building and talking to people, surveys, diary studies, usability studies, market awareness studies, and other simple UXR and Market Research techniques. In a larger organization, you should be working closely with your functional experts in qualitative and quantitative research (or hiring outside help) in order to understand the "whys." You might have a go-to-market problem (lack of awareness, improper positioning, mental blockers, poor pricing, usability problems, or missing pieces in your acquisition and engagement funnels). Or you might have a strategy problem (people not associating your product with solving their problem that the new feature is meant to solve, or people not trusting your brand to solve their problem, or people not needing to solve this kind of problem repeatedly, or people being satisfied by workarounds and alternatives and not seeing a strong need for your product). Or you might have both types of problems. To figure out which you have, and which are most important to resolve, turn to research techniques to understand "why."

Trends tend to be high risk, high reward. You have to be careful not to assume too much. There's a long list of heavily-hyped social media trends that turned out to be pretty niche and unimportant over a 5+ year timeframe: 1. Vine's 15s videos, 2. Clubhouse and audio social media, 3. Anonymous social networks like "Secret" from 2014. But sometimes there are huge opportunities due to shifts in the market. 1. New regulations like GDPR created market opportunity for a lot of companies, 2. The decline of Twitter/X is creating opportunity for competitors, 3. Music-backed short-form videos like those on TikTok are capturing an increasing share of user engagement in many social products. Think of an opportunity to pivot toward a market trend as a bet. You have to evaluate this high risk, high reward bet compared to the opportunity costs. 1. What's your likely range of outcomes, growth, & increased revenue if you make safer investments for the next year? 2. What's your potential range of outcomes if you chase the trend? If you're willing to bet on the upside of the trend being durable and your company and team can also bear the risk of it being temporary (and the wasted investment), then go for it. Otherwise wait and reevaluate in 12 months.

I like to get as much input as is reasonable both internally and externally. * Internal examples of stakeholder input: partnerships, sales, marketing, product support, operations, finance, data science, design, execs… * External sources of roadmap input: press reactions & professional product reviews, app store ratings and reviews, customers support requests, user research, inbound marketing research, and above all behavioral data from your logging/analytics. In terms of gates... during the planning process, you don't really have gates, just the process you follow that encourages all good ideas to flow in, but then rates those ideas by expected impact toward your goals, and level of effort to design/build/test/release. See my other answer to the question about the end-to-end roadmap prioritization process. When the team is executing, not planning, ideally all ideas coming in can wait for your next update or refresh cycle. This might be quarterly or even monthly in a very lightweight way... but since your teams have established goals, new projects will only bump other projects out of the roadmap if they're more likely to have higher impact in achieving the goals. Occasionally there's a disruptive change that should cause you to think through changing the team's goals or an immediate pause/change to work-in-progress in the roadmap. But these should be rare, and very important, because you're going to thrash your team to consider the change/pivot.