AMA: Dropbox Group Product Manager, DocSend Growth, Willie Tran on Growth Product Management and Experimentation

Personally, I've always used pirate metrics. * Acquisition - Getting the user to sign up * Activation - Getting the signed up user to realize the value of the product * Revenue - Getting the user to pay for the product * Retention - Getting the user to stick around * Referral - Getting the user to refer someone else to sign up These are solid high level metrics to measure the overall effect of a team. However, when running a specific experiment, these metrics aren't always the best to choose when measuring whether the experiment was effective. If you have a lot of traffic, then it's not such a problem. But most companies don't have the traffic to consistently measure Retention/Engagement. Which, then you'll want to pick a less precise metric, or rather, a more upstream metric (something that happens before the high level metric) as it will will likely have a higher conversion rate, which means you'll need less traffic required to conclude the experiment. In regards to which I find to be the most important? It really just depends on where the company is at at the time and where the biggest drop is. Most people probably say Activation, but that's not really true for every company. 

I've wrestled with this over the course of ten years now. I used to think Growth PMs and Core PMs should just be one in the same and every Core PM should be able to do Growth work. However, I don't really believe that anymore. From my experience, there's a place in the business to do optimizations (Growth) and a place to do exploration (Core) work. Core PMs generally have a stronger skillset around taking big bets which require a lot of coordination and patience. Growth PMs require high analytical acumen and rapid development, but much smaller bets, and not as much coordination. 

The number one thing first time growth PMs get wrong is only focusing on wins and not involving their team. When creating a Growth squad, your number one enemy isn't not making the number bigger, it's xfn attrition. Designers and engineers who are new to Growth have historically been rewarded for releasing new features. When joining a Growth team, you have to be able to prepare them to launch something and then delete it. This is very new behavior and goes against what they are used to doing. So what do you do? You have to redefine how they value themselves. Essentially get them to unlearn what they've been used to for the last 10+ years. It's no simple task. But one of the greatest ways I've seen this get driven is by involving your xfn partners (who are interested) in the strategy of what your team will build. In that you are trying to move the number, but really, you're trying to answer questions that will inform the strategy. By involving your xfn partners there, they will feel rewarded when we answer a question and you help proliferate that insight across the company. You want your team to feel a sense of pride not just by moving the number, but by how you all affect the strategy as a whole. Even if that means you didn't make any permanent changes to the product.

The areas I index the most when hiring someone: * Partial - Do they fall in love with their own ideas? If so, then I don't believe they will be a strong Growth PM. Given so many experiments fail, you want someone who is focused on answering questions about the user and the science of it all. * Analytical - They don't have to be a statistician, but they need to understand how to read the data and extrapolate user behavior from it * Collaborative - This is probably the most important part. A person shouldn't be hired based on their ability to generate good ideas. Even if they do a great job at the beginning, they'll run out eventually. A great growth team leans on their xfn partners to help come up with good ideas. Also when you involve them, you end up with a happier team * Logical - There's a reason why we do everything. Every change is intentional and has a further reason behind it. The ideal candidate is able to understand the decisions they're making and explain why they're doing it with rational data.

If you're trying to measure the results of an experiment, assuming they're under a binomial distribution, you just need the numerator and denominator (people exposed to the experiment) for the control and treatment. Then just throw it into your favorite chi square significance calculator (I'm a big fan of the Evan Miller calculator). In regards to other product tools, it's the same as usual. You can use Amplitude to look into user behavior. StatSig or Optimizely or all the other ones for running experiments. Really any out of the box feature flagging tool has experimentation capabilities.

I mentally use the ICE (Impact, Confidence, Ease) framework when prioritizing experiments. Impact = How many people does this affect? Confidence = How sure are you that this will work? Ease = How easy is it to implement? Impact and Ease are pretty simple to calculate. However, I find that most people don't know how to accurately assess "Confidence." This is likely because your ideation strategy consists mainly of throwing things at the wall and seeing what sticks. Or just working off of best practices. This strategy works well for the first six months to a year of an experimentation strategy, but falls short after you've picked up all the low hanging fruit. Instead, you want to create a higher level process that's focused on generating high level user problem statements. You create these problem statements by doing a lot of user research around why users are not doing the thing you want them to do (e.g. Activate). Once you do your set of interviews seeking to answer a few questions, you should have some good idea as to why people are Activating (in this case). Turn those into well written problem statements (As a [user type], I want to [do a thing], but [reason why I can't]). Then for each problem statement, do an ideation session with your team where you and your xfn partners come together and come up with ideas on how to solve this problem. Then vote on it. You'll see a lot of ideas have common themes which will give your designer a lot of material to come up with the best way to address said problem. TL;DR - Do a lot of user research to understand the problem. If you understand the problem really well, then it's very easy to come up with solutions you have a lot of confidence in.

By far the best material you could read to learn about PLG has nothing to do with PLG. PLG is based off of the Scientific Method you learned in grade school. Go back and reread the fundamentals of how to go through the Scientific Method. Lessons around experiment design/methodology are highly valuable and should not be glossed over. There are other parts though that are more focused around experiment logistics and team dynamics that the Scientific Method won't cover. But in my opinion, a good book around logistics is "The Goal" by Eliyahu M. Goldratt. I unfortunately don't have any recommendations around team dynamics. But my biggest bit of advice (as you can see from my other answers) is just over-involve your team and listen to them. Get their input on everything. It's not a democracy, but good ideas come from anywhere. The people around you are very smart, use it.

Ideally, you design experiments in a way that seek to answer a question regardless of the result. For example, "do users not adopt this feature because of discoverability or usability?" is a great question to be answered with an experiment. If the exeriment execution puts that feature front and center of the main page when a user logs in, and the experiment is flat, then you could be pretty sure it's a usability issue. If it's stat sig positive, then you can be sure it's a discoverability issue. What's important is what you do after you've identified this finding. This new information can inform your entire strategy of your team or you can pass it on to another team for them to work on.