How do you best structure and leverage beta releases to assist the product team (with iteration, feedback) and Product Marketing (positioning, messaging, enablement, onboarding)?

A bit of an “it depends” answer. Sometimes people use betas for QA: does the feature we built work end-to-end? Other times betas can help you determine if you’ve hit product-market fit with your product. And everything in between.

It’s best to get super aligned cross-functionally at the KICKOFF (or early in the development process) about how alphas and betas fit into the overall timeline, project strategy, and what the objectives and go/no-go decision are up front. Determining all that at the beginning will help you appropriately time the beta, get the learnings you need, and still hit a GA milestone on time. It also helps the team be more objective about success when the beta is actually live.

When you can work with a product team to include messaging or feature on boarding in the beta, you can usually learn a ton! Especially when paired with some sort of survey or user interviews (depending on your audience size) for both people who did and didn’t engage fully with the feature. You can use that info to determine it’s a functionality issue or messaging issue before letting your messaging rip to zillions of people.

If a beta is short and the product isn’t likely to change a lot before production, you can still do surveys or interviews to learn about product functionality and “see around corners” that can help you as you develop sales enablement content (ie a more fleshed out FAQ).

Sometimes you’ll learn from a beta that it’s best to rollout the functionality but not do any proactive product announcements around it. Have a success or sales team sell a (more limited) feature when needed, but wait until you really nail a use case (ie more dev time) before announcing it more widely.

This is actually something we're working to improve at the moment, as the process at Intercom hasn't been that standardised in the past.

Historically at Intercom, betas were driven by the product team and wholly focused on collecting product feedback (often via research interviews and feedback through our messenger). As our PMMs work closely with PMs - and are often involved in sending the messages to beta participants - the feedback from these betas would be shared with PMMs and we'd use it as relevant to inform messaging and launch plans. 

However, because we also want to leverage betas for things like customer testimonials and messaging feedback, and because our customer base has over time become more upmarket (which tends to mean beta featuress need to a bit more 'baked' before customers will use it), we've started introducing new processes to betas. This includes things like having them being more full-featured before rolling out as a beta, starting earlier so customers have more time to see the results of using a feature (which aids us with testimonials), and sales and PMM being more involved in choosing the beta audience. Things like ideal timelines and how to standardise dissemination of the information is something we're still working on - maybe we'll write a blog post when we figure it out!

The primary goal of a beta should 100% be focused on improving the product and giving valuable customer feedback to your product, design, and engineering teams. The way to make sure it’s impactful for those teams is to get really clear up front in a beta what questions we’re trying to answer. You start a beta because you have open items that need to be figured out before you’re ready to launch a product, and you hope that a group of hand selected users can help you figure those things out. Those questions and beta objectives need to be defined, documented, and agreed upon before the beta begins. I have sometimes seen products go into beta because teams felt like they “should” or that it was a required step and, at the end of the beta, they left with no real answers because there was no real strategy for the beta period. PMM can help in driving that clarity, targeting the right users for the beta, and making sure the communication with beta testers and with internal teams happens at the right cadence. If you’re lucky, like I’ve been in many of the companies I’ve worked for, you’ll have a stellar user experience research team who can lead this work or, at the very least, partner with you on it.

In terms of making sure betas are impactful for PMM, the same principles apply. What questions do you have that you’re hoping to answer? Where are you least clear in your GTM plan? What would help you run a better launch and ongoing engagement efforts? Figure out what you want to know from beta users and make sure it's reflected in the overall beta strategy so the time is impactful for everyone involved.

From there, here is some tactical advice on running betas: 

• Participants: Be very clear upfront about the level of engagement expected from beta participants and get agreement that they’re onboard. Think about offering some incentive for the beta customer to ensure they give you what you need, it can be as small as some company swag or as large as a few free months of service in exchange for their feedback. Be careful to not keep adding folks into the beta just because a sales person wants them to get added or because they complained about not being included on Twitter. If a customer doesn’t fit your target persona for the feature, they shouldn’t be in the beta.
• Communication with beta participants: I’ve seen researchers set up Slack channels with beta participants that internal stakeholders have access to so beta testers can give their feedback in real time as they experience the product. It’s also important to talk live, usually via video call, with your participants at key points in their journey with the product ofr feature (ex: right when they get access, when a significant change happens, once you think you solved a problem they have etc)so you can get more in depth feedback. 
• Communication internally: You can invite everyone to your feedback channel so they can see it live, but you should also have some cadence of summarizing findings. If you’re running a three month beta, think about a monthly update. If it’s a six week beta, maybe bi-weekly updates are best. If it’s a short, month-long beta, a weekly update will be the right cadence. Keep your updates clear and to the point, then link out to the full findings for those that want all the intel.
• Timing: I can’t give you advice on this until I know the product you’re testing. I think it depends on how much is left to build in the product when you go into beta, as well as how long it’ll take a customer to see value from the product. I will say that I have not seen a beta shorter than six weeks be very impactful for business learning and iteration.
  
  Good luck!

This will depend on your goals for the beta. Does the product team just need to test the product for bugs or are you establishing product-market fit?

If you are testing the bugs, you can run the beta for a few weeks. In this case, you likley don’t need customer feedback because you can use your own data to figure out if the product is working properly.

If you are establishing product-market fit, you’ll need a few months and explicit commitment from your customers to provide feedback. I recommend setting up a formal beta program where you invite customers, require NDAs, and set expectations that customers will need to regularly provide feedback via surveys or phone calls.

In a sales-driven org, the sales team handles the beta recruitment process and customer vetting. In self-serve org, you’ll likely be doing this yourself or in partnership with the channel marketing and product teams.

As part of the feedback collection, you can cover the following topics.

1. Does the product solve the right problem or in other words is this a problem the customer actually has?

2. Does the product actually solve the problem? If not, what would need to change.

3. Does the product meet customer’s expectations. If not, what did they expect and how should it be adjusted to meet them?

4. Does the customer enjoy using the product -- what are the pain points, what’s missing, what do they really like.

5. How would they describe what the product does and why they are using it.

My team at Atlassian has a framework for how we launch new products that I would love to share. But before I dive into more detail I'll give you a few tl;dr tips for beta programs. 

• Create hypotheses that you want to test before your beta begins: Decide what you're trying to answer about your customer, product, or design via the beta program. This is important to keep your focus and ensure you come out learning something. 
• Great for product & marketing: Beta programs can be used to gather product and design feedback, refine your target audience, your messaging, and get early insights on your pricing and packaging strategy. 
• Get notable customer references: Betas are a great way to gather notable customer references or stories before your main launch.
• Build anticipation: Before launching a beta, you can use a beta waitlist to help build anticipation for your product and gauge excitement.

I lead the new products GTM group at Atlassian and we've developed a framework for how to build, launch, and grow new products. I won't go into all the detail, but below I'll explain a bit about how we run the beta portion of the program: 

In short, each new product within the portfolio follows a series of milestones:

• Wondering - Identify the problem/pain in the market
• Exploring - Find problem-solution-fit: evidence that customers care about certain jobs, pains, and gains
• Building - Build the MVP (an Alpha or Beta customers can start to use) 
• Showing (customers)- Find product-market-fit (the value proposition is actually creating value for customers by alleviating their pains and creating the gains they desire)
• Charging - Find business-model-fit: the value proposition is embedded in a profitable and scalable business model
• Scaling - Identify a sustainable distribution model

During each stage, the team develops a set of hypotheses focused on what they are trying to answer about the product, customer, or even GTM motion. We have check-points during each phase, where the team presents their findings and process to the leadership group to move to the next milestone.

In the 'Building' phase, we will select a handful of early adopters to work closely with the team to provide feedback on the product, design, and even early marketing assets. At this stage, you'd get a valuable (but limited) set of qualitative data on your hypotheses, which you can further validate in the beta phase with a wider audience and more quantitative results.

We typically launch a beta waitlist once the product meets a quality threshold and has early signs of product-market-fit with the alpha testers. Here are a few reasons why we launch a waitlist first before opening up the floodgates:

1. Build anticipation for the product: a beta waitlist can help to gauge excitement for a product/solution and help you generate anticipation in the audeince for your product.
2. Refine your targeting and messaging: By gradually allowing customers into your beta and not opening to too many users, you can not only test your product's performance but also buy yourself time to learn about your audience, the value they experience in the product, how they talk about your product, and more. All this helps you refine your target audience, messaging, and gather customer references before the grand reveal (your GA!).
3. Design the self-serve/onboarding journey: As you let more folks into your product, you can start to understand where users may be getting stuck, have questions, or need extra support while getting started with the product. This extra time and insight will allow you to build out your customer's onboarding journey - email drips, in-app flows, getting started guides, help docs, and more.

We then open up a beta for self-serve adoption, meaning that a user can start using the beta without signing up for the waitlist or interacting with a team to be authenticated. Because we have a larger set of customers in the beta phase, we can start to validate some of the hypotheses we've earlier uncovered about the product or customer using funnel data or customer surveys.

You can learn more about how we design the zero-to-one new products group at Atlassian by listening to my episode on the Women in Product Marketing podcast.  

Beta releases or early access programs are a fantastic opportunity to get a sneak preview of how users/customers will interact with and use a product or feature. It is really important to capture feedback early and often and package the "so what" back to your internal stakeholder group. This includes the raw data and feedback, but also an opinionated recommendation on what should be done to address. Betas are a great opportunity to test out messaging and positioning (in A/B format or other) and figure out what is landing and what isn't. What really resonates with the user and what doesn't. It's a great chance to get out of your internal echo chamber. When sales is involved, getting a few high context reps together to poke holes in your plan can also be a great way to sharpen your narrative. Lastly - betas are a great opportunity to partner closely with users/customers and capture their stories to leverage in case studies/testimonial/claims later down the line when you are driving adoption.

That's an excellent question. Done right, product release stages should service not just your product lifecycle goals but also the marketing/commercial goals that go with it.

Just because something is in beta, doesn't mean you shouldn't amplify it via your marketing. Similarly, just because something is now technically ready to roll out into production, doesn't mean you should force an announcement out of the gates.

Just like product strives for and MVP (minimum viable product) to test the waters, you should make sure your marketing team is set up with minimum viable positioning to validate technical and commercial readiness.

My team and I developed the following framework at WeWork, which we then used at Matterport, to structure our beta releases through GA. It was inspired from how we thought about product launches at Google but simplified and codified so we could set the right expectations with our GTM teams from the get-go.

Notice the callouts to technical as well as commercial readiness. This is the most important thing to align on with your product, marketing, as well as leadership teams. There are many different, winning, plays you can orchestrate to highlight where exactly your product is from a capabilities and marketability standpoint. But the kiss of death is when you either over-market something that's not ready, or under-market something that is. Take a look and feel free to adapt this to your needs.