Do you ever develop quantitative insights for testing messaging (e.g. via survey, other)?

100%. It’s a great option when you either 1) don’t have a direction and need to narrow your field of view or 2) you have very similar finalists for messaging and are hoping to choose a winner.

Note that surveys rarely have crystal clear results. You’ll still need some amount of qualitative or other insights to help you make a final choice.

Yes! We can and we do - surveys, interviews, localization checks, experimentation, and instructor advisory boards. Some of these are a heavier lift than others, so it’s important to ensure you are really thoughtful to marry the content and desired outcome to the testing approach, to ensure you can glean as much actionable data as possible.

Sure do! I like to start with some qualitative research first to help get at any nuances in messaging, especially across different audience segments. Then, run a survey (max diff is a great technique) to understand what resonates most with your different segments. If you also have the budget and/or time, running your messaging by focus groups is another good option, so you can get a deeper understanding of their reactions and sentiment.

All the time! Again, I'm lucky to be a product marketing leader working on a market research product, so I have unlimited access to our own message testing solutions.

We approach quatitative message testing a couple different ways:

- Our live marketing assets provide a great testing ground for messaging: you can A/B test things like ad copy and email subject lines and copy. Click-through rates can be a great indicator of what resonates. (Keep the visuals the same so you know results are only related to the copy/messaging itself)

- We also run copy tests on high-traffic webpages, like our homepage. We have our own home-grown experimentation platform for this, but you could also use a solution like optimizely.com

- The above two methods really only work for short copy and require you to get something live in production. And depending on your traffic volume, it could take a while to reach statistical significance on your A/B tests. So we also use surveys as a faster way to test messaging. With surveys, you can get explicit intel on top challenges, most important value props, and can pit messaging statements head to head. I will almost always conduct a messaging survey when I'm building a fresh messaging framework for a new solution or target persona. An example from a couple years ago is we were starting to lean into professional services offerings, and I wanted to know which language would sound most impressive when talking about our team. I used a survey to test phrases like "market research consultants" vs "market research experts" vs "research scientists" and understand those terms across a variety of attributes like knowledgeable, trusted, and approachable. 

Here's more info on our message testing solution: https://www.momentive.ai/en/solutions/message-testing/ and a guide to doing this yourself: https://www.surveymonkey.com/mp/concept-testing-guide/

Qualitative ways to test and validate messaging would be to get in front of customers (interviews, injecting a question or two in existing research the team is conducting), and establishing a feedback loop with your sales & success teams that are actively using the messaging in conversations.

Yes! Glassdoor, under the leadership of Bonnie (Head of Market Insights), established two market research online communities (MROC): one for consumers (B2C) and one for customers (B2B). We regularly field message tests to these two research communities to get quick, detailed feedback on messaging. This is what Bonnie had to say about MROCs:

1. MROCs make message testing easy, efficient, and effective! Once your online community is fully recruited, you can send them a survey and usually get a sufficient number of responses back within 48 hours. Automatic reporting dashboards make it easy to share insights with the team. Fuel Cycle is our MROC provider and our partnership enables us to conduct far more research than we could before when we hired research vendors to launch our message tests for us.
2. One of the reasons we chose Fuel Cycle is because they’re partnered with Alchemer (formally Survey Gizmo). Alchemer is a fantastic survey platform, especially for team on a budget. In addition to all the usual survey question types, they have text and image heat maps which are some of my favorite tools to use while message testing. The PMM team loves to see which words and phrases engage the audience the most as well as words and phrases that are confusing or disliked. We can also cut the data to look at it by different demographics or audience segments to see how a message may affect groups differently.
3. When we run message tests, we’re not just looking to see which message is the best performing, we want to learn:
1. Which audience is engaging the most and why?
2. Are there messages that appeal especially well to specific target audiences, even if that message did perform the best overall?
3. Who are the product/feature acceptors and what do they think about the message linked to the product/feature?
4. Which messages are most likely to lead to the desired outcome?

Pro Tips:

• Sophia’s pro tip: Leverage the tools you have to get feedback. If you can’t field a survey, could you conduct a few in-depth interviews with customers that represent your target audience to see how they react?
• Bonnie’s pro tip: Whether you’re using qualitative or quantitative methodologies, know your target audience and make sure to test the message with them. Getting feedback from outside our target audience may lead your messaging strategy astray.
• Patti’s (Head of Consumer PMM) pro tip: In addition to surveys, when available, you can A/B test through the growth team in marketing and/or product. We recommend testing every part of the funnel to understand conversion and get a more nuanced understanding of what actually drives greater appeal (is it the color, design, certain parts of the copy/specific key phrases, etc.)

Yes, we aim to gather quantitative feedback whenever possible. While I don't believe it's necessary for great messaging, it can complement existing efforts and help reduce the risk of bias interfering with your messaging.

There are really two areas that my team thinks about research:

1. Market Research: Gathering data, either from 3rd-party sources or from customers, in order to identify market opportunities to pursue in your GTM strategy and product development.

2. Messaging Research: Gathering input from customers and prospects to better understand what matters to them and how your messaging resonates.

Both types of research are valuable. However, good research is difficult to do and it can be quite expensive. So, it's critical to go into the project with a clear idea of what you're trying to achieve. Taking on too much or spreading too thin can make effective research difficult.

Some of the things that I think are important to success are:

• Quantitative + Qualitative: Ultimately, asking a large number of people structured questions to get quantitative feedback is very valuable. But, if you can complement that with qualitative interviews from a smaller subset of customers you'll often gain contextual insights that are easily missed by pure quantitative surveys.

• Enough of the Right Customers: Be honest with yourself about what you're trying to achieve and how many customers you need. I've seen a number of situations where a decision was made to go small with the sample size. This isn't always bad, but the smaller you go the more you need to discount reliability.

• Don't Lead the Witness: The reason you do research is to validate your intuition, not prove it. If you go into research set on getting a specific outcome, you'll probably find a way to achieve it. But, research should be as much about identifying incorrect assumptions and things you've missed as it is about proving you're right.

• Focused vs Shallow: Good research can be quite expensive. But, you're likely to be much better off opting for fewer research projects that are more thorough and better conducted, as opposed to several shallow ones.

An example that I think highlights where this can go right was a research project we conducted a couple of years ago. One of the things we tested was a four-pillar framework for describing the value we deliver. Despite internal feedback that it didn't resonate with customers, we were shocked when 97% of those surveyed said the framework resonated well. However, qualitative interviews showed that one of the four terms wasn't very helpful. This helped us 1) determine we should keep the framework, and 2) set the stage for moving from four pillars to three over time. Without research, we may have landed in a much different place.

While research alone doesn't guarantee great messaging, it can significantly enhance an already effective product marketing process.

This is something we would like to get better at within our team. We aspire to be persona-led and data-driven, but don’t have processes/tools in place to perform the right testing for messaging and positioning. Today, we are able to test messaging and positioning with the sales teams(joining customer calls for customer feedback), Customer advisory board. We have identified champions across sales, presales, etc. for focus areas or use cases (by solutions/products/industries) to help us test the messaging with biweekly guild meetings. We would like to put more processes and conduct surveys to analyze impact and ROI for effective messaging.