Do you ever develop quantitative insights for testing messaging (e.g. via survey, other)?

Question

Dee Dee Wolverton · Accepted Answer

Yes! We can and we do - surveys, interviews, localization checks, experimentation, and instructor advisory boards. Some of these are a heavier lift than others, so it’s important to ensure you are really thoughtful to marry the content and desired outcome to the testing approach, to ensure you can glean as much actionable data as possible.

Morgan Molnar · Answer

All the time! Again, I'm lucky to be a product marketing leader working on a market research product, so I have unlimited access to our own message testing solutions.

We approach quatitative message testing a couple different ways:

- Our live marketing assets provide a great testing ground for messaging: you can A/B test things like ad copy and email subject lines and copy. Click-through rates can be a great indicator of what resonates. (Keep the visuals the same so you know results are only related to the copy/messaging itself)

- We also run copy tests on high-traffic webpages, like our homepage. We have our own home-grown experimentation platform for this, but you could also use a solution like optimizely.com

- The above two methods really only work for short copy and require you to get something live in production. And depending on your traffic volume, it could take a while to reach statistical significance on your A/B tests. So we also use surveys as a faster way to test messaging. With surveys, you can get explicit intel on top challenges, most important value props, and can pit messaging statements head to head. I will almost always conduct a messaging survey when I'm building a fresh messaging framework for a new solution or target persona. An example from a couple years ago is we were starting to lean into professional services offerings, and I wanted to know which language would sound most impressive when talking about our team. I used a survey to test phrases like "market research consultants" vs "market research experts" vs "research scientists" and understand those terms across a variety of attributes like knowledgeable, trusted, and approachable.

Here's more info on our message testing solution: https://www.momentive.ai/en/solutions/message-testing/ and a guide to doing this yourself: https://www.surveymonkey.com/mp/concept-testing-guide/

Qualitative ways to test and validate messaging would be to get in front of customers (interviews, injecting a question or two in existing research the team is conducting), and establishing a feedback loop with your sales & success teams that are actively using the messaging in conversations.

Sophia (Fox) Le · Answer

Yes! Glassdoor, under the leadership of Bonnie (Head of Market Insights), established two market research online communities (MROC): one for consumers (B2C) and one for customers (B2B). We regularly field message tests to these two research communities to get quick, detailed feedback on messaging. This is what Bonnie had to say about MROCs:

1. MROCs make message testing easy, efficient, and effective! Once your online community is fully recruited, you can send them a survey and usually get a sufficient number of responses back within 48 hours. Automatic reporting dashboards make it easy to share insights with the team. Fuel Cycle is our MROC provider and our partnership enables us to conduct far more research than we could before when we hired research vendors to launch our message tests for us.
 2. One of the reasons we chose Fuel Cycle is because they’re partnered with Alchemer (formally Survey Gizmo). Alchemer is a fantastic survey platform, especially for team on a budget. In addition to all the usual survey question types, they have text and image heat maps which are some of my favorite tools to use while message testing. The PMM team loves to see which words and phrases engage the audience the most as well as words and phrases that are confusing or disliked. We can also cut the data to look at it by different demographics or audience segments to see how a message may affect groups differently.
 3. When we run message tests, we’re not just looking to see which message is the best performing, we want to learn:
     1. Which audience is engaging the most and why?
     2. Are there messages that appeal especially well to specific target audiences, even if that message did perform the best overall?
     3. Who are the product/feature acceptors and what do they think about the message linked to the product/feature?
     4. Which messages are most likely to lead to the desired outcome?

Pro Tips:

* Sophia’s pro tip: Leverage the tools you have to get feedback. If you can’t field a survey, could you conduct a few in-depth interviews with customers that represent your target audience to see how they react?
 * Bonnie’s pro tip: Whether you’re using qualitative or quantitative methodologies, know your target audience and make sure to test the message with them. Getting feedback from outside our target audience may lead your messaging strategy astray.
 * Patti’s (Head of Consumer PMM) pro tip: In addition to surveys, when available, you can A/B test through the growth team in marketing and/or product. We recommend testing every part of the funnel to understand conversion and get a more nuanced understanding of what actually drives greater appeal (is it the color, design, certain parts of the copy/specific key phrases, etc.)

Kevin Garcia · Answer

100%. It’s a great option when you either 1) don’t have a direction and need to narrow your field of view or 2) you have very similar finalists for messaging and are hoping to choose a winner.

Note that surveys rarely have crystal clear results. You’ll still need some amount of qualitative or other insights to help you make a final choice.

Div Manickam · Answer

This is something we would like to get better at within our team. We aspire to be persona-led and data-driven, but don’t have processes/tools in place to perform the right testing for messaging and positioning. Today, we are able to test messaging and positioning with the sales teams(joining customer calls for customer feedback), Customer advisory board. We have identified champions across sales, presales, etc. for focus areas or use cases (by solutions/products/industries) to help us test the messaging with biweekly guild meetings. We would like to put more processes and conduct surveys to analyze impact and ROI for effective messaging.

Katie Levinson · Answer

Sure do! I like to start with some qualitative research first to help get at any nuances in messaging, especially across different audience segments. Then, run a survey (max diff is a great technique) to understand what resonates most with your different segments. If you also have the budget and/or time, running your messaging by focus groups is another good option, so you can get a deeper understanding of their reactions and sentiment.

Related Ask Me Anything Sessions

Unity Vice President Product Marketing, Jon Rooney on Market Research

Google Head of Marketing, Cybersecurity, Apurva Davé on Platform and Solutions Product Marketing

Calendly Head of Product Marketing, Jeff Hardison on Messaging

Related Questions