Product Testing for Beginners: The Complete Guide (2021)

Before we get into a Q+A with our CEO Dana, let's start with the basics of product testing. So, what exactly is product testing?

Simply put, product testing is the process of gathering feedback directly from potential customers about your product. This could be targeted to look for a wide range of specific insights like usage behavior, consumer preferences or reactions to a product. Product testing usually occurs when launching a new product, but can be done at any point, helping you refine, optimize or validate your product. 

Why is product testing important?

When launching a product, it is critical to test how your potential customers will engage with your product. While your product might be interesting in concept, or test well across a smaller cohort of consumers, your product will only be successful if it performs across your target market. This is where product testing proves to be so important. By testing your product, you collect critical, structured data on how your target customers interact with your product. You are conducting customer journey and experience research. Product testing enables data-driven innovation, to ultimately make your product better. 

Innovators know all about the high-profile product failures that could’ve been avoided with proper research ahead of launch. Even Coca Cola has had some misses.

On the other hand, product research can define, accelerate, or magnify the success of a product. For example, we’ve worked with brand partners where we’ve discovered missed opportunities (this energy bar should be called cacao, not chocolate!)  

Bottom line - product testing is an important investment. 

What are the benefits? What can I learn?

While this may seem all good and well in theory, we know that as a new product developer you need to understand the ROI of investing in product research and early-stage product testing.

Here are some of the most value-creating and beneficial uses cases that our brand partners have used the Highlight^TM Agile Product Testing platform for: 

• Sensory, Flavor, and Pack Testing

  • Flavor/Formulation Testing: We enable data-driven R+D. Gather product feedback from target consumers to ensure you're creating products consumers want.
  • Package/Form Testing: Get in-context feedback on your packaging, rooted in how consumers organically interact with and use your product.
  • Cost-Savings/Formulation Matching: Test lower cost formulations to ensure performance and maintained flavor/product experience.
  • Quality Control: Ensure your product quality is up to par across manufacturers, production runs, etc.
  • Line Extension Inspiration: Understand potential product innovations, rooted in core product trial and potential extension ideation.

• Usage Testing

  • Overall Experience: Dig into the product and consumer experience, when and why your product is being consumed, and whether it's living up to its claims.
  • Preference/Competitive Analysis: Bundle your own products, or with competitors, to understand preferences and relative performance.
  • Efficacy Testing: Test performance, perceived efficacy, and whether it’s delivering against its promises.
  • Ethno-‘lites’: Video research that’s in-home and in-context means you get a holistic understanding of your consumer and the context around your product. consumption – watch them unbox and try your sample live.

• Message Testing

  • Claims Innovation & Renovation: Stat-forward, quantitative research to develop and refresh product claims.
  • Positioning & Benefits Hierarchies: Optimize positioning & messaging, taglines, descriptors, and more, rooted in data-driven product experience.
  • Naming & Messaging: Ensure and optimize product-name and product-messaging fit, to ensure proper expectations are set.

Q + A with Highlight's CEO

Understanding the movement towards agile, in-home usage tests (or IHUTs, for short)

How do you think product testing is evolving?

The way that people think about it today is much different than even a year ago. In a pre-covid world, the primary forms of product testing were always in person, so you can distribute the products, and watch as consumers test and experience those products. And that's super compelling. There's definitely a time and a place for that. But where that gets less optimal and where that falls short, is that it's in a very different environment. You're in a focus group facility. You're not in the comfort of your home. You're not consuming a product or trying a toothpaste or drinking a beverage the way you would, as if you naturally grabbed it in your home and in your space. So I think people are really starting to appreciate that, brands are really starting to appreciate that. 

The closer you can get to the most organic, natural consumer experience, the more valuable that feedback is, because that's how consumers will actually experience your product in their realistic day to day, not in a sterile facility or in over-observed situations.

I also think there's some really cool movement around testing product that's not in final packaging and not yet on shelf. You're making sure that you're getting feedback in the R&D process and actually using consumer feedback to help inform product development. This has definitely been done before, but it's being done more often, earlier in the process. It's awesome that brands are better able to ensure product success, by making sure that they have data-driven product development processes. Consumers are now able to actually have a hand in the innovation process. As a CPG or as a consumer product company, all you can ask for is to be as close to the consumer as possible at all phases, and this approach really speaks to that. Putting the consumer at the center is a win-win for all.

How important and valuable is constructive or negative feedback to product development and testing?

I love this question. There's a big a challenge in traditional market research where survey respondents are wrongly motivated, and think they need to say what you want to hear. They're incentivized to take the surveys by gift cards or prizes. Sometimes it's gamified by a point system. But at the end of the day, participants want to be chosen again for the next survey and they want to be rewarded for their survey responses. So, oftentimes they tell you what they think you want to hear. And that just results in A, skewed data, and B, just an echo chamber of "Yes, this is great. Love it. Thank you for the opportunity," which is far less valuable than "This is great but X could be better." It's so important to reinforce that 'honesty is the best policy' - as an unbiased research platform, we're really proud that we enable honest, critical feedback - good or bad. The genuine desire to make products better, that incentive alignment, that motivation to be constructive and co-create better products alongside brands - those are the things we're really proud of about our community of Highlighters.

We always hammer home that fact that we want to hear what you really think, the good and the bad. What do you like about it? What do you not like about it? Honesty is the best policy, and really critical to product development and better products for all.

The way we position the survey is always, "you have a hand in helping make this better." So by couching the entire feedback process in that mission, it really speaks to consumers by saying to themselves, 'How can I make this better and how can I help the brand succeed or improve?' It's funny, coming from market research agencies and the traditional market research side and being a client-facing person, I've always dreaded those meetings when you have to present results to clients and tell them they didn't perform so great. But, those always end up being the most rewarding research efforts. It's because of that research that brands avoid making really big "wrong" decisions or place big bets on the wrong things. Calling out the critical feedback is where you create a lot of value and really unlock opportunity.

When is product testing important? Where does Highlight fit in?

Product testing is a very general term, but really where Highlight comes in is when there's a need to get physical product in hand, when there's a prototype or a product. Prior to that, there's so much higher-level diligence on the consumer product manufacturer’s side around "What grand ideas can we pursue? What product categories should we approach? What is the market size of those? Is there a consumer need or a job to be done?"

Ok, now that we have, you know, 10, 15 ideas, really just concepts at this phase, let's test those with consumers and get down to the five that people are really interested in. That whole phase is concept testing and development. The next phase is: now that we have narrowed down to a few product ideas and concepts that we're excited about, let's see if we can bring them to life, create prototypes and actually execute on these ideas that we've invested a lot in. That's where Highlight comes in and where things get really interesting. You've done all this hard legwork and invested in research around making sure the concept is perfect. But, if what's on screen or what's on paper doesn't translate to the physical product, then all that work was for nothing.

A product not living up to its claims or its expectations is disastrous because you might get that purchase, or capture that first interest. The consumer opens the bag and takes a bite, or takes a sip, or whatever it might be. If they don't like it, they never buy it again. And that repurchasing is oftentimes key to a product’s success. You can't afford to let a consumer down.

There’s a framework General Mills and P&G use where they define consumer's experiences with products as "moments of truth." The first moment of truth between a consumer and a product is when they see a product on shelf or on a screen. At that point, you need to somehow stand out. Is it a cool flavor? Unique benefit? Beautiful branding? Interesting functionality? Whatever it might be, that first moment of truth, you need to win because you need to have someone give you a chance - reach over, click through, take a bet and both consider and try you. The second moment of truth is when they actually consume the product - when they open it and actually experience it. That's where we come in - we're making sure the product lives up to expectations. It's complex and nuanced - that delicate cycle of consumer trial and repeat, tied to that first and second moment of truth. There's a lot to think about and get right.

How large of an audience can you reach? How do you do make sense of all the data you get back?

Yeah, a core differentiator for us as a platform is our ability to offer product testing is scale. Formerly, you could bring people into a room, have them surround your newest products, try each one, answer questions. Central Location Tests (otherwise known as CLTs, or product focus groups) were core to product research, but also so restricted in scale. On average, you get 6 people at a time with those approaches. But we can get product feedback from hundreds, thousands of consumers in days.

In terms of the data get-back - if you're asking more scoring, rating, ranking quantitative questions, that's super easily synthesized. If you're asking more open-endeds and collecting verbatims, it's of course less so. You're like, great, now I have three hundred "Why did you like this product?" responses, what do I do with them? This is an age old challenge. There's always been an overwhelming problem of, “How do I make sense of these all of this free-form text?”

We do the typical coding and sentiment analysis to help with this understanding. It comes down to some really basic things, like word clouds and things that help us synthesize all that data. What our product team is working on, that I think is really interesting, is building out more robust algorithms around how to identify the most helpful responses and therefore how to highlight the most actionable responses. Our algorithm can then tag those "most helpful responses" and actually keep intact the verbatims that are most useful and representative and actionable, and only spotlight a few of them. As we grow in terms of project scale, we don't want to sacrifice that qualitative, open-ended response quality, and want to be able to collect unstructured thoughts from our audience. They're so engaged - they give paragraphs of really great responses to our open ended questions. We want to keep encouraging that, but to make it more digestible for our brand partners. So we will continue to play with those different ways of synthesizing all of that data in really interesting ways. It is a fun part of our internal product development. We want to practice what we preach and ensure all Highlight platform innovation continues to create value for our community!

To understand more about how Highlight^TM Agile Product Testing, drop us a note here, and an Insights Associate will reach out!