RyanKantor.com

Listed as one of the six worst mistakes of marketing research according to one marketing research methodologist’s blog, is getting bound by statistical significance. It is just so simple, agreeable, and “clean” to base recommendations purely on statistical significance. It’s so nice to tell a client that Product A is better than Product B because it is “significantly” better on a specific measure, even if it is “almost” worse in every other area.

Consider this:

Product 1

Purchase Intent: 3.9/5

Attribute A: 5.1/7

Attribute B: 5.0/7

Attribute C: 4.4/7

Product 2

Purchase Intent: 3.4/5

Attribute A: 4.6/7

Attribute B: 4.5/7

Attribute C: 5.0/7 (Significantly greater than Product A)

In this case, Product 2 is the only one with a statistically significant advantage, but is it really the better product? If not, how do you explain this with any authority? This is where telling a story becomes part of the research process and it moves beyond numbers. Graphics that quickly illustrate these “insignificant” differences throughout the report paint the picture and the recommendations must caveat the one significant difference with the larger story told by the other data points. Becoming beholden to significant differences can make findings more dependent on sample size than desirable. Please share your opinions in the comment below.

As always, please subscribe to this blog by clicking the “Follow” button at the top of the right sidebar. If you don’t have a WordPress account, you’ll have to enter your email address. You can share your opinions in the comment section below or by tweeting to @Ryan_Kantor. Thanks for reading!