Blogs

Note from the Editor: As CX professionals, we’re tasked with demonstrating the impact of our work through metrics that matter to the business. That starts with having confidence in the data we’re using to make decisions. In this article, Rachel Cope, CCXP, who brings deep expertise from the market research world, walks us through concepts like sample size, significance testing, and margin of error. While the focus is on survey design, the implications are far-reaching: better survey practices lead to better insights, which in turn drive smarter business decisions that enable us and our colleagues to improve retention, refine products, and identify growth opportunities.

I’ve been involved in discussions recently about interpreting customer survey results, so I decided it might be helpful to share a series of articles about some key market research concepts.  By sharing best practice from the market research industry, I hope to help CX practitioners design better survey experiences and interpret their results accurately.  

How many responses do you need for your survey results to be meaningful?

In the world of surveys, results can be misleading unless interpreted with a solid grasp of statistical principles.  Imagine you’re trying to figure out what percentage of customers are dissatisfied. You run a few surveys and each one gives you a slightly different answer.  That’s completely normal. It’s not that something went wrong—it’s just ‘sampling error’ at work.

What is a sample?

Because we can’t survey everyone, we work with samples.  Naturally, different samples give different results.  There are two critical elements to good sampling:

1.        Sample size: if we collect enough data and plot those results, we start to see a distribution. The middle of that distribution gives us a good estimate of the true value.

2.        Sample representativeness: the more your sample represents the population you are trying to represent, the more accurate your results will be.

A few other terms that will help you to design a robust approach to surveying….

Significance testing

Significance testing helps us figure out whether differences in our data actually matter. In other words, is this number really different from that one, or is it just a fluke of sampling?  It’s this significance testing that tells you whether you’re making real change over time.  This kind of test adds statistical weight to your findings. If the difference between two groups is "significant," we can say with confidence it’s unlikely to have happened by chance.

Margin of error

Error bounds give us a range around our result—basically, how much the number might be off just due to the randomness of sampling. These margins depend on the size of your sample: the larger the sample, the tighter the error bounds.

For example:

• A sample of 1,000 gives you about ±3.1% error at a 95% confidence level.  This means that the ‘margin of error’ is + or – 3.1%.
• A sample of 500 increases that to ±4.4%.
• With just 100 people, it jumps to ±9.8%.

So, when planning a survey, it’s crucial to think about how much uncertainty you’re willing to accept.  That’s not just about the overall margin of error though, but the extent you want to analyse your data at the sub-group level[1].

Confidence intervals

A confidence interval tells you how often your survey result is likely to fall within the error bounds. A 95% confidence level means that if you ran the same survey 100 times, you'd expect the results to fall within that range 95 times. The other 5 times? They’d fall outside it—that’s the margin of error in action.

To summarise…

You don’t need a statistics degree to understand whether your survey results are solid. Just keep these key ideas in mind:

• Sampling error is normal—don’t read too much into small differences.
• Bigger samples give more precise results.
• Significance testing helps confirm whether differences are real or just random.
• And perhaps most importantly: how well you sample matters even more than how many people you include.

Understanding these concepts helps you and your stakeholders to have both confidence in your approach and in your results.

Rachel Cope is Head of CX and Growth at 2CV, an established market research agency working with global brands to help them use insights intelligently to make business decisions.  She has over two decades experience in the industry across the full breadth of sectors, markets and audiences.