How to determine the optimal sample size for your survey

Arriving at an optimal sample size is a question often pondered upon when designing quantitative surveys. Choosing the right sample size can be tricky as a number of factors come into play. The sample size should neither be too small that you question the reliability, nor should it be too large such that you put a strain on your resources.

So how do you arrive at that magical sweet spot?

In this article, we outline five factors that you should take into consideration to determine the optimal sample size for your study.

1. Margin of error

The first question to ask yourself is what is the margin of error that you’re willing to accept?

What’s the margin of error though, you ask? Here is a brief explanation.

Online survey panels work by surveying a small group of people (a sample) that belong to a bigger group (population). The population would be the target audience you want to study, say the general population of Singapore, or, perhaps, all Singapore residents who have a university degree.

Surveying the entire target population is ideal but a near-impossible task. Therefore, market research typically relies on the results of a sample to draw conclusions about the target population.

However, in every quantitative study, no matter how hard you strive to make your sample representative of the target population, there are unavoidable errors bound to occur during sampling (i.e. while selecting the group that you will actually collect data from). This is known as sampling error. In statistical terms this means that the estimates you derive from your sample (e.g., mean, percentages, etc.) will differ from the “true value” of the target population. Sampling error occurs because we only survey a sample of the population and not the entire population itself.

This is where margin of error comes in. The margin of error, typically written as +/- X%, tells you how much the results of your study might differ from that of the actual population. It is typically accompanied by the confidence level, usually at 90%, 95% or 99%. So if your survey has a margin of error of +/-3% at a confidence level of 95%, this means that if the survey was repeated 100 times, 95 of those times your results would be in the +/- 3% range.

Margin of error and sample size have an inverse relationship. Common sense would tell you that if you increase the sample size, the scope for errors will be less (smaller margin of error) because you are taking a greater sample of the population. Conversely, the smaller the sample size the wider the interval and the less precise your sample results may be. In order to minimise the deviation from the actual population estimates, a larger sample size can help to reduce the margin of error and be more confident of the results.

However, an important point to take note of is that while the reduction in the margin of error is considerable as you increase the sample size till N = 1000, there is a diminishing return when you go further up.

Here is a table that illustrates the relationship between sample size and margin of error (at a 95% confidence interval) :

In sum, a factor to take into consideration when deciding the sample size is to understand the threshold of error you are comfortable with in your results.

Although a higher sample size is always ideal with respect to giving you greater precision, it may not always be important depending on the specific research aims.

For instance, a quick dipstick study or a quick study on early stage advertising ideas to test whether it is worthy of pursuing further could work with smaller sample sizes. However, a survey commissioned by the government with policy-making implications would warrant a larger sample size for more accurate results.

2. Analysis plan

It is important to take into consideration how you plan to analyse your data when making the decision on sample size. For instance, there are statistical methods (e.g. regression analysis, cluster analysis) that require a certain sample size per variable.

A common method of analysing your study’s results would be to slice your dataset into different segments. These segments could be by demographic groups such as age, gender, household income, or some unique categories specific to your study e.g. by users of a product vs non-users etc.

If you already plan to analyse your dataset by segments, then it would be best to aim for a sample size that will allow you to have a robust base sample size per segment. The industry standard is to have at least n=30 samples to read a segment, but here at Milieu we generally recommend no less than n=50.

Something to keep in mind is that if you intend to slice your data by segments and then zero in on a subsegment to do further analysis within that pool, a higher margin of error kicks in because you’re working with a lower sample base.

Take for example a study among n=500 respondents from the general population of Singapore. The margin of error would be roughly +/-4% at the 95% confidence level. Assuming the proportion of males to females in the general population is 1:1, then we would have collected n=250 males and n=250 females in the study.

Now, if you’re looking at just overall gender differences in your results then you’re still at a 4% margin of error. However, if you want to zoom into the male subsegment further and analyse within this group, the margin of error for n=250 males would then be +/-6% at the same confidence level.

In sum, an important factor in determining the sample size would be to consider the number of segments you want to slice the dataset by and ensure that each segment has a decent base sample size for reliable results.

3. Population size / Incidence rate for niche samples

Another important aspect to consider would be the actual population size for a certain group where you want to draw your samples from. This is what we call the incidence rate. If you’d like to understand the travel preferences of Singaporeans, then very simply the population you would draw your samples from would be the general population of Singapore.

If you’d like to obtain consumer opinions on, say, your new plant-based meat product versus an already well-known one like Impossible meat, you might look at surveying vegans or vegetarians as they might be your main target audience. The population you can then draw samples from becomes much smaller i.e. they would be a niche group of people.

The criteria for niche samples can be awfully specific, and the pool of respondents who fulfil your criteria (incidence rate) would be much fewer than if we were to draw samples from the general population. Generally, for lower incidence rates (e.g. around 10-15% or less of your target population who fulfil your criteria), most market research panels would only be able to obtain a small number of samples that fulfil certain specific criteria, perhaps n=100, sometimes less. Thus, it is important to take into account the population size/ the incidence rate of any niche samples.

4. Costs

Conducting online surveys costs time and money, and most online panels work by incentivising respondents to participate in surveys through providing monetary rewards. More effort is also required for other research processes such as cleaning data, time taken to survey sufficient respondents and so on. Thus, with higher sample sizes typically comes higher survey costs. For those with budget constraints, costs could be a limiting factor that determines the sample size.

A balancing act

Determining the optimal sample size for your study can be tricky, and it is a balancing act between all the factors we’ve just discussed.

A higher sample size can be more beneficial in terms of accuracy (lower margin of error) and the ability to analyse the data by segments, but a caveat is that it might come at a higher cost and time.

Surveying respondents from a niche population can provide specific insights into your target audience, but they may be harder to reach thus lowering the sample size you can work with.

The most crucial thing to remember is that every survey is different depending on the research objectives, budget constraints and limitations of conducting it, so it’s important to prioritise what is most important to you. At Milieu, we strive to understand the objectives of our clients, and recommend options that will help them achieve their goals in a cost-effective, timely, and reliable manner.

We hope that this article has provided you with better insights on how you can determine the optimal sample size for your study!

‍

Milieu Insight is an esteemed online survey tool and market research agency, partnering with businesses to harness the power of data.

‍

Brand health research is essential for discerning consumer perceptions, evaluating competitive landscapes, and devising impactful marketing strategies, as outlined in our brand health research guide.