Long survey lengths lead to bad data - Here are 11 ways to keep your survey short

Let’s start with a simple question. Would you take an hour-long or even a 30-minute-long survey on your phone, with unwavering attention? The likely answer is no, and that’s only human.

The troubling reality though is that several brands continue to run really long survey engagements. The intention is not misplaced i.e. to get a deeply rich profile on consumers. However, the quality of data you get from these long surveys is often compromised.

Excessively long surveys alienate people who are time poor. With various activities and apps competing for a respondent's attention, it is important that your survey is short enough to catch them during their little pockets of available time - on their way to work, while queuing for food, between chores, or just before bed time. Notably, those who go on to complete long surveys risk being coupon warriors (motivated by rewards) or an unrepresentative fraction of the target population.

We ran a set of controlled experiments and our data shows that 10 minutes (appx 40-50 questions depending on the platform) is the breaking point in an online survey setting beyond which you will start to see response quality deteriorate in the following ways :

1. Increased randomisation of responses : Likelihood of respondent picking a random response (instead of careful consideration) doubles
2. Poorer quality of answers in open-ended questions : An increase in gibberish answers, non-committal answers (e.g., NIL, no comments), and decrease in length of responses
3. Less opinionated, more neutral responses : Increased likelihood in picking a "neutral" response option
4. Lower respondent satisfaction : Sharp decline in survey satisfaction ratings by respondents

Given the impact of survey length on data quality, we were curious to find out how Milieu’s surveys fare in their lengths. And the results are reassuring.

Data shows that across 4,103 surveys run on Milieu’s platform in 6 markets since 2017, the majority (76%) of surveys consist of 30 or less questions. In terms of time, this translates to the majority surveys being 6 minutes or less in duration. On average, a Milieu survey is about 25 questions long (~5 minutes) - just nice to finish during a coffee run!

The learning here is that most of our clients get actionable insights by running less than 30 questions in a survey - and all this without the worry of data quality looming over their minds.

If you’re thinking about designing a survey, we can’t stress enough on the importance of keeping it succinct. But fret not, here are some handy tips on how you can keep the question count in check.

11 best practice tips on how to keep your surveys short :

1. Follow the 10-minute rule.

‍A good rule of thumb is to ensure your survey does not exceed 10 minutes (~ 50 questions on our platform). Ideally, you should limit it to 20-30 questions for the most optimal quality of results. However, if you need to go beyond that we recommend a hard stop at 50 questions. Anything longer and you run the risk of respondent fatigue kicking in and losing their attention - the perfect recipe for bad data.

2. List your objectives clearly and stick to them.

When designing a questionnaire, it is natural to be tempted to pack in as many questions to make it comprehensive. To avoid falling into this trap, start by listing down the objectives and outcomes that are crucial to obtain from the survey. Think about 3-4 broad objectives you can thematically divide your questionnaire into, and populate each theme with only relevant questions. Each question should fight for its place in the survey. If a question doesn't serve your objective and/or is redundant, strike it off. This way you’ll be more likely to stay on track.

3. Tag your questions to a priority level.

‍Your first draft is often going to exceed the target question count. It’s always a good idea to assign each question a priority level so that you can differentiate those that are "absolutely important" from those that are “nice to know”. This makes it easier to know which questions can be sacrificed on the chopping block later.

4. Avoid having too many cooks.

‍Circulating the questionnaire among various teams and stakeholders can often lead to a ballooning question count. To avoid this, have a clear team or a person-in-charge who has the final say. If you have to involve others, it is good to gather feedback from various stakeholders at the outset before working on the questionnaire. However, limit the number of people you send the questionnaire for reviews and comments once the questions are drawn up. Trust us, we know how hard it is to say no!

5. Choose the best question type.

There may be multiple ways to get to an insight you need. The question you need to consider is: are you asking it the best possible way? We often work with clients to polish up a questionnaire and a common occurrence is to ask multiple questions around the same topic when you can club them into one well-thought out one.

For instance, we had a client who wanted to know people’s sentiments towards brand advertisements. They had five hypotheses on how advertisements are perceived, and wanted to gauge how popular they were relative to each other.

They were initially crafted as five separate agreement scale questions. For instance,

Q. To what extent do you agree or disagree with the statement : My gender is often not represented accurately in advertising and brand content. 

• Strongly agree
• Agree
• Disagree
• Strongly disagree

While this is not wrong, it could have been rolled into one question as follows.

Q. Which of the statements below do you agree with when it comes to advertisements and brand content?

1. My gender is often not represented accurately
2. Statement 2
3. Statement 3
4. Statement 4
5. None of the above

This helped them arrive at their required insight i.e how popular are the perceptions compared to each other without getting too granular. However, it is important to not pack in too many response options into one multi-select question as this would increase the cognitive burden on the respondent, defeating the purpose of reducing question count.

6. Reduce cognitive load in questions.

‍Question counts can often belie the effort needed to complete a survey. For instance, a survey with rating scales is cognitively less burdensome relative to one replete with open-end questions. Hence, beyond the question count it is worth examining how “demanding” each question is. Avoid having too many open-end questions, long and wordy response lists in close-ended questions, and grid/matrix questions since they require a lot of mental effort and the survey may end up feel longer (read here on how grid questions can be bad for your survey data).

7. Use skip logic / survey routing.

To ensure respondents get only questions relevant to them, introduce some sort of survey routing, also known as skip-logic or branching. For instance, if someone indicates that they are not aware of a brand, you can skip asking them questions related to that brand. This not only shortens the survey length, but it also provides a much better user experience compared to having them go through all the irrelevant questions and respond to them with "not applicable". This will also help increase data quality by reducing the chances of conflicting answers.

8. Split a long questionnaire into multiple surveys.

‍If your questionnaire is leaning towards being long and tedious, you can consider modularizing your questionnaire into bite-sized surveys instead of sending it out as a 1 'big-whopper' survey. Consider splitting it by categories as this will make it easier for respondents to answer. At Milieu, we often help clients adapt traditionally run large surveys into smaller bite-sized surveys. This helps shorten timelines, arrive at better quality data, and you can run them more frequently and regularly throughout the year rather than 1 single drawn out field work period.

9. Be selective about the demographic questions.

‍Clients often preface core questions in a survey with demographic questions to get a deeper understanding of the respondents. For instance, if you’re running a customer satisfaction survey, beyond their feedback on your product/service, you might also want to know their age, gender, location, etc. Here too it is important to exercise restraint and limit them to variables that will be crucial for your analyses. At Milieu, we are able to keep average survey lengths short because we offer clients a solution to skip asking basic demographic questions and a host of other profiling data. The reason being Milieu has pre-collected data on more than 1,000 profiling variables for every respondent who joins our panel. These data range from demographics, general attitudes, traits, media behaviours as well as sector based behaviour. This allows clients to cut to the chase when it comes to asking questions in a survey. However, while viewing the results, the data comes attached with various other pre-collected profiling data, thus not comprising the richness of consumer data clients desire.

10. Use available pre-designed questions or survey templates.

‍Designing surveys from scratch can be a daunting task for many. Where possible, you can leverage on available standardized questions (e.g., NPS questions, brand funnel metric questions) or questionnaire templates from various survey platforms. Milieu too offers an extensive library of pre-designed survey templates that one can choose from and customize for their own study. This will not only assure you that you are asking the right questions in the right manner, but it will also ensure the questions are optimized to measure what you desire effectively.

11. Test your survey.

Last but far from the least, put yourself in the shoes of the respondents. There’s no better way to sense-check your survey length than by doing it yourself. If you find your attention and motivation levels dwindling towards the end, there's a good chance respondents are going to feel the same. Testing a survey will help you remove questions or response options that feel repetitive or non-essential to your objectives.

That's all for now! We hope these tips help you in designing a short and effective surveys in order to arrive at high quality data.

‍

Feel free to check out more survey design best practices under our Learn Section.

If you have questions related to Milieu's survey platform and/or other product/service offerings feel free to reach out to sales@mili.eu.

‍

Milieu Insight is a renowned survey software and panel research company, supporting businesses in their data-driven endeavors.