Questionnaire Design: Common Issues

Understanding your audience is key but being able to implement this in a way that allows you to get the right information from the audience is vital. Surveys typically measure both factual information and subjective states through different question types which all have their specific issues. More broadly the flow and method of how the survey is delivered to participants as well as its visual appearance are important to ensure quality data.

1. Design: both the way the survey is structured online and more visual elements can have a real impact on how participants view and therefore answer online surveys

2. Response device: a number of device types (laptops, smart phones, tablets etc) will be used to take surveys

3. Question order: questions can be differently interpreted and answered depending on what precedes and follows them

Let’s look in more detail at each of the areas, and what you should do to combat common issues.

1. Design

The first thing to remember about design is that there are many elements that essentially come down to personal taste. Therefore, what you are looking to do is create clear and neutral designs, which allow for easy reading.

1. Back/foreground design: Keep it clear and readable, with enough white space. Lack of contrast between the background and text can make it difficult to read especially if the font size is small – it’s also good for accessibility. If you are going to use color be mindful of color associations, especially across cultures. If in doubt, stick to neutral colors.

2. Font/typeface: When choosing a font your first consideration should be readability. Beyond that, neutrality is key. Whilst some people like cursive script fonts, they’ll make some people cringe to the point of closing out of a survey.

3. The layout of questions: In most (but not all) cases avoid grids, but if you have to, try to ensure no scrolling is required. Scrolling vertically or horizontally is very taxing and leads to more dropouts, especially on mobiles. Try to keep multicode lists to one screen, or split them across screens, rather than leaving a list of 50 options on one page.

2. Response Device 

Surveys were built for desktops, and while we have come a long way in adapting them for laptops, tablets, and smartphones it’s often not enough. Long-running trackers in particular are hard to adapt and, as an industry, we don’t always make enough effort to do that. New surveys should be written with the end device in mind and if this means having 3 versions of the survey, so be it. Sometimes mobile surveys need to be shorter, so this practice can be a great way to focus on what you really need to ask, leaving out the surplus “nice to knows”.

3. Question Order

Before we get down to the detail of how to ask specific questions in an optimal way, we need to consider the order in which we question. You’re probably familiar with the funnel technique (moving from general, less personal topics to more specific, potentially more sensitive ones), but how often do you consciously review the order of your questions?

The sequence in which questions are asked can influence participant answers in two ways:

A. Serial order effect (location in a sequence of items): Serial order can operate in at least three ways: by affecting motivation, promoting learning, and producing fatigue. 

For example, the beginning of the questionnaire may influence the willingness to respond to the survey, as they shape a participants’ understanding of what the survey is about.

B. Semantic order effect (location in a sequence of meanings): Semantic order requires the questionnaire to flow coherently, which usually requires that items on related topics be grouped together. 

For example, consider how your answer to both these questions would change, dependent on the order: 

 

 

Someone who is unhappy with their career might well rate their life happiness level lower if they see the career question first. In some cases, you might want to fix the order of your questions, but in cases like this, where there is a clear bias both ways, then you would probably rotate the order they are asked in (dependent on the rest of the questionnaire).

Although these topics might seem obvious and basic, think about when was the last time you reviewed a tracker or other existing questionnaires consciously and thoroughly? Maybe it’s time for an audit.  

These are the three main areas to consider before you start looking at individual questions. We hope you’ve learned something from this post that you can put into practice. These posts are based on an intensive workshop that we run for clients (often for free!). If you are interested in hearing more about this workshop, and others, please contact us.