Questionnaire design is one of the core skills of every researcher, and most of us think we’re pretty good at it. However, when you’ve been doing something for a long time, it’s easy to fall into habits and take some things for granted. Response rates are falling at an alarming rate, which means we need to take more care of our participants.
Participants are often overlooked in research - in many cases, surveys are bordering on a traumatic experience for the people taking them. When designing a survey, we often focus on giving our clients what they want (which of course we should focus on, as we all want to get paid). Our next priority will be making lives easier for ourselves (think massive grids for easy DP and programming). Then, if we have time, we might consider participants. Of course, there are always trade-offs to be made, but we should always remember survey design is a balancing act between 3 groups of people:
Attention: we cannot process everything we see at the same time - the mind only processes the most important parts
Calculation: we cannot accurately calculate all of the outcomes and consequences of our choices
Myopia constraint: we focus on the here and now, meaning we can only act on how we’re feeling in the present moment.
2. Research barriers influence participants’ responses in the context of research:
Self-knowledge: we don’t always know everything about ourselves to accurately answer survey questions.
Honesty: we might not want to tell a researcher everything about ourselves because it’s embarrassing or painful.
Memory: our recall of past events and experiences can be influenced by characteristics of both the event being recalled and the question asks:
3. Survey specific barriers influence participants’ answers in online surveys:
Comprehension: the language used in questions, as well as people’s cognitive abilities, determine how well people understand survey questions
Cognitive effort: participants may tire and stop investing cognitive effort into surveys, resorting to 'satisficing' (just giving the bare minimum) in the responses they provide.
It is important to be aware of these constraints and barriers when designing questionnaires.
These will help you think about the audience you are writing for. These questions may seem obvious but when we look at many of the surveys out there, it’s clear our industry is not giving participants the thought and consideration we should.
Who is your audience? What sorts of constraints and barriers will apply to that group in particular?
Is the survey too long? Will it realistically hold the attention of your audience?
Can they understand the questions you are asking?
Can the participant be realistically expected to know the answer, and to know themselves well enough?
Does the question ask participants to predict future behavior?
Does it rely on participant memory?
Will the participant tell the truth?
Would you want to answer these questions? Would you spend 15 minutes of your time doing this?
We hope you’ve learned something from this post that you can put into practice. These posts are based on an intensive workshop that we run for clients (often for free!). If you are interested in hearing more about this workshop, and others, please contact us. You can also read more on some common issues in questionnaire design in this article.