Questionnaire Design Process: Create Surveys that get Filled

Any research requires primary data collection for analysis, which is where the skill of designing a questionnaire for market research becomes handy. The Questionnaire Design Process which I will elaborately discuss with you today will be meaningful in whichever primary research and data analysis task you take up.

You might be in your graduation studies, designing your very first survey questionnaire on Google Forms, or might be a research analyst at a leading organization. Our discussion will help you cover all bases, at whichever level of detail you wish to achieve, in this comprehensive pack on tips and tricks for Questionnaire Design Process.

Why do you need a questionnaire?

Every market research study, be it exploratory, descriptive, or causal, needs you to collect significant amount of data for analysis. How will you collect this primary data? And how will you ensure the collected data is in a shape where it can be easily used to generate interesting insights in line with your research objective?

You will realize that, if not in touch with this crucial skill of designing questionnaires, you will struggle to get things right. To ease your anxiety and worries, I am here with a compendium of the most detailed and yet, most concise ‘set of protocols’ to ensure a decent questionnaire design. ‘Decent’, because questionnaire design process does not involve direct calculations or logic. It is an art that needs to be appreciated in its full form to gain the magical results.

I will share with you the basics of how to design a good questionnaire, along with various steps involved in questionnaire construction, in depth. But first, let us understand in detail, the aspects that we will cover in this discussion.

Quick Navigation

So, without much ado, let us dive deep into the rich concepts of a successful questionnaire design process.

Also Read: Difference between Market Research and Market Analysis

Why must you design the questionnaire just right?

While qualitative analysis involves data collection through Focus Group Discussions, Depth Interviews, and Ethnography studies, quantitative analysis needs to be collected based on the following specifications, to achieve meaningful results.

Data Collection Technique should be -

Structured
Standardized set of procedures (across multiple stakeholders of data collection and analysis processes)
Reliable and scalable (to collect significant amount of data)

Data should be -

Coherent
Consistent (across all collection points)
Uniform (in values collected under each question)

Consider this sample questionnaire for market research PDF, to understand how questionnaires typically look like.

We will understand the steps needed to create a questionnaire that is complete in itself. But, first, let us briefly understand the overall flow of the process of designing and delivering an effective questionnaire, through this interesting video -

Now we will discuss the questionnaire design process which will guide you at each stage of the process to finally launch a well-reasoned questionnaire.

Also Read: How to become a Market Research Analyst

Steps to make a wholesome Questionnaire

There’s no magic number or protocol to create the most suitable questionnaire for your audience. The number of questions, survey structure, type of responses expected, everything is subjective, based on research purpose, target audience, and analysis objective.

Still, the process can be categorized into various steps involved in questionnaire construction. Remember, fine-tuning the questionnaire is the creativity of a skilled researcher, but comes easily with practice.

1 Specify Information Needed

This gathers pre-requisite understanding of the research design process. As your project progresses, your understanding of information required gets clearer. But, these few points should be affirmed before moving to the next step –

1
Identify components of problem to analyze, mainly research question or hypotheses
2
Identify type and format of information needed to analyze
3
To ensure the information identified in point #2 is sufficient to answer the research question, create a dummy table. A dummy table will depict the questionnaire outcome which will be used for data analysis.
4
Identify target population of respondents (this will influence many following steps such as questions wording, and type of opinion to collect). Ensure you choose a very specific target group, to avoid making a broad questionnaire that doesn’t give meaningful depth in responses (such as more no-opinion answers).

2 Specify Type of Interviewing Method

Questionnaire design process has delivery method based on the complexity, duration and variation of questions. For example, a face-to-face interview can have lengthy, scenario-based, complex questions to be answered by respondents (due to respondents’ physical presence) But, for mail or internet surveys, simple questions and detailed instructions are needed. Computerized or Computer-assisted interviews can have complex skip patterns and randomization of questions to eliminate order bias (bias in answering a question based on the previous sequence of questions and the resulting thought-process). Further, based on mode of questionnaire delivery, questions language can be decided as conversational or direct.

3 Individual Question Content

1
Is the question necessary? – Ensure that the response from the question contributes directly and significantly to your data collection (the Dummy Table), and ensure that a particular piece of information is captured only once in a questionnaire (no repetition).
2
Are multiple questions needed instead of one? – Avoid double-barrel questions. What are double-barrel questions? A single question that tries to cover two issues. For instance, “Do you think Amazon Pantry is a cheap and convenient option to purchase groceries?” is a double-barrel question, whose answer can have multiple parts. Hence, the response’s reliability and validity are at stake. Instead, break the question into multiple parts, and collect responses for each component in an individual question. So, "Do you think Amazon Pantry is a cheap option for groceries?" and "Do you think Amazon Pantry offers convenient purchase options for groceries?" are now two distinct questions.

4 Overcoming Inability to Answer

My mother was asked to fill a customer feedback form at a retail outlet while exiting post a purchase on a busy Sunday evening. The form was 4 pages long and had many sections of open-ended questions. She ended up partially filling the survey, and submitted it for consideration. Imagine a sizeable ratio of such submissions for your survey, and the amount of data cleaning each inaccurate response would require.

You need to identify common problems faced by respondents while filling up your questionnaire –

Uninformed
Unable to remember
Unable to articulate
Not willing or disinterested

Uninformed

To ensure quality of responses (or that respondents are aware of what you are asking), add filter questions in the beginning. Filter questions are added to screen respondents so that only the target audience actually send the responses for consideration.

Unable to remember on cue

How many times did you wash your hands this week?

When was the last time you called any customer care?

Were you able to answer these questions? Didn’t they send you off at a tangent, and ended up frustrating and distracting you? Instead of collecting factual information, which more often than not, is incorrect by over 100% of the actual answer, information can be collected in a range of time duration, or qualitatively. For example, now consider answering the questions again based on these options –

A. Never

B. Rarely

C. Sometimes

D. Often

E. Numerous times

Inability to remember leads to any one of these -

Omission - Inability to recall an event that actually took place in a particular time in the past (Maybe it wasn’t significant or memorable, like checking your Facebook last month).
Telescoping - Psychologically, we compress time (or telescope) by remembering an event’s occurrence more recently than reality. For example, you mentioned visiting the nearby grocery store twice in the last two weeks, when in reality you visited it the first time more than 20 days back.
Creation Error - We might even remember events that never actually happened, but seem they did happen upon coming across the relevant question. For instance, you might feel that you received a delayed delivery by Amazon if the question asks, since you might perceive it to be a frequent event without ever having experienced it before.

Unable to articulate

You might find it difficult to describe the atmosphere in an airline while flying, but if given options to choose from, you will be in a better position to choose the best fitting answer. It is good to facilitate options with visual cues such as pictures, maps, or even detailed descriptions.

Unwilling or Disinterested

This is my personal observation that people who are busy or will be occupied in work soon tend to avoid filling questionnaires or rush through to get done. Often times, while delivering the questionnaire to respondents, timing plays a key role in getting the survey filled with the desirable attention you expect. Few pointers to hold respondents’ interest –

Ensure least effort in filling the form, with more information given as options to choose from, lesser options to decide among, mutually exclusive options, options with the right level of details expected (neither too specific nor too generic).
Ensure to maintain context of the survey. People will be comfortable to provide specific information to you only if you are a reliable source and a subject matter expert at the topic. For example, people will be more comfortable telling a medical research institute about their blood cholesterol levels than a fast-food joint. So, the fast-food joint can present this question with a mention of an attempt to offer cleaner meal options to their customers.
If as a cereal-manufacturing company, you want to know people’s occupation or income level, provide details of why the information is necessary for your analysis. A simple statement saying “To analyze the variation in consumption of cereal across different demographics of consumers, we need information on…” will ensure legitimacy.

5 Choosing Questionnaire Structure

Unstructured

Open-ended questions are ones where respondents are free to answer in any way, with any type of information. While they capture general attitudes and behaviour of respondents and are much less biased in responses, unstructured questions pose major disadvantages. Coding the responses is one of the major issues, where contextual and subjective answers are difficult to accommodate within the analysis framework. Also, they may involve interviewer bias where the interviewer may selectively include or exclude certain pieces of information.

Structured

These have a specific set of response alternatives to choose from while responding. These are of three types –

1
Multiple Choice - Respondent is given choice of responses to choose one or multiple from. For instance, a question asking “Do you intend to buy a smartphone within the coming 6 months?” can have response options as - 'Definitely will not buy', 'Probably will not buy', 'Undecided', 'Probably will buy', and 'Definitely will buy'.

MCQ-type questions face two issues -
1 Number of Alternatives - Response choices must include all possible variations and must be mutually exclusive. If the question becomes too complex (involving high information-processing), consider splitting the question into two parts.
2 Order or position bias - Respondents tend to check the first and last options to observe the range of variations, and often, among numeric choices, tend to answer around the mean. To eliminate this bias, randomize the order of options.
2
Dichotomous - Response choices come in pairs, with ‘either’ ‘or’ options (Yes/No). These must be applied very selectively, and should be avoided in case there is any grey-area in answering. One major issue still persists about the influence of the questionnaire structure in indicating response. To overcome this, the split ballot technique can be used with questions having biased connotations in language are given to 50% respondents in its double-negation form and the other 50% as is.Consider the following example -
“Individuals are more to blame than social conditions for crime and lawlessness in this country” produced 59.6% agreement. However, the opposite statement “Social conditions are more to blame than individuals for crime and lawlessness in this country” produced a 43.2% agreement, as opposed to the expected 40.4%. How can we overcome this? Both the formats of the question must be sent to half the randomized audience.
3
Scale - Consider the sample question of purchasing a smartphone in the coming 6 months, and consider the following response choices -Definitely will not buy - 1
Probably will not buy - 2
Undecided - 3
Probably will buy - 4
Definitely will buy - 5
This is another format of offering question responses, similar to the multiple-choice technique, and directly offers coded responses for data analysis.

6 Choosing Question Wording

Language in which questions are framed are known to make or break the deal when it comes to respondents investing time and effort in responding to your questionnaire. It is a crucial step because –

Response error occurs when a questionnaire is incorrectly responded, due to multiple factors, as already seen in previous section on ‘Overcoming inability to answer’
Item non-response happens when a wrongly worded questionnaire doesn’t get responded at all.

What can we as researchers do to tackle such problems? A 7-step guide will help you –

1
Define the issue - Complete the context and meaning of the question clearly by the six W's concept – who, what, when, where, why, and the way.
2
Use ordinary words - You might be a Shakespeare in Literature, but a survey isn’t your canvas of creativity and flamboyance. Match the vocabulary level of the respondents, especially in cases where your target respondents are responding to a very domain-specific questionnaire.
3
Use unambiguous words - Words in the question and choices must have only one meaning in the context. For instance, consider the two sets of response options for the question “In a typical month, how often do you hop at departmental stores?” -
'Never', 'Occasionally', 'Sometimes', 'Often', and 'Regularly'
'Less than once', '1 or 2 times', '3 or 4 times', '4 to 6 times', and 'More than 6 times a month'

Which options seem clearer and more intuitive to select from? The second one, right? It is because there is no perception bias from the respondent leading to inconsistent data. Remember that certain words which are all-inclusive (“all”, “always”, “every”) and all-exclusive (“any”, “none”) might be perceived at different degrees by people.
4
Avoid leading questions - Leading questions are ones which clue the respondent about the desirable answer, which results in yea-saying or acquiescence bias. A question saying “Will you consume fast-food knowing the associated health problems?” will attract more ‘No’s since the associated ill-effect mentioned right next to it tends people to say/believe that they won’t indulge. Try asking the question as “Will you consume fast-food?” and It will collect a more realistic response.
5
Avoid implicit alternatives and assumptions - Implicit alternatives make it unclear, and increase the probability of choosing the alternative mentioned in the question. For instance, “Would you like to fly when travelling short distances?” will answer more inaccurate ‘Yes’ than “Would you like to fly when travelling short distances, or would you rather drive?”. Close preferences can be missed while framing the question, but not in most cases.
6
Avoid generalizations and estimates - Ask specific questions, which seek direct answers, and frame questions such that respondents create no generalizations or own assumptions while answering. To find out domain-specific numeric information from respondents, do not ask “What is your family’s per capita expenditure?”. Instead, ask direct and easy questions like average expenditure per week, and number of family members, and calculate per capita expenditure by self based on responses.
7
Use both positive and negative statements - For attitude and personality-recording questionnaires, often questions are worded as statements, with responses to be collected as degrees of agreements or disagreements. In such cases, it is better to have dual statements randomly (both positive and negative) such that each questionnaire has half positive and half negative interspersed statements.

7 Determining Order of Questions

The opening question must gain confidence in respondents, ask their opinion, and seek assistance in a simple, interesting, and non-threatening way. This is also the section to qualify and segment target respondents and check their eligibility to answer the questionnaire.
Classify the type of information obtained in the questionnaire into – Basic information (answers directly related to the research problem), Classification information (socio-economic and demographic characteristics), and Identification information (used to verify respondents and/or remit proposed incentives). Obtain basic information first, followed by classification information and finally identification information. Basic information, being most crucial for the research, should be asked first to avoid alienating the respondent later on with a series of personal questions.
Your questionnaire might have sensitive, complex, embarrassing, or in general difficult questions to ask. Place them late in sequence, so that a rapport with the respondent gets established.
Questions asked early in a sequence can influence responses to subsequent questions. So, general questions must precede specific questions, as a rule. Consider following the funnel approach to sequence questions beginning with general questions and followed progressively by specific questions, so that specific questions do not bias the general questions.
The overall questionnaire must follow a logical order with smooth transitions from one section to another. Questions about a particular topic must be grouped together, by funnel approach, and transitional phrases must be used to help respondents switch their train of thoughts.

Designing the flow of the questionnaire accurately is important. Try to create a flowchart clearly sequencing the questions that lead to the respondent being navigated to different sections of the flowchart (called branching questions).

8 Form and Layout

General guidelines of maintaining uniform formatting, spacing and positioning of questions affect the accuracy of end outcome of the questionnaire design process. Precoding of questionnaire responses must be done prior to launch. Precoding means assigning a code to every conceivable response for data collection. Questions and sections must be numbered serially, to easily control the directional flow of information from the questionnaire.

9 Reproducing the Questionnaire

Carefully reproducing the questionnaire for administration is another critical step in the questionnaire design process to ensure it actually gets filled with the right intentions. Visually, it should be de-cluttered, should appear professional and the questions and section breaks must be clearly visible and distinctly demarcated. Improve readability through visual cues such as instructions to questions in different font style, and sections of the questionnaire color-coded differently.

Online questionnaires must ensure access permissions to respondents and must include appropriate settings about including or excluding the respondents’ email IDs.

10 Pre-Testing

Even after such as detailed scrutiny of the questionnaire design process, there might be elements whose representation might create some unknown issues in the mind of the respondents while responding. Keep it a habit of pretesting the questionnaire on a smaller group of respondents with an aim to improve the questionnaire by identifying and eliminating potential problems that they face. You will also get a view of the data being populated and will be able to make changes even in the responses. It is good to analyze data collected here to ensure your research objective or hypotheses will be answered through the data collected.

Two ways of proper pretesting are protocol analysis and debriefing.

Protocol analysis makes the respondent think out loud while answering the questionnaire. The remarks are recorded and form the basis for editing later on.
Debriefing occurs after the questionnaire is filled when respondents are told the pretesting purpose and are then asked to narrate their understanding of and state the issues faced during answering the questionnaire.

Finally, edit the questionnaire based on inputs obtained from pretesting. Your questionnaire is now finally ready to hit the road!

Also Read: How to do insightful Secondary Market Research?

Conclusion

While an all-encompassing summary of the questionnaire design process would offer a good summary to the vast yet wholesome topic of how to design a good questionnaire, few steps to keep in mind are –

Know what information is needed as an outcome to the questionnaire
Identify the type of interviewing method
Know the content to individual questions and to each section
Overcome the respondent’s inability and/or unwillingness to answer
Decide the question structure and appropriate wording
Sequence questions and sections in right order, ensure right branching layout
Identify form and layout
Reproduce and ensure the format of administration meets the reproduced questionnaire
Eliminate bugs by pretesting

So, the next time you are designing a questionnaire for market research, follow this checklist of items to achieve in a smooth roll-out of your survey questionnaire.

Found the article interesting? Share it with your friends: