1st Feb, 2017: The foundation of good research is good data and the foundation of good data is asking the right questions in the right way.  

Most questionnaires are developed by researchers far removed from the field. No matter how much experience a researcher has, their frame of reference is limited - there’s only so much you can read or imagine about a local context. You must, therefore, test the questionnaire on the field before data collection begins. This exercise is known as a pre-test.  

What is the difference between piloting and pre-testing? 

The two terms are often used interchangeably, but there is one critical difference between the two. In a pre-test, you only test one or a few components of the research study on a small fraction of your intended sample size. During a pilot, you conduct the research study in its entirety, but on a smaller sample size. For example, if you are conducting a mixed-methods study that includes a structured questionnaire, open-ended in-depth interviews and focus group discussions, you may only pre-test the structured questionnaire. However, in a pilot, you would conduct each component of the study but with a smaller sample size.  

Pre-testing in the field 

Pre-testing a questionnaire can help identify problems in the language, tone, structure, and design of a questionnaire. It also provides the research team a brief glimpse of the kind of data they can expect at the end of fieldwork. For instance, recently, while pre-testing a questionnaire on hand hygiene in rural India, the OI team noticed that all the respondents in a village claimed to use water to clean their hands. We then realized, that we had primed the respondents by asking how they “washed” their hands. Using the word “wash” automatically created an association with water in the respondent’s mind! The solution was straight forward, we substituted the word “clean” for “wash” everywhere in the questionnaire.  

At times, respondents fail to grasp nuanced differences between similar sounding questions. While the research team believes that the questions are fundamentally different, the respondents are simply annoyed by having to answer what seems like the same question over and over again. The language of these questions then has to be reworked and the questions have to be contextualized to accentuate their differences. 

The language of the questionnaire needs to be localized. The meaning of every word changes from one region to another, especially in a country as diverse as India. Pre-testing the questionnaire in the areas where you plan to conduct fieldwork can help bring these local differences to the fore. For example, in a questionnaire on schools, middle schools were defined as having grades 1 to 8, whereas, middle schools in Karnataka only had grades 1 to 7.  

How to Pre-test? 

Typically, a pre-test is conducted with 2% of the total sample, with respondents who are characteristically similar to the sample respondent but not a part of the sample pool themselves. A small team including researchers and an enumerator should conduct one round of surveys with researchers taking notes on consistency and reception of questions. This should be followed by a debriefing and modifications in the questionnaire. Ideally, the questionnaire should be tested again with another round of surveys after the changes have been made.

Do you need to pre-test if you are using CAPI? 

Computer-assisted personal interviewing (CAPI) allows the research team to make small changes to the questionnaire in real time even as field work is going on. Do you really need to conduct a pre-test if you are using CAPI to conduct your surveys?

The short answer is yes. The quality of data that you collect and its comparability are compromised if the survey tool is changed significantly from one day of data collection to the next. If the data collection is being done by enumerators, the changes in the questionnaire will confuse them and the quality of data collection will worsen.