 What cognitive interviewing is and why we might use this as a research method. In finishing this section, we will learn about the theory which underpins the cognitive interviewing technique. Cognitive interviewing is a set of qualitative techniques which can be used to check if questions are working or not. As this method is rooted in cognitive theory, this method explores individuals' thought processes when presented with a task or information. In particular, this method looks into how individuals interpret information, how they understand what is being asked of them, and how this information is recalled. As a method, cognitive interviewing is traditionally used in the pretesting of surveys, but as we will discover in this resource, it can be used in many other ways. Cognitive interviewing is the practice of administering a survey questionnaire while collecting additional verbal information about the survey responses. This additional information is used to evaluate the quality of the response, or to help determine whether the question is generating the sorts of information that its author intends. There are a number of reasons why we might want to use the cognitive interviewing method to test things like our survey questions. When designing a questionnaire or survey, we want to make sure that it is a good instrument. To have a good survey or questionnaire, it needs to be the following. Firstly, our questions need to be valid in that the questions measure what we want them to measure and only that. Cognitive interviewing can help us ensure our questions are valid. In a cognitive interview, we can ask individuals what they think the question is asking them about. To give you an example, imagine if I asked you to rate your health on a scale from 1 to 10, with 1 being of poor health and 10 being excellent health. It might be easy to give me an answer, but in a cognitive interview, we would be wanting to know more about how you got to this answer. In a cognitive interview, we might ask individuals what they understand by the term health. Doing this is beneficial, as some people might think of health in purely physical terms, whereas others may think of health as a combination of their physical and mental health. How people interpret the term health may affect the score individuals give for this question, which could compromise the validity of this question. Secondly, we want to make sure our questions are reliable, in that the responses do not contain too much random variability, and if a survey was to be repeated over time, we would be likely to achieve consistent results. We can use the cognitive interviewing method to ensure that our questions are consistent, and would therefore produce reliable answers. Imagine if you were asked the question, does your workplace adhere to the European Working Time Directive, with the response options being yes, no, or don't know? Chances are, most people would not know what the European Working Time Directive was, and would therefore select don't know, not because they do not know if their workplace does this, but because they do not know what the term means. In a cognitive interview, we could explore the reasons why people selected don't know, and make changes to the question which would allow for a more reliable answer. For example, we may suggest using a simpler term, such as, does your workplace allow you to take at least four weeks paid holiday a year? Thirdly, we want to make sure our questions are sensitive, in that they are appropriate to ask, and measure real change and differences. Imagine if we wanted to measure, on an average day, how long individuals spent watching television, with the response options being less than an hour, more than one hour but less than two hours, and more than two hours. In a cognitive interview, we could explore this question to see how sensitive this question was. For example, would the option, more than two hours, be suitable enough to capture how much television people watch in an average day, or would it need to be higher? Fourthly, we want to make sure our questions and response options are unbiased. When we design survey questions and response options, we always want to ensure that our questions are balanced and not leading. However, sometimes biases are not always obvious. Remember when I asked you to rate your health on a scale from one to ten. Depending on your background and your personal experiences, some of you may have rated based on physical health alone, whereas some may have rated based on overall health, including mental health within this score. If you ask this question within a general population, we may expect to see some biases, in that younger people may be more likely to consider their mental health, whereas older populations may be more likely to consider the physical health only. Through using cognitive interviews, we can explore individuals' perceptions of the questions, and can fully explore if these questions are biased. Finally, we want to make sure our survey questions are complete. By this, we mean that missing data is reduced, and that the questions are designed in a way which allows for full data. There are a number of ways in which cognitive interviewing can be of benefit here. In a cognitive interview, we can explore if there are any questions which may cause an individual discomfort in answering. Imagine if you were asked, do you have a criminal record? If the survey was about crime, individuals may feel more comfortable in answering this question. However, if individuals perceive this sensitive question to be unrelated to the survey topic, this may cause individuals to not answer this question, or worse, to withdraw from the survey completely. In cognitive interviews, we can explore if any questions have any particular concerns about them to participants. We can see how individuals interpret these survey questions, and we can also explore if any questions would likely lead to missing data in that participants would not answer these questions, or worse, they would withdraw from the survey. So we've now discussed what cognitive interviewing actually is and why we should use it, but what actually happens within a cognitive interview? So cognitive interviews are typically one-on-one sessions between a trained intuer and a participant. Cognitive interviews traditionally have taken place in a participant's home, workplace or within a neutral interview location. However, since the pandemic, interviews now typically take place online. Participants are asked the survey questions or the survey materials we want to test in an environment that best mirrors how they would be asked in a real-life setting. For example, if the survey is meant to be administered by an interviewer in a face-to-face setting, the interviewer will read the questions allowed to the participant. However, if the survey is meant to be self-completed by the participant, the participant will complete the survey without the interviewer reading aloud the questions or providing any assistance as to how they should respond. When the participant is answering the survey questions, in a cognitive interview we're more interested in focusing on the mental processes used to come up with the answer rather than their actual answer to the question itself. During a cognitive interview, we're interested in exploring the following mental processes. Firstly, we want to explore how an individual interprets the survey question and any other important terms within that question. Secondly, we want to know how an individual retrieves and recalls the information needed to answer the question, including any estimation strategies they may use. Thirdly, we want to know how an individual judges and perceives the question and their comfort level associated with answering. And finally, we want to know how confident an individual feels in answering a question, including how accurate they think their answers are. The model on this slide is a visual representation of how these mental processes work together. What's important to remember is that these stages are not necessarily sequential and that they may overlap and cross over. To illustrate how these models work in practice, I'm going to imagine that I was asked the following question. So imagine I was asked the question, how many hours do you work in a week? Firstly, I would have to comprehend the question, so I have to think about what this question is asking of me. And as I've just read, I can see that this question is asking me how many hours I work in a week. Next, I must retrieve this information. So for me, I may be thinking about the number of hours I worked in the last week, as this is easier for me to think about and remember as I have flexible working hours. To do this, I might be thinking about other activities I did in the last week, both before and after work, such as doing shopping, going on walks and going out for dinner. Then I have to make a judgement about how accurate this question is requiring me to be. For example, could I make a guess of around 20 hours per week, as this is what I'm contracted to do? Or would I have to really make the effort to be specific and think about the exact number of hours I worked last week? And to what level would they be requiring this information from me? Would it be to the nearest kind of 20 minutes, half an hour, or could I go to the nearest hour? Finally, I must respond in the format that the response options are given to me in. This might mean that I have to adjust my in-mind answer if it doesn't quite fit the response options that I'm presented with. And it might mean that I have to pick the most suitable response option of the ones that I'm offered. We are now at the end of the section on cognitive interviewing theory. Hopefully you will now have an understanding of what the cognitive interviewing method is and why we use it. In the other sections of this resource, you can learn about how to conduct a cognitive interview and other situations where the cognitive interviewing technique is beneficial.