 By the end of this presentation, you will be able to understand the different techniques you can use during a cognitive interview to help stress test your survey questions. You'll know how to produce effective cognitive interview protocols. These are also known as topic guides. We'll give our top tips on how to write these to get the most out of your testing. Finally, in this session, we'll describe one approach for the analysis of your cognitive interview data. Before we start, let's have a quick recap of what we mean by the term cognitive interviewing. Cognitive interviews are qualitative interviews where we attempt to gain insight into the mental processes people go through when they are completing a task. Cognitive interviews are often used as part of questionnaire development. The aim of cognitive interviews in this context is to establish whether survey questions are fit for purpose. For example, are questions understood in the way we intend by respondents? Are respondents able to give accurate answers? If not, questionnaires may collect poor quality or biased data. There are four main techniques used during cognitive interviews. These are administering the survey questions, making observations, collecting think-aloud data, and collecting further data from qualitative probing. The first part of the cognitive interview is to administer the survey questions that you wish to test. It is important that, as far as possible, these questions are administered in a context that replicates real-life survey conditions as far as possible. For example, if you are testing a question that would be asked in a face-to-face interview or a telephone interview, then the question should be read out aloud by an interviewer. In contrast, if you are testing a self-completion questionnaire, participants should read the questions themselves. Self-completion questions should ideally mimic the intended mode of administration. For instance, web questionnaires should be presented online, and paper questionnaires should be given to respondents in a formatted paper prototype. Interviewers should not attempt to help participants answer the questions during this stage of the interview. Whilst participants are completing the test questions, cognitive interviews should observe participants closely. Do participants ever hesitate when they are answering any of the questions? Are there ever overt signs of frustration or boredom, such as sighing or eye-rolling, and so on? Do participants ever directly ask you for help? Do participants ever revisit questions and change their answers? Do participants ever miss out questions or skip any part of the task set? The purpose of making observations is for interviewers to get non-verbal cues about the mental processes participants are going through, especially cues that may indicate a problem. An experienced interviewer can then follow up on non-verbal cues with so-called observational probes. These observational probes encourage participants to verbalize their thought processes. For example, if an interviewer observes someone hesitating or eye-rolling, they can inquire, can you tell me more about what you're thinking about now? If an interviewer observes a participant changing an answer to a question they can ask, I notice you changed your answer. Why is that? If a participant goes back to an earlier section of a questionnaire, or accesses a help screen, an interviewer can ask, what are you looking for now? Can you find the information you're looking for? If participants skip a question, interviewers can ask, why did you miss out on that particular question? And have a technique used in cognitive interviews is the think-allowed technique. To use this technique, participants are told at the start of the interview that we want them to verbalize all of their thought processes when completing the survey questions. This does not come naturally to everyone, so prior to admissing the test questions, some training in think-allowed is required. During this training, the interviewer demonstrates the think-allowed technique themselves, i.e. by completing a task while verbalizing their own thought processes. After the interviewer has thought-allowed for the practice task, participants are asked to do the same practice task. One practice task we use in think-allowed training is called the Windows example. Then I was asked to say how many windows there are in my home. If I was answering this question while thinking-allowed, I would say, I'm picturing my front door, there's a pane of glass but I don't know whether to include this, there is a window in the living room, and a glass door in the kitchen which I will include, and there's a window in the bedroom and in the bathroom, so approximately four windows. What about in your home? Please tell me what you're thinking about as you answer. As a participant has the hang of thinking-allowed, you must remind and encourage this behavior throughout the entire interview. Don't forget to tell me what you're thinking. The final technique in our cognitive interviewing toolbox is probing. The majority of cognitive interviewing data is likely to come from probing. Probes are basically questions about the product we are trying to test. As with other techniques, the purpose of probing is to elicit verbal information about what respondents are thinking about when they are exposed to our survey questions. We would typically start by asking general probes. An example of a general probe would be, how easy or difficult did you find that? Why do you say that? We would then ask probes based on each stage of Tarangio's four-stage question and answer model. We would probe on comprehension, retrieval, judgment, and response. For example, if we wanted to explore comprehension, we could ask the probe, in your own words, what is this question asking? Or we could ask, what does this phrase mean to you? Retrieval probes explore how easy or difficult people find the calling information. Judgment probes explore whether people are giving precise answers or just estimates. Response probes explore whether our list of response options are clear and complete. Example of all these types of probe are shown on this slide. Next we will talk about how to produce your cognitive interview protocols. Your protocol is a document that describes, in detail, how cognitive interviews should be conducted. This includes information on how the study should be introduced, what the test questions are, and how these will be tested. Protocols also provide a list of suggested probes to use. An example protocol has been provided as a resource for you to use, in addition to this presentation. It is important to remember that protocols are designed to be guides, not scripts. We have already described how observation and observational probes are an important technique in cognitive interviews. While some probes are spontaneous rather than scripted, these probes are based on what interviewers can see in the interview. Likewise, it may not be necessary for interviewers to ask all the probes listed in the protocol. For example, you may find that a participant who is good at thinking out loud will naturally talk about comprehension, retrieval and so on, without you having to probe specifically on these areas. Here are our top tips for designing an effective cognitive interview protocol. 1. Always provide a clear introduction. Explain the purpose of the interview, for example that you wish to check your questions are working as intended prior to their being asked of hundreds of people. Encourage criticism. Explain that if the participant finds anything difficult, others will too. If you are using the think out loud technique, you need to provide your demonstration and training of this technique as part of the introduction. 2. Document your aims of testing in the protocol. Obviously, you will need to include your test questions in the protocol. We recommend that you also write down the aims of these questions, that is exactly what they are attempting to measure. We also recommend that you document what specific features of the questions you are hoping to test. Documenting your aims in the protocol will help if interviewers need to include spontaneous probes. Documenting your aims will also keep your cognitive interviews relevant and focused. Be realistic about the number of aims you include in your test. We have found that we can test around 20-30 questions in a one hour interview. You may need to prioritise which questions you test if your questionnaire is longer than this. 3. Always write probes that are neutral. For example, the probe, how did you find that? It's a neutral probe. In contrast, was that difficult? Is a biased probe. Ideally probes should be open rather than closed, as open probes will elicit more feedback. Asking follow-up probes like why did you say that, or can you give me some examples are useful ways of following up on a closed probe to gain more information and further insight. 4. You need to decide at what points in the interview you are going to probe. Probing can either be done concurrently after each test question, or retrospectively after all questions are complete. Concurrent probes and retrospective probes have different pros and cons. If you probe concurrently after each question, participants are more likely to remember issues and will be more able to tell you exactly what they were thinking about when they were exposed to each test question. However, if you probe concurrently, participants may lose the flow of the questions due to repeated interruptions from the interviewer. Therefore, concurrent probing may inadvertently make questionnaire completion more difficult. The advantage of retrospective probing is that interviewers minimise how much they interrupt people during questionnaire completion. However, with retrospective probing, participants may be less able to remember what issues they encountered or why they had problems. And retrospective probing participants may create artificial problems that they thought of after the event, but that did not occur to them in practice at the time they were initially answering. Whether you lean towards concurrent probes or retrospective probes will be partially dependent on the nature of the questionnaire you are testing. For example, if you are testing a very short questionnaire, retrospective probes may be better. If you are testing a longer questionnaire, consider probing as you go along, or maybe probing at the end of each subsection to avoid issues with interrupting flow. 5. You need to be mindful of the mode of interview when developing protocols. Which mode you should use will be determined by what questions you are trying to test, as well as the resources you have available to you. Face-to-face interviews may be more appropriate for participant groups with lower levels of digital literacy. Face-to-face interviews may also be better if we want to make detailed observations of how people interact with a specific type of products. For example, if we wish to test an information pack containing paper consent forms, leaflets and instructions, we may wish to conduct face-to-face interviews, so we can watch how people handle the document packs, observe what documents they pull out first to read, and observe which pages they look at first. It is difficult for interviewers to observe these types of behaviors during a virtual interview. In contrast, during virtual or remote interviews we can request that participants share their screen, and we can record exactly what they are doing on each web page. Screen recording is a particularly useful tool when testing web surveys or mobile applications. Remote interviews are also a pragmatic option if you need to interview a geographically dispersed sample or if you have limited resources to spend on travel. Once you have conducted your cognitive interviews, you will need to analyse your data. Next, we are going to discuss one process that you could use to do this. As the first step, we would recommend producing a case summary or a set of field notes for each interview you have conducted. It will be highly difficult if not impossible to make detailed notes during the interview itself whilst making observations and actively listening to the participant in front of you. Therefore we recommend that these case summaries are made after the interview by reviewing the interview recording or an interview transcript. For each case summary you should provide the answer the respondent initially gave to the survey question, any relevant observations, for example if a respondent changed their answer or asked for help, and the case study should then describe findings from the think aloud and the probing. The next step is to combine case summaries for all participants into a matrix format. We call this process charting. We tend to use Excel to produce these matrices, like the example on this slide, however Envivo or other qualitative data management software could also be used. Each column in the matrix represents a field from the case summary, for example all survey answers to the first test question could be included in column A, all observations in column B or findings from think aloud or probing in subsequent columns. Each subsequent column represents a theme under investigation, for example column D could include evidence of comprehension issues, column E could include evidence of potential issues and so on. By organising our data in this way we can read the data horizontally. This will allow us to review all information collected from one particular interview. Alternatively, we can read columns vertically. This will allow us to read all evidence about how well a question is working by a particular theme. For example, we could look at all evidence on question one comprehension issues by reading down the appropriate column. This allows for systematic interrogation of the qualitative data collected. Once we have completed our data management phase, that is we have a set of completed matrices, we can start to categorise and label what issues have occurred. Spend some time familiarising yourself with your charts and highlight and summarise what you think the main issues are that occur with each question tested. When looking at the charts we are attempting to cluster similar issues together and label them. You should be asking yourself, is the issue I see in case one the same type of problem observed in case two, or is it a different or separate issue? By the end of this process the goal is to have a comprehensive list of the different types of problems that arose for each question. Please note, the aim of the exercise is not to quantify each issue uncovered. We are not interested in how often an issue occurs. This is because when we conduct cognitive interviews we typically only talk through small numbers of purposefully selected people, between 8 and 20. Therefore we count and infer how common problems will be in the general population by how often they occur in our cognitive interviewing sample. I repeat, the aim of the thematic analysis is to map out all the types of problems encountered, but not to quantify how often each problem occurred. After you have completed your thematic analysis you need to draw your conclusions. First off you need to ask why each problem has occurred. Hopefully your interviews will already have captured why type information. Let's imagine your testing has identified three types of problem. Firstly, it has been observed that people ask for the question to be repeated. Secondly, one of the terms in the question is not consistently understood. And thirdly, participants have stated the list of response options is not complete. Looking for evidence of why a problem has occurred is important. Perhaps the requests for repetition are occurring because the question is too long-winded or it contains too many clauses. Perhaps participants don't understand a particular term because this is not column and parlance. Perhaps the language is too technical or specialised for the target audience. Perhaps a relevant answer option is not included in the response option list. Once you have decided the cause of a problem you need to decide what the impact of the issue is. Will this issue create a potential problem or bias in the data you collect in your questionnaire? Will this issue make your questionnaire unnecessarily burdensome for respondents? Or is it a minor odd issue that you and your respondents can live with? Finally, once you have assessed the impact of an issue, you can consider options for solutions. For example, the first problem where requests for repetition are being made because a question is too long-winded could be fixed by shorting the question or breaking it up into smaller, simpler sentences. The second problem, lack of understanding of a specific term due to technical language could be fixed by using a different term or by including a definition. See what language your respondents used in the interviews to describe the concept of interest. This will help to make sure the language you use matches their mental models. For the third problem, missing response options, see what response option was considered missing and add it to your question. Thank you for listening to this presentation. Now you have learnt about how to conduct cognitive interviews to test your survey questions. We have discussed different interview techniques, how to develop protocols and how to analyse your data. The next section of this course will tell you about other situations where cognitive interviews can be used beyond testing your survey questions.