 Hello, my name is Debbie Collins and I am going to introduce you to a method of pre-testing your survey questions called cognitive interviewing. In this video I will describe what cognitive interviewing is and what it entails. Cognitive interviewing is a set of qualitative techniques that can be used to explore people's thought processes when presented with a task, such as answering a survey question. Some of the techniques are listed on this slide. Many of these rely on verbal reports, though some involve observation. Think aloud and probing are the most widely used and we are going to focus on these two techniques in the rest of this video. You can find out more about the other techniques by looking at the resources that go with the film. So how does cognitive interviewing work? This slide illustrates the process. Typically it involves you as the interviewer and a participant in a one-to-one setting. The survey questions to be tested are administered in as close as possible away to how they would be administered in the actual survey. Let's imagine that the questions you are testing are going to form part of a web survey. In testing them, respondents will be asked to fill in a test web questionnaire using their own device. The respondent may be asked to think aloud as they attempt to answer the survey questions. The researcher may note the respondent's behaviour. For example, were they frustrated or puzzled at a particular question? After each question has been asked, or perhaps after a series of questions, or maybe at the end of the questionnaire, the respondent may be asked some further questions, verbal probes, about their thinking as they attempt to answer the survey questions. Let's look a bit more at think aloud and probing now. As the name suggests, the purpose of think aloud is to encourage respondents to articulate their thought processes spontaneously as they are presented with a survey question or task. At the start of the interview, respondents are trained in how to think aloud. I am going to demonstrate that now. Let me explain a little bit more about how the interview will work. I am going to ask you some questions. I want you to tell me whatever comes into your mind as you read the questions and try to answer them. This is called thinking aloud. We found it helps to have some practice doing this, so let me give you an example. Let's say I was asked how many windows are there in my home. If I was to think aloud, I might do it like this. So, I come into my house and the front door actually has two windows, either side of it, but they don't open, so I am not really sure that they are windows, so I am not going to count those. I go into my front room and that's got a large bay window in it. It's actually made up of several different windows, but I am just going to call that one window. Then we go into the bedroom and that's got another bay window in it. Again, so that's another set of windows. Then I go into the kitchen and there's a window in the kitchen. There's also the door which is made of glass, but actually that's a door, so I am not going to count that. Then finally, there's the bathroom and that's got a window in it too. So that's what, front room, bedroom, kitchen, bathroom. So I've got four windows. Now let me ask you the same question. How many windows are there in your home? As you count the windows, tell me what you're thinking. During the interview, the researcher encourages the respondent to think aloud using non-directive open probing, such as those shown on this slide. Verbal probing used to explore specific aspects of the survey question, such as the question and answer process. The probes can be scripted or spontaneous. Examples of scripted probes are shown on this slide. Spontaneous probes are used to follow up on respondent's answers to the scripted probes, encouraging further expansion or clarification. For example, can you tell me a bit more about that? Program can take place concurrently or retrospectively. Concurrently happening immediately after the question has been asked, retrospectively can be after a series of questions have been answered or at the end of the questionnaire. What do you think are the pros and cons of concurrent versus retrospective probing? You might want to pause the video now and reflect on that. Let's see how we might probe about this question, which asks about days spent outside in the past two weeks. It formed part of a series of questions designed to measure sunlight exposure of the general population as part of a national survey. The number of days is recorded. Try to answer the question yourself. How easy or difficult did you find it to answer? We could ask respondents to think aloud as they try to answer this question. Were you aware that you were doing this as you started to answer the question? Here are a couple of cognitive probes that could be used to explore respondent's thought processes when presented with this question. What sorts of information might you get from these probes? Is there anything else you might want to explore? Have a go at writing some of your own probes to test your own survey questions. Have a go at doing a cognitive interview with a friend or colleague to test your own survey questions or test the questions provided. Reflect on how you found the experience as the interviewer and how your participant found it. Finally, let's consider a few practical issues. Cognitive interviews are typically audio or video recorded with participants consent. You will need to think about how many questions you can test with anyone respondent. You don't want your interview to be longer than about an hour, which means that you'll probably only be able to test around 15 to 20 survey questions. Though, of course, that depends on the extent of probing that you may need to do and the complexity of the questions that you're testing. Also think about how many people you'll need to test your questions with. This will depend on the aims of the test and the intended target audience of the survey questions. You're testing. In the literature, you will see cognitive testing involving anywhere from five to several hundred participants, though 10 to 30 in any one round is more common. The resources that go with this video contain more information on sampling and recruitment. So that's a brief overview of cognitive interviewing methods.