 Yn ysgrifennu, gwneud y gwneud cyfnodd y ff Davies Celloodau Cynraedd cwestiynau, yw rhai ddyddiau cael ei bannu i gyfnodd haedda i'w rai wneud arall. Mae flynyddau Celloodau Celloodau Cells canw i yn cyd- convincing o'r bobl cardeb arddangos yma yn y gweithgafol, mae'r bywch mnesadau hiwn yn addysg eu bod i'n gwaith peth llefydd, mae'r gweithgaf ei ysgolant, ni chi gweithgaf sy'n gweithgaf ein gweithgaf, with someone or consent to data linkage. Cognitive interviewer methods can also be combined with usability testing to look at web questionnaires, websites or other documents. What types of survey materials can be tested? A number of survey materials can be tested using cognitive interviewing methods. This can include advanced letters, information leaflets and opt-in or opt-out How is this done? Usually the testing is done alongside testing of the survey questions. Advanced materials are tested at the start of the interview. This may mean that you can test fewer questions due to timing. Documents are placed with the participants, sometimes in an envelope. Participants are asked to think aloud while looking at the documents and asked to look at the documents as if they had arrived in the post and the interviewer was not present. Probes are then asked about whether the information was clear, if anything could be amended and hypothetically whether they would opt in or want to take part in the study and what could be done to encourage this. What are the benefits of testing these materials? Similarly to testing survey questions, testing these types of documents will allow us to try and create the best letters and leaflets that we can. We can collect feedback on whether the information is clear, if anything is missing and what can be done to encourage people to take part in the study. Is there anything that should be kept in mind? It's important to remember that this testing can't predict response rates. We have small sample sizes and it's to get a more general feel for how participants might react to materials and their understanding of them. An example of some testing we did for the National Survey for Wales on survey materials can be read in the case study. What types of other messages can be tested? We've tested public health information, for example where leaflet inviting people to get their cervical screening and return to player messages which appear before you place lock machines. How does testing these messages work? Using the cognitive testing techniques we can explore people's understanding of the messages. For the return to player messages we had a variety of message wording printed onto showcards. We asked people to look at them and tell us what they thought the message is meant. We also used probing questions to explore whether they had noticed the messages on the slot machines, if they read them and if they understood them. This can help to ensure that the messages tested are clear and fit for purpose. What types of consent questions can be tested? Any types of consent questions can be tested. We've tested consent to data linkage for government departments, consent to a follow-up physical survey on a property, consent to Twitter data being linked to survey data. How are these consent questions tested? We usually try and have some context. It's often just a single question that we're testing so we would ask it among some of the other survey questions. They're usually asked at the end of surveys so they would be placed at the end of the interview as well. We get participants to think aloud whilst answering them and ask them about the information provided in the question. Was it clear? Was it off-putting or encouraging? Would they consent in a real-life survey situation? Why or why not? And can anything be done to encourage consent? So how could I combine cognitive testing with usability testing? The aim here is to ensure that the layout and format of the survey is working as intended. You would ask the participants and complete a web questionnaire whilst you observe. Nowadays we tend to get someone to share their screen on Zoom. The interviewer observes how the participant is using the web survey. If they trigger error messages, if the format of the questions look okay on the screen, if they're able to navigate from page to page and give their response. Participants are asked to think aloud as they complete the web questionnaire. The focus is on the format of the questions, not necessarily the content. Either after a small set of questions or once a survey has been completed, the interviewer will then use probing questions to ask how they found navigating, what they thought of the design and how the survey worked for them. Can you use this as well as testing question wording? You can just have a user testing interview where the sole focus is on the design of the survey. You could ask if they noticed certain elements such as extra information buttons, or you could combine and just have a section at the end of a normal cognitive interview where you're asking about the content of the question wording, where you ask for general feedback on the layout and ease of use of the survey.