 Hello, I'm Phil Newton from Swansea University. Very excited to hear your talk today about tips and tricks for running survey-based research in academic integrity. So yeah, tell us more about it. You have not done quite a few surveys yourself. I have and that's really where this comes from. Before I share my slides, I think I might just elaborate on that point actually, explain how it is I've come to be sat here on this warm and sunny day with you all on the CNAI webinar. We are going to talk about how to run survey-based research in academic integrity. The reason I offered to run this webinar is because I have been involved in lots of survey-based research and not all of it has been done very well. And I have personally made a number of mistakes in the way that we've done survey-based research and what I'm hopeful is that by sharing my experiences, I can make it easier for the people not to go through the same things that I've been through. I think it's worth also taking a step back and thinking about the big picture about where this all sits. We're going to talk obviously practically about how to do surveys. But the reason we're doing that is because we want to understand academic integrity and academic integrity is really important. I'm sure I don't have to make much of a case to the audience for any NAI webinar about the importance of academic integrity. But if we don't have academic integrity or if it's not working, then the educational systems we're involved in don't do the thing that they're supposed to do. Education is really important, perhaps more important now than it's ever been in my lifetime. Training people to become independent, critical thinkers to become doctors, nurses, lawyers, engineers, pharmacists, and so on. And unless society can trust the qualifications that we award to our students and the research outputs that we produce, then society won't trust us to do the things that it's asked us to do. And the whole thing falls apart quite quickly. So to understand academic integrity, we need to ask lots of questions. Why are people doing the things that they're doing? Why are they not doing the things they're doing? What are they doing? When? Why? How? How can we design policies and procedures to make academic integrity work, to promote it, to maintain it, to nurture it? All of those sorts of things we can answer using research and the most common way that we do research in academic integrity is through surveys. And so that's why I think it's really important for us to try and do that as well as we can. So if I share my screen, so I know you could just give me the thumbs up that you can see the slides. Yes, we can. And it's not the notes page, as you can see, it's the slides. No, it's not. It's the real slides. Perfect. Okay. So this is what we're going to cover. I've already told you a bit about who I am and why I'm here. I've done lots of research in this area and as I said just now, I've learned lots of things the hard way. And I want to share some of those lessons with you. I've also learned some things the easy way. I've done three soon to be four systematic reviews of survey based research in education, mostly in academic integrity. And what I found is on a personal level, quite consoling, reassuring that lots of us when we do research in academic integrity, we don't necessarily follow best practice in how to do surveys. And so it's not just me that's had some of these challenges, but at the same time it means lots of us are doing things when we're conducting our survey based research in a way that we could perhaps do a little bit better. I can see that someone's raised a hand. Okay. And I'll press on. Yes, that's absolutely fine. I want to be clear that the things that I've learned to do the hard way are because in part, I wasn't a social scientist. I didn't come to this with training and how to do surveys. And like many of us, I thought, how hard can it be to run a survey? We put some questions together, we bunged it out, and we looked at what the results were. And many, many of us who do surveys in academic integrity do the same thing. There is a whole literature about how to do surveys, and that's part of what we're going to focus on. I've shared some of my experiences in a forthcoming chapter of edition two of the Springer Handbook of Academic Integrity, thanks to the generous curation by Sarah Laneyton. I've been able to contribute a chapter, which explains some of this in more detail. And what we're going to do today is just go over some headlines, really from the contents of this chapter, that we could possibly cover everything about how to do surveys in the time we have together. So I'll give you 30, 40 minutes on some quick wins, some easy things that we can do to make surveys more effective. And then there'll be some time for questions, hopefully as long as I don't go on for too long. And then if you want more, we can share copies of the chapter. So as I said, surveys can be easy to do, they can be cheap, they can be quick. We can get a lot of data quite easily. But there are things that we can do that can make them better. And that's what we're going to focus on some quick wins. We should spend a second just making sure we're all clear what a survey is. It turns out that's not as easy to define as I had thought it was. This particular chapter, which is a very good review of how to analyze the findings from survey-based research by Fink defines a survey as a system for collecting information from or about people to describe, compare, or explain their knowledge, attitudes, and behavior. There's a huge variation potentially in there in what a survey is and what a survey is not. Technically and officially, there are multiple different survey methodologies. The most common one is questionnaires. But when you get stopped by someone in the street outside a shop, or when somebody calls you up and asks your opinions on something, interviews you, they are doing a survey. When we can survey records, existing records in a structured and organized way, and we can observe members of the population going about a particular behavior, for example. And those are all technically forms of survey-based research. But what we're going to focus on for the purposes of this webinar are really questionnaires. Questionnaires can collect data that are quantitative or qualitative. We're going to talk a little bit about both. And this is by far and away, in my experience, at least, the most common method that's used in survey-based and academic integrity research. So we're going to talk about how to do a survey. If you're thinking about doing some research in academic integrity, you're probably thinking at least in part about doing a survey, what are some simple things that you can do to improve and enhance and guarantee the quality of the data that you get? And we're going to touch upon, then, some ways in which we can analyze survey-based research that's been done by others. And this is something that I think we don't do enough of in the field of academic integrity, is do systematic reviews of existing survey-based research. We won't explicitly focus on that, but many of the things that we'll cover will be useful. So what we're going to do is break that down into why, what, and who. Why do we do surveys? What do we actually do when we do them? And who do we get to fill them in? And all of these things are really fundamentally important. The first thing to ask, which alas, I don't think many enough of us do, is define what the question is that we're trying to answer with a survey. What are we actually trying to find out? And then is a survey the best way to answer it? And it's really worth spending a bit of time on this if you're doing this research as a team, brainstorming it, sitting around and kicking around specifically and clearly and objectively. What are you trying to find out? What questions are you trying to answer? And then how could you answer them? And is a survey the best way to answer them? There are some other ways that we can collect data that relate to academic integrity. And where possible, it tends to be a better way to answer a research question if we can collect objective data, data on the behaviour or performance of individuals involved in the scenario that relates to our research question. If we can collect objective data about, let's say, rates of plagiarism or rates of research misconduct, that can be one way, possibly an additional and possibly a better way of answering our research question than sending out a survey and asking people questions about their behaviour. If we are going to send them questions, it's better wherever we can to try and ask questions that give us objective answers. It's better if we can to ask people, have you ever done something, then would you ever do something? Because have you ever done something is a more gives a more objective response? People can, although with some limitations, point to when they may or may not have done a particular thing, asking people to imagine a situation and then project and predict their own behaviour is much more prone to all sorts of errors and biases, some of which we'll cover. Having part of deciding what your research question is, is trying to decide what sort of data would be useful in answering the question. And very, very importantly, and again, something that we perhaps don't think about enough beforehand, do we have the skills necessarily to analyse the data? If we're collecting qualitative data, have we got the skills to analyse that properly, or do we have collaborators who might be able to do that for us? If we're going to collect significant volumes of quantitative data, do we know how to analyse that properly? And or do we have access to people who do? One thing that I certainly think we could make a lot more of in our field of academic integrity is to look back at existing research and ask, if we're interested in a particular question, has that question been asked before? And not in a cursory way to give us some information for our literature review, but in a systematic way, how many times has the question been asked before? And was it asked with a survey? And are there enough surveys that we could review research that's been done by other people in order to give us an answer to that question? So we think about questions that we commonly ask in the field of academic integrity. We're often trying to find out how common a particular behaviour is. How common is it for students to commit plagiarism? How common is it for researchers to commit a certain type of research misconduct or to report it? Whatever it might be, there are hundreds, if not thousands, of existing survey-based research studies that have asked questions that are quite similar to that. And before embarking on asking a new set of questions to a new set of people, it's worth thinking, can I answer my question or get a part of my answer to my question by systematically reviewing the existing research that has been done in this particular field? Okay, so we decided we want to do a survey. We've decided it's the best way to answer a question. What specifically are we going to do? One tip and trick that really can help improve the quality and meaningfulness of the findings from your survey-based research is to ask the fewest questions possible. Most of the time, as we'll see, the people who are going to fill in our survey are doing it voluntarily. They're doing it as a favour or because they're good citizens and they're giving us the gift of their time and their contributions to the answers to our surveys. We want to make the best use of their time and there's an abundance of research which shows that people's patience wears out very quickly when we're asking them to fill in lengthy surveys. Can they answer in completely all of the questions in your survey in just a few minutes? And if not, maybe we need to rethink the question and narrow its focus so that the questions can be answered in just a few minutes. One way to go to practically attack this and work out how to ask the fewest questions possible is to draft a list of questions and then go through each of those questions and hypothesise what the answers might be. And then on the basis of what we think the answers might be, what would that tell us in terms of the answer to the research question? So if we're asking students, what discipline are they studying? Are they studying business, are they studying computer science, are they studying biology? And they give us the answer. Does that help us? Is that important for helping us answering the question? Are we interested in whether or not students from different disciplines commit plagiarism at different rates, for example? If we're not, then we don't need to ask them what discipline they're studying. Going through that process rigorously and systematically with all of the questions that we're asking and being brutal with ourselves and saying the answer to this question is not going to help me answer my research question, we don't need to answer it. There to ask it. What this does then, if we've limited the number of questions we asked, is it does then give a space to ask important things twice or more than twice? Once we've distilled down the very clear specific questions that we want to ask, it can be for reasons I'll explain important to ask those questions more than once. When you ask somebody something for the first time, they will give you an immediate response that might be different if you had given them a few minutes to think about it. Even if you just give them a few minutes to do something else and then ask them the same question again, they are subconsciously reflecting on the question you've asked them and they might give you a different answer. The phrasing that we use for questions and surveys profoundly influences the responses that people give us and I'll give you a few examples of this in a second. So it's worth thinking these questions, identifying the key questions that you want to ask and then for some of them, particularly where the answers are subjective, asking them more than once, framing and phrasing and ordering them slightly differently and then either averaging across the responses or identifying an advance that you're only going to analyze the data from the last iteration of that question. Give you some examples of this. Most of what we do when we ask people questions and surveys is we ask them to make a judgment or we're asking them for their perception. Even if we're asking them to report their own behavior, they have to make a judgment, a decision, a perception based answer to that question. When they make that judgment or they report their perception or their opinion, the decision that they're making, the judgment that they're making can be made consciously or subconsciously and I'm going to give you some examples of this which I hope will illustrate the point in a profound way. The first example I'm going to give you is I apologize in advance because I cannot find the original source for this particular example but I've seen it come up in multiple different textbooks and methodology training and it really works very well. So consider if you want to ask somebody this particular question. You're asking somebody, is it okay to smoke cigarettes while you pray? Now, most of us, when we're asked this question, regardless of our religious persuasion, would probably say, no, it's not okay to smoke while you pray. When asked to explain why we might find that challenging but in general our response to a question like this or answer to a question like this would be no. If we take the exact same question and just turn it around and say, is it okay to pray while you smoke? Then most of us would say, yes, it is okay. In fact, we might encourage it if you are a religious person. Smoking generally is considered to be bad for you and anything that you can do if it involves divine inspiration to reduce the amount that you smoke or to connect with whichever spiritual entity is important to you is a good thing. Now, the behavior smoking and praying at the same time is almost the same in both of these situations. Because we've changed the way that we phrase the question, our answers to them, certainly our initial answers before we think about them through in detail are going to be very, very different. The judgment that someone makes when answering this question is going to be different simply depending upon the order of the words in the question. Let me give you a second example then. This is to do with the way the order in which the questions are framed within the survey itself. This is an example from the literature on criminology, which is a related field often to academic integrity. This is a study from 2016. In the study, the participants in this survey-based study were asked to recommend a sentence for a person in a scenario who had committed a violent robbery. So there was a scenario depicted of a violent, difficult and unpleasant crime and the perpetrator had been caught and the people who were answering the survey were asked to recommend a sentence. I watched it happen, which would be the punishment for this person. Now, the people answering the survey were split into two groups, A and B. The people in group A answered four questions before they answered the question about the violent robbery and the questions that they answered were asking them to make recommendations for the sentencing of a series of other violent crimes. The final question that they answered was to make a recommendation for the sentencing of a person who had committed a murder. The second group of participants, group B, excuse me, they were asked a different set of scenario-based questions. They were also asked to make recommendations for sentencing for other crimes, but the crimes that they were asked about were much less serious, speeding tickets and other crimes, still things that are against the laws in the scenario were in the place where the survey was being carried out, but clearly less serious than crimes like murder. So both groups asked five questions. Group A were asked questions about sentencing for four very serious crimes and then asked a question about sentencing for violent robbery. Group B asked four different questions about very minor and moderate crimes and then asked that same question about sentencing for a violent robbery. And what happened was that in group A, the recommendation that was made for the punishment that should be administered to someone who had perpetrated a violent robbery was far less serious than the punishment that was recommended from group B, because the comparison between murder and sentencing that would be the outcome for that versus the contrast between speeding tickets and robbery is much greater. And so the ordering of the questions within our survey, obviously we're not asking people normally about murder and speeding and violent robbery, but we're asking them about things that are normally sensitive and difficult and challenging and we need to consider the contrast, the judgments that they will make and the contrast between the questions in terms of the order in which the questions are asked. Related to that then, because of people's limited attention span, limited patience, and normally the voluntary way in which they're giving us the time to participate in our survey, it's really important that we ask the most important things first. Related to that then is to ask the demographic questions last. Now you don't want to ask irrelevant and trivial demographic questions, as I explained earlier on, but all things being equal in general on average, the demographic questions or the answers to them are less important than the answers to the specific questions that we're asking in the survey. So we want to put those first. Now there may be situations where we don't want to launch straight in with questions about difficult and challenging scenarios. We want to set the scene for our participants. We may want to make them feel comfortable, reassure them with anonymity, etc. But as soon as we've done that, if we need to do it, we need to get straight on with asking them the most important questions. The questions that will give us most quickly and meaningfully the answers to our research question in general, unless we're very specifically interested in demographics, that's not the demographic questions. Okay, so on then to the nature of the questions that we might ask in a survey. This is something that I've seen many, many times. It's something that I've done myself when designing and delivering survey-based research. It's a common, I'm going to say, mistake that we ask more than one thing in our questions. A good survey question asks only one thing at a time. It is then very clear what is being asked. And the thing that's being asked is very specific and it's very objective. You don't want to leave the survey taker in any doubt what information you want from them. You don't want there to be any room for ambiguity in the answer to the question. Let me give you an example. Now, this is more or less a real example. I have modified it slightly because, as I said, many of us have asked questions like this and it's unfair and unkind to pick out individual studies. But we have seen many survey-based studies in academic integrity that ask questions that look something like this. In the last year, have you ever cheated on an assignment, test, or quiz? The answer to this question in the survey that I reviewed, in the particular survey that I was reading, was a yes-no answer. And what we're going to do then is break this question down and see how we can improve it. The answer to this is a good question is absolutely not. And there are four key ways in which we can modify this question so that it still gets to the same point. But it asks very clear, specific, and objective, very clear, specific, and objective answers. So let's start at the end of the question then. So in the last year, have you ever cheated on an assignment, test, or quiz? Assignment tests, assignments, tests, and quiz, each of those can mean different things to different people. This is a question that was asked of university students in a particular domain. For most of us, most of the time, a quiz is a formative activity, doesn't normally carry any course credit. And so the implications, the consequences of cheating on a quiz, are going to be different to cheating on a formal test that carries course credit. We may be interested in both of these, but if we are, then we want to separate them out into different questions. So if we reword the question ever so slightly, let's say we're interested more in credit bearing assessments, summative assessments. And of course, there are many, many other things that can affect the nature of an assessment. They can be online, they can be in person, they can be invigilated, they can be uninvigilated, et cetera. So we've added some more detail here. And now our question is, in the last year, have you ever cheated in an online exam taken for course credit? That's a much more specific example of a scenario. Let's move back up, then. This is a word that obviously crops up a lot in research and academic integrity. It's a difficult and challenging word. It's a controversial word, in part because it means different things to different people. What does it actually mean to cheat? There's lots of research in the field of academic integrity that shows that students fall foul of regulations on academic integrity because they didn't know what those regulations were. The same is true in research integrity. People's papers may be flagged up for having engaged in an unethical behavior because the researcher didn't know that they weren't supposed to do things a particular way. So that's just one example of a situation where if you are somebody, have they ever cheated, what you're actually interested in is whether they've ever engaged in a particular behavior. But if they didn't know that behavior was cheating, they're going to say no. But actually, according to the definitions of the policies and procedures, they may well have engaged in a behavior that we might recognize as cheating. So we need to be much more specific. We also perhaps, where we can, want to try and take some of the heat and controversy and emotion out of our survey questions. Try and make them cold and dry and boring and objective. Cheating is clearly something that's not permitted. If instead we are something that's more specific and objective, have you ever gone online to look up answers in an online exam taken for course credit, despite this being prohibited by university regulations? It clearly still does describe the behavior that is not permitted, but it's less emotive than the word cheating. And more importantly, it's much more specific. It describes one particular type of unauthorized behavior. Now, there are lots of other ways that students can cheat in online exams. If we are interested in those, then we need to ask specific questions about those behaviors. It's a very common motivation for us doing this sort of research, particularly because resources are often hard to come by, funds and time to do this sort of research are often limited. So we want to get the most out of our survey based questions. So we ask broad questions that cover a range of behaviors, but that undermines the quality and validity of our answers. Okay, the third issue then with this question that we're going to work on is this word ever. Have you ever cheated? This is very common in survey based research in integrity. And yet of course it can mean very many different things. A student who cheats on every single summative assessment is qualitatively different to a student who maybe just wants possibly accidentally without really realizing it did something that would be considered cheating. Do we really want to put all those students together? Sometimes we do, but it may just be a simple modification to say, how often did you do it? How often did you do it? If the answer is yes, if you've done this, how often have you done it? And then when we're asking them that question, we need to be very careful about the range of options that we give them. Very many survey studies will give students or participants the option of, I've never done this, I've done it once or twice, I've done it more than twice. Or some other variation of a scale. By providing them a scale, you provide them an anchor. Our natural inclination is to pick the midpoint of a scale. It's far easier and more objective to just give them free numerical entry and say, how many times have you done this? And then you're going to get a more specific answer and a more objective answer and one that's not anchored. When we are asking these sorts of questions, it can be really helpful then to give people the opportunity to explain, to collect some qualitative data alongside a quantitative data. If you have done this, if our participants have cheated in an online exam, of course, credit when they were supposed to, why did they do it? Give them a paragraph to write why. If they haven't done it, again, give them a paragraph to write why. This obviously takes a bit more time, but then you're going to get a much better answer to this specific research question than if you just skip on to another behavior and another behavior and another behavior. Okay, the final element of this question that we want to reconsider then is the first part in the last year. Now, this seems, on the face of it, fairly straightforward. We know what the last year means, but there's a lot of work done in survey based research in the social sciences to show that this phrase in the last year means different things at different times to different people. So if you ask somebody in January or February, in the last year, they might think about the 12 months back to the January, February of the year before. If you ask somebody in October or November, they might think about only that calendar year. So the last eight, nine months, of course, then we're often interested in participants who work in an educational setting, your university students, academics, if you ask them in the last year, have you ever done something they may think about the academic year, which is the third iteration of that same question. We may be interested in all of these. We may be interested in only one of these, but again, it helps to be clear and specific. In this case, then we're interested in the last 12 months. That also then means we can ask the question over a broader range of time, which may be important when we're trying to maximize the response rate to our survey. Okay, just some simple tips and tricks for how to frame the questions that you ask in a way that can really make a profound difference to how useful the answers to those questions are to your research question. There are lots of different what the psychologists would call cognitive biases that can influence how people answer survey based questions. We're not going to have time to go through all of them, but I'm just going to mention one of them because it's really important in the field of academic integrity and that's called social desirability bias. But then that into plain language, what that means is when somebody is asked a question in a survey, for example, a lot of what influences the answer that is given by the participants is them trying to work out consciously or subconsciously. What am I supposed to say? What do they want me to say in response to this question? This could be a profound issue, of course, when we're trying to ask people questions about behaviors that are challenging than when they've committed academic misconduct or research misconduct, for example. There are some measures that we can take, some steps that we can take to help mitigate, reduce the influence of social desirability bias. One is to maintain our objectivity and our specificity. When you're asking people about very specific things, it becomes less vague what the answer is supposed to be. Perhaps more importantly is to guarantee the anonymity of the participants, the anonymity of the people who are answering the survey. I can't emphasize enough how important this is. It relates to that one little word there in the center of the question about what social desirability bias is. It's the participants trying to work out what do they want me to say. Who is doing the asking can profoundly influence the answers that people give to survey-based questions, in particular, where there is a power imbalance between or perceived power imbalance between the people who are asking the questions and the people that are answering. For example, university academic researchers are asking students whether they've ever done things that the students are not supposed to do. It's clearly going to influence the way that people respond to the questions. Reassuring the participants that the questions are being asked by someone who is independent and that the response is going to be anonymous is really, really important to making sure that you get meaningful, valid responses to your survey-based questions. Importantly, though, it's not just about telling people that you are independent and that the responses are going to be anonymous. Very small, simple cues can profoundly influence participants' perception of their security, their anonymity, and your independence. For example, if you are conducting a research study at your university or place of our education provider, you put an online survey together and you put that then on your learning management system, on Blackboard or Canvas or Moodle or whatever it might be. You can give students all the reassurance that you want, that their response is going to be anonymous and won't affect their progression or they won't get in trouble. But if they have to log in to the learning management system in order to access the survey and then answer it, that is going to profoundly influence the likelihood that they all respond and then also the likelihood that they will give answers that are an accurate representation of their behavior. The final thing I want to talk about then is who are we going to ask to fill our survey in? I've saved this bit till the end because in a way, in many ways, it is the most important thing when it comes to survey-based research and it's something that we and I have not spent anywhere near enough time considering when it comes to survey-based research in academic integrity. Let me give you some key terms that will help me explain the particular issues that affect us in this field. When we're doing a survey-based study, we need to define the answers to all of these key terms, in fact. The first of them is the population. Whose behavior are we seeking to represent, to describe, to explain, to understand? For example, do we want to try and explain the behaviors of university students, university students in a particular country, at a particular country, at a particular university, policy makers, academics, whoever it might be, when we're writing our results in our discussion and presenting our findings, what's the population that we're trying to explain or understand here? The second key term then is the sample. The sample specifically describes those individuals from the population whom we will ask to fill out the survey. It doesn't mean the people who do fill it out. It means the people who we ask to fill it out, the people who we send the survey to and say, please will you fill this out? I'm going to say this again later, but I'll say it again now. Ideally, these two things should be the same. That's a really good way to improve the rigor and validity of our findings is to say, look, our population is this particular group, and we're going to send the survey to all of them and then try and get as many of them as possible to fill it out. That brings us then to the third key term, which is response rate. Response rate is the percentage of the sample who complete the survey, and we want that to be as high as possible. I'll explain why that is in a second. A final term that I've put in brackets because it's perhaps not a key term is then can be given a whole range of different terms, but we're just going to call it N, although some of us myself included will refer to it as sample or sample size, but it's not really. N is simply the number of people who actually fill it in. We collect our data at the end. How many responses are we analyzing? There is a tendency in survey based research in all domains to focus very much on this fourth term, the actual raw number, how many people have filled it in. Obviously, we want that number to be as high as possible, but it is not all that important compared to the importance of some of the other considerations that we make when defining the people who filled in the survey. Having lots of people fill in your survey can be a good thing up until a point, but unless we've maximized response rate, defined our sample on a population, then actually having lots of people fill in the survey can be worse than having only a few people fill it in. I'll explain why that is. Let's say you make an error on your slide. Let's go to the next slide. Let's say you get 1,000 people to fill in your survey. 1,000 people sounds like a lot, and in the field of academic integrity, it is a lot. An N of 1,000 is very rarely seen in survey based research in academic integrity. It's more normally in the low hundreds. That feels like a lot. We can be pleased with having 1,000 people fill in our survey, but if we've sent the survey to 100,000 people, then only actually 1% of our sample have filled it in. If our sample is meant to represent a population of say a million university students, let's say we're trying to explain the behavior of university students in our country, and there are a million university students in our country, then actually we've only got 0.1% of those students filling in that survey. For reasons I'll explain in the next few slides, the results we get from surveys like that, particularly where we're asking people about challenging and difficult behaviors are very limited validity, because the sorts of people who are going to have filled that survey in are probably not the sorts of people who are going to have engaged in the challenging and difficult behaviors that we're interested in, and actually then we're going to get not only an unrepresentative response, we're going to get an inaccurate response because we're going to have more people filling in the survey who haven't engaged in the behaviors that we're interested in compared to those that have. I'll explain that again in a second, let's give you another example. Let's say we only have 500 people fill in our survey. It sounds far less impressive than having a thousand people fill in your survey, but if you've defined your population much more specifically, let's say there are 550 people studying a particular topic at a particular university and you've sent the survey to all of them, and then you've got 500 of those 550 to fill it in, now you've got more than 90% of your population who've completed the survey, and so the findings that you've got are a much more accurate and valid reflection of the population, even though the absolute number of people who've filled it in is less. Ease to getting this right are to define the population in the sample beforehand. We often don't do this or we don't really do this to a level of detail and specificity that can help us mitigate some of these challenges, so we might think about sending a sample out to university students in our country. There are normally lots of those, we're not going to be able to reach most of them. More, better if we can, and if it's relevant to our research question, to define a specific group of students, for example, those studying X or Y. Academics employ that in a certain department in a certain place or in a certain country, for example, it would be another way of defining a specific population. The narrower our population is, the easier it is for us to then send the survey to all of them, and then the sample and the population are the same. If we can't do that, if we are genuinely interested in, let's say, all university students in a particular country, then we need to take steps to make sure that our sample is representative of our population, and this is the most important part of the most important part, the most important part of the messages that I want to get over this afternoon. Think about a group of individuals, you know, I can't remember how many are on this slide, I think it's 40 or 50. We think about them as an anonymous whole, but of course the truth is that people vary in all sorts of profoundly, significantly different ways. Our basic ways like our age, our gender, the things that we're interested in, the things that we do at the weekend, and then some ways that may not be important for our research question, our food preferences, our height, for example. Wherever we can, we want to try and define what those differences are and what differences are important for helping us answer our research question, and I'll talk about some common demographic differences that affect questions of academic integrity in a second. Before I do that, though, I just want to add a further layer of complexity to this challenge, which is then the way that we collect our responses that the terminology would call the sampling method. Almost all survey-based research and academic integrity uses what we might call a convenience sample, and that's a technical term that means very basically we send out our survey, hopefully we've defined who we're sending it to, or we put it on Twitter or on TikTok, and we say to people, please, can you fill this in? And then people fill it in voluntarily, and we analyze the data from them. This is very common, it's very easy, it has lots of intuitive appeal, but it can introduce some profound errors into the analysis of our results, or at the very least some profound challenges to the validity of our findings, even though it is very commonly done. Let me explain what that means. There's a huge amount of research that's been done on the characteristics, the demographic profile of people who voluntarily fill in surveys, and I reviewed some of these in a paper a few years ago, but I haven't done this work myself, I want to be absolutely clear, I'm just briefly reviewed for some work that's been done by other people. Here are some of those characteristics. People who, let me take a step back. In general, on average, women are more likely to fill in surveys than men, when you ask people to voluntarily do it. In general, on average, men are more likely to commit academic or research misconduct. The findings tell us these things quite clearly. In general, on average, people who voluntarily fill in surveys tend to be older than the men or the median of the population as a whole, and yet academic misconduct or research misconduct tends to be committed by people who are younger. People who voluntarily fill in surveys tend to be high achievers academically, and yet we know that one of the motivations for people to commit academic misconduct is that they may be struggling with their academic studies. In general, on average, people who voluntarily fill in surveys tend to be wealthier than people who don't, and yet people who commit academic misconduct or any other type of so-called deviant behavior tend to be of lower socioeconomic status than the population mean or medium. People who voluntarily fill in surveys tend to be native speakers of the language in which the survey is administered. Again, from lots of survey-based research, we know that academic misconduct, one of the driving factors for that is that people are studying a language that's not their native language, and that makes their academic studies much more difficult. You can see quite clearly if we are using a convenient sample of people who are voluntarily filling in surveys, particularly where we're interested in questions of academic misconduct, but anything to do really with academic integrity, we are going to have a sample, or at least responses, I should say, from people who are not necessarily representative of people who may have engaged in or observed or have an understanding of the particular behaviors that we're interested in. If we're trying to quantify a particular behavior, then we're almost certainly going to under-represent the prevalence of that behavior where we're using voluntary convenient samples. As I said, I've done some reviews on survey-based research and academic integrity, and I'll just bring together then these two really important concepts of response rate and sampling method. Let me just change my pointer. These are two studies I've done, which are this one with a collaborator, Keone Asics, where we did systematic reviews of other people's survey-based research, trying to quantify some questions in academic integrity. We probed those surveys to look for markers of survey quality, one of which is the response rate. What is a good response rate is a question which doesn't really have an absolute answer, but clearly you want it to be as high as possible. You can see from the vast majority of studies, not only was the response rate very low, the response rate wasn't even reported. What that means more commonly is that the basic criteria needed for us to be able to calculate the response rate wasn't reported in the survey-based, in the survey study itself. Who's the population? What size is it? What sample size? What was the sample? How many people was it sent to? How many people filled it in? And then what steps were taken to improve the response rate? Within the same group of studies, 87% of them used convenient samples and the rest of them didn't give us enough information to be able to decide, to be able to make a judgment, I should say, about what type of sampling was used. These two things together mean that the data that are being analyzed in lots of survey-based research and academic integrity are from respondents who are not necessarily representative of the sorts of people who have engaged in the behaviors that we're interested in, in a profound way. What can we do to improve that? Define our population and sample in advance. Define the key features which affect your research question. I've given you some of them there, basic things like gender, socioeconomic status, academic achievement, and so on. Whatever the particular influences may be, can you realistically reach all of the people in your sample? If you can't, maybe reconsider the size of the sample and the focus and specificity of the question. And then once we've got our population and our sample identified in the key characteristics, chasing up the people who we've sent the survey to. Knowing how many people we've sent it to and then how many people have responded is obviously key to this and then chasing people up with reminders and occasionally potentially also incentives to help them to fill in the study. Monitoring then the response rate and keeping an eye on whether the number of people who filled it in truly does represent the sample. If only five percent of the sample are filled it in, what does that actually mean? And then when we're doing our analysis, considering whether the final sample actually represents the population. Now these things are not, some of these things are quite straightforward to do. Doing all of them together and getting high response rates from a representative sample does take more effort, does take more work, and I want to be clear that if we haven't done those things, then at the very least we need to be mindful of that when we're analyzing our results, when we're discussing the meaningfulness and significance of our results and the impact of them. The final thing, which I won't talk about in great detail, simply because there isn't time, is the means by which we analyze the data. I said earlier on, we want to define that in advance and we want to make sure we have the skills within our team to be able to analyze the data. Some very common things that we fall foul of where we don't necessarily have the skills is we might do descriptive statistics or t-tests on multiple different comparisons. We can confuse correlation or causation and lots of survey-based research is done using what's called liquid scales. You'll be familiar with these. You're asked, given a scenario, and asked whether you completely agree, somewhat agree, now they agree or disagree, somewhat disagree, completely disagree. When we're reporting the findings of survey-based research like that, report all of it. Commonly collapse then, agree and disagree from five points into two, and that then chucks out a whole bunch of the data from which we might find something useful. The key thing, as I said, is to plan in advance. All right, summary then of the key things we've covered and then I'll stop and answer. I can see I've got some questions here. When we're designing a survey-based research, start perhaps most importantly with defining what is the actual question that we're trying to answer. Spend a huge amount of time on this relative to some of the other things because you may find that surveys are not the best way to do it, and it will certainly, if you do decide that survey is the best way to do it, help you define how best to do that. When we're asking the questions then in our survey, we want it to be as simple as possible, possibly as boring as possible, specific, short, and dry, unbiased, and objective. And then wherever we can, we want to at least consider how representative the findings are of the population whose behavior we seek to understand or explain. And if we can take steps to ensure that our sample is representative. These are some of the references I've used. I'll just leave that there for a second as you are watching the recording then and if you want to access any of these. We've got a few minutes for questions. If we don't have time to get to your question, or if you're watching this at some point in the future, hello in the future. If you've got any questions about what we've covered here is my email address, please do feel free to send me many questions you have. I am a nerd, unashamedly and unapologetically, and I could happily answer your questions and talk about this stuff for the days. Okay, let's stop sharing. And okay, I can see I've got a question from Rita. Thank you so much for this amazing presentation. This is really a masterclass in what to think about. I think I'm going to, if I ever get a student who wants to write to do a survey, I'm definitely going to give them this recording to watch before they start. So here is a question from Rita. Thank you so much for your presentation. I have a few questions and thoughts to ask you. When you're serving in multiple countries with students from multiple cultural backgrounds and multiple institutional practices, how do you fit such complexity in a single question? Also, from your experience, how likely are people more likely to be more honest on admitting misconduct behaviors observed on others rather than admitting their own engagement in academic misconduct? Okay, so those are all really good questions. The answers to Richard took out of the presentation simply in the interest of time. Let me start with the last one. Are people more likely to be honest when admitting misconduct behaviors observed on others than admitting engaging in their own academic misconduct? There is something to that, but it throws up a whole different set of challenges. If I give you an example, if you ask a participant, have you ever observed somebody else or are you aware of somebody else doing this, buying an essay, for example? Let's say you ask a class of 100 students that question. If one student has bought an essay but everybody knows about it, then you're going to get a response of 100%. If one student's bought an essay but nobody knows about it, you're going to get zero and they're going to get all possible options in between. The number of students buying an essay is the same, but you can get between zero and 100%. If we want to incentivize people to tell the truth, there are some slightly more sophisticated things that we can do to incentivize that. Glycurtis and his colleagues did a fantastic paper using what they called, I can't remember the exact term for it, I think it was truth serum, incentivized truth telling. There are methodologies where you ask people questions with guarantees of their anonymity and their answers are much, it's obvious to them that their answers are less likely to be trackable and so they're more likely to tell you the truth. The complexity of that is a bit beyond the bounds of what we had time for this afternoon, but I'd certainly recommend Guy's paper and then the references that he and his colleagues cite in there. I think what's easier than that though is to give people, is to set the survey up so that people don't have any doubt, the participants don't have any doubt that their responses will be anonymous, that you have no influence in their career, in their progression, in their reward decision, whatever it might be, and no interest in that either. Allowing them to be as anonymous as possible, for example, using one of the existing survey-based tools and sending them out a code to answer the survey so they don't have to log in, don't collect any personal information, those are things that people might be more reassured by. In terms of surveying multiple countries and students with multiple backgrounds and multiple institution practices, how do you fit such complexity into a single question? Honestly, I don't think you can. I think that's part of the challenge. If you're not interested in those complexities, if you're just interested, let's say, in how many students have cheated in an online exam in Europe, then you can ask all the students, if your sample is representative, you'll get a representative response. If you're interested in what are the factors that influence that, and whether cultural backgrounds, institutional practices in different countries influence that, then you obviously have to collect those data and make sure that the sample is representative of data from all of those different demographic characteristics, and it's not easy to do without access to time and resources. I know that some people, and Richard included, are engaged in research that's doing this and doing this well, but if you're thinking yourself, anyone watching this, of just doing a survey yourself for a dissertation or a short research project, I would instead advise to focus your research question much down, much more narrowly to a single country or even a single department in a single institution, and get meaningful results on that narrow focus rather than trying to do something bigger. Yes. At the same time, it is interesting to try to compare countries, but it's definitely hard. Yes. Yeah. I'm thinking about one more application you might have with surveys, actually. If you use it in a specific context, it is that they in itself can be an educational tool, because asking questions, basically, you can make people think more about, in this case, academic integrity, and in that way, if you ask them on procedures and they're not above it, you can raise awareness and basically create a culture of academic integrity. I'm so glad you said that, because that's the first survey-based study I ever did in academic integrity, is I asked students who are new to university some objective questions about the correct formats for referencing how confident they were in academic integrity and what the outcomes should be, the penalties from particular forms of academic misconduct. The way that I got lots of people to respond was I offered all of the course leaders and the academic, the program leaders, a free session on academic integrity for their students, and I asked the students these questions, I collected their responses, and then I explained to them what the correct answer was as a way of showing them their own data about what they as a group thought, which was often that they didn't fully understand what the scenario represented, and then explaining to them what the correct answer was, and then I used that, those data, and then what was my master's dissertation more years ago now than I care to remember. So, yes, that's a really good point, and I would certainly advocate for where you can use a survey to raise awareness of a particular issue. It can be a very powerful way of killing two birds with one stone, as we would say. It's a good point. So, thank you so much. It was really very interesting, and next time, next E9 monthly webinar is on June 9th. We will have another guest from the UK, Alamed Apollah, who is going to talk about do-it-yourself forensic linguistic techniques for authorship identification, ghost writing, and plagiarism detection. So, very welcome. It is going to be at two o'clock Central European time. You will be able to find this recording on our website and on the YouTube channel of the European Network for Academic Integrity, together with all of our previous webinars. Phil Newton, thank you very much. Thanks everyone. Good luck surveying. See you in the future. Bye.