 Hi, everyone. It's good to see you and thank you for attending the quantitative analysis and survey research workshop that will be conducted this afternoon by Kevin Fomalant. I'm Sue Boffman from ARL and really pleased to welcome everyone here this afternoon. It's good to see you all, especially after our summer hiatus from our workshops. Today's workshop is part of our series of training opportunities for the Research Library Impact Framework Initiative, and those of you who have attended some of our earlier sessions have heard me say this. Our overall goal for this series is to help all of us develop or improve our skills in conducting research and assessment in our libraries. We have a few remaining workshops in this series that will address reporting our research. When we share the materials from this workshop with everyone, we will include registration information for those workshops as well. We are recording this session and we will share the recording and slides and other documentation that Kevin has prepared with everyone. You're welcome to share this information and these materials with colleagues who could not attend today, and we do have a few people on our list that we knew couldn't attend but wanted the materials. So Kevin, welcome. Thank you again and let me turn the podium over to you. Okay. Well thank you, Sue, for the introduction. And today's session, today's workshop is about quantitative analysis and survey research. But before I get started, I did want to ask that if anyone has any questions to feel free to break in verbally, I will be monitoring the chat, but it's a bit easier if you just go ahead and interrupt me and I'll answer them as they come. And so just to get oriented to where we are in the process of survey development, we've had earlier workshops on designing a survey and then also doing visualization and tableau after the data collection period. So now we're into the preliminary quantitative analysis part of our survey research, which includes exploratory data analysis and hypothesis testing. And so survey research is a little bit different from some other methodologies that I've worked with in that your hypotheses actually change and probably should change after you finished your data collection and you've done your visualizations and you see some initial patterns in the data. So of course, when you're doing your survey design, you're thinking about what questions you want to answer, what themes you're interested in learning about from your respondents and their opinions about those themes. And you design the questions to get at those themes. But then you get your results and you may find out that your respondents actually think about those themes in different ways than you anticipated. And it's totally okay and encouraged to adjust your data analysis methods in response to that information and to include that in your statistical work. So in survey research, we're not only testing respondents' answers, but also testing respondents' interpretation of our survey itself and using that information to redesign and reevaluate our hypotheses after data collection has ended. And so I'll take you through the process that I go through after data collection, which I think is one of the most important parts of survey research since that's really the guide that sets up the statistical analyses that you will do at the end of your research. So it's kind of the neck that turns the head of your research project. So today I'll take you through several aspects of quantitative analysis. The first most important part is data cleaning. And data cleaning is essential because it helps preserve the reliability and the validity of the survey research project that you've just completed, the data collection that you've just completed. So you need to be eliminating responses that are extraneous and that are actually detrimental to learning about your respondents' opinions on your work. There are several methods that I'll go into with that part of quantitative analysis. And then a more flexible, so we go through pretty strict methods for data cleaning and then we move into more flexible methods in exploratory data analysis that help you revisit initial questions and assumptions by looking at response rates, actual answers to your questions and to cross tabs, which I think are actually an undervalued part of survey research. And I'll explain to you what a cross tab is and give you a visual, just to see what it looks like. Although you may have seen it if you ever looked at a polling memo around the election, they tend to publish those. I'll go into item analysis as a part of composite creation. Some people pronounce it composite, depending on which person you're listening to. And so in item analysis, what you're really doing is figuring out which items are most important and which items fit into which themes. And it's those themes that are we use to create a composite. Scoring methods, which are just quantitative methods to calculate what your respondents' scores are, how they're responding to your questions. And if you just approach it simply, it might just be a mean score, but there are actually several different methods to calculate end scores, depending on what you are emphasizing over other methods. And I'll take you and do some statistics work in hypothesis testing comparisons of means and then Pearson correlations, which I think are a key statistical test to use both to evaluate your results on your own, but also as a way to share with stakeholders who might be interested in your work. It's a particularly intuitive statistic that I find helps not only to just report results, but to drive change whatever program it is that you're asking about in your survey, whatever project that you want to evaluate to actually implement for the next year changes in whatever organizational structure or do any other different methods that you want to use to use the survey results that you get from your project and some conclusions about best practices in quantitative analysis. So data cleaning is the first stop on our tour through quantitative analysis. So data cleaning, I don't actually have this on here, but it does come up sometimes depending on what sort of web design you're using. So if you're using a web survey, maybe it's not so likely that you actually get someone from outside your population responding to the survey. It's common for people to email their surveys to specific email addresses. So you have a pretty good idea that those are the only people who are responding to your survey. At the same time, it's possible that someone else could get access to the survey and answer it even though they're outside of the population of your survey. So say you have a survey that goes out on a listserv and you really just want people who are on that listserv to answer that survey. But you find that when you get your survey results back that some of the email addresses that are listed in the respondents answers actually aren't on that listserv. So it could be your decision that because they're not a part of your population that you want to study to remove those responses. At the same time, this sort of removal does require typically for you to collect some sort of identifiable information about the respondent in order to eliminate them. So that is something that I do when I'm doing survey work for clients. I use email address to confirm that that's the respondent that we think it is or other information linking them including birth year. But for your work, you might have a more expansive definition of your population that you want to work with. So it's up to you whether you want to be strict about excluding people who weren't necessarily in the population that you want to research. But one thing I definitely recommend for any survey research project is a deduplication of responses. This can be a problem with paper surveys or with web surveys as well in that a respondent may respond twice. They may, looks like I may have a question. So let me just pause there. Got a question. In your analysis, do you include partially completed surveys? So that's a that's a good question. And part of you know part of what I'll go into is what in a few slides is what constitutes a completed survey. In some cases I have created, I have declared that a partial survey is a complete survey. But a lot of it depends on what you value. So in some survey research projects I've set a minimum number of questions that make a respondent qualify as a complete survey. In other surveys, it's the number of gate questions. And so as a reminder, a gate question is typically a yes or no question that asks if a respondent has experience to service if they've gone to the help desk, if they visited the library. And if they answer, say I've had one survey project where if they answer a majority of those questions, just the gate questions, and then qualify as a complete survey. So the point is to understand what it is that you want out of your survey. So if you need people to experience a certain number of services that you'll offer, and you're evaluating that through these gate questions, where you get a yes or no answer, and a respondent only answers, yes to one of them, you may want to eliminate that respondent. And any respondent who answers too few of those to reject that as an incomplete survey. Or you could use a mean number of questions. So if there are 40 questions, you might want 20 questions to be answered before including that as a complete survey. So yeah, just make sure that you're setting the standard, the same standard for the entire number of respondents to your survey. There are some other methods beyond just deduplication for data cleaning. There's also disqualification, which not all surveys use. Sometimes when I'm working with a client, I'm required to include all survey responses no matter the responses that are given since they're for regulatory purposes. And other ones I have more leeway, I have more leeway, excuse me, to eliminate surveys that I believe have been answered inconsistently, or just with a lack of attention. So dealing with any population, some respondents will answer questions randomly. You can tell this because they will answer. So the example I have here on the slide on the slide is that two questions are really asked about the same topic. If they had a positive experience about this library, the Petit Library at Penn State, and then sort of the repeated questions asked in a different way if they've ever had a negative experience at the Petit Library. And so if they answer in contradiction in both of those questions, you might feel that you should eliminate just those questions for that respondent, or you can also eliminate the entire survey. I recommend using those last resort. I'm more concerned about beyond just inconsistent responses is when respondents will answer the same answer choice to every question, or it's clear that they're just picking a certain letter pattern like ABCD over and over again. Those are the things you want to be looking at when you're doing your data cleaning to eliminate those responses. So the next step in data cleaning is actually skip logic enforcement. So if you remember from our earlier workshops, skip logic is the structure that's designed in your survey to take respondents from one question to another question. So this typically involves a gate question. So if they have a yes or a no to a question about a service, if they've gone to help the help desk, and they say yes, then they can go on to the next series of questions that are about help desk service or if they answer no, then the point would be to have them skip that series of questions because they wouldn't have a relevant answer to any of those questions. And you want to eliminate those extraneous remarks. Now typically in a web survey, you can enforce skip logic so that the person who answers no to that gate question never sees those questions. But things can go wrong. Sometimes even in web surveys, I've seen people somehow answering those questions. Sometimes it has to do with them leaving the web page and coming back and then the the web survey programming forgets their their response and they're somehow allowed to answer those questions. So it's good to check this at the end of your data collection, you go into data cleaning. And the easy way to do that is to actually look at frequencies. And so I have a frequency example here. Questions one through three of the survey represented. So you can see that there are 160 respondents in question one and say there's no skip logic. Everyone who answers question one goes to question two. So you would expect 160 responses for question two, which there are. But say I put a skip logic restriction that only people who answer question two with answer choice a would be allowed to answer question three. So that would mean that there could only be a maximum of 80 responses in question three. But I can see from my frequencies that there's many more than 80, 80 responses. So I know that something has gone wrong with my skip logic. So the first thing in data cleaning that you can do is to look at these frequencies and then check them against your skip logic to see if anything has gone wrong. And then once you're satisfied that the frequencies make sense that it's mathematically impossible for the skip logic to have not been enforced by your frequencies, then you can you can move on. For finalization, we're moving the extraneous responses. So those responses that the skip logic doesn't allow. We're checking for completeness of administrative data. You may, this is something I didn't answer before. So if you may require the respondent to at least provide some identifying information, whether it's their name or maybe just a pin code that you sent them, if you want, if you're anonymizing your survey. And that way, you know that this is a person you wanted to answer the survey. You have only one survey complete from each respondent. And then finally, we're using some metrics to determine what is a complete survey. So I have a few slides about that. So what is considered a complete survey? Another example is that maybe a respondent hasn't answered a lot of your questions, but if you want them to at least have seen all of your questions, you can require them to have at least progressed to the final question, even if they've skipped them. You can require that all answered questions have been answered. It's been dangerous since most people will skip some questions. Or you can set a certain threshold of questions that have been answered. Or you can even calculate a score based on weighted importance of questions. I think that's a little bit beyond what a lot of us need to do, but it can be done. Or set a few questions that you think are important as the bare minimum for survey completes. So I plot the number of questions answered by the number of respondents that have an understanding of how many people are getting to all of the questions. Of course, I want them to answer as many questions as possible. And just to point out, in this chart, a question is considered answered if they were forced by the skip logic to skip it. So I'm not penalizing them if they have been forced by the skip logic to skip it. It's only those questions they have skipped on their own. They've admitted for reasons that aren't clear. If they just were busier, didn't choose to answer that question. So in this survey, there's actually quite a high number that didn't answer that many questions. And so I don't want to probably put the barrier too strict on this because I'll lose an enormous amount of information. So for this, I would probably put, I would probably actually favor a gate question method just to make sure that they're answering yes or no to a service. Be pretty lenient about the number of responses that they're giving to the other questions. And so I do the same thing for the gate questions to map it out. Same survey. And I'm finding that a pretty big majority of respondents are answering at least two gate questions. So five, seven services. And I want to have people who are at least able to answer about some of the services in the library. Then I would probably set the limit at at least two gate questions answered to be considered a complete survey so that all of the respondents who have only answered zero to one gate questions would be eliminated from the survey and it would not be considered a complete survey. Let's answer the question about the complete surveys or any other questions that may have come up. A lot of times in survey research, the rules are flexible depending on what your intentions are. So if you really want to encourage opinions from people who haven't had a lot of interaction with your services but you are interested in their opinions on that specific service that they've encountered, then you can set pretty loose restrictions on the survey complete whereas if you really only want your responses for people who have visited every single one of your services or you know are committed to your library or spend a lot of time there, then you can set more strict regulations on what is considered a survey complete. So to continue into exploratory data analysis, we're approaching getting getting a final data set. We've established our survey completes. We know how many people have answered our survey. We've done, we've calculated you know how many questions each respondent has answered. So now we're thinking about actual themes in the questions and what trends we should investigate. So we may look at scores immediately. We're going to look at the themes of the questions and how they fit together and then some initial statistical analyses to determine what sorts of differences we have amongst the respondents that have answered our survey. So it's really at this point that we revisit our initial research questions. So if it's something that we prioritize, I would plot answers to the different questions based on the demographic of identification of each survey respondent. So you may be interested in different answer patterns amongst genders, amongst races or ethnicities. And this is really opportunity to start to do that before you go into statistical analyses. Even more just as important I think is that we're looking at the experiences that you're evaluating. When you put your survey design together, you probably had in mind the certain surveys, the certain services that you offer and what you want to learn about them. And it actually may be the case that your respondents think about those services in a different way and that you need to reformulate your hypothesis about how your respondents are interacting with your services. And I'm thinking of a time when I actually had to do this. I had to reformulate a hypothesis from a client, a health insurer who came to us with some data from a different survey and asked us to run a new survey in the following year. And they really were focused on customer service and overall satisfaction. So sure we've all had an experience of talking to a health plan over the telephone so that they were very focused on the telephone service operators, the customer service assistants on the phone who answer questions about the health plan and how that affected overall satisfaction. And so we ran our initial analysis, the initial mean scores and then we started to group the questions together into composites, which I'll describe more later. And this is when we can start inferring things about what's happening in the data. So we noticed that a lot of the customer service questions were heavily correlated with the questions about claims processing. So people were answering the same way to questions about the people who pick up the telephone answer questions about the health plan as the same as the questions that we're asking about how their claims are processed in a timely manner. And so for the health plan those are two separate services, sort of the administrative and policy of responding to those claims and then the customer service, which is really a communication about those services. And so what we were telling the health plan is that the way that the survey was designed is that the opinions about the claims processing session, the claims processing services was bleeding over into the customer service question. So people's opinions was colored much more by the claims processing than by the customer service interaction over the telephone. And so we encouraged, we worked with the health plan to actually redesign questions so that we're asking more specifically about customer service on the phone and separated them structurally from questions in the survey about claims processing so that we could actually tease out the differences in those services. And although that's, you know, it's an example from a health plan. This can happen in a lot of different venues. Sometimes one service offered is much, much more important or just more in the forefront of the minds of your respondents. And you may see at the end of your research project that a lot of those answers are correlated altogether. And that's likely because one experience is coloring everything else, not necessarily that they're all thematically and organically related. And so that could prompt you to redesign your survey to try and reword it and really get the respondent to think about the specific mechanics of the service provided, whether it's the actual interaction at the help desk or whether it's the software they use in the library. So I just, I use that as an example, because this is really the time when you're able to start to see those patterns. And you can think about the ideas will start to pop up about, you know, redesigning the survey for next year or just not overinterpreting your results. So if you get similar answers across questions, you don't want to, you want to get in the mind of the respondent and think about what is their most important, what is the most important priority for them? And I'll get into some statistics that actually help you identify that. So for, you know, for subgroup analysis, you know, I like to actually, I like to track by age group and do response rates. So it's pretty common for younger respondents to be less responsive to a survey. You'll just get fewer responses. So it's a good idea to plot that to see if you're looking at your overall results in the survey. If a large number of your respondents are older, it may not be as representative of younger opinions than you would have thought. And so that's something to keep in mind when you're reporting your results. And I will get into a discussion about waiting results as a method to, as a method to deal with that, although there are some concerns about using that method as well. So another thing that comes up during exploratory data analysis is time sensitive evaluation. A lot of survey projects are run to evaluate a program that has happened over the last year. And so when you're looking at your data, you can focus in on those time specific questions. If there's been a change, a new initiative that you're measuring, you can see if your respondents are answering different to those questions. Maybe they're more positive about the new initiative, or there's just more responses to that question. So you know that there's interest in it. So just to sum up, when we're revisiting our initial research questions, we're thinking about which components of our services are most important. What information can we gain from the survey that is actionable? And this is important for working with stakeholders. I find that when I report, you know, cross tabs by themselves, they're just mean scores to a bunch of questions. Service can tend to, can be pretty long. They're getting longer. So it's hard for people to orient to 30 questions on a survey. And so if you just, if I just report the mean scores, it's going to be hard for my stakeholder, for my client to know what to do with that information. And so the statistical analysis that we do will help you present that information. And how do we gain information from our survey to create or modify a new program? There's the time sensitive project. We can use information from those questions to touch on that project to redesign the survey or to, you know, reorganize our initiative for the next year and test the change in that initiative and how it was implemented and the respondents reaction to it, which is something that I think has come up in a lot of your surveys with respect to the pandemic and how your services have changed. So when there's a big event like that or an initiative, survey research can really illuminate your respondents reaction to that and how they perceive the implementation, particularly when there's a fast change. You may not have had a lot of interaction with your population. So particularly when people aren't seeing each other, survey research can be even more important to gain information about events like that. So I think you mentioned once or twice about revising the survey. Is there a caveat, I guess, percentage of revision that would impact your longitude no exploratory analysis? Yeah, that's a great question. So yes, it is true that when you're redesigning your survey for the next year, you're changing your wording. It always comes with a caveat that you may not be able to make a direct comparison between those individual questions. However, I'll get into this a bit later, when we talk about composites, I'm putting together questions that are related to a specific theme. So if you're reporting just composite scores, according to one theme, say customer service, and the individual questions change within that composite, you can still report the customer service metric composite score. And that can be a thematic comparison between your change survey from one year to the previous year. But it may, yes, it may have an effect on, it will have an effect on the individual questions, which is sort of why I push back against reporting individual questions, results, and relying on them since you may find out that you get rid of that question the next year. But if you create a composite, you're really killing two birds with one stone, you're making it more interpretable, and you're also making your survey more reliable and stable over time. So it's really, if you're testing the same themes year over year, I don't think there's a big question about the fairness of doing a longitudinal comparison. But whenever I'm doing a survey report, and the question wording has changed, I do make a note of it to alert stakeholders that part of the reason that the response opinion has changed may be because of the wording. So I just want to make sure I'm good on timing. But yeah, these questions are great, so feel free to break in. For severe response rates, I don't want to spend too much time on this. It does have to do with waiting, which I'll get in two minutes, but yes, yes, please. Oh, I just had a question around just sort of a follow up to that. When it's apparent in the responses that the interpretation of the question was obviously, there was a high degree of variation, right? And so you see, and you're planning to do a follow up survey along sort of along the two lines to sort of track a trend, let's say like asking a question around, you know, has your library hired within a particular area? And you try not to provide an example, so not to be leading, but then obviously the interpretation. I'm thinking, for example, around research data management, there's many different types of ways that, and job types. And so there are obviously respondents that respond in a certain way, others respond in another way. You see that that's a potentially fruitful area and yet, you know, where would you go with that in terms of, you know, comparison, you know, adjusting for say the next survey? Yeah, that's a good question. So if you've already conducted the survey, you know, I would follow up with a few of the respondents, if you're able to, if it's not anonymous, from sort of the two different interpretations or maybe the different interpretation they have of research meant to see what they are actually thinking. If you have, if you see this problem coming, the other thing you can do is to allow them to write in a response that explains in more detail what they're thinking about that topic, so that when you redesign your survey, you sort of know the, the points of differentiation in their opinions. But I think, yeah, when there's, when there's confusion about the question itself, and the survey has already been completed, the best thing to do is to follow up and do some interviewing to understand how the respondents have responded, have interacted with your question. Is that something that's possible in your case? Our, the particular survey I'm thinking about was actually a national survey and we asked, like, so the responses were collated around by institution and, and then for anonymization at the institutional level. So, we did have a textual answer, but as you can probably imagine, even within the textual answer, I mean, people are reporting, you know, fairly broad job titles. Job titles are not necessarily descriptive, depending on what you're trying to answer. So, I mean, we definitely saw it as a place to explore, you know, for the next go round and sort of getting maybe a gate question with like logic based on that. But, yeah, as a comparison, we struggled with that as a group. And so I was just kind of interested to hear. Yeah, I mean, that can be a challenge. It is, it can be, it can be difficult to write a question that you don't want to lead them into giving you specific responses, but then you get responses that are too disparate to really work with. So you can use, you know, typically, and since our language is pretty regulated, this is something we deal with quite a lot is to have a specific definition that, that tries not to be leading. So we'll have some expository text. So we might describe what research management is without mentioning any titles, and then, and then guide them into answering questions about that field. But I think at the same time, you have to, there's the element where you have to take the responses at their word. So if, if there's sort of a different differential interpretation of what research management is, then that's actually sort of an answer that I think is actionable, that problem isn't so much, the problem may not be the responses to research management is that, that some institutions may not have a concept of what that is for them necessarily. And so you can create a follow up survey to go into more detail if you have that ability. Okay, yes. So survey response rates have a breakdown here by race, since I want to see the difference between the response rate by, by racial subgroup. So I know the, the natural makeup of the population itself by, by race. And so in this survey that I did for a, for a client, African American respondents were underrepresented relative to the population and Asian respondents were overrepresented, while white respondents were about, were about, we're responding at about the same rate as you would expect from their number in the population. And so this is useful information to know that when you're reporting your results, that there is a slight difference in the demographic makeup of your respondents compared to the population itself. Now what to, what to do about that is something that's very tricky in, in survey research. And that can involve waiting. Let's see. Yeah. So for, for waiting, and I do want to talk a little bit about waiting because it comes up as a, because I know that people get frustrated when they don't have an exact when their sample isn't the same as the population by demographic makeup, which is pretty typical since it's very difficult to get that to match perfectly. It's, it's common. And some groups just have different response rates than others. One of the challenges with waiting is that, so for, for, for with waiting, what you're doing is you're taking the responses from that, that population. So for the African American group, for example, in this, in the survey project, and then you're reweighting them as a percentage of the entire population. So you're taking the responses from that small, that group, and then expanding them as more important than the number of respondents, but as important as their demographic makeup in the population itself. So you're using that smaller than expected response group and expanding it to its proportion, the population itself. So this controversial, controversial because what you're, what you end up doing is you're sort of doing a mini survey within a survey. And the fact that you've gotten smaller responses and then you're extrapolate extrapolating means that you're, you're taking a smaller subgroup with a higher error. And you're actually expanding that error by increasing that weight within the population itself. And so that can have real life consequences. And we've actually had a good example in the 2020 election. So in the 2016 election, a lot of pollsters made a correction when they, after finding out that their results were erroneous, that voters without a college education were less likely to respond to surveys, but nonetheless go to the polls. And so their answer was to actually weight those voters more heavily. So in the 2020 election, when they got the same response pattern, those the same voters weren't responding to their surveys. And then inflating that population to its projected proportion of the voting population itself. The problem was is that they're the people who didn't respond to surveys were characteristically different from college to voters without a college education. And that actually made the problem worse in a lot of states. So you're actually taking that error from the, the, the characteristically different people who didn't respond to the survey and expand and expanding it because you're just taking the people who did want to respond to the survey, which has ended up being the cause of the error in the 2020 election. So it's one of the challenges when dealing with weighting is that if you think that there is a characteristic difference about of the group that's not responding to the survey, then weighting it, weighting without their responses may actually make the problem worse. And it's better just to go with the demographic makeup that you've received. However, if you think that there is no characteristic difference between the people who have responded to the survey and who haven't within that group, then it is safer to do a weighting protocol. But you know, there's, there's a pretty big controversy in survey research. So there's no easy answer to this. So I encourage you, I encourage people who are running survey research projects to be open about the different demographic makeup of their survey respondents compared to the population as a part of your discussion and then to weight carefully. Now, for a lot of you, this may not actually come into play since you're just getting responses from everyone and you're doing the best you can. There's not a whole lot of things that you could do, whereas pollsters have millions of people that they can, they can reach out to in addition. So the better solution in a lot of ways is to oversample, to talk to more people so that you can actually get more respondents from that specific subgroup, whereas that may not be an option to use since you're already surveying the entire population. So answer choice counts. It's a simple way to look over your responses to the individual questions of your survey research project and get a sense of your respondent preferences as you're going through your exploratory data analysis, which I'm sure many of you have done just naturally. But then cross tabs, which I mentioned before, I think are an underutilized tactic in looking at survey results and exploratory data analysis, which I'll describe here. And so what a cross tab is, is you're looking at how one group was responded to a question in one way, how they've responded also to a different question. So for example, you can look at the people who responded yes to question one and see how they answered question eight. So if they answered positively to a question about customer service, how are they also thinking about software? For example, software in your library. So that can reveal hidden subgroups and preferences within your data. So you might find that across tab with younger respondents, they're really interested. They have positive opinions about the software or maybe your older respondents don't. We can also find that people who respond positively to customer service are just responding positively to everything, they're just a positive group. So you basically just put the, you pit the two questions against each other and look for trends for when you're doing your exploratory analysis before you go into a statistical analysis. So one of the things that I am looking for and doing cross tabs analysis, you know, I'm looking for errors and sorry, design because it can help you look at the counts and you get one more chance to fix skip logic. If the frequencies are not matching up with what they should be, should be based on the skip logic. And then something that I like to look at is sort of confounds and demographic variables. So maybe you're interested in ethnic subgroups. So I work with a lot with survey with clients who are interested in opinions from Latino respondents. And so one of the issues that I always have to deal with is that Latino respondents tend to be younger than other demographics. And so it's not always clear. And I can see this in the, in the data, whether I think whether the statistical analysis will determine that the effect is due to the, the youth of the respondent or because of the ethnicity. And so it's good to know those confounds in advance before you go into a regression analysis, which will actually be in the next workshop that we have, or before we even do some correlation work. Let me show you what, just a visually, visually what a cross tab is really just a matrix. And so for here, you can, you might want to zero in on the answers to question one. So the most frequent users of your library and to see how satisfied they are with your library services. So from these cross tabs, I can already tell that those who have visited the least, who have visited the library the least, are the least satisfied, whereas those who have visited a lot are most satisfied. So you might actually think that that is a good thing that the more that they come to your library, the more satisfied they are, because you, those are your, you know, that's your key audience, the people you want to to, whose attention is most important to you. And then also affect, affective age can be important too, whereas in this question, I'm seeing that there's an effect of age, whereas older respondents are less satisfied with library services. And from this, I might, you know, I can re-evaluate my hypothesis of the hypotheses. When I do my correlation work, I can try to see what it is that older respondents think is important compared to younger respondents. Maybe older respondents are more interested in opening hours, or younger respondents are more interested in software. So you can do, so it's really when you're doing your cross tabs work that you're, you're thinking of questions that you can test with your correlation work. And item analysis. So item, an item is just a survey question. We're going in and identifying survey questions and putting them together into composites that are, that have a specific theme. So this was a little bit covered in earlier workshops. I just want to go over it again, survey validity and reliability. The two keys to, to a successful research project, validity being a survey that intends, that measures what we intend for it to measure. So we're looking at wording questions in the way that we'll get at the concepts that we're interested in, and not something unrelated to what the concept we're interested in is. Reliability, which is the ability to get the same result from the same survey if it's given a second time. And then the right reliability in the sense will each respond and understand the question in the same way. And so when reliability statistic that's become very common in survey research is, is known as some crown box alpha. It's not typically used for survey overall, survey reliability, but it is used for composite development. And the history of this is that surveys used to be shorter. So they used to cover just one concept. So the alpha statistic was used to determine the reliability for that entire survey measure, whereas now surveys are longer, so that we're looking at multiple themes within a survey itself. And so the alpha is actually used to put together series of questions to develop composites. So I already covered this a bit, but so composites are there to, for, for a number of reasons, it improves interpretability. Instead of reporting single questions, you can report themes that people can latch on to. Customer service opening hour software and other things I mentioned already, I'm sure you can come up with more. It's easier to track changes over time. Since if you substitute one question for another, if you change the wording, and your crown box alpha still establishes the statistic establishes that it should be a part of that, of that same composite, like you wanted it to, then you can safely report through the composite results without having to, you know, talk about individual question wordings as much, since you've already established that this composite is testing the theme that you wanted to test. So, which is really an improvement in construct validity. The key thing is, I want to point this out, you have an idea of what you want your composite to be when you're doing your survey design, but these composites are generated after data collection. So when I put together a survey, you know, I might have four questions that I think are customer service questions. And then I find out that maybe only two or three of the four here are related to each other, or thematically related to each other, using the alpha statistic. And those three two or three questions will get reported together as a customer service score. And so the alpha statistic itself, it's not a statistical test. So it's not, you're not doing a test for significance. You're looking at the variance among the questions that you that you think are in a composite. So you might put together 10 questions you think are in a customer service composite. And the alpha statistics statistic, which I've calculated in SPSS. Next slide. We'll give you a statistic that tells you the degree of variance amongst those questions. So the similarity and variance. And it's really you are heuristic to select questions for thematic composite. So you'll typically a statistical program will give you a an overall alpha score for the the questions that you've identified are what you think are in that composite. So as if it's a mini survey itself, and then it will give you a list of the alpha scores if you eliminate a question. And so on the example here that I've provided, you know, it's a five, I thought that six, six questions would be in the composite. But it turns out if I eliminate question for the alpha statistic increases to a pretty high number, almost point nine. So I'm willing to drop that question from the composite, since it's not as related to the other questions. And so I would probably have a pretty solid five question composite here. And so the heuristic that I'm the statisticians usually go with is anything over point seven is good. But if there's a major drop in your alpha, that that sort of that interferes with your your alpha statistic, then you can drop that from the question. And so the alpha is really looking at so the correlation between the questions, but it's a function of both the number of questions in the composite itself, and their inter item, their inter item correlation. So the larger number of questions you have, the more, the more leeway you have. But if there's only a few questions in the composite, you know, there'll be a large change in the alpha if you drop one, compared to if you have ten you only drop one. So what's it what's important for me? I like to do Pearson correlations of my my survey research project is to differentiate these these two, these two topics, these two concepts. So crownbacks alpha is a measure of reliability, it's not really it's not used for hypothesis testing, you can't run a significant test on it and it's used for making a comparison amongst multiple questions. You'll get a single metric for a series of questions, whereas the Pearson correlation is I'm trying to determine the covariance amongst two questions. It is used in hypothesis testing, so you can figure out if two questions are significantly correlated with each other. And it's key that it's only two questions. So this is important for my work, because I want to figure out typically how different concepts in the survey are related to overall satisfaction, which aspects of the services that my client is offering are most important to driving overall satisfaction, since that's what they want to pay attention to the most. And something that's important to stakeholders is to find out, okay, we have these resources. And we have this amount of time, what can we focus on to improve our scores? What can we do to improve our respondents opinion to make them more satisfied with what we're offering them? So I'm running a little bit of time. So I may come back to the scoring methods. I think you're familiar with the T test, but I do want to show the Pearson coefficient. It's mentioned, yeah, it's useful in hypothesis testing, but I want to show you what it looks like when I'm running my correlations. So each question, I've given a name that's related to the concept that it's getting at. And in this case, it's these are individual questions. I've just given a different name, although you can do a Pearson correlation for the composite scores as well. And so if I'm interested in this typical, what is the driver of overall satisfaction? I want to see which question has the highest Pearson coefficient. And so in that case, that'll be customer service is the strongest driver overall satisfaction with opening hours being farther behind and software availability being among the least important. But you can also see other correlations between the questions. So the answers to visit frequency are similar to the answers on opening hours and also for study rooms and opening hours. Okay. So I'll touch on some scoring methods since we have a little bit of time, but I do want to leave some time open for questions. So there actually are different ways to approach scoring answers from your respondents. Probably the most intuitive ways to do proportional scoring. And this scoring is dependent on the number of answer choices to your question. So in this case that I'm showing you, it's a never sometimes usually always question. So there's four answers. And so you just divide them into the zero to one scale, zero for never and always for one. And so this is to translate the words into a numeric valence. So that's a pretty intuitive way of doing it. However, this is also pretty common. It's called top box scoring. And this is when you're prioritizing, when you're sort of, you're dichotomizing your responses. Say at the end of your survey, you realize that a lot of people are answering very satisfied, satisfied and either dissatisfied or satisfied. You're not getting a lot of other responses. You may want to separate those two and you're kind of, you've looked at your responses and you realize there just isn't enough variance in those answers. So instead, you're doing top box scoring. So you can tease out the difference between a satisfied respondent and not very satisfied respondent. And this way, you can just assign a one to the two top scores as if that's a perfect score for you. Whereas everything else is a zero, which is the lowest score possible. And that way, you're, you're teasing out minor differences and sort of making up for an imperfect question design. Because you didn't anticipate in advance. And it's not always possible to that your answers would be skewed from your respondents in a certain way towards being very satisfied and satisfied compared to not satisfied. And so top box scoring is a way to separate those respondents. So composite scoring. So the most intuitive way to do composite scoring is simply to take the proportional score for each question. So for the example here, you can see at the top, at the top right, there's the answer in the numeric equivalent, and you're just taking an average based on the number of questions in the, in that, in that composite. Problem is that, you know, some respondents will skip a question. And so you have to figure out what to do with that skip question, whether it's to simply remove that question from the average itself and then just put the, the total over, over three, because that because of that one question. The problem is, is that when you do that, what you're doing effectively is imputing the answers to those three questions, the fourth question, so that when you're reporting the composite score, you're effectively reporting that all the questions, when you're, when you're doing an interpretation and someone's reading your report, it's going to look like all four questions or were answered for everyone, and that each question had an equal contribution. So that might be okay if the respondents are all answering the questions in similar fashion. But if there's one question that's getting skipped a lot, you might want to try a different method. So what you can do is actually just, you can wait to the questions in a different fashion. So if one answer, so you can actually wait the question that hasn't been answered as much more, so that to compensate for the fact that it has been omitted so many times. So that's one of the corrections that you can do, or you can actually do some imputation methods, which I will actually get to in the next, our next workshop to assign a to assign a response that's been omitted based on their responses to other questions in the composite to the survey itself, which is something you can do if you're allowed to, you know, if you, if you want to learn more about your respondents, but are also allowed to impute responses sometimes for compliance reasons, it's not possible to, to do that. Okay, okay, so we just have a few minutes left. So taking you through the quantitative analysis, which is the approach you use when you have your, your survey data collection, you've established what is a complete survey, you've done your deduplication, you've done your exploratory data analysis to find trends, and to reformulate your hypotheses. You form a composite to improve your, your survey reliability and your validity and also to improve the interpretability, especially if you're doing a repeat survey over time, longitudinal analysis, and you're doing hypothesis testing based on your expert knowledge of your, of your population itself. So if you have intuition about how different subgroups are answering, you can answer those questions with some, with some work with, with the Pearson core, the Pearson coefficient statistic. So just cover what I wanted to talk about for quantitative analysis, there will be a workshop in October on survey reporting, which we'll get into regression analysis, survey preparation for future cycles, future iterations of your survey work, report structure and development, and then developing reports and dashboarding for stakeholders who are interested in your work. So just want to thank you for your attention and for your questions, but we do have some time for, for questions now. So feel free to, to jump in. Just clarify, you said you were using SPSS here for, for all of your examples. For some of the, some of the graphics are in R. R, okay. Even the, the, the race tables from R or, or SPS, but I've used data a long time ago, and I know it used to give you some pretty crude graphs too. All the, all the visualizations are from, from R, just the, yes, the, the statistical output is from SPSS. Okay. Yes, I just have a general question, Kevin. And there's a lot of decision making in terms of handling the data as, as we've talked about, you know, what's complete, what's not, are there ways that you found to kind of counter any bias on your part when you have to make decisions like that? Yeah. Yeah. That's an important point. I think it helps to, you know, to use your, your professional experience to sort of write out what it is, the analysis that you want to conduct, and then to evaluate which of those analysis could lead, analyses could lead to, to bias. So if you're interested in demographic subgroups, you may actually decide that you, you want to limit your exploratory data analysis to, to other topics, so that you're not, you're not developing preconceived notions about what you see in the data. You know, at the same time, when you're doing your exploratory data analysis, you're, you're still early enough in your work that you're not, you're not doing the sort of regression work and the reporting work, which will come in the next step of the process. So there's still some separation from what we're doing here to reporting. So there can be another, there can be another time period if you think about the work that you've done and to see, you know, how bias could have entered into your analyses. But that's when we're, that's when I'm typically talking to other colleagues in my, in my group in the cert, who are, who are working with me on the survey project itself, to see, you know, how, how, how else they would be conducting those analyses. There might be a different method for mine and to work out, which would be the best approach as we move towards the reporting period. So I will, I will go ahead and add my, my email address in case you all don't have it. If you have any questions about, you know, the presentation today or any questions about survey research that I could, I could assist you with, whether it's dashboarding or anything you heard in earlier presentations. Great. Thank you, Kevin. I would also encourage our colleagues here to reach out to Kevin, because he is available to, to provide some consultation service and advice, especially to our teams as, as you begin to analyze your data and begin to think about how to interpret what, what you have collected. So please do reach out to Kevin. Kevin, it looks, it looks like we may be at the end of questions. So I would like to add my thanks again to you, Kevin, for leading this workshop. And I look forward to seeing you in October for the other workshops that you'll be offering. And just as I said before, we'll send materials out to everyone from today and encourage everyone to sign up for the next set of workshops. So thanks everyone for attending. We really appreciate it. Thank you, Kevin. Thanks so much, everyone.