 Before we begin I'd like to acknowledge that we are gathered today on traditional ancestral and unceded territory of the Coast Salish peoples Including the territories of the Musqueam, Squamish, and Swift two nations. So I'm a little bit of housekeeping before we get started Um, so slides for this presentation will be available after the session. So don't worry too much about taking notes They will be there for you. Um, so the first half of this presentation Um, or for of the session so the presentation part will be recorded And then afterwards after the presentation will be having some breakout room activities and some discussions in q&a So though those those activities in the back half will not be recorded So if you don't feel comfortable asking your question while the presentation Part of the session is still in progress and being recorded Please save your questions until after that in the unrecorded q&a part of the session Thank you. Javi Welcome all to the student experience of instruction workshop My name is Abdullah Zayn and I'm an institution with the planning and institutional research office I'll try to keep my presentation under half an hour to allow enough time for a short activity And for question and answer This is an outline of what we hope to cover today Uh, we'll talk about the type of data that we collect in those in the student experience of instruction surveys And what that means in terms of what we could do with the data and the measures that we compute from the data Um, in particular the three statistics that are reported in the instructor report Uh, finally, we'll also look at ways in which we uh, those statistics could be combined to summarize the student's scores Next slide, please By the end of this session, uh, we hope to have a better understanding of the new metrics particular the three statistics interpolated median percent favorable and disparate an index And also be able to use simple graphics to examine the relationship between the reported statistics Next slide, please in 2018 we started to transition to the new metrics During that transition the old metrics the mean and standard deviation Were reported along the new metrics In I think 2019 UBC made a competitive switch to the new metrics In 2021 changes were made to the sixth university module item questions The modified and new questions were implemented in the form of 2021 But these changes are beyond the scope of this workshop So I won't be talking about them and we will be focused exclusively on the statistics in the in the instructor report slide, please um The type of data that we collect in the student experience of in the sexual surveys is categorical in nature That's to say that the student responses were captured in categories of ranging from strongly agree to strongly disagree But these categories have sense of sense of order and so the data is actually ordinal Because we think of strongly agree as being higher or better than agree. This is higher than neutral The university module item question the six questions the umi's and most faculty and department questions use a five-point scale But there are some faculty questions that use seven-point scale Um a bit on the scales in the next slide, please so Both scales have odd number categories whether five or seven and they are balanced around a neutral response So in the case of the five points, we have two responses agree and strongly agree that are above or higher than neutral These are favorable. We have two that are unfavorable, which is below neutral And in the case of the seven, we have three categories on both sides of the neutral The methods that we will be talking about apply to actually, uh, this type of fiscal as Like a type of scale is odd number categories Next slide, please So we'll take a look at a sample instructor report This one, uh, so just I'm sure most of you either have received this report as instructors or if you're a staff and you're working with those reports You have seen them, but Just to refresh our memories. We'll just take a look at what's in the report We'll take and then we will continue with the talking about what's in the report. Next slide, please So in the Uh, students in the instructor report, uh, there is the description of the section and instructor information at the top along with the the audience or the number of students in the in the section And the responses they receive and the response rate Uh, and the response rate is basically the percentage of the responses they received as a percent of the Of the student invited to participate in the survey and then there is a table that's provided By class size that give the recommended minimum response rate And for example, uh, in this particular small session, this is just a test with two students Both of them respond with handed it to a self-responsory, but I say if we have a class that has Uh, 55 students so that would be in the class size 50 to 74 And we would require a minimum 35 percent We try to work with the students to increase response rates And of course the higher the response rate the more confidence we have in in the data that we in the responses Next slide, please Next you we have in the instructor report We have the university model questions the six umi's And for each question, we have the capital n Upper case n is the number of students that were invited in this case two The lower case n is the number that responded to each question And then we have the distribution of the responses by the five categories strongly the disagree to strongly agree There is none applicable which doesn't apply to the umi questions Then there is the interpolated medium Prevated as i m dispersion index di and percent favorable, which is given below And these are the three statistics that we will focus on In this presentation Next slide, please So, uh, we'll start with percent favorable in a palace like type scale the favorable responses are both that are higher than neutral Excuse me Um, so agree and strongly agree on a five point uh on the five point scale And percent favorable is the proportion of responses that are higher than neutral suppressed as a percent of the total received responses This measure is simple. It's intuitive. It's informative and it's easy to calculate um, for example, if we have 20 students in a section that responded to the survey if sickest of them responded with agree and strongly agree then the percent favorable would be 16 out of 20 or 80 percent I think it is important to mention that uh In student surveys by and large and this is in the literature Students tend to rate their instructor favorably more often than not So for example at ubc Upwards of 75 to 80 percent of all responses are favorable Next slide, please The dispersion index, uh, this is a measure of the data spread How variable the students responses are? um And dispersion index that we use range in value from zero to one A value of zero indicate that all respondents Rated their experience of instruction the same so they all use the same response. There is no variation in the data Uh, a value of one is obtained when the respondent is split evenly between the two extremes strongly agree and strongly disagree um It's uh, I think it's also worth mentioning that uh in our ubc student experience of instruction data Dispersion index rarely exceed 0.8 And usually such high dispersion is associated with section that did not meet the minimum uh recommended response rate Next slide, please So these are actually, uh, three examples of the uh, how the distribution of the responses Uh, uh Which yield at this particular dispersion index if we look at the one at the bottom of the of your screen the last one Here we have sickest responses They were split evenly 30 strongly disagree and 30 strongly agree and that results in the maximum dispersion of one That's at the bottom right of the of that table. Don't worry about how it's calculated. This is this example of how the distribution How the distribution? Affect the dispersion index if we look at down at the top We see that the out of the sickest the responses to 40 are In the category of agree 20 in the strongly disagree strongly agree and because of the uh, the data being A it's the majority of the responses are in one category and the next category is actually not far off from that category We have a low dispersion of 0.2 In the example in the middle We have 100 responses and you can see that they are Spread throughout the five categories 22 strongly disagree 27 disagree and so on and that results in a relatively high dispersion of almost 0.8 So these are just examples of how the distribution of the responses Relate to the to the dispersion index Next slide, please Before I get into the interpolated medium I would like to talk about the distribution of the scores students scores for a given question and also Look at the medium the customary medium, which is the 50th person type. So we have two sections here the one on the left a has 12 student responses So one response is Disagree, which has a numerical score of three of two. There are six And maybe if you have a highlighter, uh, this is a touch you can highlight So there are six, uh neutral numerical value of three And there are four agree numerical values of four. We start the one in blue and one five strongly agree Um, so the median is actually the 50th percentile is three, which is the average of those two threes And we have equal number of responses to the left and to the right of that medium However, if we just look at the value of the median being three, there is one response. That's below that median, which is two And there are five responses higher than the median. This is one of the right And so this distribution This description of the distribution relates to how we calculate the interpolated median and how we how we do that I'll explain it in the next slide, but for now Uh, I just want you to keep those two distribution in mind. The first one has one response That's less than the median five responses that are greater than the median And six responses that are equal to the median the distribution to the right b as a median of four There are 15 responses still the responses that we receive Is a median of four. There are four responses that are below that median value But when you read There are two that are higher than the median value The two fives and there are nine that are equal to the the median value of four So keep those two distributions. We'll work with them in the next couple of slides next slide, please Here again, we have the two distributions Uh, we indicate the number greater than the median by n plus sorry in distribution a we have five of those The number that's less than the median as n minus we have one If you look at the top the interpolated median formula is simply the median and we adjusted by a certain factor That factor depend on the distribution Of the scores below and above And above the median So this in this particular case because we have more responses higher than the median The Median is adjusted or interpolated upwards by about three tenths of a point. So I will interpolated median if three point three in the distribution to the right b There are more responses that are below the value of the median And so the interpolate the the median is actually interpolated or adjusted Downwards by one tenth of a point And I will interpolated median in this case at three point nine If you are interested in the interpolated median formula how it's calculated You can drop me an email I'll share my email later and I will send you a reference for that Next slide, please So these are the again the two distribution. This is just a histogram of the two distribution We see that the two distributions are markedly different. They both have the same mean of three point four So the mean doesn't really Reflect the distribution The first one has a percent favorable Of 42 percent These are the responses of four and five the despite of them out of 12 42 percent The one on the right has 73 percent if we look at the interpolated median of 3.3 and 3.9 We see that they are closely associated with the percent percent favorable Next slide, please So the interpolated median Given that students actually read the Experience of instruction favorably More often than not The interpolated median is preferred to the mean Or to the median for that matter Because a it better reflects the distribution of the responses Better than the mean of the median and also It has It's closely associated with the percent favorable and by Closely associated I don't mean just a high statistical correlation. There is actually a unique relationship between the interpolated median and percent favorable Which we will get into in a minute And those two statistics can actually be combined to summarize the data Presented in a tabular form or in a graphical form Next slide, please So this is the this is a simple scatter plot of the percent favorable And an interpolated median each point represent This is for urm i question number three. So each point represent the Interpolated median and percent favorable from an instructor report. So each point here is actually A section instructor Combination of a section instructor. So for For example, if we look at The point and the upper Quadrant at the bottom Down here to the right We yeah If you take for example an interpolated middle of 4.5 and a percent favorable of 60 percent That would be represented by that point down Down No, same idea the last one Yeah, so that's an interpolated middle of 4.5. So this this point here represents an instructor for this question The students rated their experience because an interpolated middle of 4.5 and a percent favorable 60 percent The relationship has a pivot point at 3.5 and 50 percent And such that for interpolated median values of less than 3.5 percent favorable does not exceed 50 percent and for Interpolated median values that are greater than 3.5 The percent favorable does not Is above 50 percent 50 percent plus Is greater than or equal to 50 percent And the dirt the data is only counting those two sub quadrant The upper left and the bottom right sub quadrant would have no data And if we another thing to notice is that the relationship is quite linear around the pivot point 3.5 And then further from that point they might be spread in the data And that could be explained through the dispersion index as we will see in the in the in some of the examples that we look at next slide please The same this is actually the same scatterplot except this is for a seven point The methodology applies to both scales So this is a seven point the difference between the two is that on the x-axis the point has shifted to 4.5 So the pivot point is actually at 4.5 and 50 percent As opposed to 3.5 and 50 percent in the case of the five point scale Next slide, please So What I have here is for the same question for the same period. This is one academic unit. So this is one department And again, this is a simplest scatterplot of the inter of the percent favorable and interpolated median Again, the relationship goes through the pivot point of 3.5 and 50 percent With no data in the upper left and the bottom right quadrant indicated with the red x The red dot and maybe you can point to the visitage if you can point to the No sorry I thought I lost you for a minute. So the red dot Is actually the aggregate for the academic unit I do not use the word evidence because I don't want people to confuse it with the arithmetic evidence I mean, so this is the aggregate point for the and it has an interpolated median of 4.2 And a percent favorable of 76 percent percent with a dispersion of 0.52 Now we have four instructors in that unit that are highlighted as a b c and d if we look at instructor a We see that they have an interpolated median and the data is given in the table to the left of the graph We see that they have an interpolated median of 3.9 and an 80 percent favorable rating with relatively lower dispersion of 0.35 If we relate that to the aggregate, we see that the 3.9 is a slightly below the aggregate value of 4.2 But the percent favorable is actually four Four percent is points higher than the Higher than the aggregate if we look at the instructor c We see that they have an interpolated median of 4.3 Which is comparable to the aggregate of 4.2, but they have a percent favorable of 100 percent And that's reflected in the lower dispersion of 0.24 And this point will become clear when you look at instructor d and b They both have almost the same interpolated median 4.6 4.5. So they are almost the same vertical line in terms of the interpolated median Instructor d has a sorry instructor d has a 100 percent favorable responses and they have 100 percent And the dispersion of 0.25 whereas instructor b Down there has Percent favorable of 73 percent. So about 1 in 4 student did not read their experience favorably And we can see the difference between the two is in the dispersion index Dispersion index for b is 0.57 and for d is 0.25 So we see here for a given interpolated median in this upper quadrant For a given interpolated median the higher the dispersion the lower will be the percent favorable In the lower sub in the lower quadrant The the relationship is actually the opposite the higher the dispersion the higher will be the percent favorable So if we look at the an interpolated median of three, we see that there is a few points ranging from just over 30 percent about 40 243 percent So the higher the dispersion index the higher would be the The percent favorable in this sub quadrant By and large And from our experience in the last few years about 90 to 95 percent of Responses are in the upper right quadrant And about 5 to 10 percent in the lower In the lower quadrant. This brings me to the end of of my presentation