 I'd like to welcome you all to the Student Experience of Instruction Workshop. My name is Abdulazim. I'm a statistician with the Planning and Institutional Research Office here. And I'm going to turn it over before I continue with the presentation. I'll turn it over to my colleague Jovi for the land acknowledgement as I was giving. Hi. So my name is Jovi. I am a support analyst for the State Team at Pair. I work a little bit more closely with the actual administration of the reports rather than on the statistical side, which is Abdulazim's expertise. So before we begin, I'd like to acknowledge that we are gathered today on the traditional ancestral and unceded territories of the Kosoj peoples, including the territories of the Musqueam, Swamish, and Slava Tuth nations. Now, a few housekeeping notes before we begin. So the slides for this presentation will be available after the session, and only the presentation part of this session will be recorded. Everything else that includes breakout room activities, discussions, and Q&A will not be recorded. So if you don't feel comfortable asking your questions while the presentation part of the session is still being in progress and still being recorded, please save your questions for after that when we get into the unrecorded Q&A part of the session. And now back to Abdulazim. Thank you, Jovi. Again, welcome you all to this workshop. And I'll try to keep my part of the presentation under half an hour to allow enough time for short activity and also for question and answer. I'm joined by my colleagues, Jovi, who just did the introduction, Alisson, Kizytash, and Gavin. And they will be with you in the breakout rooms for the activity after the presentation. The next slide, please. This is an outline of what we hope to cover today. We will talk about the type of data that we collect in the student experience instruction surveys. And what does that mean in terms of what we could do with the data? We will look at the measures that we compute from the data. In particular, we will look at the three statistics that are in the instructor report. And finally, we will look at ways in which we can use those statistics, combine them to meaningfully interrogate and understand the students' scores. A group activity will be assigned to breakout rooms and joined by us to go through the activity. And then we'll have a discussion and question and answer that then. Next slide, please. So by the end of this session, we hope to have a better understanding of the new metrics. That's the instructor report. And also be able to use simple graphics to examine the relationship between reported statistics for a better interrogation of the data. Next slide, please. In 2018, UBC started a transition to this new metrics. During the transition, both the old metrics, which included the mean and the standard deviation, were reported along with the new metrics. And in 2019-2020, UBC made a full switch to the new metrics that we have now. In 2021, changes were made to the six university module items, the UMI questions. And the new and modified questions were implemented in the winter of 2021, Term 1. But these changes to the questions are beyond the scope of this workshop, as we will be focused exclusively on the statistics in the instructor report. Next slide, please. So the student experience of instruction data is categorical in nature, meaning that the student responses are captured in categories. So for the UMI questions, for example, we have five categories ranging from strongly agreed to strongly disagree. However, these categories have some sense of order, which make the data ordinal in nature. So it's an ordinal data. And we see that, for example, strongly agree is a higher or better response than agree for a given question, which is higher than neutral and so on. The UMI question, the university module items, as well as most of the faculty and department questions, use a five-point scale, but some faculty questions use seven-point scale. So a bit more on the scales in the next slide, please. So in the five-point scale, I mean, both scales have an odd number of responses, response categories, and they are balanced around a neutral response. So in the case of the five-point scale, we have two responses, strongly disagree and disagree, which are represented by the numerical values of one and two. These are below neutral. They are considered unfavorable responses. Agree and strongly agree are higher than neutral and they are considered favorable. In the seven-point scale, we have three responses on both sides of the neutral. Next slide, please. So we will take a look at the sample instructor report just to see what's reported, and then we will continue talking about the information in the report. So many of you are familiar with this report, which the instructor received at the end of the term. At the top of the report, we have a description of the section and the instructor information. This is just a test report. And then you have the number of students that were invited to the survey. In this case, we have two in this test one and the number of who responded to the survey. So we have two responses for a hundred percent response rate. Because blue, or explorance blue, doesn't have the capability to highlight or flag surveys that did not meet the recommended minimum response rate. We have this table at the top of the report where the instructor can look it up depending on the number of students in the responses. So the class side, the number of students that were invited, then they can compare their response rate to the recommended minimum to see if the survey met the minimum recommended response. So that's the first part of the survey. In the second part, next slide, please. We see the university module item, the six UMI questions. And we have a breakdown. So the upper case N is the number of students invited. The lower case is the number of students who responded. And then we have a breakdown of the responses by the five categories from strongly disagree to strongly agree. And then we have the NA, which is not applicable. And in this case, it doesn't apply to the UMI question. Then we have the interpolated median indicated by the abbreviated by as I am. The dispersion index abbreviated as D I. And then below that we have the same questions repeated with the percent favorable indicated. And I'll talk about those, those measures in more details. Next slide, please. So starting with the percent favorable that we saw in the last slide. The favorable responses are for any odd number scale. The favorable responses are though that are higher than the user. So in the five, in the case of the five point scale, we have agree and strongly agree, which are higher than neutral. So those are considered favorable responses in the seven point scale. We have three favorable and three unfavorable responses around the neutral response. And so percent favorable is is the proportion of responses that are higher than the user expressed as as a percentage of the total received responses. It is very simple measure, very intuitive and quite informative and it's easy to calculate. So for example, if 20 students completed the survey and 16 out of the 20 students responded with agree or strongly agree to a specific question, then for that question, the percent favorable would be 16 out of 20 or 8 out of 10, which is 80 percent. I think it is worth noting that in the student experience with instruction surveys, a student by a large tend to read their instructor experience of instruction favorably, more often than not. And for example, at UBC overall greater than 75 to 80 percent of the student responses overall were favorable. Next slide please. The dispersion index that we report is a measure of data spread. So it tells us how spread the responses are in those categories. The value of the dispersion index ranges from zero to one. A value of zero is obtained when all respondents agree on a response. So they all give the same rating to their experience of instruction. On the other hand, the value of one, which is the maximum possible dispersion value, is obtained when the students or the respondents are split evenly between the two extremes. So half of them responded strongly agree. The other half is strongly disagree as in the five point scale that will result in a dispersion index of one. Again, I think it is important to note that the dispersion index rarely exceeds point eight. And when it does, it's usually associated with the small sections where the surveys did not or the survey did not meet the recommended minimum response rate. Next slide please. So these are actually examples of the dispersion index. So we look at the first, we just focus on the first one. We look at the first example. We have sickest the total responses. This is the column that says count. Second column, if you can highlight it, you can point to it. The count column, second column. Yeah, so we have 40 responses of agree, 20 of strongly agree for a total of sickest the responses. So we see that the majority of the responses are in one category, which is agree for that specific question. And the remaining responses are in the next category. So they're not far off from that category. And don't worry about the calculation, but this results at the far right in the dispersion index of point two, two. So that's a low dispersion index. If we look at the second example, we have a total of 100 responses. So if we look at the count and we see that the hundred are spread throughout the five categories starting from 22 for the strongly disagree down to 27 on the column of the count and all the way down to 17 which are strongly agree. So we see that the standard responses are spread throughout the categories. And this results in a high dispersion of almost point eight. And if we look next, and then we have the last example and this is the theoretical maximum. I never seen it happen, but this is the theoretical maximum response. We have sickest responses, and they will split even between 30 of them strongly disagree and 30 that are strongly agree, agreeing with the statement of the question. And this results in the maximum possible dispersion of one of one. So this is a don't worry about how it's calculated. This is just to show the range of the of the dispersion index and how that relates to the distribution of the responses. Next slide please. So before I get into the interpolated median, which is the last statistic I'm going to talk about. I just want to talk about distribution of scores and the median. And these are two examples, two small sections. And within an example A we have 12 responses. So the first, if you look at the example A, the first response, we have 12 responses. One of the responses is disagree with the numerical value of two. There are six responses with the neutral. If you can highlight the example A. There are six of them that are neutral numerical value of three. There are four that are agree with the numerical value of four and one strongly agree with the value of five. So what we see here is that the median, which is the 50th percentile, as one of my colleagues, the number in the middle is three, which is the average of those two threes, because we do have an even number of responses. So the median is three. And if we look at the distribution of the responses, we see that we have one response, which is the red value of two, which is lower than the value of the median. We have five responses, which are greater than the median. These are the four than the five. And we have six responses that are equal to the median. These are the six threes. If we look at the second example, which is a section with 15 responses, we see that we have three student responded is strongly disagree on the numerical value of one. One is student responded with disagree. That's a numerical value of two. We have nine responses of agree, which is the value of four. And we have two responses with strongly agree. This is a value of five. Again, when you look at the distribution, we see that four of those responses are below the value of the median. They are red. Two are greater than the median value. This is the two fives. And nine responses are equal to them. So I want you, as we proceed, please keep those two examples in mind. Those two distributions and we would use them to demonstrate the computation of the interpolated median and have a better understanding of what the interpolated media represents. Next slide, please. So I want you to recall the two examples that we saw in the last slide. Example A, the median was three. And we have one response. We call it n minus. That's one response which is below the value of the median. We have five that are above the value of the median. The interpolated median, if you look at the formula at the top, the interpolated median is simply the median m, the customary median m. And it's adjusted by that amount to the right. Plus that amount, that amount could be positive or negative and will adjust the median upwards or downwards depending on the distribution. So in this case here, we have, in example A, we have more responses that are higher than the value of the median of three compared to only one response that's lower than the median. And the interpolated median of 3.3. So the median was adjusted or interpolated upwards by three times of a point, resulting in an interpolated median of 3.3. In the second example B, we have four responses that are below the median. Two responses that are higher than the median, as we saw before. And we have a large number of the responses that are actually equal to the median, which is four. Because we have more responses to the left of the median of the value of four, it's smaller than the value of four. The interpolated, the median is adjusted downwards, but not by much in this case, only by one tenth of a point and we have an interpolated median of 3.9. So this is actually just a quick introduction just to show that how the interpolated median relate to the distribution of the scores. So again, this is actually, these are still the two examples, A and B. This is just the histogram, we have the, as we can see it in picture. And we can see that the two distributions are markedly different. They both have the same mean, if we look at the values of red down, those have the same mean of 3.4. So the mean is 3.4 for both of them. The percent favorable, which is the percentage of the favorable responses. In example A, we have four agree and one strongly agree. So there is five out of 12, and that's 42%. In example B, we have nine responses of agree, nine fours, and two responses of strongly agree, category five. And that gives us 11 responses out of 15, which is 73%. And we see that the interpolated median, which we saw in the last slide of 3.3 and 3.9, is kind of related to the percent favorable. It reflects the increase in the percent favorable in example B, even though the average of the mean is the same. Next slide please. So given that the students tend to read their experience of instruction favorably more often than not, and this is actually in the literature, the interpolated median is preferred, and that's why UBC started to use it. Because it better reflects the distribution of the scores, the student responses better than the mean of the median. It is also closely associated with the percent favorable. And by closely associated, I don't mean that it's just a high statistical correlation. There is actually an interesting relationship between the interpolated median and the percent favorable, which is the subject of a draft manuscript that's actually going to print soon. Working with one of my colleagues on it, it has been submitted for publication. And this relationship actually ended up in the whole idea behind using this new matrix. So if we look at the next slide, we actually see this is a simple scatter plot of the interpolated, the percent favorable on the y-axis and the interpolated median on the x-axis. Each point represents the interpolated median and percent favorable from an instructor report. So each point here is actually an instructor representing the values from an instructor. So you can think of each point as an instructor. This is from 2020 term one, winter term one, and this is UMI question number five. So what we see here is that at a glance, we can see that the vertical line and the horizontal line were added just for emphasis. So we can see the picture better. If we look at this relationship, we see that for an interpolated median that's less than 3.5, there is no percent favorable exceeding 50%. Sorry about that. And for an interpolated median greater than 3.5, there is no percent favorable below 50%. So basically the data, when we plot those two statistics, the data is now limited to the upper right quadrant and the bottom left quadrant, and there will be no data in the upper left or the bottom right quadrant. The other thing that we note about this relationship is that it actually goes through a pivot point, and that pivot point is at an interpolated median value of 3.5 and a percent favorable of 50%. Also, the relationship in the vicinity of that pivot point is fairly linear. And you can see some spread of the data as you go away from that vicinity of the pivot point. In this particular case, we have 96% of the responses for that particular question. 96% of the instructor were in that upper quadrant and only 4%. By and large, for most questions, the upper quadrant have about 90 to 95%, above 90% for most cases. Of the instructor reports would be in the upper quadrant and about 5% to 10% would be in the lower quadrant. I'll talk more about this in the next example. This is actually just a quick, this is the same relationship, the same scatter plot of the percent favorable and interpolated median, but for a 7-point scale. Everything that I said on the 5-point scale holds here in terms of the relationship between the two statistics. The only difference is that on the x-axis, we see that the vertical line shift to 4.5. And so our pivot point for a 7-point scale is at 4.5 and 50%. Everything else holds just like in the 5-point scale. The next example we will get into more detail. This is actually an example from one academic unit. This is all the instructors in that academic unit. Again, this is the interpolated median and percent favorable for one question. We see that the data as we saw before is only in the upper right and the lower bottom quadrant. There is no data in the other two quadrants. The data, we can see that at a glance, it splits between those two categories. Those who have an interpolated median of greater than 3.5, those are with a percent favorable greater than 50%. And those with an interpolated median of less than 3.5 with a percent favorable, not exceeding 50%. Again, to highlight how we interpret this information, we're going to look at the highlighted point in the upper quadrant. The red dot in the center of that is actually the aggregate point for that academic unit. This is the overall statistics in interpolated median of 4.2 and you can see the data on the left of your screen. The academic unit has an aggregated value for the interpolated median of 4.2 with about 76% favorable and moderated dispersion of 0.5. We are now going to take a look at foreign instructors in this unit. They labeled A, B, C, and D. If we look at instructor A, and again the values are written, the values are left, we see that this instructor have an interpolated median of 3.9, which is slightly lower than the aggregate. But a percent favorable of 80%, which is about 4% point higher than the aggregate and a relatively lower dispersion of 0.35. The picture will get more clearer when we look at the other instructor. If we look at the instructor C, instructor C has an interpolated median of 4.3. Again, you see the values on the left, 4.3, and a 100% favorable rate. So the interpolated median is quite comparable to the aggregate for that unit, but the percent favorable rate is a perfect 100%. And the reason that this particular instructor has 100% favorable rating with an interpolated median comparable to the aggregate is because of the low dispersion. They have a low dispersion of 0.24. And if we look at the instructors B and D, sorry, if we look at an instructor B at the bottom, instructor B has an interpolated median of 4.6, which is, would be by many measures considered to be high. But they have a percent favorable of 73%, which means that one out of four respondents did not read the experience of an instruction favorable. The dispersion index is almost close to 0.6, 0.57 for this instructor. If we look at the instructor D, we see that the interpolated median is comparable to B. This is about 4.5. So they are both almost on the same vertical line, having similar values of the intermediate median. But instructor D has a 100% favorable rating. All the students in all respondents in that section rated their experience of instruction favorable with scores of responses of 4.5. And the difference between instructor D and B could be seen in the value of the dispersion index. So instructor D has a low dispersion of 0.25, as opposed to instructor B with a dispersion of 0.57. So in the upper quadrant, as we go down from instructor D down to B, we see that as the dispersion increases, for a given value of the interpolated median, as dispersion increases, percent favorable decreases. Now, in the lower quadrant, the relationship is actually the opposite. If we look at, for example, interpolated median of 3 and we go up, we see that 3, interpolated median of 3, that's 2. Yes. And if we go up, we see that the percent favorable ranges from about 30% to just over 40%. In this lower quadrant, the higher the dispersion, the higher will be the percent favorable. It's just the opposite of the relationship in the upper quadrant. So this is just a quick example to show us how we can use those statistics, and especially in a graphical form, to have a better, maybe more meaningful interrogation of the data, as opposite to in the past where people were relying on a singular statistic, which is the mean, and they just compare the instructor mean to that of an academic unit, for example. So this is a more holistic approach in the sense that we look at all three statistics, and we can see that graphically or in a tabular form, and allows us to have a better meaningful interrogation of the data. I think this brings me to the end of my presentation.