 Good afternoon and I would like to again welcome all of you to this workshop. I'll take the first part of the presentation just to talk about the metrics that's reported in the instructor report of the student experience of instruction. And then I'll turn it over to my colleagues to introduce the instructor dashboard, which is under development. And then we would have a question and answer and some activity before the end of the workshop. So next slide please. So in the next 90 minutes, we will go through what we expect to achieve. We'll talk about changes to the instructor report, the type of data that we are dealing with, which is the student experience of instruction data. And then I'll go into more details about the new metrics that is reported in the instructor report. There will be an introduction to the demo of the instructor dashboard, which is under development. And we will have an opportunity for question and answer and we will end with a group activity at the end. And possibly some more time for question and answer. So by the end of this session, we hope that you will be the participant will be able to will be have will have a better understanding of the report metrics in the instructor report. And be able to explain how two of those statistics interpolated median and percent free available can be used along with a measure of dispersion dispersion index to interrogate the data in a more meaningful and fairer way. And also be able to use graphics as simple scatter plots to explain these concepts behind the new metrics. The changes started back in 2018. And through a transition phase, we first switched from the standard deviation as a measure of variation to the dispersion index. During that transition, we were reporting the old metrics along with the new. And in the 2019-20 academic year, we switched to the new metrics. The key statistics reported and the report include the response rate in interpolated medium percent favorable and dispersion index. And this is just by the way there are some changes that were introduced in 2021 where the six university module items six questions were updated with the new modified questions. But this is beyond the scope of this workshop, so I won't be talking about this. Just a bit about the student experience of instruction data. The data is categorical in nature, but it is also ordinal. And by ordinal, we mean that the categories in which we capture the data are have a some sense of order. For example, on a five-point scale, strongly agree is better higher than agree, which is higher than neutral and so on. And at UBC, we collect this data by using a balance like a type rating scale. Now a balance scale would have equal number of favorable and favorable responses, which may or may not be, which may or may not include a neutral response. So as long as there is equal number of favorable responses and unfavorable responses, the scale would be considered balanced. So the UMI question use a balance five-point scale that's balanced around a neutral response to a favorable and to unfavorable. There are some faculty question that use a balance seven point like a type of scale again with three responses that are unfavorable, three that are favorable and a neutral response. Next slide, please. So these are the examples of the two response, the two balance scales that we use at UBC five-point. And as I said, some faculty use a seven-point scale. Both of them have a neutral category with equal number of responses categories on both sides of the neutral. Next slide, please. So we're going to take a quick look at the instructor report. You're probably familiar with this report. On the first page, you have a description of the section instructor. This is just a test run with only two students and two of them responding with a hundred percent response rate. So this information is in the dark blue. And then blue doesn't have the capability to tag or to flag surveys that did not meet the recommended minimum response rate. So we provide this table so that instructor can look to see if their particular survey has met the recommended minimum. For example, if the class has 40 students, that would be in the certified to the 49 category and 40 percent would be the recommended minimum. Does that mean if there is 39 or 38 percent response rate that the data is not useful? No, it doesn't mean that. It just means that if we do not meet the recommended minimum, then we need to look at other factors and interpret the data with caution. Next slide, please. Next, we have the six item, the six university module item, the six questions. And we have the uppercase N is the number of invited students. In this case, we have 42 in this test example. We have two that responded, 100 percent response rate. And then we have a breakdown of the responses by the five categories ranging from strongly disagree to agree. And you can point to them as it is followed by another applicable category and then the interpolated median and dispersion index. I just want to say that the interpolated median is reported to the nearest one tenth to the nearest one decimal. The dispersion index ideally should be two because of the range of it. We'll talk about that later. But unfortunately blue doesn't have that. I think it only allows you to select the number of decimals for all the statistics. At least for those ones. The percent favorable is reported below and it is to the nearest one percent. Next slide, please. So I'm going to talk now about the three statistics in more details. The first one, which is the most straight for one is percent favorable. On a five point scale, the favorable responses are those that are higher that neutral agree and strongly agree. And percent favorable would be the proportion of the responses that are higher than neutral expressed as a percentage of the total received responses. So, for example, if out of 20 responses, we have 18 of agree and strongly agree, then that would be a 90 percent favorable rating for that particular question. This is a simple statistic. It is intuitive. It's informative, but it is blunt in the sense that it does not distinguish between a four and five or between the one and two. And so that's why we need to use it with other statistics to have a better meaningful look into the evaluation. Next slide, please. The particular dispersion index, which is a measure of the variation in the data that we use at UBC. This index is actually very suitable for ordinal data and it ranges in value between zero and one. A value of zero indicate that all the students respondent rated their experience using the same category. So they are responded with agree or strongly agree, whatever the case may be, resulting in a dispersion of zero. There's no dispersion in the data. One would be the extreme and the other end. And this happened if we have an even number of responses and the respondent is split evenly between the two extremes of strongly disagree and strongly agree. So we have a very polarized rating of their experience and that would result in a dispersion index of one. In our data at UBC, dispersion index rarely exceeds 0.8 and when it does, so if we have a dispersion index greater than 0.8, usually that comes from the small sections where the minimum recommended response rate was not met. I will talk more about those statistics in more detail. So here there's actually this example of the dispersion. I just want to focus on the response categories, the first column, the count, second column, and the dispersion index rate. Don't worry about the rest of the calculations. So what we have here is we have secrecy responses and they were split between two neighbouring cells, two neighbouring categories. So in this case, 30 in neutral and 30 in agree. When the responses are split evenly, split between two neighbouring categories, the maximum dispersion will be 0.25. Let me say that red. And if we change this distribution between the two cells, the dispersion index will go lower than 0.25 as long as it is in two neighbouring cells. In the second example, we have the responses between two categories that are one category apart. So this agree and agree with neutral in between them. So they are not neighbouring cells. There is actually one category separating them. In that case, the maximum dispersion would be 0.5. And if they are separated by two categories, as in the third example, the maximum dispersion would be 0.75. So that's just a theoretical aspect of this particular measure. Next slide, please. And here we see if you can click off three of them. So the first one is an example of a lower dispersion. And we see that the responses in this particular case is in two cells. Most of them 40 is in one cell. 20 is in the neighbouring category. And that results in a lower dispersion. In the second example, we have the responses are spread across the five categories. If you look at the count column, they are spread across the five categories. And this results in a high dispersion of almost 0.9. And then the last example is the theoretical maximum when the student responses are split between the extreme values of strongly disagree and strongly agree. And that results in a dispersion index of one. Next slide, please. So before I get into the third statistic, which is the interpolated median. I just want to talk about medians and distributions in general. And we're going to follow those two examples for instructor A and B. These are the responses for a particular question. And they are ranked from lowest to highest. And we, these are actually 18 responses. Sorry, 19 responses. So they are odd number. And so the median would be the 10th response with nine responses above it and nine responses below it. So the median here is four. But if we look at the value of the median of four, we see that there are actually seven responses, you see the light blue, the fives that are higher than the value of the median. And there are three responses in red that are below the value of the median. And there are nine responses that are equal to the median. So I just want you to keep this, you're going to see this distribution again. Instructor B, the median is also four. And if we look at the distribution, we see that again, we have nine responses that are equal to the median. So there is nine fours. There are only two responses of light blue that are higher than the median. And there are eight responses that are below the median. And we're going to see how we're going to take that into consideration when we compute the interpolated median in the next slide please. So the interpolated median is simply the median with the 50th percentile and it is adjusted. So it's an adjusted median. Think of the interpolated median as an adjusted median. And the adjustment depends on the number of responses that are less than the median plus responses that are greater than the median and those that are equal to the median. So in the case of an odd number of responses, one value would be the median and we adjusted by the quantity on the right. That equation I am equal to m plus n plus minus n minus divided by two. So the number of responses greater than the median is n plus and minus would be the one that's the number of responses below the median and n is the number of responses equal to the median. So we use that equation to calculate the interpolated median. If we have even number of responses, that would be the case of equation two. And in that case, the interpolated median is simply equal to the median. And we will continue with the examples in the next slide. So these are the two instructors A and B that we saw before and we saw the distribution relative to the value of the median. So in the first case, case A, we have nine responses that are equal to the median or seven that are greater than the median. And three that are below the median. So because we have more responses higher than the value of the median relative to those lower than the value of the median, the median is adjusted or interpolated or adjusted upwards resulting in an interpolated median of 4.2. In the case of an instructor B, we have more responses that are higher than that are lower than the value of the median. We have the same number of responses nine equal to the median, nine fourths. But there is only two that are higher than the median because we have more responses below the median value. The median is adjusted or interpolated downwards. And in this case, the interpolated median is three points. So next, we're going to look at those two distribution in pictures. So this is the histogram tells us that these are two markedly different distributions. They both have the same median of four. One have an interpolated median of 4.2. The other is 3.7. And we see the associated percent favorable are 84 percent and 58 percent. And the use of this matrix in a publication that we have submitted for publication is based on a unique relationship between the interpolated median and percent favorable, which I'll get into more in details. And that's actually formed the basis for the use of this matrix at UBC. Next slide. So this is the case of equation two where we have even number of responses. And in this case, the median will actually be the average of the two values in the middle because we have even number of responses. And here in this case, it is the value between five and two. And the median in both cases is actually in both examples C and D is 3.5. And below, I give the expected range of the distribution index. So if the data is all fives and two in the first example, if this is all the data we have in this example, the distribution index would be exactly 0.75. But it is potential. There is a potential that the two, which is the nines value, could be the only two and the rest could be ones. So in that case, the distribution would be higher. And the third approach. In this case here between if all the values are fours and threes, the distribution index would be 0.25. But if we have fives and twos and ones in the data which is quite possible, then the distribution will be higher than 0.25. And we could approach one, if all the values, if the four and the three that we have in the center of the only four and three, and the rest of the data is fives and ones, the distribution index will approach one. So this is the case for the interpretation. If the two middle values are both favorable responses, let's say if the median is between a four and five, then percent favorable will necessarily be greater than 50%. If the median is the value between two unfavorable responses, like a two and a three or a three and one, then percent favorable will necessarily be less than 50%. When the median is the middle value or the average between a favorable response and an unfavorable response as in those cases, then percent favorable will be exactly 50%. Next slide please. So when in the case of an even number of responses on a five point scale where the median is the midpoint between a favorable and unfavorable response, these are all the possible outcomes. So the mid discourse could be a three and a four or a three and five, two and four, et cetera. And this will be the median or the interpolated median because they are one and the same. And the expected range of the distribution index is given. I just want you to note the last one. If the median is the mid value between a value of one and five, that means there is no twos, threes or fours. The data is made of ones and five. And the distribution index would be the maximum one as we talked about before. So given that the students tend to read their experience of instruction favorably more often than not. And this is well established in the literature. The interpolated median is actually preferred to the mean and to the median for that matter because it reflects the distribution of responses a lot better. It is also closely related to the percent favorable and by closely related, we don't mean just a simple statistical correlation. There is actually a unique and interesting relationship between the interpolated median and percent favorable that has not been previously reported in the literature. And we are actually reporting it in our publication and using it as a basis for this matrix which we have been using in UBC in the last few years. As a general rule, for a five point like a scale, an interpolated median of 3.5 corresponds to a percent favorable rating of exactly 50 percent and this is mathematically exactness for exactly 50 percent. And it divides the evaluations or the instructor evaluations into two distinct classes that are above 50 percent and those are below 50 percent. And we can see that in the next slide when we show the relationship in a graph. So this is UMI Question 5 from 2020. So these are all questions about this instructor showing concern for the student learning. By and large for all the questions, 90 percent plus of the instructors would be in the upper right quadrant. And 5 to 10 percent would be in the lower left quadrant. Mathematically, there should be no values, no dots in the upper left quadrant or the lower right, meaning that if the interpolated median is less than 3.5, the percent favorable cannot exceed 50 percent. Nor when the interpolated median is greater than 3.5, percent favorable will not go below 50 percent. And so if you go back to that previous slide, please. So each dot in this graph represent an instructor for that question. So we have the interpolated median for that. So if you point to the next one, the one next to it, this is the next one. Next to it, yes. So TZ-tash is pointing to this instructor here. This is actually campus-wide for that question. This is all the instructors at UBC. This particular instructor has an interpolated median of 4.5. If you go vertically down. And if we go across, we see that they have a percent favorable of about 66 or 77 percent between 65 and 70. And so this is how we read this scatter plot. I'll talk more about this relationship and show how the three statistics, even though we are plotting the percent favorable across the interpolated median, this page in index plays an important role in interpreting the interpretation of these results. And we will see that in more details. But next, we will see how this relationship extend to a balanced seven-point scale. And if we see here, the only difference is that on the x-axis, the pivot point of this relationship shift to an interpolated median of 4.5. But again, the data is divided into two upper right and lower left quadrant with no data in the other two quadrants. In the next slide, we'll take an example of one academic unit. And if you can click so that we can see the data. So this is one academic unit, a particular school or department. Again, this is from 2020. And what we have here is all the instructors for that particular question five in this unit. The red dot is the aggregate for that academic unit. It has an interpolated median of, the data is on the left, an interpolated median of 4.2 and a percent favorable of 76 percent and a dispersion that's moderate to high of 0.5. Now we're going to look at four instructors. A, B, C, and D. And we'll start with instructor A. Instructor A has, if you look at the data to the left, it has, for this particular question, an interpolated median of 3.9, which is about slightly lower than the aggregate of 4.2. But the percent favorable is 80 percent, which is 4 percent is points higher than the aggregate. And the reason being that it has a slightly lower dispersion of 0.3.5. So if we use the mean or the median or even the interpolated median, this instructor could be rated as being lower than average, slightly lower or lower than average. But if we look at all three statistics, we see that because of the low dispersion, this instructor actually has a higher percentage of a student that rated their experience favorable. The picture will get clearer as we look at the other three instructors. If we look at instructor C, we see that instructor C has an interpolated median of 4.3, which is comparable to the aggregate of 4.2 for that unit. But because of the low dispersion of 0.24, and that's a little considered low, they have a percent favorable of 100 percent, meaning that all the students who responded rated their experience favorably. Again, if we just use the mean on their interpolated median, this instructor would have been rated average, but there is really nothing average about 100 percent favorable responses from the students. Now we're going to look at instructor D and B. Both of them have a relatively high on a scale of 1 to 5 interpolated median, 4.6, 4.5, which would be considered high on this scale of 1 to 5. If we look at instructor D, because of the low dispersion in their data of 0.25, there is a 100 percent favorable response, meaning that all the students rated their experience favorably. Whereas in instructor B, even though this instructor has a slightly higher interpolated median of 4.6, because of the high dispersion in their data, the percent favorable is 73 percent, which means that more than one out of four of the students in this section who responded did not rate their experience favorably. This example is meant to show that when we look at the three statistics together, we will be able to meaningfully and in a more fair way look at the interpolate and look at the evaluation of instructors more so than if we just look at one and run a statistic. And this is the basis actually on which we switch to this new metrics and this is the basis for us proposing this metrics in a publication that we hope to see the light soon. And I believe this brings me to the end of this presentation and I'm going to turn it over to Alison to talk about the instructor dashboard. Sure. Thanks. I just have a quick introduction to the dashboard. It is something that our team has been working on in the last year. So mostly just cash working on building it and developing it. And we've been working on this in the last year and what we are hoping to do is get us some feedback in the next couple of months and with the goal to release the dashboard to instructors as we release the winter term one results and that is in January. So that's our current target timeline and I'll just turn it over to Stash and what you're about to see is a very still in the development stage quite pilot. We're kind of still working on it and making changes as we go. So feel free to let us know what you think and provide feedback. This would be great for us as we work on this. Thank you. Thank you Alison and Abdulazim. So I will go through the dashboard. Let me know, you can see the dashboard, right? Yes. Okay, good. So this is the landing page of the dashboard that is working in progress and the landing page has some information about the different measures and it has some navigation buttons. As you can see, this is the contact link. If anyone wants to contact the SEA team and there is also a more detailed contact here when you click in it will open a new page so that you can get more information about the SEA and there are also navigation buttons here and image so that you can click and jump into the different visualizations we have. So on this dashboard we have two visualizations and one data summary table and the first visualization is the scatter plot and on this scatter plot I will go through the different filters and the points that you can see here. So the first filter is to select campus as UBCO and UBC Vancouver and the second filter has two options for the pre and post, pre-2021 summer and post-2021 summer that shows where the UMI questions, the UMI module questions are revised and on the second we can select the year now it is selected pre-2021 summer so when we select one year it will show the respected individual instructor's data points here and this filter is for the sessions for this demo it is only winter 2 session is included that's why it shows only winter 2 and then the next filter is for the UMI questions and all the six UMI questions are listed here and for better visualization it's advisable that to select two or three maximum of three UMI questions at a time and now we selected UMI question one, two and three and when you come here the dark blue dots indicates the courses taught by a given instructor and when you hover over it shows the instructor name the year in session, the course name whether the course meets the minimum requirement the interpolated median percent favorable and dispersion index and the background light blue dot shows all courses and all instructors in the respective campus so an instructor can see how the courses that he or she taught in context with the overall UBC campus course and instructor and these red dots indicates the campus wide measures for the IM and percent favorable then here when you switch to the pre and post the question description also shows changes depending on which period you selected and here let me check some points to elaborate and give example as to what Abdulazim shows in the slides before when we take two points as you can see there is no point let me select only two UMI as he mentioned there is no point here on these two and you can see only on the lower left corner and on the upper right corner of the quadrant and if we take two points here for a given interpolated median as we go down and as the variation increases the percent favorable will go down here and this is just as an example as to what he mentioned there so the upper here point shows interpolated median of 4.5 percent favorable 95 and the dispersion index of 0.34 but when you go down the percent favorable will go down while the dispersion index is high and we can also see for another UMI question here and let me exclude the first one the same example here if we take these two points as we go down for the same interpolated median you can see the percent favorable is 76% 76% in dispersion index is 0.48 then when we go as the dispersion index increases the percent favorable will decline from 76 to 67 and let's see if we can get an example to see how it behaves in the lower quadrant let me switch to the other category and take a year in 2021 and if we can have an example here okay for UMI 2 we have points on the lower quadrant and here the behavior is reversed from what we see as we go from this point to this point you can see that the percent favorable and the dispersion index behaves the same for the higher dispersion index the higher percent favorable and the lower dispersion index the lower percent favorable then this is for the scatterplot and let's jump into the second visualization that is the trained line there is a question in the chat okay let me see okay when an instructor has more than one section or course will there be a filter for the course section on the current as we mentioned before this is this dashboard is still under development but it is possible we can add a filter here but currently it is showing all courses that are given that are taught by that instructor or the same academy for this academic year and for a given section so here it can be changed to like section session winter or summer or winter one or winter two but we can add a course filter here that it is a good comment thank you then so let's go back to the landing page in order to navigate to the second visualization so this is a trend line and in order to see the visualization here first we need to select a specific course because we are going to see the behavior of that course throughout time and we will have the same filter here for the pre and post 2021 summer then we will have an academic session here here because we are looking for a trend more than two at least two or more than two academic years needs to be selected the same for the session and for the UMI question we need to select like two or maximum of three UMI questions at a time and here the red lines show the person favorable and the blue lines show the interpolated median so you can see here and when we go to the second filter definitely as more data are added in the coming periods these lines will be a bit longer than usual so you can see here this is for the person favorable and this one is for interpolated median so this visualization so how the lines behave for different courses and different academic periods then I go back to the landing page and the last part on this dashboard is the summary data and this shows a summary of the data for the courses that are selected on the scatterplot and it shows like which period the courses the question with the full description whether it meets the minimum requirement meet or not in the three measures and it will have these filters so I think by this I will finish the dashboard part if you have any question I will bring you back to the main presentation and thank you