 What I wanted to tell you about today is we did actually do a study. We implemented what we came out of the study and I want to hear what we're now going on to. So first of all thank the National Vet E-Learn Strategy and the New Generation Technology Funding which actually funded the study a couple of years ago now. What we did for the study is we looked at a lot of different learning analytic tools, how they were going to interact with our Moodle system and which ones we thought would actually be able to implement in the long run and actually use with Moodle. So it was fun, I'd like to say, most of the time getting the different plugins into Moodle or using them around Moodle was often a little bit trickier than we thought even though we weren't using it in a live environment. So that was something that we did sort of stumble over a little bit. Getting data into a dummy environment to see how it's going to work was also something that sort of was a little bit tricky. But in the end we had some great successes with a couple of the plugins. So in the end we've gone with the Progress Bar which we've chosen because the students, one of the things that we got with the feedback with them was they really wanted to have something visual to show what they were doing. So we do have the course content lock-in and so students can see from that what the teacher set up to actually show them what they're doing and how they're doing with it but they wanted something that was visual which is another one on from that. The other one that we're using is Gizmo and so Gizmo's been an interesting plugin. One of the biggest reactions we had from it was we've got a teacher who is into all the animated presentations. So Animoto, something tunes. And he makes really long ones, inappropriate ones that you get like four lines of information out of watching 15 minutes and for years we've been telling him it's not a good way to teach. And finally he looked at the Gizmo and went, how come they've watched the first two videos and they don't watch the rest? Because we're telling you they don't like them. So it was really good for him that he could actually have some evidence to critically reflect and then move on from that as well. So what we found especially with the Progress Bar was the teachers were very innovative and used it in different ways and we sort of thought they might. At CRT because we are a vet institution we do tend to do something that we call cross-listing or clustering of courses. So we might have say 15 different competencies that are involved in a qualification and they're teaching that in a semester and we'll put them all into one Moodle course to actually teach it. Okay so that can be absolute hell for the teachers in the long run to actually work out what it is as well as the students because it's a little bit harder for them to track their progress. So it was actually one of our plumbing teachers that went, hmm, can I have more than one progress bar? And we went, yeah I suppose so. And suddenly he was putting one in for each of the competencies. So then the students knew exactly what they had to do for each competency and they could tick it off as they were going. The students actually really loved it because they could see how they were going and what they had to do in each one and they don't often do them linear. They do actually jump forward and back because it'll depend who the trade teacher is on the floor, which practical component they can do. So they might be doing theory from something else because the teacher who's practical on the floor isn't doing what they're up to and so they can jump back and forwards like that. So they found it was of great benefit to them to find out to know where they're up to. Now the gizmo picture there is the darker the colour is the more the students use the resource. It's been interesting for teachers to see how they actually, how many people are looking at resources. When we're doing the study you can see down the side there's the grade. So it's a graded subject so it goes from a P down to a distinction. A WA is that they attended but didn't actually finish but did actually engage. The darker the colour is on the gizmo the more that resource was looked at. So one of the things that we were doing was looking at how many times people hit on certain resources and whether that affected their grade. It did a little bit. The people who failed obviously didn't do enough to pass and obviously didn't look at the right resources or they might have actually had a more successful outcome. And between a pass, a credit and distinction there was a little bit of difference. So distinction and credit students often did look at resources more often. But then again it was also some pass students that probably if they had a little bit more support from teachers would have actually been more successful. So it did also show that it's not always how often you look at the object it's how you actually interact with the objects and how much you actually get out of it. So that was interesting as well for the teachers to be able to target students who were looking at something but still not getting the grades. Why aren't they getting the grades? Is it a problem with the resources? Is it a problem with the student? One of our canning teachers had a beautiful course. She had so much content in it she tried to download her whole brain into the course. Lots and lots of different resources. They didn't need three quarters of it to actually pass. As she worked out when she looked at her gizmo for the semester. So she could see that the students were passing and they were looking at about a third of the material that she had up there. So she went through and paid back her course to exactly what they needed to know to be able to pass. And she was suddenly getting rave reviews about her course. So it was really good for her to self reflect on what she's got in there and to actually move forward from there. So this is the plumbing teacher that had all the different things in there. So please excuse the student names. Whoops. But yeah so you can actually see how they can actually go through and have a look at what they're doing and where they're doing things and if they've been successfully complained or not. One of the problems we had with the teachers is they wanted to put absolutely everything as a must do. So the progress bars were looking really busy and unless you actually moused over them you couldn't tell what you still needed to do because it was such a busy little toolbar there. The toolbar doesn't get any bigger. The little gaps just get smaller, the more resources that you have in there. And so that was a little bit of a cultural change the teachers had to do because they had to then make decisions about what they thought was the important things of their courses. And it's amazing how quickly once they've worked out the important things they can actually get rid of a lot of the other content because if it's not really important do the students really need it to pass. So the student comments have been that makes me feel good when it goes green, which is always nice. Most people have that every time they're sitting in a red light. They also love it so they can see what they're doing and what they have to do next. And so that was, it's a really important thing for the students to have that visual because we've had it with the course completion block but mostly students didn't even know that they could tick on that or click on that to actually get more information. We haven't made the progress bar as something that is automatically in the courses yet. We did do that with the course completion block. So that's an automatically now system-wide in every course. Teachers actually have to get rid of it if they don't want to use it rather than having to add it in. But the progress bar is something that we're looking for probably next year that will actually make as a standard item in every course. So if the teachers don't want to use it, they'll actually have to take it out. So with the teachers in Gizmo, made a big difference to their course design which I said before and yeah, I can't believe I spent all that time on those resources. I need two students in the class used it and the majority of students passed. One of the things that we're going to do from here is and I'm actually hoping that you guys might be able to give me some ideas and some help with this, is because we've worked out that a lot of the resources that teachers are developing probably aren't the best type of resource for the course that they're teaching and they don't need a lot because they're all over teaching, they're over assessing, they're spending way too much time actually marking and everything as well and we need to get that so it's actually more realistic. So what we're going to be doing this semester is we're going to be trying to benchmark what a good course is and what a good resource is. So we're doing another bit of a research project that we're trying to get all the students involved in as well as teachers in set areas to actually look at how we can benchmark the courses and the resources to then advise teachers how to actually go about developing. So we're hoping that the research we do this semester will actually enforce all of our teacher PD that we do next semester. So anybody got any really good ideas how we can actually go about benchmarking resources? I'll tell you how we're planning to go about it. We've done a little bit of a research already, just a literature research on it and there doesn't seem to be too many things out there that give us a great idea of benchmarking a resource. It all tells you how to develop a really great resource which isn't really what we need because we need a good resource for that. We're going to be doing focus groups with the students and we're going to do those in specific areas so it would be like the ICT students would do one, the hospitality students would do another one. We're not sure how many students would actually need to make that viable. One of the things that we were doing when we rolled out the learning management for the first time, the learning management system, is we got hundreds of students through to actually look at the system and tell us what they liked, what they didn't like about it and do you know what, we could have done it with one. They all told us the same thing. It was just in different words. So we probably actually could have done that with one student rather than hundreds. We're also going to be doing the same thing with teachers. So getting teachers together, showing us their good resources, showing us the ones that they think are fantastic, the ones that need work, showing us those as well. So we've got something for the students to compare and we're also going to do a desktop audit which is my team's going to absolutely hate me for because it'll mean they're going to get probably a couple of hundred courses each that they'll have to have a quick look through to decide what is good and bad about the courses and what's currently in courses so then we can know how we're going to actually upskill teachers from them. So has anybody done it before? We've done not so much on the general course but on the video resources. We actually moved to YouTube and one of the things I noticed once we started using YouTube was the stats give you very detailed information on how much of a particular video we watched. We found some videos people watched on average it was like 180% so they've watched it more than once. Again, it's such a good video. And other lecturers who thought they were really, really good had like 12% on average of their video and so that was really good for us to work out in terms of how people were shooting videos or how good the lecturers were in that kind of stuff. So what sort of information did you get out of that? Was it the length of the video that's good, the content that's desirable? There's a bit of it. I mean the length did have a factor on the drop off but also people who we thought were good, when people are only watching 12% of your video, something's wrong. Yeah, so that was good. So that gave you what you need to change. Yeah, just as far as video resources. I can't really offer anything on that. That's alright. Somebody else was, yeah. We actually just go back to the basis and use the assessment requirements that are provided on training.gov.au because our trainers will give us tons of information and they'll say put this, put this in. But our instructional designers go, it's really nice to know what do they need to know. Because our trainers work on a different model too, where we go out into the workforce and actually work within their workplace environment. And there they can tell them they're nice to know. But on their online learning, they need to know that they can actually do the assessment so they can be marked as competent and if it goes through an audit, then it can say yes, they're actually marked off the assessment requirements and the performance criteria. Yeah, we don't have too much problem with them during the right assessment. We don't have instructional designers, though, to develop our resources or make our courses. That's what the teachers do. Because we've got, I know, 54,000 courses, a couple of hundred, well, a couple of thousand teachers, as well as 74,000 students or something. So, yeah, that would be very nice. Take the performance criteria back to the teachers and they can just go, hey. Yeah. Which is good. Which is good. He's actually provided a good assessment tool to connect your unit in the mood or go, here's the elements and performance criteria of that unit. Do they map questions? Yeah. We do that. So, we already do that with the teachers and it's amazing how they say, no, no, this has to inform that. So, yeah, they're very good at sneaking little bits in that they probably haven't been counselled on doing it. I was going to say, you know, eventually I don't have anything on that subject. I had a quick look through their databases and I couldn't find anything that tells me a good methodology even to actually go about benchmarking for resources and courses. So, we all think we know what's a good course and we've recently just updated our subject guide, which is, you know, what's in the course and I took that to the student association as well with a group of students and said, well, you know, what do the students want? And they went, well, we don't want it. All we really want is we want to know our assessment and our teacher name other than that. We don't really care what we're doing weekly as long as we know where to go. We don't care about all the support systems that everybody says there must have information has to be in this guide. So, it was quite interesting that they went, yep, just pop it up on e-learn, which is our Moodle and we'll look at it there. Mind you, we need that to report back to ASQA when they don't complete, but anyway. One of the things that we've started doing is putting a feedback with 11 compulsory questions at every course across all of our Moodles must have. So, what we found is we had feedback before but we were comparing apples with oranges. So, we've got 11 compulsory questions that is just about the course and the functioning and the resources and the things that they use within a course. And we compared, so now we're comparing apples with apples. So, I can go to hospitality or business or community services and see what resources the students are using and why they liked them and why they didn't think of that. Yeah, and we've also got the Student Association going around to the end of courses. Does anybody know if in Moodle, you can, because we have the end of course feedback in every course as well, how can I get them from every course into one pool? Develop them all in one course and link them out to each resource. Oh, no. Thank you for your time in the series. Somehow, I remember there was a study just a simple thumbs up or thumbs down with each resource. I don't think Moodle's got it at this time but just getting a vote from each student on the resource. That's not a bad idea. I bet Facebook likes a lot of them would probably like that. Yep. Oh, sorry. You might be able to use just a simple poll just one question as like a yes or no button so if you've just got to agree or not agree yes or no just with the whole question at the bottom. Oh, the bottom of the course or the bottom of the resource? Of the resource. And in that, yeah. Yeah. That might be quicker. A quicker way to do it. Yeah, but teachers might not tell us though when they get those results back. They might hide them if they're not good. Don't you already have a lot of the data? You've got engagement data. Yep, that's part of the desktop order that we're trying to do to actually do that. I suppose the thing that we're after for more of is how to get the buy-in from the teachers because having like we've changed our culture within eLearning and with online learning from where people used it as a repository to now they are actually using it as they're building courses so they're interactive and the students are getting a lot more out of them. But we're looking for information that's going to change the way that the culture is so it's got to be really driven. So just saying like, well, the students all passed this one and this was the good resources that went into it is probably a good place to start but I don't know how well that'll go in the long run and all pulling them out of every course. How am I going to get all that data across Moodle? Add help reports. Yeah? Yeah. Sorry? We just want to ask people have got it working where we've got feedback tools being centrally. Okay, so every feedback tool feeds back centrally or can you turn that off and on? It's based on using conditional questions in the feedback tool. Okay. So I'm not the one who's subscribed but she's in the other lecture but she's in the other lecture. Okay, where are you from? Sorry? Riverenite. Riverenite. Oh, this is good. I'm developing my methodology through my presentation. Sorry. Yeah, the one that's looking behind in the purple. Yep. Were you going to say something before? Yeah. Why did you choose Gizmo? Why did you choose Gizmo? I was looking at a few other things. There's a couple of ones that look at forums only which is actually an add-on to your browser that works and at CIT we do not use forums. Well, we use them as a question and answer for teachers. It's no community building a community of learning or community of practice within the course at all in forums. The students don't seem interested in that and the teachers probably don't have the skills to moderate that and so anything that was looking at just one resource I knocked out pretty quickly. So Gizmo looked across all which was quite good. So I do have a... I suppose a matter of time, am I? I was going to say I do have a little bit of a demo just to show you what it looks like in the course. So you can actually go into... If you go into the program I do have... I've just put a very small course up there on the Moodle Moot site and you can see here that I've actually got a couple of resources in it and some students have actually already come in and done some things which is very nice of them. Other than Michael, he's very slack and has done nothing. So I'll have to have a talk to him about that. So this actually does print off quite well as well. One of the things we like about the progress bar is when it prints off it actually puts the date as well. One of the things that the auditor's really like is the date that they were successful. And so you can see here that some people have done it. So it's very easy to see at a glance how well or how not well students are doing and you can see here I've got more than one. So I've just put progress bars in there for every block which would be doing if we were a cluster delivery. So I'll just see if the gizmo will get up. Gizmo's interesting too because it doesn't give you a live feed of the information so it does have to wait 24 hours because whenever they do something at 2 o'clock in the morning it actually gets all the gizmo data to come in. Gizmo's not something I'd probably use in the future. So like when we actually did it there was nothing else around that we could find from the other research. It's not coming in very well at the moment is it? And it's a bit slow. Live data would be so much better. We're probably going to be looking at doing a data warehouse so we can get the information from our student management system, our learning management system and actually get some better data out of that. So at the moment we can't do that because our student management system doesn't talk to anybody because we've got banner. And I shouldn't say that like it's an apology should I but yeah. Any other banner users will understand. Okay it doesn't look like it's going to pull that in. Beautiful. Thank you very much. Thank you.