 Thank you. Oh, can you hear me? Yeah. Good. It's working. I'll try and entertain you before lunch as much as possible This is a slightly longer session. So I'll try and make it fun From Monash, I'm Cliff Ashford. I'm from the ICT function. So I'm not an educator but I came into Monash about a year and a half ago and my previous boss used to call me a Troublemaker and I sort of take that as a badgie of honour in sort of things I try and do at the university Let's see if the slides are working They are so Quick slide about Monash. This is the one that we also have to go Monash is quite big But there's a reason I'm putting it up here is because of the little Malaysia dot down here, which is Quite important in some of the things that we're trying to do with this space, but I'll get to that in a little bit Okay, and obviously we have lots of students so Does anyone know what this image is showing? So welding practice I put this slide in because I want people to understand where we're trying to go with what we're doing at Monash So this is a virtual welding simulator Which teaches and assesses welding better than using a welding torch Because you get much better feedback. You can see what's going on It can tell if you're going too fast if you're going too close all these various features the The technology is better than reality and it's something that you get great analytics back from my background is analytics I want to capture as much data about what's going on. This has very rich data I want you to bear that in mind as I go through the rest of the presentation By the way, that's been around for at least 10 years that technology We're still building enormous welding labs at universities that cost us Hundreds of thousands of dollars when we could be doing a lot more of this sort of thing You still do the practical in reality, but yeah, this is good to do too Okay I can't remember Monash administers 360,000 exams every year This is equivalent to 8.5 million sheets of A4 paper or a thousand trees Thousand trees from a plantation takes one acre and In virgin forest, it's even more each year We have thousands of students undertaking paper-based exams. It's the only time we ask students to sit down Silently in a huge room and just hand write for hours. The best thing about the whole platform is probably being able to see the Exam paper on one side and have my exam on the other side in the split screen function and just have everything in front of me as I work I thought that was really good in terms of setting up that that's how I set up my Physical desk for a handwritten exam and it's sort of it was really intuitive Typing something out is a lot more flexible than writing something out in terms of being able to change the order of my paragraphs Or go back and edit something without it giving too messy So I won't go through the whole video that's up there on YouTube if anyone wants to look at it and that was from last year so When I came into university a year and a half ago, I knew nothing about assessment I discovered obviously I went to university 30 some years ago and I remember doing some but not much about it lots of handwriting as I say what We're trying to do here is reimagine what we're doing with handwriting So you saw Chris Ryan in there who's the the head of the learning innovation department at Monash Chris and I have worked very closely on a variety of different things But one of the sort of most challenging things was the assessment program of work We originally sat down and just brainstormed this about little over a year ago about how to solve this The thing about assessment is there's lots of components And I couldn't have given you this list a year and a half ago Now I've been immersed in this for a long time There's huge complexity to what you do, especially when you're talking about the scale that's happening at Monash that 360,000 is about 250,000 take place every year on Prem if you like we take over Coldfield Racecourse, which we'll talk about in a bit and we invigilate all those students Literally thousands of students coming through the first time I saw the paper exam. I Almost sort of gave up the project. It was terrifying There's rooms with a thousand students sitting there scribbling way on sheets of paper and thought what on earth Do I think I'm doing trying to get into this space Obviously each of these stages that we have to address there's the creation the setting up of a new exam Finding questions novel ways of sort of assessing students Staging the exam setting up as a we take on Coldfield Racecourse and we have this mammoth procedure Twice a year primarily with some other sort of sessions outside of that Sitting the exam the advantage of Coldfield Racecourse is it's right by a railway station And we can have those thousands of students pouring through and then finally close to the hearts of an awful lot of our academics is marking improving the marking process That's far more deep and complicated than I ever imagined it could be and I've got a whole presentation just about marking Which I won't talk about today particularly This project has a lot of components to it In terms of what we're looking at here, there were a few sort of key things that we had about this So when I first discussed with the University what happened when we ran these huge paper exams Literally sacks of paper and I went and had a look at this and I couldn't believe this there's bin liners full of papers There are whiteboards with lots of things being checked out and checked back in if you've done marking at scale You'll be very familiar with this Shipping off these papers getting the mark shipping the back making sure we got them all making sure they're all marked There is a lot of error in that process We have one horror story at the University which I usually pull up when people are saying electronic assessment Isn't that dangerous? You end up in the news and all sorts of things go wrong One of our senior academics had two of these bin bags full of papers in the corner of his office Cleaners turned up thought it was recycling recycled them You can't see the bottom of realize the bottom of the screen has moodle moot in front of it Some of the advantages about getting things on to computerized system analytics again my background understanding What's going on with the analysis at fine-grained detail? We got a lot of data out of moodle We've got a lot more as a side effect of this Authenticity that bit at the start where I showed you the the welding simulator That's where we're trying to go but to do that We've got to have a platform to hook all this stuff into in the first place and student experience it's whenever you roll out a system at the university We have a lot of academics talking just about all the things that they want And we're going yeah, but there's a lot more students than academics So we'd quite like to make sure we got the student bit right First we're going to do lots of things but we're going to focus on student feedback and the student experience Sitting an exam is the most stressful thing. It's a horrible experience We're putting them on a new platform We want to try and make it as easy and pleasant and nice and helpful for the students as possible So we're actually assessing what they know rather than how they deal with a stressful situation Okay, so we started this really at the start of last year That very thin line at the left hand side Was our pilot group that we ran in semester one last year That was just 600 students We built this system I suppose it's no secret we built the system on moodle Why did we do that? I'll talk about that in a second, but in terms of the growth we have here 600 students We tried to grow it as fast as possible So our Vice Chancellor Margaret Gardner has made a couple of sort of statements about since you took over Well, lots of statements, but specifically about this at Monash Monash is one university Anything we roll out in terms of systems have to be rolled out enterprise-wide at the university There's no more pockets of this and pockets of that. We run stuff across the whole university Her Deputy Vice Chancellor in charge of education Sue Elliott We went to her and said we can do a couple of things with the system We can add lots and lots and lots of bells and whistles and lots of different ways of assessing just now and focus on that Or we can focus on pushing into as many students as possible first and then do the bells and whistles And she said I'll have more students please as soon as possible. We'll talk about the other stuff afterwards So we went on this rapid growth curve that you might be able to see so Semester one last year. We did 600 students semester one this year. We just completed last week. I think it was we did 37,000 students 37,000 students here in Australia And In the Malaysia campus combined with synchronized assessment taking place in both areas That's why the map was up earlier to tell you about that We we have another mandate at the university, which is equity We want to deliver exactly the same educational experience in Malaysia as we deliver in Australia And one of the things to prove this system would work in both those places flawlessly and have it synchronized We expect to do 40,000 seats in semester two this year That will give us about a third of our seats for the sort of on campus process that we run through Next year we want to double that we want to hit to about sort of 70 percent There's a lot of technical challenges in doing that But we'll get to those later But we are doing extremely ambitious growth curve because we want to stay ahead of Stay ahead of the wolves if you like people who try and drag us down One of the things about this project and what's worked so far is it isn't an it project It's not one coming from exams alone. It's not one coming just purely based on pedagogy From the very first stages Chris. So you saw earlier on the video Um Vern over there who runs exam services and runs these enormous exams and is thankfully the calmest man you've ever met in the world And me at the top we have this combination of the three of us working on this project together Any of the three of us can veto any decision at any point if we think we're pushing things too hard Raising the risk too much we can veto what's happening So we're trying to balance that rapid movement forward with good technology a good solution without having too much risk for the university That said we have a risk register of I imagine something at 100 different things each of those things has three or four at least mitigations on them If this thing happens then we fall back to this we fall back to this we fall back to this we fall back to this The last thing we want is to be on the cover of the newspapers on the tv for all the wrong reasons That happens around electronic assessment. We will not let that happen at monash For instance this last semester we still printed all the exams. We had them there if anything went wrong We've taken the paper copy roll it out to all the students in that and they'd run a paper exam instead We want to fall back. We don't want students flooding out of their interiors and having to redo their exams Go on go. Yes It's a much bigger team than Vern Chris and I of course This is the call field race course where you just finished the exams. I think that's the final day That's the team who were on site at the time and we did this big photograph of them all There's a lot of people involved in this project from an it point of view. It cuts across every discipline of it Okay, there's a timeline. I was talking about originally like everybody else. We looked at different systems off the market back in 2015 Those systems weren't providing what we needed Because one of the core things about a university or any educational establishment isn't so much the teaching It's the assessment. What we're what we're licensed to do is assess people It's good if we do the education that's tied into the assessment process and it should be tightly tied But what we're trying to do is assess We decided that a core function of the university as fundamental the future of Monash is possible We had to have complete control of Move forward to 2017 when I started at university We did this market scan. We looked at all different tools Chris and I sat down in a room and said how about we do this a different way We already have Moodle. We've had Moodle for six years. Why don't we look at using Moodle to be the basis of what we're going to do And I think is Andrew here somewhere We met in a coffee shop about the same time and said we've got this really weird idea. Do you think it's a good idea? And you went yeah, it'll be fine. No problem at all So with that level of confidence, we thought we'd go for it Okay 2018 We hammered into this so we hired a team. I imagine Darren is somewhere in the room as well Ah, there's Darren So Darren was the core of our team The good sort of he possibly convinced of it was a good idea as well because he'd previously been at the college of surgeons And had done a similar thing with the college of surgeons And we might think things are stressful at university, but the college of surgeons Significantly more stressful. So Having a couple of clear voices saying yeah, this is doable Yeah, we thought let's go for it Okay This is one of Chris Ryan's slides. Um, so I'll let you read a lot of it Pretty much it's saying that we want The unit cost of what we do when we run paper exams is very low Because we do so many students at the same time. We've ground it down over the years to be as efficient as possible What we want to do is do the same thing with electronic exams So we need to get scale as fast as possible because we get savings a side effect of scale Scales difficult with solutions like this But this is how we save the money Currently we have a fleet of people or a horrid of people that we pull in to read students writing And the writing is often A stream of consciousness that jumps up and down because it's oh, I forgot to mention that I'll put this I've got an arrow to go here. Oh, I need to write this a different way It's a hideous thing to mark if you tried to OCR that the the machine would go insane Just moving from the written to having a sensibly formatted long four months are ignoring all the multiple choice Ignoring all the auto marking stuff just that aspect of our exams If we hit 80 percent of our exams Compared to the statistics out there. It's twice as fast to mark a well typed exam At monash because the scale that saves us seven million bucks a year That pretty much pays for the project in one hit Getting that money back out of the faculties and into central front is another problem But we'll just deal with that one separately But it was a good it was a good argument to have with the vice chancellor She liked that number. She went with it We also my background is it obviously we want to run. This is an agile project The whole concept of those monolithic pieces of work that you do from start to finish and then roll out and everybody goes That isn't what I asked for that's dead Thankfully, so we worked through this project. We were trying to be agile We were trying to be best practice We were trying to make sure we addressed all the correct things that we had decent user interface and user experience That we made the system accessible for all the students. We have the 75,000 students Every variation on the theme you can possibly imagine with those numbers and the system has to support every single student We also wanted analytics at the far end of this because you want to understand what we're doing Most of our assessment process just now you can't see the detail of what's happening There's our architecture aws underneath Moodle on top catalyst on top lot making sure that whole process works For the first year we bought laptops The university doesn't have a bring your own device policy across the whole university So we couldn't just say bring your laptop to the exam and do this We needed to buy laptops in the first year. We bought 1500 laptops. You can do the arithmetic. That's a large number Sitting on the laptops we rolled out Sort of throughout last year octa, which is two factor authentication The idea of that is to try and make the university as secure as possible We want to ensure that the people logging into our network are the right people So we have octa in place and we have that's the symbol of safe exam browser I don't know if you've used it. It's a lockdown browser which takes control of your machine You can't do anything unless we configured it to happen. So we can whitelist websites We can whitelist applications, but by default it locks everything down That combination Is what we consider to be your assessment at the moment at the university Um, this is Is it clock counting? Yes, it is. This is actually a video. This is what it looks like So we've had various experience with Moodle over the years academics Some areas love it some hate anything that's in front of them What we want to do is just do some work which separated this from the the main Moodle So we have an enterprise Moodle, which may or may not be the biggest one in Australia If you went to an early presentation We have this one entirely separately hosted We've taken a lot of the features out what this is doing is running pretty much quiz We changed the look and feel of it At the time we were working on this we were still at version 3.1 I think we're at version 3.5 at the university on the enterprise Moodle Some of the things that we've done in this actually exist in the sort of core stream now So this should be familiar, but there are various features in here you can see You might have seen that one turn orange up that that's for the students who say I'm unsure about this question. I want to come back to it later Lots of things you've seen before, we can do various formats We've got six question types We know there's lots of question types in Moodle We don't want to do them all We wanted to keep it to six and make sure we did those really well And worked at the whole experience for the students So those types of questions are elegantly delivered I should be able to list them all But we've also got MCQ, we've got short form, we've got long form We've got sort of fill in the blanks We've got a drag and drop feature and possibly another one You can ask Darren later, he'll know these things But we tried to wrap a lot more around this to make sure it was a complete solution So you probably see at the top that little colored thing That's accessibility for color blindness So you click on that, it will give you different color sets So that you can discriminate what you're seeing on the screen We've got things up here like the highlighting mode Our first collaborators with the law faculty And what they do in law is they have a huge case study Is they want to be able to highlight bits of text Move them into the answer and then address what they're doing So essentially we replicated what they're currently doing With a large pile of highlighters when they're doing their exam Let me have a look at this We have a notes function which drops in So that another thing that came from the law faculty Is they're sort of essentially trained to take notes as much as possible Do all your notes and then turn it into your answer So that features there, it can collapse or expand as required It's everywhere just now, but it's primarily I think For use if you're doing the sort of long form answers of which they have a lot Left hand side, you'll recognize that navigation But we sort of pimped it up a bit from what was already in Moodle quiz So you can see the very situation is the one I've done The things that I'm unsure of We have some multiple sort of nested question types So you see that 15ABC in there We have the marks up there We have Pro, we got sort of strange feedbacks from academics Saying that you shouldn't be able to see the whole exam Because students will then game the exam And only answer the questions that they know Which I thought was a bit bizarre in terms of feedback Like again, we stopped listening to some of those academics And said what do the students want in terms of the experience As you can see it's quite a clean interface Now a lot of this stuff is taken away So it doesn't detract from what you're looking at We have the clock counting down at the top What I cut off as it started is the front page that you sit at And the exam comes live just before you're about to kick off Here is the long form answers So we put in a rich text entry box with various features It used to have icons all the way over here And the students said can you take away most of them Because they're confusing Can you just keep the ones that we want The other thing that was in there was this Which is an undo and a redo feature Which is really simple and something like word It's not simple when you get into something like this That was actually quite hard to implement And we had some interesting bugs associated with that But from that point of view we knew that the system Had to behave as much like a word experience as possible We don't want the students having to learn new behaviors Just when they do exams That's what we're trying to get away from So we put a lot of effort into this This is the widescreen view that we built for the students As well because when they're doing a case study They want a lot of room So this, you can have the case study over here And we have some very extensive ones With a notes area that will collapse down there And then they get this large editing point You saw in the video earlier they said This has had to lay out a desk during the exam So it felt familiar for them I'm probably going to skip You can see this, like I say, we'll release this whole Presentation so you can watch these videos to the end They are quite large What it eventually does at the end of this Is takes you to the summary page So the students can go in there And they can look through all the questions they've answered And then dive back to the ones that they want to look at Oh, word count We put that in because we didn't want a hard word limit In there because we don't want a student having to Reduce five words So they have exactly 3,000 words in their essay Okay A couple of words about the implementation That's our Caulfield campus That's Caulfield Racecourse Where we run the exams We run it because it's so big We had to take our network From there and stick it in there Never ever say to network engineers I don't care about the price Just triple gold plate it Because the original quote They gave me It was five times that when we installed it But we put multiple dedicated connections Between the campus and Caulfield Racecourse We put a complete new network in Caulfield Racecourse And the densest wi-fi implementation I believe in Australia And much broader than that potentially It took us a long time to design it Some of the stats When we built that We used 56 kilometers of network cable We put in video cameras We haven't done that before But we had several million dollars worth of laptop sitting on desk So we thought we should probably have some security We put those fiber optic links in $80,000 worth of patch leads I asked the guys to calculate the other day Because we went through and clipped off All these cable ties that we cabled up all those desks 850 meters just of cable ties And 1,500 laptops in 40 charging trolleys And again you can do the arithmetic for how much this cost Luckily we'll be saving money out of this Okay Every day the students were asked to do a survey At the end of the exam And we collated that as we went through This was the final one on the final day Very positive across the board from the students The interesting one is this question over here That said essentially e-exams are less stressful than paper exams Day one that was the other way round And throughout the two and a half weeks of the exams That slowly flipped until the students prefer the electronic to paper Okay We ran various focus groups And we had lots of feedback So we are trying to engage across the whole university as much as possible There's a very holistic piece of work Lots of people involved I'll run through some of these Because you might want to ask questions How are we doing for time Yeah we're pretty good Okay This was one of the ones that was our most prized feedback Because we had a senior academic It was very cynical about this And then chose to run some of his exams on it And then we got this feedback from him If you can read that My goodness this is absolutely wonderful Where has this been all my life We put this up every time Because it was a huge relief that we got this feedback We also have Some other interesting things we did So we got the students in If you've been to Monash campus That's our big learning in the round space In the learning and teaching building We just pulled students in and we said Tell us what's going on about this What do you think about the environment Lots of good feedback Students generally have been going Why do I have to do this writing thing For such a long time It's completely irrational that you asked me to do this You teach me through the whole process To use electronic techniques And then you make me write on a sheet of paper for three hours Okay, where next Harking back to that first slide This isn't about getting rid of paper What it was about was building a platform We chose Moodle as the platform Because Moodle is open source And extensible and solid and reliable And all our academics were familiar with it We found some good partners to work with us And some great developers to use that as our basis But we are taking it somewhere else Couple of things There's an awful lot of assessment at the university That needs some sort of free forum input It's not handwriting as such We want to get away from handwriting Because it's hard to read But what we do need to be able to do Is draw pictures and annotate things Also in things like pure maths We need to be able to do a mathematical formula And do it quickly You don't want someone to have to be Fighting with the experience to do that The other thing over here That's sort of a slightly ominous picture of a laptop That does two things One, we're going to bring your own device next year So we have a new policy at the university We're at limit to how big we can get I'm not buying another 1,500 laptops Could it get insane So what we're going to do is move to Primarily bring your own device at scale next year So to hit that 70% That'll be BYOD at the university We'll still have our 1,500 laptops Because we have to be able to swap laptops out For the students So if their laptop fails We bang another one in really fast So we need that process to work We also have a virtual delivery Of application program Which is one of the other things I've delivered this year And we want to be able to put the actual tools The students use into the application So rather than just a rich text box at the far side You would have Excel The entire faculty of accountancy said If you can do that We'll get everything on your system like that For IT We want to be able to put a full development IDE Into that So you code, you compile Everything takes place within the exam So it's authentic A couple of years time We want to be doing assessment of people's behavior We want to roll augmented and virtual reality into that So that's where we're going We're not about We're about Once we built the platform It's about what we do with it after the fact So this is running at speed And running quite ambitiously Okay Key takeaways There's lots of stats for you That you can take away from this Lots of those numbers Again, you get these slides afterwards But our focus was on students We're adding lots more functionality this afterwards The whole marketing experience Is like a project in its own right And then finally On the very last day We flew a drone down Caulfield Racecourse Just to have a look at the scale of it And it still horrifies me to some extent What we did We have another slide of the breakdown With time lapse But I didn't manage to get that for you in time But it is a large venue So that's our journey so far And we're hopefully going to go to in the future Any questions Presentation I did have a couple of questions Is there automatic saving in place Or are the students required to save As they're working through their assessment questions Good question One of the things When you originally set something like this That you can go two ways You can go with a client That has the entire assessment downloaded onto it And it's essentially local So that if anything happens to network It will still run We decided it's 2019 And networks are pretty good So we think we can rely on the network So from that We need to take advantage of the fact we have a network Every 30 seconds Whether the student does it or not So we're forever storing this back to the cloud Chink, chink, chink, chink, chink We've had situations where a student has Accidentally done the select all From a 3,000 word Sort of response that they've had in the long form And then press space And they've suddenly lost 3,000 words and panic We can catch them packing And we can go there And we can roll it straight back in for them And they have everything back in a few moments The other aspect of this Is we can see how they actually Write their assessment We can see how it builds up How the thinking process goes Because we have 30 second timestamps Of everything they've done through that whole period So the analytics about how people respond To long form answers We have sitting on the servers We haven't done that thing with it yet Because we've been so busy But we think there's educational advantages to this To see how people form an argument Does that cover that one off? Any more questions? Very good, thanks Cliff Students are bringing their own device How do you plan on locking down those devices And deploying per faculty or per assessment A machine, like you said, Excel or something So the lockdown will still be safe exam browser So one of our faculty is faculty of pharmacy They do a lot of in semester assessment As well as end of semester They volunteered Because they already had a de facto BYD policy To roll this out last year They were running slightly faster than we were They said the system works perfectly We'll just run it in the faculty rooms And we'll use that We will force the students To put a safe exam browser onto their own laptops And we will go through the lockdown process At the start of every one of these assessments Pharmacies quite bullish about these things They tell the students to bring laptops And they tell them how they're going to use them That is not the case of the whole university But they were a good place to start from Second part was Okay So currently we use We've got a program called Move Which uses Citrix to deliver All those sort of high powered apps to the students So they don't have to go to PC labs Because we think the PC labs are As they stand a waste of good flat floor space At the university We want to have fewer of them But the fewer labs have more sort of High tech equipment in them And larger screens and things like that So we get rid of those generic labs If you see the learning and teaching building To the new buildings at Monash The students sit everywhere Through these new sort of communal spaces With their laptops studying We want more of that And less of that formalized situation Did that cover that one off? Cool Anybody else? Good evening, Garon I'm really interested in their learning analytics Especially about learning styles Have you set into place the ability To judge the different type of question types? So kinesthetic learner might really enjoy A drag and drop question Have you put into place analytics To determine that as opposed to the paper based And the effectiveness of that teaching? So that's another really good one Because it's On the paper based side we have some analytics The whole trouble this is baseline The time to see how well things are going We expect to see some changes And we have seen some changes in assessment Since these things were done on paper This is an ongoing piece of work Because it's quite The devil's in the detail With this sort of analytics Initial position was to capture As much as possible And then see what we could do And understand where it's going We don't want to We want this to enhance The learning experience We want to roll this into semester We don't want to do end of semester exam We see the platform as a way of saying It's easy to run an invigilated exam It's no longer with paper You can do it in semester like pharmacy does Then with the other tools you already have In Moodle and some of the stuff That obviously open university put in As you're saying earlier That feature you do your multiple choice You get multiple hits at it Or you do your long form answer And you get the feedback And you've got all of that available to you So you can see you had to do things Better in the future That's I'm assured far better Educationally That's where we need to get to This stressful falling off of cliff At the end of semester is not the way We should be treating our students to my mind Any other question? Hi, have you started looking into Doing maybe advanced mathematics or physics And those sort of complicated formula On line assessments We have So we're trying to skin these Counts several different ways That's sort of the free hand approach And we know some things just need free hand You just need to build the right formula We do have various tools That will let you use an iPad Or a similar thing to write the formula in That's sort of working But it doesn't work very well for us at scale Again, so we have to We're actually attacking this problem In a variety of different ways There's nothing off the shelf You can buy that makes this work What we're doing for 2020 I suppose it is This year already we do some hybrid exams So if you've got some formula you've got to do Or you have a diagram you've got to do For example, sheets are still delivered Out to the students So they do most of it electronically That sheet is there That gets gathered up It gets scanned And then it gets given back to the academic for marking Next year we're moving to industrial scanners So it's not so manual We'll have essentially printed sheets With QR codes and various other things on there Those will be collected up And just dropped into the hopper on the scanners Darren's going to kill me about this in a bit We will script on the back end of that Gather up those scanned sheets And tie them straight back to the original Moodle quids So it's available in the same place for marking I am convinced it'll all be some sort of hybrid on this It's not like there is no single solution to any of this Some of the sort of drawing aspects That if you look in IT They're really pulling together flowcharts Or they're pulling together mind maps and things That stuff can be delivered via a graphics tool Really quite well It doesn't need to be freehand In fact it's nicer to do that I even hate drawing flowcharts by hand I'd rather do it in a tool So we're essentially trialling lots and lots Lots of different technologies To look at these really gnarly complicated things And not just saying We'll try one site's fits all for university That would be madness Anybody else? I've got a question in regards to You're bringing your own device policy How do you relate that with Or how do you associate that with your Wi-Fi policy? I appreciate that the assessment component You provide that But so I'm just wondering beyond the assessment Your Wi-Fi associated policies with it Do they provide their own? This is something we're looking at in our workspace Okay So Wi-Fi is obviously ubiquitous all over Monash What we had to do around assessment Is we couldn't have the two-factor authentication component Which requires you to have your phone When they were doing assessment So in that situation We create a separate Wi-Fi zone Around where it was taking place And we disable some of the two-factor authentication So you just go into your password The students are already connecting All the time to this And that process of coming in Through the Wi-Fi and having to go through our authentication area Locks the machine at that point We know which machine is being used We know where it's being used physically So it's been a sort of ongoing process With our Wi-Fi teams to work at How best to segment the spaces So we do it at Caulfield Racecourse We don't necessarily do it at the in-semester stuff Because you can see all the students In the room together using their laptop That said, we did have one student Trying to cheat with their mobile phone On our Wi-Fi In this last round of exams Where she would slide her phone out Take a photograph of the screen And then someone would text the results And she'd type the results in When we caught her, she denied doing it And we went, you see those six cameras right here We have you from every angle doing this And we found a lot of cheating from that point of view About as much as we get on paper Same sort of amount But we have clear evidence of it And it's one of those things We want to discourage people to even attempt to cheat Because we tend to catch it anyway But bringing that equipment around sort of dels it down Does that cover it? Or do you want more information about it? The only one Yeah Yeah Yeah So Monash has one giant Wi-Fi They have free Wi-Fi and logged on Wi-Fi Edge your own It's actually the same Wi-Fi We don't really separate it We spread the rumor that the free Wi-Fi is slower That the other one it's not really It's exactly the same We have time for one more very quick question Over here And then it'll be time for our lunch break Hold on for one more question Who writes the questions? And do you have any special custom tools to write the questions Or are there any templates for us Different types of questions Okay That comes to the question bank function So the academics are writing the questions As you know when you have quiz The sort of question bank is tied to the quiz What we've done is we torn the question bank away from quiz So question bank is freestanding That was a significant piece of development But we needed that so that we could We could have a bank of questions that can be used And reused over the years And refined and modified And we'd have all the analytics about the use of those questions Stored in the question bank And from that question bank we can then construct new exams We can produce randomized ones If we don't need to have an invigilated environment And also Monash College Obviously is a part of Monash And they do an awful lot of exams That are essentially built digitally But then delivered on paper Because the network conditions in various countries around the world Aren't good enough to do this So we can extract any given test as a PDF So we can pull the basic exam Or we can pull the answered exam out as a PDF Because we obviously have to store the exams In perpetuity and to our records management system as well So the academics do it They can either do it straight into the system And yes, following templates in the question bank Or a lot of our academics sort of scroll it out somewhere And give it to admin staff And say you go and put that in the system So we support both Thank you very much for the thoughtful questions And for attending this session And thank you very much for an incredible presentation Please join me in a round of applause