 He's got a lot of fun telling that, and they're like, it's a good day for me to go to school today. So, you know, you're supposed to make it a year, so I think it's about a year. I think it's about a year. It's about a year. It's about a year. I'm doing it for you. Oh, I don't have a passion on all of that. But it depends on time. It's the only thing that's going to last for the year. Sometimes it's the other side. But if it's not... Yes. OK. LAUGHTER LAUGHTER I don't have so many reasons. Yes, I know, but if we just check these words now, we'll be starting it from the top. There's some sits down there at the front and there's some sits up there I'd like to move down a little bit. LAUGHTER LAUGHTER LAUGHTER LAUGHTER Good afternoon, everyone. We're going to be starting in a couple of minutes. People who are arriving now, there are a couple of chairs down there, I think a couple at the front. Otherwise, you might have to stand at the back. We're very full, very popular yet again. And just a reminder, if you can make sure all your mobile phones are at least on silent, so we don't hear lots of beeps of notifications like yesterday afternoon. And can you be careful with your drinks? I didn't notice it before, I was really not allowed to have glasses or any kinds of drinks because one was broken and I don't want to be responsible for someone's injury because they didn't see it. Can you tell I used to be a teacher? LAUGHTER I'm trying very hard to control my frustrations. LAUGHTER Find any more seats? I think there's one there. There's some room in the middle. Do we have a meaner from the University of Minnesota? Oh, that's good, OK. Do we have Anna from the Swiss Federal Institute? Oh, that's you? Yeah, excellent. So all our speakers are here. So I think we're going to make a start. And we've got presentations all lasting 15 minutes and then five minutes for questions. And so for our first presentation, I'd like to introduce you to Tim and Sharon from the UK Open University, and they're going to be talking about looking after the Open University's Moodle site. Thank you. Good afternoon, everyone. I'm Sharon, this is Tim. And we're just going to talk to you a little bit about what we do to look after our Moodle-based learning systems. So Tim introduced me to the Free as a Puppy metaphor a little while ago. A bit louder. So, lean over. Tim introduced me to the Free as a Puppy metaphor a little while ago. So when Tim gives me my new puppy for Christmas, I take it home, install it in a comfy bed, give it some food and leave it. It's not going to last very long. It won't be a very usable pet. It will soon catch some bugs and it may well fall over. We rely on Moodle as the base platform for our learning systems. Our license is free. Our investments in people and technologies need a lot of management. We've been working with Moodle for a long time and our processes and working are always evolving. So we're going to share with you a little bit of information about the people, systems and tools that work for us. So where do we start managing Moodle? I work in the online student experience team who alongside the learning systems team are responsible for our product management. Tim works in the IT development team. We have very distinct job roles and responsibilities, but Moodle admins, like Tim and myself, span across all of those teams. We have numerous hats from research to development from design to support. We work really closely to make sure we share our knowledge and don't have any single points of failure. And our Moodle admins always partner up when we're making live system changes, basically an extra pair of eyes. Most of our administration does sit within IT. So we do work closely together every day. We have daily stands-ups. Our primary delivery is our online teaching and learning, where we have qualification websites, our individual courses, tutor-only spaces, various other community and content websites. We've got a really flexible curriculum, and it's important that all our websites are consistent, they're dynamic, they're familiar to our users, and that they are easy to use. We develop and enhance all our systems using our user-centered design process. At every stage, this incorporates gathering user feedback, usability testing, and accessibility testing. We use this for all of our projects, so whether it's a large project like developing a new theme, releasing a new plugin like Student Quiz, or even quite a small feature. We keep on everything new as it's released, and this is vital that we can track what's going on, so we can see where things are being used and how many people are using them. The ad hoc database queries plugin is something we use a lot. I've got several automated daily and weekly reports that I now rely on in order to do my job. There is more to looking after our systems than just the administration. We've got several instances of our learning platform from community and staff-only servers, our VLE for Students in Secure Environments, and the Open Science Lab. Learn2 is our core teaching and learning server. We have 55 roles on that server alone, which we use to manage who goes where and does what for about 180,000 active users in a year on over 450 live courses. 55 roles does sound a lot, but I'll go on to how we use those. We have roles assigned at system, category, course, and activity level. Staff in the system roles range from our managers looking at website configuration and settings through to specific roles, for example, for managing our assessment and library content. We have non-editing roles for our help desk and our support staff, and within the course areas, we've got a wide variety of staff with editing access so they can look after all the content and update it. Most of our users are in multiple roles at multiple levels within the system, making sure they have the right permissions in the right context. For example, our tutors have an automated system-wide role that gives them read-only access to all of our courses and their content. They will also have an automated role on specific course websites and automated group allocations. This gives them their full access to the collaborative tools and moderator features. Most of our manual role assignments are looking at administration at the system level and website editing at the course level. This number of roles means we can really tailor the permissions to design the perfect learning activities that we need. I do love roles, capabilities and permissions, and I may annoy our developers at times where every time we develop a new feature, I do ask, is there a capability for that? So we review all of these roles and permissions as part of our quarterly release cycle. This is now semi-automated, so we exchange emails and we've got a shared changelog. We used to use lots of Excel spreadsheets playing spot the difference across all our servers, so things are getting better. We use the Edit Roles by Capability plugin. This allows us to update a single capability for all roles at one time. This means we can quickly and accurately keep our roles and permissions up to date, especially when we're making changes on the live server. So moving quickly onto our website. Learning systems have grown around Moodle into quite an ecosystem. We've got a lot of plugins, some are Moodle, some are our own. We've got a number of integrations with internal systems and also third-party systems, and we have real-time data integration 24 hours a day. We've got a number of key tools and processes that help us manage this at scale. Our students have their course website for an additional three years after the course finishes. It gets put into read-only mode. So for each new cohort of students for each course, we need a new course website. So we've developed a one-step roll-forward process that automatically backs up the website, restores it, but it removes the users and their content. It keeps the groups, renames them, and attaches them to the new version of the activity for us. The roll-forward also adds a new workflow to the website, and this is where I'm going to hand over to Tim. Thank you, Sharon. So with all these roles and different people involved in producing a website, we need some way to coordinate that without being too heavy-handed. So we developed the workflow block, and you can see it there sitting at the side of the course, making it clear which user is responsible for doing something to this website next and reminding them what they should be doing at that point. All the steps in that workflow are freely configurable by the administrators, so really the teams working in a particular area will define their own workflow. So you have a bunch of steps, each one leading to the next. And as well as the instructions that you can see, the workflow system is also capable of performing some automatic actions. For example, when one step finishes, that might be the moment when the site is automatically made visible to students. And also, the workflow system will send people an email to tell them it's their time to do the next task, and those emails are all fully configurable as well. And for the admin staff who are not helping to set up, for example, all the websites in a particular faculty, then we have a report so they can see at a glance all the websites they're responsible for and where the workflow has got to. So the general trend of this talk has been to start from the kind of people end and gradually head in a more and more technical end. So let's now start to look at how we enhance the system with development and customizing Moodle. Actually, there was an even better description that I'm about to get of how agile... Sorry, I've forgotten your name. This morning in the global room. But if you're not familiar with modern software development practices, you might be surprised to hear it's all basically powered by post-it notes. As Sharon said, they come up and we discuss the requirements, the things we want to change, and if we have very large changes, we try and break them down into a lot of smaller pieces, and then each separate thing we want to change becomes a post-it note on our board and goes through the process where we discuss it, we make sure we clear on what we're going to do, we do the development, we test it, and then it's ready for release. And we don't just do it with sticky notes on the board, we also have a bug database. And the way we work, we develop a whole bunch of features in this way, but because Moodle needs to be taken offline to upgrade, we do only a few major releases each year. And at our university, we have the problem that we are always teaching. There are always some modules in exam periods when there are assessment deadlines, just the amount of expected traffic we expect on the system on any days. And eventually, if you stare at that hard enough, we find that we can do a release every three months. So actually, we're doing one in a couple of weeks' time at the start of December. And in the lead-up to the release, we sort of pause our agile development cycle and go into a sort of five-week testing period where we take all the new developments, we use them to upgrade a copy of our live site, and we test them, and during that time, we only do bug fixing. And at the end of that period, we should be confident that when we do the release, we can release it. And although I say there's downtime, most of our releases only take under an hour of the system being offline, because it's fully automated. And, of course, because we're releasing every three months, and Moodle puts out a new version every six months, every other of our releases has to include a new Moodle version. And this is where Moodle has really fantastic automated testing tools, and these are basically what save us. This graph showed what happened to our automated tests after we merged Moodle 3.6 earlier this year. And basically, it looks like huge amounts of stuff broke. Actually, the stuff that broke is 70 tests out of about 1,900, so that's not bad. And because we know specifically these tests have broken, this tells us immediately that these are the places we're going to have to go and fix. And the x-axis on this graph is days, so you can see that after a couple of days we had fixed almost all the issues caused by the code merge. The automated tests are also really helpful for managing, just as you're doing development, as you try and improve one thing, your changes might inadvertently break something else that you didn't think of, and the automated tests will catch that more often than not. So if you're doing Moodle development and not running the automated tests, you're missing a trick. So heading in the ever more techy direction, let's look at the servers on which we host this thing, we didn't start with this many, but over the years our usage has grown so we've needed more and more servers. These are all virtual machines, we have lots of web servers, the kind of bottleneck in Moodle, but actually it's not much of a limit for us these days, is you can only have really a single database we use to postgres, and we have master slave or applications, one database falls over to the other. You also need a cache system and you need a big file store pool. But that's it really. It's no good just having those servers sitting there, you need to know at every moment that they're working reliably. So it's important to have various sorts of monitoring, and we have monitoring at several levels. At the top you have some very low level monitoring using a tool called Nagios, that's basically the CPU load on the servers all day every day, that's showing the last week's data when I did the screen grab. And you can see that during the day when our students are studying, at the bottom we have some measurements of the performance from more the end user point of view, the graphs, they're basically average page load times, and mostly it's fine, but you can see there was an incident at that point, there are a whole bunch of pages being really slow to load, that's what the red happens to mean on that graph, and then you have to go and investigate. So we have this automated monitoring, it will alert to us within a few minutes if something starts to go wrong, but often our users are even quicker than that, they will notice, and it's nice to have users that care, really it is. So again here's these graphs, so the average page load time we manage is just under half a second, which I think we're quite pleased with. That's when it's working smoothly, when it stops working smoothly we have a major incident process within the whole IT department, so responsible for all systems so when something starts to go wrong it's her job to get all the right people in the room and get them just focused on fixing the problem and that works quite well. Actually Gabrielle is lovely, but when you see her walking towards your desk it's like the angel of doom, it's like oh no, what does she want from me now? But she is lovely and does a very good job. And as well as the system level monitoring, this is another place we use the ad hoc database queries, we've got various daily or weekly ad hoc queries that just analyse things in the logs to check that things are running smoothly. So to summarise, if you look after your Moodle well it will be a happy, thriving, banding along pet. And basically sort of fundamentally it's simple, it's about having the right people, the right approaches and the right tools. Now, I hope we haven't just completely scared you with this because we've been doing this for I think 14 years and all these processes, when we started it wasn't anything like this sophisticated. You know, you're doing stuff it's basically working, but then something that goes wrong and then you sort of sit down and think about it and then you improve your processes so it doesn't happen again. So really you get to this level of sophistication just by one step at a time making things a little bit better and really I think anyone could do it and the other component is tools there are three of the tools that we have created and shared with the community so if you think they would help you, you can go and get them. Thank you very much. Thank you very much Tim and Sharon. We've some time for a few questions if you'd like to put your hands up please can you make sure you get the microphone before you answer so that our interpreters can interpret? First of all thank you very much for your presentation and for the plugins that you share on the database as well which we've made extensive use of over the years so thank you for sharing those with the community. I just wanted to ask how in terms of what you've shown us today the processes around this from a staff point of view and a procedural point of view how much of that has developed over time in the 14 years that you've been using Moodle and how much of it was kind of in place already or existed previously? I'd say it probably changes slightly every year the three main changes in our processes are where we went from Moodle version 1 to Moodle version 2 to Moodle version 3 we learnt a lot in all of that and in each of those changes we actually changed servers between Moodle 1G's we actually changed servers between Moodle 1 and 2 so completely separate installations it was a really good opportunity to look at what people were doing I don't think I've done 14 years maybe 13 but over when we initially started there wasn't a lot of interactivity online it was some course materials and things so as we develop more tools as the students want to do more as the pedagogy changes the learning designers want to do things in a different way as well as updating the systems for the features and functionality that may come with new roles and permissions it may come the workflows are brilliant because you can just create new ones and drop them on websites and it will email people and it will send editing access to the people in the right roles at the right time so it's definitely a fluid process I think we're at a point now where it's working but we'll just wait for the next change I think The question the checklist I understood is for doing things on the course level now I've been asked by lecturers like we do want to do exam and have a checklist that everything has to go with so could that be everything has to go with so could that be done or abused to do some things also for a process based thank you that is a very good question because I should have said that during the talk and I forgot the workflow block doesn't just apply to courses in Moodle we also use it for producing assessed quizzes we don't bother for formative quizzes but for assessed quizzes we also definitely use it so that works for I think any activity type I mean we just use it for courses and quizzes thank you for the question we have time for one quick question so any other question thank you Tim thank you I'm going to be in the middle because it's hard to see me let's stay there in the middle wait my hand fine, I'm one minute let's go is that a mix of right everybody it's just about time for our second 15 minute presentation from Nina from the University of Minster and this is user life cycle management so thanks for the introduction I was wondering how many of you do delete their users automatically okay so that's actually part of what this presentation is about because we thought with our instance about how we are managing our users and what different stages they can have and well that's why the presentation is called user life cycle because we have kind of in the middle this cycle of first you create a user then you have active users which might be suspended but also might be reactivated and since I'm from the University the case for example if someone takes a semester abroad then they are not longer ascribed in our University and they are inactive but they might get active again and finally if the student or the teacher is no longer at the University they should get deleted so that was our use case and of course Moodle also already has functions to suspend or delete users okay that's actually if you try to use the Powerpoint online editor it does not work that well at the top you can see the normal user table which has those two icons of deleting users and suspending users the garbage bin is the deletion and the eye is the suspension part and from the Moodle documentation for those of you who do not know what happens when a user is suspended they are not longer able to access their courses but their data is preserved so if you reactivate them they can again access their courses and well the goal of the deletion is to remove all the information of the user so that's what's there in Moodle but our motivation to do something about that was that we have approximately 100,000 users by now but 50,000 of them could be deleted because they're not longer at universities so it does not look really nice and since we do not want to do that manually over and over again we thought about some way to make that automatically and as part of that we also thought we might adjust the suspension and deletion part so what additional features might be beneficial for us and out of all of that thoughts we have course developed and out of all of that thoughts we have course developed something oh no first sorry first we have the how we want to adjust the suspension part which the first thing we thought about was anonymizing the user which does not work completely so it's not the goal of this is not that it's completely that it's not possible to backtrack the user but that you get an idea of okay that user might be not longer active especially if you think about a teacher who has users in their course he might want to know okay that user is somehow not active at the moment if there's an assignment or something else so what we want to have this is the simple example of having the list of all users from a course and what we wanted to have is that there's not the name but something like inactive user and that that also is displayed for forum posts or assignments or something like that and for the deletion part we want to remove really all information of the user which is currently not really the case and also delete the data from the user table so he's completely gone and yeah if you're thinking about how plugins might delete data from user there's a pre-user delete function which you can implement in each plugin and then it can take care of users being deleted so now we developed something which you can find which is a public repository and it's admin tool but there's one problem about the whole life cycle of the users because each instance has of course custom requirements when to delete a user and when to suspend the user so some might want to delete them if they did not access their instance for 60 days and some might want to delete them based on some external information that they're getting in so we have a lot of different requirements and therefore we split our implementation in two parts and the first one is the admin tool which basically takes care of how our users deleted so what happens if they're suspended what happens if they're deleted which implements a cron job so that's all of how we're doing that and there is a sub plugin type which we introduce which is user status who where you can implement your own sub plugin with your custom rules which gives back the users which should be deleted or should be suspended so the admin tool has a cron job which by default once every night and which changes the status of the user based on the sub plugin which is activated and you can choose from the sub plugin which you have installed for the anonymization part how we implemented that is that we have a shadow table of the user table so in case a user gets suspended his original data from the user table is copied to that shadow table and the data in the user table is replaced so for example first name is not longer Nina but it is inactive user so when you have a course it is referenced to the original user table and you no longer see my name but the inactive username and the sub plugin is based on an interface which has to implement four functions where they have a function to return the users which should be suspended, which should be deleted, which should be reactivated and users who never looked in for the sub plugin we have implemented two different types which we could think of but later it would be really interesting to know what requirements you would have so how you would know which users should be reactivated the first plugin which we thought of which is the simple cases users who did not access a long time should maybe be deleted so that plugin is called time checker and you can just enter the time until a user should not in this example if for 90 days the user did not lock in he is suspended and if nothing happens for a year then he is deleted and the second sub plugin which we implemented is based on what requirements we had but it might be you might also be able to use that since it's basically just on a file which lists all the users and if the user is not longer in that file he is suspended and if he does not see again on that file he is deleted so that's a really simple one and you just have to enter the path of the file and what we are doing we are unfortunately still in beta phase so we are testing that with some test instance and what I did because I was interested in the run times was that I started the cron job for 35,000 users at how long it took and the suspension part is really fast I was really happy about that it only took three minutes and I could start work again so that was nice but the deletion part takes quite long so when we are putting that into action we will batch our deletion since it took like for the 35,000 users it took approximately 6 hours which is still if you think about it it's less than a second for a user so if you run it on a daily basis it's totally fine but for the first run it will take quite a long time and I had a look that's actually what the graph is about which functions take how much time and you can see that the most time is taken by EnrollUserDelete which is a core function and enrolls the user from all their courses and this takes quite a long time yeah, I talked really really fast but as I said I would be really interested in what requirements you have because we are still at that development phase it's still things are still adjustable and how you think about the underanimization part because we thought it's really a good feature but some others might not like that thank you so we have plenty of time for plenty of questions but again please make sure you get the microphone first so I guess unless I'm missing something when you delete the data goes away do you have any data retention requirements at your university what is retention? you have to hold on to student data we are required to keep our data for 4 years so any student grades we have to keep for 4 years we are not using the instance for keeping grades so there is something else responsible for that yeah that would be an additional requirement because it would be possible sorry it would be the admin tool what happens when you delete the users is that they of course go through all the databases through all the tables and start to delete that so you could extract all the items from the grade table I guess for that you spoke about anonymization since some time we have the functionality that are related to the GDPR aspects and they are deleting most of the information in the database there are some leaving there is this not an alternative for your process? I think deleting the data would more be about also deleting the user because the thought about suspension is that if the user comes back he still has all his data so as part of the deletion process we also use the model core function for deletion so I guess it already takes it already uses the same functions as the GDPR users for deletion I think they are different this was one of the gaps in the deletion process is this core deletion process before we have the GDPR process the GDPR process is deleting the data but the typical deleting process of a user is only deleting some information in the user table so this is not a question but a comment regarding the data retention because I'm with the same university as Nina while we do not have any data retention requirements for our Moodle I think you could still basically realize data retention things by using the suspension period so you could suspend users when they are unenrolled from your university or your institution but you do not delete until after the data retention period has ended so there is a place for something used that the user data is anonymized for the other users but not for user administrators so if there is something coming up you could still access it I think in relation to that deletion Unimaster is part of Moodle Dachre I think they keep a backup of the courses for a period of 5 years so if they want to come back to the data they come back to it after 5 years they just get rid of it this is for the German speaking part so I'm sure this plugin has no connection to the backups but someone else is having a backup for this data in relation to the data GDPR and our plugins with privacy they are terminal basically there is no data being taken out if there is anything else to be deleted with that it's usually per plugin it's not per whole Moodle usually so it's just modular idea unless I get your point in different direction no one else we have a few more minutes hi so you can delete the student account after he is inactive for a certain amount of time how does it count the moment he is inactive by the last login to the system what happens with the students who never login to the system and I want them to delete from the moment they are created is it clear you create a user and is it running the time from the moment the user is created the account is created or from the moment the last login so the time checker plugin only considers user who have a last access so who wants access the platform if you also want to delete user who never logged in you could do that in an own sub plugin the reason why we did that is we have some technical users who never logged in and we do not want to delete them because they are important thank you thank you again Nina a couple of minutes before we do our next presentation if people need to go out or come in from the computer this is your client you can come in I don't want you to stop do you have any video? somebody listen to the sound and make sure it is not too loud when I start and if it is the sound is not my hero I am not there maybe you can come to the house are you talking about I just I think this is this is this is really like he knows what he is doing he knows what he is doing he knows he knows he knows he knows he knows you do it yes that is where And if I do happen to have any time I see questions, I may need you to be my ear. Okay. Yeah. Well, you're still seeing it. Okay. We're ready for our next presentation now by Rick from the University of Iowa. And he's talking about using calculated multi-choice questions, which I'm very interested in because it's not question types that I've used myself. Okay, Rick. And Buena Stardis. I think. Just a little bit of background. I'm a professor at the University of Iowa. That's in the United States. I teach business. And an interesting fact about me and my school is that my school is a canvas site. And I'm the only professor who runs Moodle. But on the positive side of that, I do teach primarily online courses. In the United States, there is an organization known as Quality Matters that certifies online courses. And it so happens that I'm the only professor out of about a couple of thousand with a Quality Matters course at the University of Iowa using Moodle. Okay. I'm happy to be here, share a few ideas with you. I've been using Moodle for quite a few years and I like to talk about it and share the knowledge that I have gained through the years. Yesterday I gave a short presentation on how to install Moodle on your PC or Mac. I mentioned that that presentation that I presented that at Mountain Mood this year and they gave me about three hours. That's seven minutes yesterday for that one. This particular presentation I did at Mountain Mood and they gave me two hours and I have 15 minutes here or less. And when confronted with this, I always like to make videos that I can share with you and then share with people around the world after the presentation. So this particular topic using Calculated Multi-Choice Questions, that's a core question type. So we're not talking about add-ins here. It's a little bit technical and I'm pleased to see a nice audience here for a technical presentation. So, estamos listos? Rick Juer is from the University of Iowa. I want to explore Moodle's Calculated Multi-Choice Question Type, which is a complex yet powerful question type for quantitative courses. I will explain what Calculated Multiple-Choice Questions are. It describes some of its advantages and disadvantages, provide an example, show you one of my real applications of this question type and provide a few summary points. By its title, you might recognize that this is a form of the standard multiple-choice question. There are some folks who would argue that multiple-choice questions are a superficial exercise, one that requires little thought, less insight, and no understanding. But others have argued that multiple-choice questions can be devised, so as to emphasize an understanding of a broad principles of a subject, I will leave this debate up to you to decide what is true or false. However, I like using multiple-choice questions in my courses for several reasons. They can be automatically graded by Moodle. This saves me time and makes me more productive. I can easily allow reattempts of quizzes using randomly selected questions, giving students more practice. Problems with potential ambiguity, student typing mistakes, close answers, wrong units, etc., are eliminated because you have one correct answer. Students not only have to explore the correct answer, but often will explore wrong answers just to make sure. What is unique to calculated multi-choice questions is that they can use variables in both the question and in answers. Some people refer to this as algorithmically generated questions. This allows the instructor to provide many versions of the same question. It helps avoid the problem of students sharing answers. In the United States, we call this cheating. Variables can be synchronized across multiple questions. This is an extremely powerful feature. Here are some characteristics of calculated multi-choice questions. Obviously, you can create the correct answer. You can have some choices contain constant values such as 35.2. Answers can have texts such as none of these are correct. You can also calculate incorrect answers. Words can be included both before and after the answer, such as the distance is 32.5 miles as the crow flies. You can have many choices more than the commonly used four choices. There are many functions available. Here is a list of functions from Moodle's documentation. You can include bogus variables, variables that are not needed in the equation. And this last important point, variables can be shared across multiple questions. I will illustrate these advantages in just a moment. The disadvantage of calculated multi-choice questions is that they are more complex to understand and set up, which is why some instructors might not use them. I will demonstrate calculated multi-choice questions with an example. This example will involve geometry, calculating the area and perimeter of a rectangle. In this simple example, let's have two variables length and width. Let's use the letters X to represent length and Y to represent the width. I am using X and Y just to keep my example simple and easy to type. The words length and width could be used as variable names in Moodle. I should also mention that variable names are case sensitive, so little X is different than big X. Then I'm going to create two problems. The first problem is to calculate the area of a rectangle, and the second problem is to calculate the perimeter of a rectangle. For both of these problems, I am purposely going to use the same X and Y variables so that I can demonstrate the concept of private and shared datasets. Here I am in Moodle 3.7.2, boost theme, and a course that I have called geometry. I will begin by creating a quiz in topic one. Eventually, I will add these two questions to this quiz. I'll call this quiz global mood and put all questions on one page. Now I go to Moodle's question bank to create my two questions. I create a category called global mood quiz so that I can keep my questions organized. Now the fun begins. I will create a new question and select calculated multi-choice. For question name, I use area of rectangle. For question text, I type what is the area of a rectangle whose length is X and width is Y. But this is not correct because X and Y are just letters. I need them to be variables. To do this, surround them with brackets. There we go. I scroll down to answers to create my choices. Moodle's default is to provide five choices. I always like to make choice one be the correct answer. For the answer, I type the following equation. Note the use of brackets and also note that I use an asterisk to represent multiplication and not an X. I also make this answer worth 100%. Okay, so remember area is length times width. Area is length times width. Now let's create the other choices. I'm going to illustrate some of the characteristics that I mentioned earlier. For choice two, I will enter a constant value of 18.5. However, this is not a number. It is text. To force it to be a number, I have to surround it with brackets. Like so. This will force the proper display of decimals. For choice three, I will enter the text. None of these are correct. For choice four, I will create an incorrect answer X divided by Y. For choice five, I will demonstrate the addition of words. The area of the rectangle is X inches. I mentioned earlier that you can have more than four choices. The default is five, but I will add a few more. For choice six, I will show the use of functions. Let me see. Cosine is a good one, so is Pi. I will use this equation. For choice seven, I will demonstrate the use of what I call a bogus variable. I will use Z. Z is not going to be used in either the calculation of area or perimeter. However, just as I will eventually vary X and Y, I will also vary Z. You will see how I do this in just a moment. Okay, I will save these changes. Moodle presents a second setup page. On this page, we see three wild cards, X, Y, and Z. For now, let's not change anything. Click next page. On this third and final setup page, we get to define our variable's ranges. Notice that Moodle provides defaults. Let's not change X. Let's make Y go from 20 to 30 and Z 300 to 400. I also enter values within these ranges. The challenge of defining these variable ranges is to do so, making sure that incorrect answers will never be correct. This takes some time and thought. I will update my defaults by clicking on Update the dataset parameters. Below this section, we see answers, tolerance, parameters. So, I'm saying in the video, but just to emphasize, you get a chance to see the correct answer and all of the incorrect answers on this screen. It shows us our seven choices, what we entered in the answer box for each choice, and the answers. At this point, we have only one version of this question. Let's add 20, drop down to 20, then click Add. Well, what happened? At the bottom, we see Set 20. Where are the other sets? Drop down the value next to display, pick something big, like 30, and click on Display. Now, we scroll down and we see all 20 versions. Wow! Here's a very important point. When we are creating these questions that use variables, Moodle does not generate the variable values on the fly. Instead, you force Moodle to create the values ahead of time. In our case, we have generated 20 versions of this question and only 20 versions that Moodle will randomly pick from when the question is provided to the student. You can display all versions and what might be extremely important, verify each version to make sure it makes sense. In this display, you can change values if you need to. For example, if I don't like a value for X, I can simply change it. When you save this page, the new values are also saved. We have done a lot of work. Let's save this and see our results. I can preview this question to see how it looks. Pick the correct answer, submit and finish, and yep, it looks good. I close this window and generate another preview. Yep, seems to be working. Let's add our second question for perimeter. I'm going to make my life easy by duplicating this first question. Then I edit it. I change the question name, the question text, and change the choice one correct answers equation. Look carefully at how I am using parentheses and brackets. Of course, we need to know how to write correct equations. Otherwise, well, don't be an instructor of geometry. I keep all the other choices the same and save changes. On this second page, I will accept the defaults for right now. On this third page, I'm going to make X go from 20 to 30 and Y from 1 to 10, just the opposite of what I did before. I will update the dataset parameters and then add 20 questions. Everything is looking good, so I save changes and preview. Yep, this question looks good. Now we have two questions. Let's add them to our quiz. I go to this quiz, edit quiz, add questions from Question Bank and pick these two questions. Then I click on this quiz to preview it. What I see are these two questions, and each question has its own set of X and Y values. But what if we want to use the same... So you'll notice the differences in X and Y between the two questions. Sorry, Mary. X and Y values for both questions. Can this be done? The answer to this is, of course. We are using Moodle, the most powerful LMS, so of course we can do this. To synchronize the X and Y values across multiple questions, three conditions must be met. First, the questions must be in the same Question Bank category. Since we created these two questions within the same category, this condition has been met. The second condition is that we must have the same number of versions for each question. We will take care of this in a moment. The third condition is that we must tell Moodle that we want to synchronize these datasets. I will show you how. Begin by editing the first question. No changes are necessary on the first setup page. On the second setup page, instead of using private datasets, we are going to switch to shared datasets. We also pick Synchronize. These changes wipe out our previous 20 versions, so we have to establish our variable ranges and re-add 20 questions, then Save. Next, edit the second question. Again, no changes are made to the first setup page. On the second page, we again share the datasets and synchronize. On this third setup page, we do not have to make any changes since this question will be sharing the 20 datasets from our first question. Finally, let's go back and preview the quiz. Wow, look at this. The values for X and Y are not... Sorry that the captions covered the second setup data. That are a little bit harder to see. You can watch the video on your own again and the uncaptioned version and see that they're the same. They're the same. All of the same for both questions. Cool. I will demonstrate how I use this question type in my Business Analytics course. I have an exam containing the topic Descriptive Statistics. I provide the students a dataset and they must use this dataset to calculate various descriptive statistics, such as the average, the standard deviation, the range, minimum value, maximum value, and other statistics. This is how the quiz appears in Moodle to the students. To create the set of questions, I established three variables within the dataset starting with a question about the mean. Here is the dataset, here is the equation for the correct answer, and here are the other choices. On the next screen, I make sure to synchronize variables. On the third setup screen, I establish each variable's data range and 21 question sets. 21 sets seems adequate for my 40 to 45 students. This is a common practice of using the same dataset for a series of questions. It would be a bit weird and clumsy to use a different dataset for each question. Moodle's calculated multi-choice question type provides an elegant solution. Yes, this takes time to set up. I probably worked 12 or more hours on this. However, it seems to work fine and I have been using this set of questions for about four to five years. In summary, calculated multi-choice questions are a powerful question type. They are a bit complex and take time to learn and master. They work well to avoid cheating. This question type doesn't exist in other LMSs, giving Moodle a powerful question type. Here is a list of some identified limitations. This is not Excel. More functions are needed. Navigating backwards and forwards between setup screens is tricky. I always start at the beginning and step forward. Removing one specific answer set is difficult. You can modify any set, which is what I do, but removing the 15th set, for example, is not provided. You can't use variables in a description question type. I would like to begin a quiz question set with a description question, then follow it with shuffled, calculated multi-choice questions. But you can't do this yet. Documentation needs to be expanded. Learning to use calculated multi-choice questions requires experimentation and practice. This takes time. I have enjoyed presenting this topic to you. I hope you enjoyed this presentation and learned a little bit. You can find more support, including this video, on my website. Hello, I am Rick Juris from the University of Iowa, location of this question type. I knew you didn't catch it the first time, but if you do want to review, you go to my website, my homepage, scroll down, and you'll see a variety of my Moodle Mood presentations, including this one with some additional resources. Thank you. We only have time for one quick question. Any questions? No. Yeah, Tim. One very quick question while we're setting up for our next presentation. Can you answer on the microphone? The calculated description noted. Actually, you should put it in the Moodle Bug Tracker, the feature request. The calculated description put it in the tracker. Then it's an official request. We are fortunate to have the quiz guru with us, and Tim knows that I do make a lot of suggestions in Moodle Tracker, so this one will end up there. Tim, thank you very much. Rick, I think I have a very nice phone here, but maybe you'd like to have it back. Okay, I'll just wait a few seconds for people to come in and go out, and then we'll have our final presentation before the coffee break. Okay, everyone, and our final presentation before we have a short break is by Amir, and it is on LTI question type. Begin. Hello. Moodle, as we know, has a module activity for LTI, but we could never get it to work as a question type, and we thought maybe you should develop it, code expert applications and IDE for web-based programming, and we decided to add extra parameters to the simple LTI connection connected with the attempts, etc. So this question type basically is exactly like acting exactly like the module LTI also. The only difference is you can add it to a quiz. You can restore it, back it up without a problem. It's just the exercises has that's actually an external system has to understand how to be a provider for exam setup rather than just an activity by adding or accepting or understanding extra parameters. The grading is actually it's a manually grading. There's no automated grading, and those are based on the intermediate table of submissions also. One of the things is actually knowing the progress of the IDE or the external system. We actually base it on the attempt. Once you are attempting the question, we consider it in progress regardless. This is one of those technical difficulties also in integrating another system and exam setup. Those are simple flow charts for the LTI connection. There are two different ways. One of them is actually going through the module, login, navigating to course, creating quiz, creating external question or LTI question type. It's created. Through creating the question, you can add directly the link exactly like the module LTI. You can pre-create it and code expert basically and getting the link for LTI and then add it again or basically through deep linking without copying. This is actually flipping charts actually the opposite. You have the two options having the link ready or basically deep linking inside without going through login again to the external system. We have here a small video showing how it's been created. It's very quick, silent video. This is an emulator, of course. This is actually the result from the emulator, the outcome summary, consumer information also inside them. As you notice, some of these stuff are actually what they are already there. We have extra parameters whether needed or not. They are with the extension. You see the LTI ID, the manually graded or based on attempt and even the quiz details and other information. If the external system is in need for them, we have a unique connection. This unique connection stays with the question regardless where you go also. And the details are basically based on the instance code, the attempt code, the module attempt itself and the LTI itself, the information are all available. So the provider has to understand those information. It's basically as simple as that exactly like the Mod LTI with the differences being able to work perfectly with no problem with the quiz setup. We have the future work, of course. There's a lot of things to do but the main things are implementing LTI 1.3 and further flexibility and mapping tweaks because outside systems, not all of them would actually back up or restore the same way Mod LTI is doing and for grading we're trying to make it slightly easier for the teachers to grade rather than grading a different system coming back and pressing grade buttons. We thought about it through web services and events but it's seen from the end user's point of view to be complicated, at least in where I work. This is actually considered for the future changes and enhancements for the question type LTI. The Q-type LTI is available actually in Moodle.org and also this is the Github repo for it if anybody is interested and if they are interested in the development IDE, web development IDE we also can't share the information. It's in-house software, not published yet. Any questions? Any concerns? Thank you very much, first of all and secondly we've got plenty of time for plenty of questions. I'm not sure I got this right but the question I have is can you use any NTI provider in your question type or does it have to implement these extra parameters? You actually have to understand those extra parameters because as a provider the only thing is just from organization point of view because the quiz behavior if the attempt is based on last if the highest best grade etc. they have to understand it so they send you the right information. However, if you don't understand them it will be technically like the normal model TI and this does not serve the purpose. We have also a documentation, we'll publish it in docs.model.org for the providers if someone is interested to connect the two systems. This is just a medium and middle. Thank you for the presentation. You said something about needing to regrade the quiz to get the grades and I didn't quite follow that bit. Could you say a little bit more about that please? The grades are sent from the provider they come into intermediate table, it's the submissions great submissions and then to get this because it's a manually graded question you have to correct and regrade we were hoping actually to do it through a more automated manner so the regrading. Just to say if you are looking for how to do it looking for the way to make it a bit more automatic, possibly the answer is to write your own question behavior plugin. That's true and actually we thought even about Evans it's just if something changes we can send it directly through web service because this question type has enabled web services where actually if the provider is more advanced they can send the grade directly instead of going through the grade system. However we don't publish this because simply we're not sure yet that if someone understands the basics first. Sure okay but based on my experience of developing this sort of thing I would guess a question behavior is likely to be the cleanest way to implement it. That's true and it's only a matter of how they decide usually from administration point of view. Anyone else? Who can give them a question? I had one other question which is not about this presentation. In the video I spotted when you just flashed up in the video there was something called the freehand drawing question type. That's true. In the plugins database. This is actually Victor drawing and we're right now working on it and it should be ready by end of January and it's a very advanced one. We worked on two different versions before and they were canvas. This is a vector based so to give extra features for advanced students or studies basically we will make sure that it will be published. We have many others unfortunately I have to do it on my personal laptop where I have this question with me but I can't show it to you after the meeting if you'd like. Two pounds though. Someone who's not team wanted to ask any question. Okay, thank you again very much. If I've read the program right we now, well we're rather early but that's a good thing for a short coffee break and then we'll be back here at 20 to 4 I think. That's right, yeah. So I'll see you in the next session. We have to yeah yeah no yes yeah yeah yeah yeah and yeah yeah yeah yeah