 Yes, thank you very much. See, hello. Hi everyone, thank you all for coming here. So my notice, there's no title slide, that's because I prefer to begin directly with the problem that we wish to solve at virus, and that is what you see here. So if one of your math teachers are involved in the math or science departments in some way, then hopefully this image speaks to you. Can we do a show of hands if anybody actually is involved in math or science in their institution? Okay, we got some. So yeah, this is what we are trying to solve. The problem of being able to have STEM content in Moodle as represented by this image, which is something that is traditionally difficult, and well, we have proposed a set of solutions to be able to solve that. So I lied, there is a title slide. This is who we are, virus. My name is Gaspar. Thank you all for coming here. We are going to talk about math type and virus quizzes, our two tools for having STEM content in Moodle. So math type solves one of the major issues, which is that people need to write things. I don't know if you've ever imagined what Moodle would be like without content. That would be pretty strange. You wouldn't really do anything. Well, I guess you could do it. You could service some sort of organizer or something, but you probably want to write things. If you're a history teacher, then you'll want to write text figures, et cetera. And if you're a math teacher, you want to write formulas, math or science or anything they use as formulas. As soon as you run into the problem of writing a formula, then you're going to run into an issue with Moodle. Because while it does have a formula editor, which does function, it does write formulas. So actually, speaking of that formula editor here, we have two very real graphs based on good data that we collected for real about comparing the usability curve of the default equation editor versus math type. So as you can see, as time goes forward, it's very difficult to make progress. You can get there, because as you can see, they both reach the same point. You can get there, but most teachers will get here and be like, I'm just going to switch to paper because it's not worth it. And math type, on the other hand, right from the beginning, it's very simple to use. Like I said, this is a scientific study that we did. That's math type. Here we can actually see it in action. So it's a simple toolbar to use. You click on the buttons, you get the symbols. Or if you're on a touch device, then you can handwrite the formula that gets recognized and turned into a typeset mathematical equation as if you had written it with the toolbar. The toolbar works for primary, secondary, higher ed, or anything you want because it's customizable. The handwriting interface, well, that's just a little closeup of it. You write the formula and it recognizes it. There's not much else to say there except that this is obviously useful for all the students who are using Moodle on their mobile devices, maybe, or tablets, so on and so forth. Or if your laptop happens to be a touchscreen, then, well, toolbars in general are not very usable on touchscreens, so this solves the problem of being able to write formulas on a touchscreen and actually have them be recognized because, of course, you can always just scribble one, but then that won't be accessible. It won't be not good. So that's math type. Math type's little sibling is chem type who solves the same issue but for chemistry. So if you happen to be in the field of chemistry, then you have a similar tool, but for everything that math type does except for chemical reactions and so on and so forth, you can use that to write chemical things. The next thing, so after being able to write things, people need to ask things. I mean, well, this is like true as a philosophical sort of thing. That's not what I'm getting at here. Wireless is not a tool for resolving existential conundrums, but the more specific interpretation that I want from this is that teachers need to create assessments in Moodle and those need to be able to be graded. Hopefully, I mean, if you grade them manually, that's fine. Math type can help you solve that problem because your students can write formulas and you can grade them manually. That's a step above sticking with paper. However, our tool wireless quizzes actually allows you to grade things automatically. So here, you'll probably recognize this as a Moodle question, but you'll see instead of a regular text box, you have a math type there so that the student who's answering it can write a formula or hand write it by just clicking on that little icon and hydrating the formula. And that allows you, and that expression is actually mathematically interpreted and graded automatically if you so wish. So then, this is a little example of what this helps solve. So say that the answer to your question is x plus 2, and the student happens to write 2 plus x. If it's just a regular text-based system, then that's not going to work because it can only recognize x plus 2. It doesn't know that 2 plus x is the same thing as someone that grades manually does. Wireless quizzes does know that, and it counts 2 plus x as correct when the teacher happens to write x plus 2, or vice versa, or maybe the student writes 1 plus 1 plus x, or maybe the answer is sine of 2x, and if you're familiar with the sine of the double angle formula then you know that it's 2 sine times cosine of x. Then it'll also recognize it as correct, and a fundamental challenge, if anybody wants, is to come to our stand and think of one expression and then the most convoluted possible way of writing the same exact thing and see if wireless quizzes recognizes it as fun. $20 says it does, not really, but it's fun. And then after that, you will probably want to ask more things of the students, so if the answer to the question is 1 half, but they arrive at it through a long series of procedures and they arrive at the answer of 5 tenths, while in general we want students to simplify fractions, not just fractions, polynomials, other types of expressions. Here are the examples of fraction. Wireless quizzes does know that 5 tenths is equal to 1 half, but it's not simplified so that you can grade it accordingly. You say, okay, you got the right answer, but it's not simplified, so we're going to give you such and such grade and it's up to you. Along with being simplified, there's a whole other range of criteria that you can ask for, factorized, whether it's a fraction, whether it's an integer, lots of different properties that you can ask for the answer to have. That's the second feature that we like to talk about with wireless quizzes. The third, but not least, is randomized questions. So if you have a bunch of students in a room and they're all taking the same test, they can have the same question but with different data for that question so that if the question was, for example, factorized the polynomial p, while one student will see x squared minus 3, another student will see negative 4x cubed plus 9, and that can be different for all the students in the room if you just, well, you can set this, any part of the question really, to be random and that allows you to solve, for example, the problem of students looking at the other students' screen. Not just that, they can also, if you're doing something for learning, then we're all doing things for learning, rather training the same type of question than you can tell the student to, they can repeat the question as many times as they want and it'll always have different data. You can customize that as much as you want to train that specific type of question. So that's basically all that we have to say about our tool tools. For now, we do have a more specific presentation on virus quizzes tomorrow at 12 noon, I believe, and that will get into much more detail, do an example of a question and so on and so forth because quizzes has so much more to explore. So we're doing another presentation on that and of course you can always come by your stand if you have any specific questions or things that you'd like to know about. If you are already familiar and you want to know about new features that we have, et cetera. So yeah, that's pretty much where I'll leave it and now we can go to questions. Yes, you can. You can say the correct answer to this is anything between negative 10 and 99. Well, not just those two numbers, also other numbers. Anything else? All right, I think we're very good on time that was probably shorter than expected, but all right, good. Thank you very much. I'll see you around. Oh, yeah. Thank you. Yeah. Okay, no problem. Is it working fine? Right. Okay. Hi, guys. My name is Alina. I'm the head of business development at Unicheck, which is plagiarism detection software. First, I'd like to ask who's speaking English. Please raise your hands and Spanish speakers. Okay, got it. Well, in any case, I'm pleased to introduce Yulia. She's my colleague. She's in charge of Latin America and Spanish speaking overall market in Spanish specifically. So we're going to present presentation. It would be very quickly, but I hope you like it. It's very interactive. So main reason why we have the presentation today and the topic specifically is like the cheating, contract plagiarism, and the things that can trick plagiarism checkers. So just in case, who knows what is plagiarism checkers? Everyone? Mostly. Okay. That's better than nothing. So yeah, so we have like different plagiarism checkers on the market, right? And different LMS systems. And of course, every student knows, nowadays every student knows that there is something, there is some plagiarism checker inside the LMS system, right? So sometimes, unfortunately, when students want to submit a paper and could get a good grade and he wants, he needs to trick system, right? So he can a, buy the essay right now online. There are tons of companies, unfortunately. And the other thing, which is pretty easier and cost efficient is trick with a text in your paperwork. So we have two sessions. The first one is today about the modifying. It's our technology, unique technology to detect such trickings. And the other one is going to be tomorrow at 11.30. It's about Emma, our ownership verification technology. So I would like also to, you let her introduce herself and tell more about what she's doing and tell briefly in Spanish, what are you going to? Hola, buenos días a todos. Aquí somos también, no solamente los equipos de Unichec de representantes de ventas, representantes del desarrollo. Aquí tenemos cubierto también el sector de para nuestros clientes para las unidades de España, instituciones educativas que nosotros soportamos. Y también aquí, bueno, estamos mostrando que la integración de model de Unichec es bastante estable, tiene cuatro años de desarrollo y también a muchos clientes que ya tienen las historias grandes de éxitos con Unichec. Y hoy día también estamos entusiasmados para dar nuestra ponencia y presentar las modificaciones textuales y los últimos avances de la herramienta anti-plagio Unichec. Por favor, atención a la charla de Onena. Gracias. Ok, guys, so, you know that the main problem is not that the student cheat, I mean, student cheats every time, unfortunately, but they want to again, get a good grade but not doing something. But the main problem is actually that we face that people like educators, instructors, teachers they don't know about modifications of text, they can skip those tricks and they just don't know it actually exists. So today's session is more about what is modification in text and actually how to spot it. So please meet our our squad of cheers. It's very bad word to say, but in this case, it's ok. So we have a copypacer, we have character replacer, we have micro spacer and a cheat man. So let's talk about everyone. So please meet that copypacer. Julia, please translate. Here we have a series of characters that have to do with the most popular text modifications that sometimes the students submit to these copies to do plagiarism. And now, our first error is the spacer copy that a person who sometimes makes some attempts to make spaces and make copies in a content. And the copypacer in my case his guy is actually what he's doing is taking a text from some repository from his friend from his brother, mother, wherever he's taking the text and he makes it as a picture to the submission, to the assignment. And the actual story, the real story it was versus a couple of months ago, the one of the teacher he said like, hey, Junichak, he's working not properly, there is some technical issues because I submitted the paper to Moodle and it says it contains less than 8 words. And we're like, hmm, there are about 12 pages in there. Can you please send this work to us? Come on. It was all in pictures so of course it contains no words, so of course it's less than 8. Like for our system it should be at least 8 words in a paper. Okay, let's move on. In here we have character replacer and we have microspacer. So the character replacer, it's a very tricky guy he's replacing characters from different alphabets, for instance Spanish, A's, O's, I's into the English text and for all of us, for human beings the text looks pretty okay because it's just a simple text but for machines, machines sees different encodings in every letter so for machines it's garbage text it cannot read the text so just for you to know there's another very, very, very popular cheating attack. Please simply. We have the second hero Encomillas. He's a person, a student who only makes a text modification by means of removing the characters from the other language if he changes his keyboard another language and he takes the same letters he pecks them in his text he presents it as if it were a good a good job, an original job but really a cheat what he's going to do is highlight these characters as agents to this text and then with a unique check for the teachers you'll see that they don't belong to this text which are the characters removed from the other language and he hid it here. Thank you. Well, the microspacer this is very this guy is very, very hard to spot unless you have the unique check, of course so the microspacer, what he does so for instance you have a word that contains six, seven characters just a simple word like hello and as a human being we see that as a word but for a machine the microspacer he made to see it as five or six different words and most probably original words because they just separate every letter with a lot of spaces and they do it micro so for our eyes it's just a regular word but for machines it's six different words which are really original one and sometimes we can not take into account but for the machine the unique check will show it and in the additional models of a unique check like the modified model it will visualize it and show that really there's something hidden here that's the text, the modified file please Thank you. And we have the last one and we're going to talk about this guy tomorrow on the tomorrow presentation because again unfortunately blind student they want to use some shortcuts they just order the papers online, they place all of the assignment details etc etc and they get this paperwork like within a couple of days and it's fully original because it's written by other people but a unique check the most important about a unique check that we want just is to punish students we don't want to waste your time the time of educators when you're evaluating paper but the paper is it's not worth to evaluate it so you should work with students you want to make students successful and it's actually our aim as well so with those tools like cheater, cheater is a very bad word so we just want to help students who face some issues who go into some problems right now we want to help it by getting the time getting the time of getting your time to free your time to work with a specific student so when you see the attention tab the warning in a model or in any other element that you use you better go and start like working with a specific student and understand what is his issue a few words about this last type of the most annoying because they don't do anything with their own hands they just hire they buy some work on the internet or they pay it for some service and they do it all their work at the end of the course and we really don't know if this work really belongs to this student who gave it to him so we are going to give a presentation about how we detect the student's own style and in this way what we launched what we now release for all our clients and it is available in the Unicheck report in Linea it only has under all the development a single objective to give an indication to the teacher that maybe there is something hidden something unconscious or with an intention in the student's work and you have to take into account not only the content that the students do but also the format and the interface in what they present their work and their work thank you that's actually that's the end of the presentation we just want to mention that we already help a lot of people all over the world and hi our newer customers in Spain and of course in the United States in different countries and we really aim to succeed we want to increase academic integrity over the world we do run a lot of social projects for the institutions who do not have some money so at Unicheck you will find people and I hope you really enjoy using Unicheck for all new features thank you so much in case you have any questions feel free to ask of course well that's the algorithm of artificial intelligence this session the presentation will be tomorrow and it is the itself of Unicheck which is the model that is called EMMA and in its base it has the algorithm printed by artificial intelligence and in this way how it is self-educated and then if it has some entries we can build a profile of this student and it is the only writing profile of each student and we can build it only in a way that we can obtain it is that correct? repeat it in English what is the algorithm of our new module EMMA? oh yeah tomorrow we are going to discuss more in details but yeah EMMA it is artificial intelligence that just reads student papers and it is chicken it is looking for formatting, punctuation specific style writing style of the student and then on the next future submission of the student she just on what she learned previously she analyzed it and tells whether the next every next paper is actually submitted by this person or not so pretty nice I mean we are not that technical person so you will not hear anything very technical details in any case any other commissions? well we are awesome there are a lot of differences between us and Ternotain and of course a lot of things as for the support and the uptime period we do provide with the uptime which is 99.9% so it gives you about less than 9 hours of downtime a year and it is usually shadow maintenance so we always prepare our customers on Saturday or Sunday very inactive time that is one of the things another thing we have a customer support team which is 24-7 and it is SLA based so when you contact the support team you receive the answer very quickly so it is about for the email it is within an hour for the chat phone it is immediate answer also we have the customer success manager team so that the girls are actually in charge of every customer account so they help you with the migration program from one solution to another they help with onboarding training sessions so we take all of the headache of the administration like on our side and we run all the trainings prepare materials that can be distributed within the institution and technology wise I would like to invite you to our booth because I can just talk and talk a lot and I would like to share more information like the next session by the way turn it in session oh we still have 5 minutes great so technology wise we provide with the online web search what does it mean so when we do search with the internet we do it in the real time so in the real time you will get all of the similarity sources like in the report you are familiar about the pleasure of the report so you have the tons of sources there and every link is very finely so you can use first side by side comparison so you have the paperwork of the student and you have the actual text from the source so it's very easy for teachers and actually it's time efficient for teachers to evaluate paper easily then there would be no 404 sources so there will be all verified so it's actual proof and you can have this paper you can generate PDF report and go to the student and ask him to revise the paper again there is the fact please revise also we respect the privacy and security policies of all of our customers so by default we do not cross check your libraries so every library of our customer is your private private database however if you want to check and cross check your library and other customers libraries we can enable this option but only after your written confirmation and yeah we do not share any details of our customers without confirmation with others and specifically no never like students data etc etc and of course like the text modification like modifying technology I'd like to show it in action again here's a boost so you can see what is hidden in the text so you can actually see what's hidden from you right more and more and more things thank you any other questions Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A Q&A similar to some strange other data, and it looks right. Well, the old system, but it's a unique check. We make it, as you mentioned, as a unique cloud. So make it unique one language, which specifically applies to this paperwork. And moreover, we show you that there is character replacement. Moreover, we find similarity with this part of the text. So in case it was used somewhere, we will show you the source, the specific source. We want to thank you, well, taking advantage of this opportunity to all the clients who are currently in this Moodle Mood event, the business owners, who are here, the Malaga owners and other clients who have also passed through our booth now. And well, thank you all for listening and for your questions. Thank you so much. I hope you like it. OK. Can everybody hear me OK? Yeah? Hi, everybody. Good afternoon. And welcome to today's great scope session. First of all, introductions. So my name's Mark Harner, and I lead on partnerships for Turner Inn in the Amir region and a couple of partnerships elsewhere around the world as well. And I've got my colleague with me here, Aaron Jawerski. He's our VP of partnerships based in North America and he heads up partnerships globally. So today, we're going to be talking about our grading platform, great scope, which is for the assessment of written work and especially in STEM related courses. So it is the leading assessment and feedback platform that modernizes the grading process and transforms grading into learning. So what we actually try and do with this service is to streamline the grading process and it's to address high demand assessment submissions, written assessment submissions in STEM related modules specifically, but it can be used in other assessment types, which we can get onto in a second. So first of all, I just want to highlight that providing quality feedback can be a challenge and especially when you're dealing with a high volume of papers and when it's actually written assessment as well. But if you take, for example, a math course or computer science course, it's not standard text that's submitted. It can be formula, numbers, et cetera. And the challenge that we're trying to address is how can you grade a high, large volume of papers in a reduced amount of time. So these are just some of the quotes and challenges that can be faced. So just to give a little bit of background on grade scope, it actually was born out with UC Berkeley and there were a group of teaching assistants on computer science course who had the pressure of marking hundreds and hundreds of students written assessments and they thought there must be an easier way to resolve this issue and also provide the quality of feedback that the students were expecting. So they created this prototype which using AI would look at the answers that were given and look to book it and organize the way that questions were answered into certain areas so that they could apply certain marks to a particular answer. So effectively if question one had a specific answer and 100 students had answered in that way they could apply a mark for that answer and then it would mark all of the students' work who had answered it in that way. So it became so successful and so many instructors across the faculty adopted it that the service grade scope was born. That was back in 2014 and then last year in 2018 turn it in actually acquired the service as we felt it was a good fit under our umbrella of academic integrity. One of the goals that grade scope addresses is the consistency of grading as well and greater coordination so as you can see there and we want to pinpoint knowledge gaps with grade scope as well so highlight where there could be gaps in learning or where problem questions where questions need to be written better or understood better by the students in the class. Increasing data security so because everything is electronic scanned in electronic you don't have all the huge mountains of papers that you have to mark and also as well it's secure so students can't try and come back at a later date and get access to the content. And also as well we integrate with existing systems so Moodle is one of those from what I understand but there's also an integration for D2L and Canvas etc. This is the timeline which was what I've just talked through so probably a bit premature in what I was saying. So yeah it was acquired by Turnitin and what we're looking to do now was really incorporated into the other services so Turnitin having the originality check products of SIM check and similarity check as well as feedback studio we're looking to incorporate the system as part of that. Right so Mark was talking about the why so what's the challenge that you have especially in these big intro STEM courses lots of paper assignments that need to be graded lots of problems many TAs in grading those papers so what are we trying to solve there is how do you quickly do that in an efficient way and how do you start grading papers and what kind of challenge you have as a TA or grading I'm grading a paper assignment and I start to grade question one a certain way but throughout the course of grading I decided that I wanted to grade it differently the way grades go will work with that is automatically if you change what you want to change for grade change the grading mechanism for each one of those problems throughout the entire exam Right so what is it really focused on? Handwritten diagrams programming projects exams and short answers right so it is paper based problems that we're talking about scanned into the system so if you think about a lot of these courses many of the exams are still done in handwritten format how do you then efficiently grade all of those different things what the software will look for initially are putting together all of the questions that are the same so that one can grade question six across the entire organization but it will also look for what is known to be the correct answer and automatically grade those and then put other answers that look similar together and that's the AI function but then it also gives the user an opportunity to go through and group those answers together that he or she wants to group together right so again it's the efficiency of grading that if I'm a TA and Mark's a TA and Max is a TA we are all looking at the same thing the same way so that you have consistency in how students papers are graded again particularly in large STEM courses where you might have several TAs or several instructors grading papers for the same class right so this is what we're talking about in producing grading time it's taking all of those answers and grouping them together in the way that needs to be and it does that first but you then have the opportunity to group it the way you want you can a little faster than I pointed so you group those answers the way you want assign grades that you want to those every answer that's like that gets assigned that grade over time again the important thing about this function here is if I go back later on and decide that that question shouldn't answer that way it shouldn't be reduced by two points but one point change that back to one point later on everything that previously graded gets the score change for that same thing it's not just what I've graded it's what Mark's graded as well right so that's part of the reducing grading time so that we can only focus on the questions that are wrong and decide what I want to do with those so in essence you as the grader are creating your own rubric as you go and it goes that way so the system's not necessarily doing it for you but you still have the art of being the instructor and the grader to put that together right and again here's how I can put the coordination together of what we're doing I can start to answer why I've taken a point away so that I and the other graders understand why I've changed that rubric right and I can assign those changes as I go through so again ease and speed of what I'm trying to put together there and the last is the pinpoint of knowledge gaps which Mark talked about before and we think about this in two ways right one is as the instructor can later understand everyone in this course got question 9 wrong I know how to go back later on and focus on point 9 more so as well if I want to I can look and see how I graded a paper versus how Mark graded a paper so I can get consistency in how my graders are actually grading what they do right so that pinpoint becomes very important a lot of different things you want to do as an instructor and the last is as Mark said it's an increasing data security so I got a thousand papers termed in at the University of Toronto right now what do I do I hand them out to several different TAs they put them in their backpack they take them home what do you work in grade scope you take it to your scanner you scan it into the system right you can then take all those papers put them in your lockbox safely away in your office now all your TAs have access to this within the grade scope application on their computer that can be accessed anywhere so as Mark said we don't have to worry about papers being lost stolen nicked away so that I can you know copy it for the next exam I do so it's very very important as well and something that a lot of our professors have asked for right and again last is integration with all the existing system so grade scope is not looking to be its own solution but a point solution within the Moodle environment or any other environment so that instructors in these courses have an easier way to grade paper-based exams and that's grade scope in our 15 minutes or less so with that you know a lot there that's the question just but that's where the idea came from initially because it was on that type of course and it was focusing on those technical type of answers that were hand-written and using the eye to to book it those and sort those out but it's not to say that it can't be used outside about that's what we're beginning to learn and that's what we're open to to see how it can be used in other assessment types yeah it's a fair room that's why it does as well that's the other way it can be utilized is that it can intelligently look at breaking up the questions not necessarily what the answers are it knows when question one starts and ends so it will put into one area one group question one group two will be question two and so on so then the Rupert can be applied just as it can be to a group of answers yeah so what I can say the product team is looking at that and bidding it just that we know right now it works effectively in STEM we're trying to come do it in humanities and other written pieces of work how in fact it works well but for a question yes yeah for two questions so first of all no we don't provide scanners that don your system your RICO, your Xerox whatever it is as long as it can email the document into grade scope you're fine so you can use whatever scanner you want to use so yes it's hardware agnostic from that perspective I don't know the exact percent of questions that don't get captured that's a great one for us to look at that is why though when you saw that diagram where certain questions automatically get bucketed where I know it's the right answer and then there's the remainders those are the ones that the system hasn't read correctly we'll have to come back to you with how many questions get there because you're right some of it's just going to be messy handwriting some of it's just going to be this answer so far out of the realm of everything else I got I don't know what to do with it as a system but that's a great question yeah right right so look grab me after I get your information and we'll follow up with the specific answer there wait, yeah are you on five questions yes can you tell us about the integration with Google like the LTI and how it works to be honest I'm not the best person to ask about that I just hope I'm not that close to it for more I understand it's integrated through LTI perfectly yeah I haven't specifically seen it in action but yeah I thought it was LTI so we have okay well again okay well I'll get you details at the end and then we can connect you with the team and they can give you more information anyone else well wonderful you get five minutes back in your life yeah thank you so much for your time I appreciate it yeah yeah this is fine I'll just go ahead yeah I'll do this again okay okay yeah you have to do that okay okay okay okay okay okay okay okay okay okay all right we'll go ahead and get started my name is Jonathan Moore I'm the director of my learning consultants and the session today we're going to talk a little bit about our journey of shopping for hosting for Moodle and then how that led us to develop a platform that a little bit different twist on what I think other people are doing out there so we probably all know what hosting is right server storage, computing deliver an application the open source world we've got kind of two modalities of how that hosting is delivered we've got the self-hosted or maybe the unmanaged hosting where the organization's responsibility to run that application and then on the other side of that spectrum we have SaaS hosting where it's a service it's kind of a sealed black box and it hopefully just works but if it doesn't work the way you want too bad so what we're trying to do with the tools that we built is to get the strengths of both of those options so have the flexibility and the power and the access and ownership of self-hosting but with the engineer design, scalability security backups that you have with the SaaS offering so just a little background of the history there we're on 16 years now with Moodle so we actually started working with Moodle before the 1.0 release my wife and I who's the other co-founder of the company we were the first two employees at one of the first middle partners we did that for eight or nine years and then in 2014 we started our own company My Learning Consultants and with a focus on consulting rather than hosting service delivery those sorts of things so so we've kind of had all sorts of roles when we got into Moodle we were self-hosting so we've done the self-hosting we've been the platform vendor provider we've been the consultant helping clients to try to implement whatever their vision is for their education so what we found was that in order to do the things that we wanted to do which is stand on top of Moodle kind of bend it to our will that we couldn't find a hosting solution that would do those things for us and so just a note on our process how do we have those needs where we wanted a lot of flexibility we would start with clients and we would say what are your goals what company do you want to spend what's the budget for this project and then we would bring in the knowledge of what are the best practices the technical practices and also the pedagogical practices and then the mission was let's make a plan that's tailored fit for this situation so here's my list when we were shopping this is what we were looking for we wanted it to be Moodle specific high quality engineered around that and maybe we get started but then it doesn't work so well later we wanted workflows that were specific to Moodle so things like what's the best practice for upgrading make a copy of the production site and then upgrade that copy test it if that works schedule the next so things to support those common workflows we needed flexibility so we needed to be able to leverage the power of I think it's 1600 contributed plugins now we wanted to be able to install any of those no questions asked and we wanted to do it on our time frame and if we found that the plugins that were out there and the core functionality didn't cover it we needed to be able to install plugins that we wrote and then more extreme cases maybe change core a little bit so and that's where we really ran into issues offerings of really high quality SAS type hosting that's Moodle specific but it's harder to find that plus yes you can do all of these other things we also didn't want to be locked into a particular vendor so we wanted to be able to move move that around again have flexibility put it on a different platform if we wanted to change the environment and we also wanted to be able to actually directly get into the infrastructure and this is about flexibility again and also we wanted to have the ability if there was a problem for our client to be able to get in there and have some control over the time that it would take to resolve it but it could also be something like as simple as well we want to generate data we want to pull particular data from the infrastructure so these are all the issues that we fought trying various offerings out in the market and really it's the necessity is the mother of invention is where we're at is we really just wanted to do our thing with our consulting custom development advice so that led us to build a set of tools that kind of cover all these things so what our unhosting tool does is we currently deploy to Amazon Web Services but it's really agnostic to the cloud so the ideas will add Azure Google Clouds down the road so it'll deploy Moodle ready infrastructure it'll create Moodle sites from the official repository it'll keep them up to date it will install plugins and keep those up to date and make sure the right ones are getting installed it'll automatically take care of those updates geographically dispersed backups and history of monitoring the server so what kind of growth projections or looking back if there's an issue report being able to see we had to spike in some type of usage or some particular bottleneck and be able to map that up with what happened at the time so again we wanted it to be really Moodle specific so it needs to scale around the way that Moodle scales and we wanted it automated so we didn't want to spend a lot what I was finding is I was installing these servers by hand over and over for clients and it was really around well we have to do this in order for them to have the flexibility to meet their goals so that's not very efficient we wanted to make it more efficient and again those common workflows that I talked about so we built that to make ourselves happy and we hope some other people like it as well and because of how we built that out what we wanted to be able to do is say we're very good at this particular piece of it but we don't have to it's not an all or nothing proposition so maybe you know a lot about Moodle already maybe you know how to run it very well but maybe you don't have a time and the resources to engineer a really scalable infrastructure to go along with it so we've kind of decomposed all the pieces that go along with traditional hosting and with leveraging the power of Moodle in mind so can we use all of those community plugins can we have our own flavor of it if we want et cetera and we wanted to be able to really control the timing of those two so that's another factor is are we being forced to take an upgrade before we're ready to take an upgrade or the flip side is I want to see how this plugin works now do I have to wait for someone else to do that or can I go into the system and it's on my time frame and so I guess to put it another way we wanted to write some tools that are very respectful of the idea behind Moodle and the open system so we wanted to be able to deploy infrastructure that models that and leverages that and that empowers users so and I don't know if I mentioned this so the idea is you can bring in your own cloud account you connect it to our set of tools it will deploy that infrastructure it will do the sites or we can also wrap it up like a traditional hosting so you don't have to worry about maybe you never log into the control panel or maybe it's in the middle it's on our cloud services and you don't want to have to worry about what's the bill going to be every month but you want to have that flexibility of I want to pick when I go and install plugins and I want to dictate when I can have a staging site that's copied off of my production or I want to try out this new plugin okay so I think that leaves us five minutes for questions did I get that right? eight minutes wow I was really efficient questions yes what do you say do a code and do customization we do so so one of the features that we've built into our control panel is by default it would install from the Moodle public repository but you can go in there and say actually I don't want to use that one I have my own code repository could be a public one or a private one if it's a private one you put in your key and then it will connect to that and you specify which branch you want to deploy and then it will track to that branch so whatever development workflows you have if you push a new bit of code to that branch the next scheduled upgrade cycle it's going to do a get pull it's going to run the Moodle upgrade from the command line and it's going to clear your cache so you have a clean environment that's ready for you to try out that new code that make sense yep and I'm happy to like actually show the UI to you at the booth and how all of that works other questions all over one of the beauties of the way that we do our deployments every client is in their own cloud account whether it's on one of our cloud accounts or whether it's on your account and that infrastructure that you deploy is dedicated to you so that makes it very easy for us to say what region do you want to deploy to so I have for instance I have one client who they have a physical presence here in Europe they have a physical presence in the US and they have a physical presence in Australia and we have infrastructure deployed in each of those continents yes no I'm happy to answer any of that so what we use on Amazon Web Services is RDS which is a reliable database for reliable database service and so we felt like they did a pretty good job of service delivery rather than just rolling our own virtual machine for that we'll deploy RDS so for instance it's very good with it automatically has backups it's a checkbox if you want to have a master-slave relationship and if you want to have it in a different region for high availability so it does a very good job of that and then the flavor that we like is MariaDB so that's what we deploy future looking we want to do some testing with Aurora of MySQL and there's a serverless offering for that that will actually kind of grow and shrink so we're very keen to see if we can get Moodle running on that because that solves a difficult problem in scalability which is the databases the size the database is yeah other questions yeah yeah yeah so we designed it so the client can pick how they do that so we deploy a stack a stack would be a load balancer at least one EC2 front end and one RDS back end and then you could choose to deploy only one Moodle site to that stack so let's say you wanted to do testing like a load overload test condition sort of thing then we'd put that on a separate we'd deploy a separate new stack which would take about 20 minutes it's very quick yeah I think about like it used to take like 6-8 weeks to start from scratch and it's 20 minutes now so but some clients would say well for our staging site we really don't need separate infrastructure we really just want to kick the tires make sure that check on a new version of the software and make sure that we don't have to go back and do some training or that the upgrade fails so in that case we'll deploy multiple Moodle sites to that same stack so like I have the stack that we use for like our staging active development project I probably have 10 or 20 sites going on a really small stack but currently we do one database per stack rather than trying to attach but the way that we deal with that is we try to right size since those pieces are split out we try to right size the database versus the EC2 instance so we can independently size those so we don't necessarily have a need where we have a benefit of aggregating a bunch of front-end stacks to the back-end database does that make sense? yeah we would do multiple front-ends in the case of like us where we're trying to scale up or dynamically scale up and down yeah other questions? yeah so that is one of our future targets that we would like to look at is BareMetal I have worked with OpenStack in the past I'm trying to think there's a Red Hat solution that's similar to that I'm forgetting the name, product name right now but so I've done on-premise scalable solutions the way that we've decomposed that a lot of the intelligence would connect to BareMetal so for instance once we deploy those instances we're actually communicating with Ansible so a lot of what's happening is actually intelligence in the control panel and so it would take a little bit of additional engineering work but it could be tied into an on-premise system like that yeah so you're working with Ansible yes so we have a lot of recipes that are very Moodle specific so we try to start with the I would call it a scripted install so we start with the AMI base image and then we layer everything on top of that through Ansible yeah other questions go ahead you now said that you could think about scaling out the job there if you're interested just in the database do you have an idea how it would be possible to scale up the database to have really multiple databases for instance yeah so that's what Aurora does in the back behind the scenes is it's actually a cluster of whatever like we work mostly with MySQL but you can also deploy instances behind the scenes it's actually doing those multi-read multi-write configurations of the database so that's where I'm really really interested in the Aurora serverless because it's basically going to bill you kind of on a per transaction so how many queries am I running to the database and the ideas in the background it's growing shrinking how many the database cluster yeah we have to write this up yes thanks everyone and I have cards too if anybody wants can everybody hear me perfect from Codecrate I'm one of the founders and Codecrate is basically a tool for programming education where you can increase the feedback to students and by automatically grading or manually grading I studied computer science at the University of Amsterdam and that's also where we developed this tool together with three other guys and so we had a lot of issues as students we never got any feedback and when we became teaching assistants we found out why because if you have to grade 50 assignments every week giving feedback on all of those so really we had some challenges in ICT education and it was the following giving formative assignments to students in ICT education is very effective but it's also very time consuming to grade because you want students to learn by doing and that's why you need to give them assignments every week but yeah if you have 500 students in total it's very hard to grade all of those the second part is that feedback has to be timely so if I go back to the example when I was a student I got my feedback and my grades after four weeks yeah and then the course is almost over so you don't really get anything out of it anymore the third part is that feedback must be accessible and qualitative and what I mean by that is that if your students have to click through five pages to access their feedback they probably won't see it because they're lazy and the quality of the feedback must be high of course as well yeah so to create a tool for that that's also some challenges and the first one is flexibility because you don't want teachers to have to change their entire workflow for like to use a tool so it should really simplify their workflow and not really completely change it the second part is a tool should be scalable and secure and that's mainly because well if you have a thousand students handing in all at once it should still work and the last part is how you can you give qualitative feedback for example with automated systems and it should really improve the quality of that so before getting to what we did to hopefully solve all of this let's look at the Moodle workflow now of making handing in and receiving feedback on coding assignments so usually when you create a coding assignment you use your favorite editor, you write some code and that's straightforward and then you'll submit this so in this case with the Moodle, so basically you upload the file in there and then the teacher can download it in some way and then the third part is you have to wait for your feedback and as I said before it usually takes a very long time and then the step four is basically viewing the feedback and that's going to the assignment then when you scroll down in Moodle you can see it great usually the comment and usually something in the line of not everything works you get a six so let's try and improve this and so the current workflow is not really optimized and to improve that we formulated two objectives the first one is to decrease the time of feedback from weeks to minutes and we do that using automation and the second part is to increase the quality of feedback and especially the manual feedback and we'll look at this in our session tomorrow because we only have 15 minutes which is also in this room so we've created an automated grading system which is called other test which makes feedback immediate and continuous and how it works is basically we integrate with Moodle using LTI basically clicking on assignment and then code grade loads you can see your rubric here and all of the automated grading works together with the rubric so you select which category you want to fill in by the automated grading system another thing you can see is handling instructions so you can give your students some instructions of which files to hand in and then you can go ahead and upload so you can either drag and drop some files in there or you can use github or github to upload and once you press submit the automated grading system will automatically run and you will get feedback within a minute and you can see that here so it started running and then you can see some tests that are going to pass or are going to fail you can also see it fills in the rubric automatically so if we close the rubric here you can see some tests passed some tests failed and then when you click on a failed test you can see why it failed so in this case it was only a simple input-output test where you can see what they expected wasn't actual if you go to difference you can see there's something wrong with this code at the moment so what the students can do now is they can hopefully fix their code and re-upload infinitely many times before the deadline to improve across through the code so let's quickly look at how this looks for the teacher because for the students the workflow doesn't really change much they still hand in through Moodle but let's look how easy it is to set up this automated grading environment basically you go to the management page of your assignment within Moodle you can go to the code-grade page automatically and here you can set up your tests and you can do that basically first connecting it to a rubric category so in that way you also increase the feedback a little bit because you're forced to group your tests and then you can create tests so you need to create a simple IO test so that's just input and what I expect is output but you can also do much more advanced tests because we basically provide an infrastructure where you can basically upload any grading scripts or unit tests you might have and just execute them and they'll return a grade and you can do that with for example a capture points test in this case so this way you can just execute your program you just upload it and capture a grade from that and in the back end we use a scalable cloud infrastructure where every student gets his own Linux server so in that way it's secure, scalable and you can execute anything that runs on Linux so also to give a little bit of a heads up of what we'll talk about tomorrow is how you can give students personalized feedback after the deadline and basically your students will see that in the following way they navigate to their assignment with a Moodle and then when they open it you'll be able to see personalized feedback comments in code grade and you can see that here and the teachers can give these very efficiently but you'll see that tomorrow so let's quickly look at a more practical example how we use code grades at Erasmus University in Waterdam to transform their education from a more formal exam to form different assignments so basically without code grade they only did a final exam at the end because they have 500 students per course and now they can do formal PIP assignments so a weekly assignment where the students get immediate and continuous feedback from code grades during the week of the assignments and then a few days after the deadline the teachers will give some personalized manual feedback to the code grade that they'll be able to see and then finally to wrap it up we have some other features so the first one is the powerful flexible automated grading environment where you can give immediate and continuous feedback to students you can give inline feedback in the code you can use rubrics which integrate with the automated grading system but also with manual grading system and that way you can also create a hybrid not creating everything automatically but also use manual grading we have plagiarism detection tool which is designed for code in fact with Moodle via LTI or any other LMS and in the spring we'll release a powerful analytics dashboard for teachers and peer feedback for code so I think that was it mostly so I'll now take some questions where are your servers located because it's running on your infrastructure so we host all the code grade servers for our clients and they are located in the Netherlands and Ireland all within the European Union to comply with the GDPR any other questions? I think that's it then thank you you So there's an optional break now. So we are going to need parts of their files, and then we are going to put them in the world class. Very nice. Very nice. Very nice.