 Okay, my name is J. Merello, M.J.J., J. Merello, in Cipán. Thank you very much for being here. This is what I'm going to talk about. This is actually a drawing made by Paola, who was a student in my class this year, an undergraduate class. This is representing pretty much the feeling of my students when they were submitting their assignments to the system. So this is a continuous interest system. It was scanning. And they were holding for their life to get half just waiting for the assignments to get through. Okay, let's pick up. This is what I'm going to talk about. I'm going to talk about testing for testing. So I'm a professor in the university. I'm not going to name the university. I'm not very proud of my university right now. So let's just say I'm in academia. And I'm talking about my experience of using continuous integration in the classroom. Right? I'm also in Granada, in Granada, but we are going to have a workshop, a fair workshop in Granada, in a couple of weeks. So if you love going to Southern Spain, you know, Guam, nice place, it's going to be an incredible place. The caliper paper is still open. If you're interested, just let me know. This is what I have been doing. It's a cloud computing class. It's an undergraduate and also master level cloud computing class. This is my actual class. So that's me. So this is not the class I have been giving this year. It was last year. In the undergraduate class, I'm having a lot of fun. In the undergraduate class, I'm having around 40 students, something like that. In the master's class, I'm having around 12, 15, something like that. So it's a smallish class. It's not too small, but it's smallish. And I'm using something called flip learning. Have you ever heard about this? Never? So it's learning as people actually learn. There's a saying that goes, you don't teach, people learn. So I think that it's pretty stupid for someone to be standing in front of a class for two hours flipping slides like I'm doing now. But the thing is that what you have to do for people to actually learn is to make people learn, autonomous learning. So you give them a series of objectives. You give them material. You give them some class notes or whatever. And then you let them learn. And you are there guiding them, helping them or whatever. So it's not like I'm standing in front of the class for two hours. It's like, I say, I put a slide with the objectives of the particular session or the week. And then I wander around the class telling people, who are you doing? What are you doing? Do you have anything to ask? Do you have any problem? I see them working with the material of the class. I say, OK, I see you are assembling onto this. Please check that out. So it's a different way of giving class. That's not really related with the rest of the thing. But I just wanted to put you a little bit in context. So it's a different way of teaching. As far as I know, there's not many people in my university who's doing that. There's not many people in other universities who are doing that. For me, it's been just wonderful. And I keep asking the students whether they want this kind of class or what is called magistral class, the traditional one. And they keep saying, 90% of them, no, no, we want this way of giving class. Of course, they can wander around. They can get in class at whatever time they want. They can get out whenever they want. So I mean, for them, it's quite relaxed. It's not so stressful. It's nice. And then another thing I wanted to do is to be as real as possible. So I don't want to give them assignments which are just boilerplate things where they have to fill in the blanks and they say, OK, this is OK. So they eventually end up filling the blanks and not learning anything at all, but trying to make the thing as real as possible. So not like these astronauts who are in a carnival parade, but real people doing real deployments in the cloud because it's what I'm teaching. It's cloud computing. So at the end of the day, they will have to go to a company or whatever to get their own company. And they will have to do deployments in the cloud. So I want them to do real deployments in the real cloud. I think that class is going to be a metaphor for real life. So when they will do deployments in the cloud, when they are developing software in a company, they're going to undergo code reviews. They're going to use continuous integration. They're going to do all kind of shit that you're doing in real life. But why not use it in class? So in class, you will have to use code reviews. You will have to have continuous integration. All that kind of thing. Then I use GitHub. I've been using GitHub for a long time. So like, I think like four years or so, I mean in class. Yes, please? You're going to ask something? Oh, OK. So GitHub is my learning management system. So everything they do, it's in GitHub. That's absolutely amazing. It's very nice. So this is the actual page they use for turning in the assignments. So what they have to do is they have all their stuff in a ripple such like this one. This is the actual page that I have been using here in master class. So they put their project. They do milestones in a project. Ito means it's not here. Somewhere. Ito is milestones. So they do milestones in a project. And they submit each milestone. When they are going to submit the milestone, what they do is they change this particular file, which is in my repository. It's in the repository in class. And they add the direction of the repository. It's always the same. Also the version. The version is important. We'll see that later on. GitHub is very good. It's amazing. It's much better than any other LMS you may think of. I'm talking about this because I think that everyone, at one point of your life or another, you're going to be a teacher. You're going to teach something. And since we are in technology, since we are in programming, we'll probably have to do something like this. We'll have to, you know, onboard new people in our company. We will just, you know, we are between jobs. And we go to some community college to give class, whatever. It's something that you have to do. So my intention was to use this kind of technology and to see what kind of things could be done to make your experience a little bit better. So I have been doing this for, as I said, quite a long time. So the people were turning in assignments. And I was checking out this page. And I was correcting it. All of a sudden I had a new beginning, 2017. You probably know that I was director of the Free Software Office in the University of Grenada. That took me around 25 hours a day, 25, 26 hours a day. I was fired. I still keep my job as a professor because, I mean, Free Software Office was an appointment. It was in a job. All of a sudden I had a lot of time in my hands. At the same time that appointment implied a reduction in the number of classes I had to do. So before I had a teaching assistant at TA. All of a sudden I had no TA at all. So I needed to do all the things that my TA was doing. My teaching assistant was doing. And then I thought, continue to say integration to the rescue. I'm going to use continuous integration. So I'm going to use the milestones as code review. So I'm actually going to review what people are sending. I was doing that a little bit before. But not very stensily. Basically what I was doing, it was checking that there was no conflict. So when people were modifying the file, they were not conflicting with each other. Maybe if I saw something coming very, very early, I took a look at it and told them maybe you should do it this way or that way. But all of a sudden, remember, I had no TA. So the TA was checking things. So I said, per, it's my TA check. So I was using per for doing continuous integration. All the checks that were done before by the teaching assistant, I'm going to do it using continuous integration, which I wasn't using before. So before, people were doing pull request. I accepted all of them, except there was no conflict. Now I was doing something completely different. I was writing tests for what people were submitting. So might fly or not. It says, all checks have failed. I'm not accepting that pull request. So that means that the student has to go back to his repository and fix whatever was being done before. What are we checking? It's important. Taking into account that the product is completely free. So they can use whatever language they want. It's not boilerplate. It's absolutely not boilerplate. In fact, people were using, most of them were using Python. But even the people who were using Python, they were using different frameworks, because they had to deploy web service to the cloud. Some of them were using Flask. Some of them were using Hug. Some other people were using Node.js, and they were using Express. Some other people were using Ruby, and they were using either Sinatra or Cuba. I think that, I'm not sure. I think that both of them were using, one of them was using Go. No framework at all. So it had all kinds of different languages. I didn't have to look at whatever they were doing, but the effect, the actual effect, the real thing. So if you're deploying a web service, I want to see if it's working. So what do we check? The first thing, and this is quite important, is that the assignment is, you have only modified a single line in the file. Because they have to modify a file, and sometimes what they did, sometimes they had an old version of the file, so they changed things, and they submitted the wrong file. So I had to check that actually, there was only one row in the table you have seen before, that was to be modified. So I could extract a single URL from that thing. That's the first thing to check. So if they had a wrong version of the file or whatever, it was caught right here. And so the repo URL is correct. So it's a GitHub repository, and it's actually correct. So people sometimes submit something with typos or whatever, and it's not working. So that's the second thing I have to check, if it's actually correct. I do it with spare, right? This is the first thing I do. It's the first test they are passing. So the first thing is I use this Git module, which is, it's a weird thing, because it comes with Git, but it's also in Cpang. So it's a module that's called Git, and what I do is I get a diff, so I get a diff between the previous version of the file and the version I have now. I check that it's only one line, right? And then I also check that they have modified the correct file, all kind of things. If it doesn't follow that regular expression, they fail. But if it follows a regular expression from the diff file, I extract the information on the repository that I'm going to check later on, right? Right now, the only thing I'm using is this Git. Second thing, is it a single line? That's important, because if it's not a single line, there's something wrong with the file. There might be a conflict. There is another subtle error that is on time. They change the codification of the file. So when they get the file, they edit it. Sometimes they change this codification from Latin 1 to UTF8 or the other ground. So they change the whole file. That happens quite a lot. Sometimes I had to fix the whole file, and it's been quite, quite, quite really bad. The worst thing was to find out who was guilty for that and maybe take out some credit or whatever. But the important thing is, is it a single line? So are they modifying a single line? Because that's kind of a sanity check. And then, of course, you have to download the repo because this is all being done in, let's say, the class repository. It's a single repository. It contains the check and everything. Then you have to download the repo and then start to do things on them. So I am using the system git. I'm using Travis. So there's a git there. I mean, it's installed already. And, well, I start the user of the name. I put everything into a temporary directory. It's Travis. I mean, it doesn't really matter what happens. I don't think about using something else next year, but we'll see. And then eventually, we create the repository and we start to get all the files in the repository using git commands, right? So far, so good. Follow with me. Any questions so far? OK. Then I'm using test more. Very simple test module. You know it already. You probably have used it. I could probably use something else, but I didn't want to put more things into the Travis because just imagine, people are physically waiting for the outcome of the thing to get out. So the more modules I use, the more they have to wait. And sometimes I wait, too. So I get an email telling me that somebody has some Apple request, and then I go there to see if it's working or not. If it's not working, I try to guide them towards doing it a little bit. But basically, those two things are the only per modules I use for the time being, right? As I said, I am using Travis. It's a relatively simple configuration file. It's not going to get any better. Anyway, below that, it's only prove-c. The only thing it does is it installs test more, test harness, git, file slab, and some other modules. It does prove because the test is included in the repository. It's not some external test. So the file, they are going to modify it to submit assignment. And the test, it's a single script. It's right there, right? So they start to check. This is the actual history, real history. You see milestone four, fail, fail, pass, pass, pass, fail, right? So they submit it. It fails sometimes. Sometimes it goes ahead. Of course, when that happens, if I am available, I see it and say, OK, it looks like this is the thing that has failed. Take into account that I write the name of the test. So sometimes I realize that the name, the test that has failed is not too informative. So I go back to the script and change the message that is issued. It's saying, OK, it's the JSON in your service is not right. Whatever. I try to make it as informative as possible, right? So that they know exactly in which step they have failed. I'm also checking presence of files and their access in the repo. They have to do something like, first, they have to deploy to a pass, like Heroku. Then they have to deploy to another different pass using containers. So I have to check whether there is a Docker file, for instance. Then eventually, they have to submit something using some deployment tool like Fabric or Capistrano or whatever. So I have to check whether there is an actual file that does that. I have to check whether there is a background file. I have to check whether there is a readme.md. All those files, I check it, right? At the very beginning, I check three things. First thing is they have a readme. Second thing is they have a license because this is all free software. So whatever they do, I tell them from the get go that it's free software. So they have to understand that it is free and they have to respect the license and everything. So there must be a license file. And then they must have a .gittig in our file because it's something saying and it's a best practice for. So that's kind of the thing to do. You can see all that in the directory. And then I get the readme and I tell them, you have to tell me where is your Docker file in Docker Hub using this format. You have to tell me where you have deployed your project using this format, right? And extract everything from that. So if they tell me, I have deployed to Heroku in this address. So I extract that using regular expression from the readme and I check it. I go to Heroku, I check it, right? Using LWP simple, right? Test files, sometimes files, for many different reasons. First reason is that usually people are not acquainted with this kind of thing. For your students, they have never used tests. They have never used Travis. They have never used Gitt, which is absolutely appalling. Actually, the first assignment is they have to open an account in GitHub and modify the file and do it with Gitt. So sometimes I have to give it. They don't know what's a web service. So they are in the last year of their undergraduates. They have no fucking idea of what's a web service. So they tell me, so it's like the web. I know PHP. I say, no. That's not so. I mean, they have all kind of things. Besides, you know, this is a parallel script. That's in a directory. And you can actually put that parallel script into your own thing, right? There is only one student that has on that. Only one. The rest hasn't done a single thing. They haven't realized that there's a per script. They can check to see everything that happens. Anyway, it's not a big deal. It's already in the description of the assignment. They say, you have to include this in the file. You need to have this thing. But, I mean, at the end of the day, they're using this as an actual test for their assignment. So they are actually throwing it, you know, like spaghetti to the ceiling to see whether it sticks or not, right? I do scraping. Why do I do scraping? Because at the beginning, something they have to do is they have to actively use milestones and issues. Milestones, issues, and commits. So they have to put every issue in a milestone. They have to close every single issue using a commit. So they have to organize the project. As I said, in the beginning, in real life, right? In the beginning, I was intending to use GitHub API. But you probably know that if you're using, if you're doing a pull request in Travis, you can't use environment variables. Because it could be a security hole. So I can't use any kind of environment variables to use my API, I mean, my GitHub API token so that I check the GitHub API. So instant API using scraping. I check milestones and issues and commits. It's relatively simple. For instance, you get the milestone info. It's just a very simple regular expression. You get if it's open or if it's closed or whatever. Then, as you can see this very well, but anyway, this is in my repository too. Anyway, what I do is also scrape the milestone page, I scrape the issues page, I scrape the commit page, I extract whether an issue has been actually closed by a commit or not. All that using very simple regular expression. Then I have to check IPs. I have to check REST APIs. I have to check URLs. As I said, they have to upload a Docker file to Docker Hub. So I have to check whether it's actually there, whether it's working, whether it's actually being built. This is actually a lot. You see that they are cloning, whatever, milestone one, milestone two, milestone three, then whatever. I think this is actually, you see that URLs, this is Amazon URL. This is site.co URL. Heroku URL must be up there. So all that is done. You have to do things like this. What I tell them is you have to deploy a web service. What they do with the web service is they need to have a root. That's a status that returns okay. Of course, most people, not most people, but a lot of people, they deploy a web service that does only that. So a web service is a single root. So we have to think about that next year. So maybe do some kind of, I don't know, maybe describe using, what's the name of that thing that you describe with services? No, it's not GraphQL, it's OpenAPI. No, OpenAPI. Swagger. Swagger, that's the thing. What I'm going probably to tell them next year is I'm going to try to do a small hackathon from the very beginning so that they program the web service, meaning different web services. They test them and everything. I will tell them to describe it using Swagger. For the time being, the only thing they have to do is to deploy it using a status. Which, I mean, it's not easy. It's not easy for them. They really have to, they have to really do that. And for them, it's something that they have never done before. I mean, my subject, my class, is not about web services, it's about the cloud. So I have to kind of, you know, teach them all kinds of different things I need to do at GraphQL. And something I do that was actually very nice. They have to use Amazon, AWS, or Azure for deploying a virtual machine. They have to tell me the IP of the virtual machine. I have to check that actually the virtual machine is working. So I use NetPing, which is incredibly nice. Just a couple of things. I use port 22 because that support that actually has to be running. You need to, at least that port that you can connect from outside. Actually, they have a limited amount of resources on Azure and AWS. And what they do is, as soon as it passes the test, they turn it down and that's it. So they save money. And they can use the rest of the money, you know, to mine Bitcoin or whatever. Thing is, it saves me a lot of time. So it actually is really, really helpful for me. When I get something that's correct, I know that it's got all the basic things that have to be there. So I can go to the beach. It's four kilometers away from my home. I can spend the day there. I can dip my toes in the water. Actually, this house over here belongs to the former queen of Belgium. It's called Villa Astrid. Belong to the former king of Belgium actually died there. And I have got an apartment there and I'm going to be not moving because I live in the city, but I go from time to time to check if it's actually being built because it's not being built yet. So I go there, I go to the beach, spend my Saturday with my wife and my daughters. I couldn't do that before. I literally couldn't do that before. I mean, it saves hours, literally hours. But I'm the professor and the whole point of doing this kind of thing is not for the professor to go to the beach. Could be, but it's not. Is it good for the student? That's a very important thing. Whenever you do something in class, that's the first thing and last thing you have to ask. Is it good for the students? Are they learning more? This is a graph of the number of students that have actually turned in a milestone. This is this year, and these were the two previous years. This is the number of students. So let's look at this one, for instance. I started with around 55 students. Second milestone goes down. Third milestone goes down. Last milestone was even downer. Students were dropping out of the class. They just said, no, this is not for me. I can't possibly do this. So it's not something I can do. Last year it was like this. I had less students because, of course, some of them said, this is impossible. I can't do that. So I got about half the students this year. This year I got even less. But I said, OK, I've got to do something. You see that the drop-out rate, the decaying rate of the number of students, is very, very mad. You are going to always have students that drop out. For reasons that are completely unrelated to your subject, because sometimes they are right. This is in Spain, the degrees are four years. I get the students, a lot of them, who still have classes and subjects from first year. Algebra, physics, because, of course, you need physics to do a web page or to do a web service. You have to first put electricity into those damn chips. All kinds of things. Some of them have some reasons I mean algebra is difficult, but they make absolutely no... I'm telling this because my daughters are doing computer science in my same school. So I know from firsthand what they do. Anyway, I get many students who have subjects from other years, so they have to go to exams, they have to turn in assignments, and they just say, OK, I just can't do this. We also have a telegram group, and I actively... OK, I'm going to put this might be... After every milestone, if I see that the student hasn't turned in his milestone, I ask him, what's the matter? Is something wrong with you? Do you need help? Please come by, get to my office, we can talk, whatever. I have been able to cite some of them using that. So I'm not going to say that this is precisely because of what I've been doing, but it probably helps. Funny thing is, I also use Perl for this. You have everything in the repository. So what I have done, I have published to Cipan this module. It's called Git repo commits. It's only to examine the commits. So what I examine is the author of the commit, the date of the commit, or whatever. So I can get very nice statistics of what's going on in my... And this is also... In general, this is called learning analytics. It's something that lots of people talk about, very few people do, precisely because the problem is that the platform that people are using, which is Mudo, usually doesn't allow you to do this. Mudo has a very nice API. Most universities, for security reasons, they close the API. In my university, it's even better because they close the API for everyone in the university, but they open it for a bank that has created the official app for my university. That's why I'm not going to name it and not proud of that. Just imagine. No single professor, no single student, no single staff in the university can access the notes in the Mudo using the Mudo API, but the database of grades of every single student in the university is open to a bank. Because, of course, that's not a security issue because banks are so fucking secure. Anyway, my point was that you can do learning analytics on Mudo, at least in my university. I can do very easily learning analytics using this parallel module, which is available in CIPAN. I just released version 0.1.0. So I got, for instance, I checked the milestone, which is basically a file. I checked the commits. I see who has been the cometer, and I can count the number of authors. I can count the date, whatever. I see things like this. How many modifications of the file have been done? This year, this particular year we have introduced this in green. The others are in blue and red. So the two previous ones. What do we see here? At the beginning, people were doing lots of changes. Every person was doing lots of changes. Why? Usually because of conflicts. They were still getting the handle of the system. They did conflict. This year, that didn't happen. There were no conflicts and I was checking that they were only doing a single line. So conflicts are almost impossible. It can happen if there is some race condition, so that there is a pull request, and there is another pull request, another pull request, I haven't accepted the first one, so the third one might be in conflict. That usually doesn't happen. In that case, usually I also fix it myself. But as you proceed with the number of milestones, this is the last one, you see that people this year, this particular year, they are doing more resubmissions. Why? Because they are submitting, they say that it fails, and they are doing it again, and again, and again, until it works, right? That's good. I mean, why not? They are learning. They are learning in the process. They are submitting something. They are saying, this doesn't work. So without compromising their grade, they are improving and they are learning. So that's something, it's a point I'm going to stress a lot. For most people, evaluation is just grading. Evaluation is learning. You have to create the process of evaluation so that people actually learn with it. You can just say, I'm going to do this exam in October and then in February I give you the grade. That's impossible. You don't learn anything out of that. Whatever you were doing there, you completely forgot. In this case, they are learning while they are, they have this epiphany, okay, I'm doing this wrong. In many cases, they are purely mechanic things like, you know, the URL is not in certain query, but in some cases, they actually start to understand what they, what's requested from them. And they are doing that through the test process. And this is a little bit difficult to explain, but let me see if I get it right. First thing, once again, green is this year, red and blue, previous years, right? Don't look at the dates. The calendar has changed anything. Don't look at the dates. Look at the shape of the card. First of this year, that means that one day before the deadline, they started to turn in the assignment. There's a few of them. The day of the assignment, they did bulk of them, were submitting it, and then after the assignment, of course, some people did it. They can do that. I'm kind of flexible with the dates, so if they turn it like on the late to the late, whatever, I say, okay, but it doesn't matter. If they do it one week late, I will put a penalty of one point, but it's no big deal. But the shapes previously were very, very, very sharp. This year, what happened? People, and this is just one milestone, it was the last one, but it pretty much happens everywhere. What you see is that people start to submit assignments way before the deadline. Why? Because they want to have time to fix it before it's actually graded. So they say, okay, I'm going to do this one week before, even two weeks before the assignment. I will just do it. If it's wrong, I don't care. The test is going to tell me if it's wrong. It's not going to go through to the professor who is going to tell me none of this fails. So you have to turn it again, right? Check it out. I mean, this is December 15th, so I think that the deadline was December 21st, so it's pretty much around here. Then during Christmas, people were doing turning in things. Then eventually, there was another peak here. So people were doing it whenever they saw it fit. So they were using it, they were incorporating the thing in the learning process. And they were also trying to do things so that they could have time to fix everything. They were learning much better. They were organizing their time much better. Learning outcomes were, for me, very, very good. And actually, students told me that. So I asked them, whenever they have to turn in a milestone, I always do a survey. I ask them what they feel, what they feel about the way of doing class, what they feel about the subject, whether they think that it's going to be useful for them or not. But in the last one, I asked them what they thought about this system. This says, it's not complete, but what it says is it helped me to submit something that's correct already. You remember when you were a student, you put it on assignment or whatever, you were waiting for the grades. Maybe one week, two weeks, one month, whatever, maybe before the final exam. That was absolutely appalling. People were stressed out because before the final exam, they didn't know whether, when they do this, except if they are actively trying to cheat, they know that they are going to get six, six or seven or eight out of them. They know that in the very moment that they have tournaments. So that's the thing that they like the most. They say, it complicates my life, four of them said that, or of 20 students, I can understand that. But sometimes it complicates the life, it's not ideal. It's because they didn't read the specifications correctly, so they put the URL in the wrong place or whatever. It helps me, I think this one, it helps me to be sure of the grade I have. So that's another of the value. So they feel empowered with this kind of thing. So they don't feel disenfranchised, like this is a machine, a machine is turning things, because after that I have to grade. But when I grade, I know that pretty much everything is in its place. The semester is over now. It's not over. Actually yesterday and today I'm still getting, because in Spain we have a regular exam and then we have a special exam. So some people who have failed because they didn't turn in their milestone on time, they're still, so yesterday I had to accept like four or five pull requests and today I saw a couple. So it got stuff to do. So I got my assignments for myself. First one, I want to create a SIPAM module. Why? Several reasons. First reason is I don't want the code to be in the same repository so that some smart student can look at the code. So they will have to go another steps, maybe to SIPAM and see what's going on. And anyway, the problem is I have two subjects and it was all hacked together. So I have a lot of common code for the master I have one escaped, for the other I had one escaped. I was moving things back and forth. Not nice. I also want to do tests of the code. So right now the test was the first student that was submitting their assignment. I told him, okay, hold on there. It's everything okay. I had to check this. It's okay. It's okay. So I usually give them a beta tester grade and also stickers because everyone loves the stickers. Everyone loves grades too. Stickers are nice. Then I want to send tests. I mean, think about it. You can do pretty much everything. You can check. You can do spell check. I don't know if I want to do this because spell check is going to be just terrible. I can actually check the commits. So I can check whether they are doing the whole thing like milestones and commits and everything. I should check using Swagger, the full rest API so that they have to declare the rules or whatever. Check for plagiarism. I have found lots of students who cheat. You know, people do that all the time. Okay. Lessons learned. You have to innovate, but only to improve student experience. You don't have to innovate to save time. It's good, but you have to do it to improve student experience. And you have to check if it does. No, you look at the students. They seem happy. They smile. They like the stickers. No, no, no. You actually have to ask them anonymously whether you are doing okay or not. The scripting goes a long way. And that's it. Thank you very much.