 So, I'm here today, my name's Ann O'Shea, and some of our partners are here too, so Frank Dochnie from Athlone, Fiona Lollis from Dundog, Christine Kelly from Mnuth, Katrina Ni Hei from Mnuth and DCU, and Ciarol Macabord from Mnuth. Our project is a partnership between the four institutions, Athlone, Mnuth, Dundog and DCU. We do have 11 people involved, people from mathematics, departments engineering, departments, and also a computer programmer, Christine and a research student. So, just to remind you for the people who weren't here before maybe, our project is about trying to combine formative assessment and technology and finding ways to do that in a sensible way in a first year maths course. We used Black and Williams' definition of formative assessment, and what we took from that really was that it was important to get good quality information to both the teacher and the student so that both could make decisions on teaching and learning quite quickly and move things forward. So, we felt that technology had advantages here to help gather information quickly, process it and give it back to the people who need it. Also, for mathematics in particular, technology is useful in removing the burden of computation that a lot of students fall under, and to try and give them opportunities to use mathematics and technology in the way mathematicians would, to try and explore and experiment. So, I'm sure you'll remember our project has two main parts. One is looking at developing an audience response system, and the other is looking at developing interactive tasks and resources for use inside and outside the classroom. A main part of our project also is to evaluate these resources, and that evaluation is going on at the moment, and we're making the resources available on our project website as we review them and make final versions. Just to let you to remind you of the audience response system, this is our uni Doodle app. The idea is that students and lecturers have tablets or smartphones that they can use in the classroom. The students download a student version of the app, and the lecturer has their lecturer version. The lecturer can send a question to the students in class, and the students can draw a graph, do a calculation, draw a diagram, send it back straight away, and the lecturer can get an idea of what the class is thinking. So, here's an example. The lecturer can display all the students' answers, or focus on one, and use that to spark the next discussion. We have this available in iOS and Android at the moment, and it's working in quite a lot of places. We started trialling last year in Manuth in the first semester, then the second semester we used this in two modules with quite a lot of students in DCU, and another module in Manuth. At the moment, it's been used in 20 classrooms, more than 20 classrooms all around the world. People in Australia are using it in South Africa, lots in Europe, UK, and in about eight or nine institutions in Ireland. So, our impact as well as the use of the Unidoodle app around the world, we've seen that we wanted to see what the students thought about this, so we gave questionnaires to the students in these modules where it was being trialled to the ones last year. We had focus group interviews with students and we also got lecturer feedback. First of all, using the app, most of the students, the vast majority, felt that it was very usable, they didn't have any kind of technological problems. We also found that we didn't have any problems in classrooms with kind of Wi-Fi or technological issues like that, at least where we were working. I know one of the questions last time was, what if people don't have a smartphone? It turns out that I'm the only person in the world that doesn't have a smartphone, but 98% of our students always have one with them at all times, and it's the most important thing to them often. We asked them about formative assessment. We didn't use the words formative assessment, but we asked them things like, did they think it allowed them to get feedback, and what did the feedback do for them? So, lots of them felt that they were liked to get the instant feedback, they found that the feedback helped their understanding, helped them figure out where they were and also developed their own understanding, and also that it allowed, they realised it allowed the lecturer to know where they were. One thing, and I'm sure it's a problem with a lot of these things, that once the students get their phones out, then they may not be so interested in what you're talking about, they might be looking at Facebook or something like that, but I think that's something we just have to build into the classes ourselves, the lecturers have to deal with that. The student focus group themes, I'm not going to say a lot about this, a lot of the issues where the students felt that it was good for engagement, that they realised that they were kind of paying more attention, they knew there was going to be a question, they were going to take part, they liked getting the feedback and they liked the lecturer being able to see whether they had problems with something or not. The anonymity, there are no names when these things go up on the screen, nobody knows who said what. They liked that because it took the fear out of being wrong and in mathematics that's actually a really big problem, but also they liked seeing that other people were wrong and they weren't the only people that had a problem, and actually I think that that's actually, we know from our own experience in our support centre, that's really important because everyone thinks I'm the only one who doesn't know this. So if they see lots of people are having trouble and the lecturer spends time then working on that, it can really help. The other thing is they felt that it gave them a voice, it gave them a chance to say what they thought or what they thought, their understanding of a specific concept. The lecturer themes are very similar, engagement, feedback, anonymity, the lecturers were saying because it's anonymous, if you ask a question in a class of 160 people ask a mathematics question, there are very few people brave enough to put up their hand and give you an answer, but if you ask a question in this form you get loads and loads of responses because people are more happy to give their response when it's not their name tagged onto it. The one thing the lecturer has said, and I think this is very important, that it's really crucial to use this properly to ask the right kinds of questions, so you can't just randomly go in and ask a question, you want to figure out what's a good thing here, what do I really want to know, how can I use, you have to know beforehand roughly what the output is going to be so that you're prepared to use it in a good way. So I think that kind of ties with the other part of our project, which is trying to design good questions and resources for use inside and outside the classroom. The other things we're doing, I've listed here through all of them because I'm going to talk about some of them about the evaluation that we've been doing, but we've been designing tasks, designing assessments of different forms. One new thing since the last time we spoke is that in the task design part of the project I've been working with some people in the US who are doing something similar and they had a game that they had designed for their own students that was just a paper based game and we thought maybe it would be good to try and develop a smartphone app again for this game that students can use to develop their conceptual understanding of functions. So Christine has been, she has a prototype of this at the moment later if you have any questions. The evaluation of these kind of resources, last time we spoke about using, we were reusing some Khan Academy materials and we were designing Moodle courses. The feedback we got from students was really useful in redesigning what we did with those, so we have been working on that and some of the things we've done has been including the numbers, using numbers that the project from Cork has been using. We thought that some of the things they did would be useful for us, we've done some of that too. We re-trialled the Khan Academy playlist tying everything much more closely to the course and to the assessment in the course and we had a much higher engagement from the students and we saw significantly, statistically significantly higher marks for the students who engaged with these materials than those who didn't and these were all correlated positively with the scores they were getting. Their final scores were correlated with the scores they got on the Khan Academy quizzes. The other projects, the screencasts, so you'll probably remember that the idea here is that students get quite a difficult problem to work on instead of giving their solution in paper form. They make a little screencast about it so they can explain it to their peers and the lecturer and that's what's assessed. We had the students in a computing and games development course doing this in Dundalk and the students did a really good job and we asked them what they thought of this whole project and they felt that making the little screencast was something that forced them to think deeper about their own solution and to have to explain, it's like the old thing that if you want to know something, teach it. I think that this was something that they were finding out that having to explain it to somebody else made them understand or think about it in a different way and also they appreciated being given a voice that they could explain their own solution and it was kind of theirs that we had some buy in. This is being currently retrialled in Dundalk at the moment and we're going to have more data from that in the next few weeks. The task-based interviews, we looked at the interactive tasks that we designed through Geotubra and we had intensive interviews with five students in DCU last summer. We've gathered a lot of data from that. We gathered the data from their screens so we know exactly what they were looking at, what they were clicking on and what they were doing. We have audio recordings and written data also. We are still working on the analysis of that. It's actually a big job but the immediate things we can see, as we can see, we gave students a pretest and then we let them work on similar topics using these tasks. We did see progression. We saw some students having real aha moments where they kind of got something. Not all students but we did see some of this and we definitely saw that being able to visualise, being able to have graphs and things moving and being able to allow the students the opportunity to do all of this exploration themselves seemed to help them. So on summary, the evaluation of these resources, we have seen the technology helps with things like immediacy, getting instant feedback, definitely with engagement in class and outside and it helps students in mathematics definitely to be able to use mathematical thinking skills like visualisation and generalisation. We're still evaluating, we're still carrying out interviews and focus groups and we will continue to do that and analyse the results but we have seen definite indications that conceptual understanding and people's disposition towards the subject can be influenced by these tasks and resources. So other forms of impact we've been giving workshops and talks in Ireland so these are some of the ones we've looked at, we've been at in the last few months and abroad over the summer there were quite a few presentations at international conferences in Europe. We have had conference proceedings published and others are in preparation and we have submitted so far three journal articles. We are preparing others on the evaluations of all of our resources. The other things we'd like to talk about are for sustainability. Well the Unidoodlop is available on the App Store and Google Play at the moment and we have seen it's actually being used in a lot of places and there was a lot of interest when we gave presentations at these European conferences in the summer. There were a lot of people who came up afterwards and were interested and some of those have actually used the resources now and are trialling them for us. We are also developing our website, it's really not developed yet but we're starting to put things up as we revise our resources. We're putting up the final version so we have some up there at the moment. We're also in the process of developing kind of a lecturer's guide to developing and using these resources and those will be ready by the end of the project. One of the things we did at the very beginning was we surveyed both lecturers and students to ask them what do they already use and what would they like to see. So some of the things that they already use and some of the things that we've kind of checked and thought they're really nice things we've linked to those on our website as well. The other thing I suppose is the Unidoodle app, last time we were here you said maybe you should think about going commercial. So Christine and Seamus who have been really responsible for developing this, they were applied for funds from Enterprise Ireland to carry out a feasibility study into commercialisation. They've got that money now and have employed a consultant who's going to look at what else is available, whether there is a chance for this to become a marketable object or whatever. That report will be ready in the next few months and at that point we will apply for more funding from Enterprise Ireland or wherever to see if we can make this a commercial project. So that's all, I'd just like to say thank you to the forum for giving us the opportunity to do this work and thank you to my team who have been amazing. Thank you.