 There were supposed to be three of us here, but Brian has got a bit delayed. We're a bit ahead of schedule as well, so he may well turn up in the middle. Catrina here is our research student and a lot of the work that we're going to talk about today has been done by Catrina herself. The partners of this, as you'll all remember, are Manuth, Athlone, Dunedoc and DCU. We're a mix of mathematicians and engineers. We have Katrina, who is a research student, and then we had employed a programmer, Christine, who finished at Christmas, but still, even though she's not part of the project anymore, is still continuing to work on aspects of the programming that's going on. So, just to give you a reminder of what this project is about, it's kind of a two-part project. So, we're looking at formative assessment using digital technology to aid formative assessment, and we had two parts, formative assessment that might take place inside classrooms, and that's really our audience response system, the Unidoodle app, and then formative assessment that could take place outside of the classroom, and that's the other part of the project, designing tasks and resources that students could use when they're not in front of a lecture. What we're working on mostly at the moment is tweaking some of our resources, but mostly evaluating what's going on and trying to do some research on the impact of them. And these things are on our website, I'll talk a little bit about that in a second. We're using Black and Williams' definition of formative assessment, so the important thing is it gives its activities a given information to both teachers and students about levels of learning, and that's used then to change the teaching or learning activities to help to provide deeper understanding. We saw that technology might be used in a few ways for formative assessment, that the technology could help gather information, process information quickly, so that students and teachers can make decisions fast. An important part for us in mathematics was that the technology might be able to remove some of the computational difficulties that students might have and allow them to just kind of explore and experiment a bit more, and also that the technology can be there when the lecturer isn't. So students can be supported, they can be doing stuff on the bus, on the way home, on their smartphone or in a lab or wherever, but it's not just when there's a person available that they can get feedback. So we designed resources using lots of different kinds of software, so we used some resources that existed already, so lots of stuff from Khan Academy. What we did there was looking at taking some resources, packaging them for specific courses and specific students, and we did that in two different ways in DCU and Athlon, looking at a course built on these materials that would be outside the curriculum for material that students might need but hadn't seen before or hadn't mastered, and then built into courses as well. In Athlon we also had our... In Dundalk we had lots of stuff about students making their own screencasts to explain what they were doing. Matlab was used with engineers in DCU. We designed tasks using Jojabra and Numbus and quizzes on Numbus as well. A lot of this was implemented or shown to students through Moodle, and then Unidoodle in the centre there is the software, is the app that our engineers in Manuth developed. So, just quickly, the idea I'm sure you all remember is that you've got an audience response system, so instead of clickers, which aren't that useful for mathematics or engineering, students and lecturers download and up. The lecturer can send students a question in the middle of class or just a blank screen. Students can answer the question, can draw something, can do a little calculation, send it back. The lecturer gets an output on their screen with the pictures of all the things that the students have sent in. Then they can see, oh yeah, everyone's getting it, or oh no, everyone isn't getting it, or they can pick one thing and use that as a stepping stone to try and move things on. So this is all available now for iOS and Android, and the Windows version is still in production. So Unidoodle was chosen as one of the 12 apps of Christmas. This is a project in DIT, but it's also got international buy-in. There was a lot of interest generated, so every day in December there was one of these apps highlighted, and Unidoodle was one of them. It got a lot of attention from the conferences and meetings that we've been at. We've also got a lot of interest from people in using this. So now there's more than 50 institutions around the world using the Unidoodle apps in the UK, in Ireland, Europe, and the rest of Europe, USA, South America, China, Australia. So it seems to have been popular, and people seem to get it and like it. I'll say more about Unidoodle later. The other part of the project then, this is kind of where we needed the extension because we wanted to spend a little bit more time on the first thing in our interactive tasks. We had done some interviews with students. We wanted to use the feedback from that to tweak the tasks that we had and maybe design one or two, some new ones. The other things are more or less finished. We've got the MATLAB practicals, the screencast projects, the Khan Academy material, the numbers quizzes, and we're just drawing that stuff together now. One thing we did at the beginning of the project was we sent out a survey to lecturers and students, and we wanted to know, first of all, what kind of areas of courses they felt they needed support on and also what kind of resources they wanted and what kind of resources they use. So we made a web inventory of things that students like to use and lecturers like to recommend. We also curated lots of other. We went off and checked ourselves what kind of things might be useful. So part of our website is not material that we've developed ourselves, but things we recommend that we think are useful. So on the project website, this is the front page, it's what it looks like, so we've got resources developed by ourselves and then resources in the kind of major topics that first-year students see. These are things that other people have developed. The resources we've developed for ourselves, they fall into the categories like our Geogebra Numbus, Khan Academy, Unidoodle, and so on. And then if you click on one of those pages, say for Geogebra, you get a long list, well, not so long maybe, but increasing list of tasks that we've designed. If you click on one of those, here's one. I'm not sure if you can see this or not, but the idea of a lot of these tasks is to use technology to take away the kind of problems that students might have with calculation. So we've got these, this task is looking at vertical asymptotes, so looking at, and this might not mean that much to you, but looking at a function like AX plus B over CX plus D. On Geogebra, they can change the numbers A, B, C and D, and they can see the behaviour of the graph changes and then you can ask them questions about, make conjectures about what you think will happen and things will happen and so on. So what about our evaluation? This is what Katrina is really spending a lot of time working on. We've tried to evaluate our resources, asking students in focus groups, using task-based interviews and questionnaires. We've got feedback from staff as well. We've also been able, in some cases, to link these, the use of resources to end-of-year grades and look at student achievement. We reported on quite a lot of this last October, and the feedback is basically quite positive. Sometimes, especially in the task-based interviews, when students were actually working on tasks, we were able to see that this would be easier for them or this would be better if we changed it. So we've spent some time trying to make those changes and the analysis of all of this data is ongoing and Katrina will be submitting her thesis in March or April, so it will definitely be finished by then. One thing we did, we designed questionnaires for each of these different kinds of areas, and we had some questions that were in common, same kind of question on these different topics. Lots of questions then were targeted at the particular thing. It wasn't really, you couldn't really give the same questionnaire to students who were using the uni-doodle-lapping class and students who were using the geogibre tasks at home. They're different things, but there were some questions that were in common, and one thing we want to do over the next two months is try and pull the common questions together to try and get a feeling. Now, we have to be careful because these weren't the same students. It wasn't the same student answering questions on all of these different resources, but we want to get a feeling of what it is students like and why they like it and what it is. Does it help their confidence? Does it help their learning? Do they think it does? What's the usability of these resources like so that we can learn from that, from the comparison? Overall, we've seen definitely there are benefits of using technology. It's not an easy thing to do, but there are benefits. The fact that you can get immediate feedback, that it can help engagement, and especially in mathematics that we can help with visualisation and generalisation, and our evaluation is still ongoing, but we have seen instances where students have had this kind of aha moment that they've kind of got something in the middle of working on one of these tasks, and so that's kind of what we were aiming for. But the thing is that as you probably realise, our project is kind of quite spread, so we have the in-class stuff, we have the out-of-class stuff, the out-of-class stuff has lots of different components. We wanted to find some way of pulling them together. So we wanted to look at a framework that would do that, and I had been working on FASMED, which is an EU FP7 project, which was looking at formative assessment and technology in secondary schools for science and mathematics, and part of the work of that project was to develop a framework, and that was for secondary schools. We thought that it was useful, but that for university level needed to have some tweaks, so we've developed what we've called the termed, or termed, we're not really sure how we're going to pronounce this framework. So technology-enhanced resources for maths education framework. Now this framework is to try and pull our resources together to try and give us a picture of what they are doing and what kind of dimensions we need to get good resources. So it's a treaty framework, and these are the dimensions. So the dimensions really are the people, are the things that are giving feedback, they're the agents, and they're taken from Black and William with our tweak is to add this kind of digital element. So there's a student who can be giving feedback, they can look at self-reflection, they can be peer feedback, or they could be teacher feedback, obviously, but in our project a lot of the feedback wasn't given by a person at all who was given by some software or some application like that, so we thought that this needed to be added. Then William and Thompson, their famous paper, had these nice formative assessment strategies. What does formative assessment, what might it look like or what should it be for, and you can see there there's things like clarifying learning intentions and criteria for success, engineering, classroom discussion, or looking at tasks that give evidence of understanding, providing feedback, obviously, that moves learners forward, activating students as resources for each other, and activating students as owners of their own learning. So we thought that these five, and FASMED also thought that these five strategies were important and that we should take these into account, and then FASMED included functionalities of technology. What is the technology bringing to this? So we have the people, we have the formative assessment, we need to link this to technology, so there's things like sending and displaying information, processing and analysing information, providing an interactive environment, and we're thinking we're still not 100% sure whether we should add a fourth functionality and that's computational power. A lot of what we're doing is that it's the computers are doing all the computations for us and allowing the students then to gain benefit from that, so maybe we need to add that in too. Thanks. So we have this 3D framework and these are just the things that we've been talking about here and we want to locate each of our resources in this and then try and match that back to the students' evaluation of those and that's the work that Katrina has been doing and will be doing, she's developed this and will be doing over the next two months writing it up. So dissemination, I think we spoke about a lot of this last October, these are the kind of national and international conferences we've been to and plan to go to this year and we've given talks and presentations at these. We've got six papers in conference proceedings to have been published in journals and they've appeared online very recently. These were both written mostly by Katrina and about the survey that we gave out at the beginning at the moment preparing other papers on design and evaluation of our resources and there is a paper on Unidoodle that's been submitted as well. What about sustainability? Well, luckily our website is hosted by the National Forum so luckily we don't have to worry about sustainability of that. We're happy to put our resources and to continue to do that in the future. Any new resources we develop we're happy to put up there. We need to, we have a section on reflections, expert guides so that lecturers can see what other people have done and the kind of pitfalls to watch out for and so on. We need to finish those off. We haven't put our publications up there yet but we will and we have some links to external resources. Our resources themselves, we're trying to disseminate through contacts at conferences and also practitioner networks, things like the Irish Maths Support Network and the Irish Math Society and the Engineers Society and so on and then lots of the resources that we've developed have actually been mainstreamed into courses in the institutions so they're actually part of those modules now and part of how the module is thought and we hope that more of that will happen as we move on. Unidoodle, as I said, in October it got funding from Enterprise Ireland to look at making it a commercial product and that report was supposed to be ready at the end of December but there was some delay and now apparently it's going to be ready at the end of February and then a decision will be made whether we need to apply for more money to carry on with this commercialisation or whether it should stand as it is and just be freely available as it is at the moment. OK, so, ddang sy'n meddyn.