 Good afternoon and welcome to our afternoon Parallel Sessions at OutSea 2021. My name is Roger Emery, I'm chairing this session and I'm delighted to welcome James Howell and a video bagdasser with us today to present their presentation on numeracy skills training for all with Elevate My Maths and they will be providing a demonstration at the end. Please put your comments or questions in the chat area on YouTube. Over to you James and a video. Thank you. Thank you very much Roger. We are delighted to present the annual conference of the Association for Learning Technology. We are going to present our findings of and our experience of implementing numeracy skills training for the students at the University of Derby. I'm an associate professor at the University of Derby. I'm teaching mathematics modules and James is the director of international programs at VRETA. Thank you over to you. So I'm just going to frame the conversation for today's discussion really briefly. The agenda is to discuss the challenge that brought us to Elevate My Math. Then we're going to take a look at some of the data in Canada as well as the UK. Just very briefly VRETA is a Canadian company. Then we're going to look at the solution which is Elevate My Maths and how we brought Elevate My Maths to the University of Derby. We're going to get into the details from inception to the end and then we're going to look at some of the data and results for the students there as well as have a brief demo of the platform itself, Elevate My Maths and hopefully we can have some time for some Q&A after that. Okay. So the first thing we want to talk about here is the challenge that is the numeracy gap. Now a lot of folks would often talk about literacy and the need for literacy worldwide. That's not often the case with numeracy but there is indeed a numeracy gap and it is widening. The definition of the numeracy gap here is going to be the present levels of numeracy in the workforce and the level of numeracy desired or required for full participation in today's ever ever more technological society, more and more technological. Now some of the causes of this numeracy gap, it's very hard to get into each and every one but we just wanted to frame some very very brief ones here. Those are societal attitudes, the idea that some people just cannot do math while others can and the over-reliance on calculators, these are some of the common ones brought up. However at VREDA and with our partners at University of Derby, we do believe everyone could be numerate and everyone needs to be numerate in today's society. Sorry about that, I went one slide too fast. Now we just want to look at some of the numeracy trends in Canada and this led to something like Elevate My Math. So back in 2004 with the results from the Organization of Economic Cooperation and Development, they test adult competency in numeracy as well as 15 year olds. They survey every year and then they release the results every three years. So a very important statistic is over here on the left 55% that's more than half of adults in the Canadian population are below the desired level of numeracy and then we can also see the trend here of the 15 year olds from 2009 to 2018, the last year that they reported the 21 data is not yet out and it looks like Canada was on a downtrend and now it's increasing. However I put a small note beside here that there's five Asian Pacific countries that typically report in top 10 that have not been reporting in the last set of data for various political reasons. Now let's take a quick look at how the trends in numeracy with adults as well as 15 year olds has an effect on higher education in Canada. So these are results we're looking at here from something called the College Student Achievement Project. This was in the province of Ontario, it was a 10-year research project and in their findings they found that across all programs in higher education about a third of students were at risk of not completing their program solely due to their first semester math course and their grade in that course. So you can see here the different subgroups that were looked at, pretty common to all higher ed programs as well as you know the success rates, good grades is defined as 60% or higher here in Canada and then at risk is anything below that and it's about a third of the students. In response to this data from the College Student Achievement Project the Ministry of Education as well as the Ministry of Training Colleges and Universities in collaboration with all 24 colleges here in the province of Ontario got together to create a system called the OCMT which was the Ontario College Math test and the goal of this was to diagnose upscale pragmatically and then reassess the level of mastery prior to students coming into the higher education system to improve student success as well as retention. There was an open RFP at that time and RETA was selected as the technology partner using Elevate My Math which has been rebranded the OCMT here in Ontario but it's also available outside of the province of Ontario. So just very quickly and forgive me every time I say Elevate My Math instead of Elevate My Maths I am from Canada so I have trouble pluralizing maths sometimes but RETA is an award-winning Canadian educational technology our vision is a world where everyone enjoys mathematics and the Elevate My Math product was designed to diagnose and identify the skills where there are potential gaps, provide targeted upgrading and remediation wherever possible and then reassess to the level of mastery prior to students getting into you know their college course. So some of the things that this is used for is either placement or streaming but it's primarily a gap-finding tool to assess the prior knowledge and then fill those gaps before any at-risk trends come in their studies. Forgive me. Now very similar to Canada we have a similar trend in the United Kingdom actually interestingly enough it's slightly reversed in the UK it's less than half of adults who are below the desired level of numeracy however the 15 year olds are not as consistent in a trend so you can see here it moving from 22nd to the 13th which is a large increase but it is always around for the UK the OCAD average again there is that note beside for 2018 where five Asian Pacific countries did not report so take that for what you will. All of these results are available publicly and it's also something we can share with you guys as well. I'll pass it off to you over to you here. Thank you very much James. My colleague has already mentioned that 49% of the working-age population of England has numeracy level expected of primary school children this is alarming and the economic impact is evaluated at around 20 billion pounds per year. This is why numeracy should be the focus I think for every educator and this has deep impact on the work for on the jobs someone can take it's expected that one percent one increase one point increase in numeracy's course can lead to 19% wage increase. Numeracy is a requirement in many STEM jobs and anyway people working in STEM fields tend to earn more. Why should we bother about this at university? Because we have to support all students to access highly skilled jobs and James already mentioned the situation in Canada when we're a third of the students were in danger of not completing their course. As a math teacher in my class I'll show you in a moment what I'm teaching in my in my classes but I really benefited from this and I hear I think in the UK especially we hear this claim very often oh I've always been bad at maths if someone cares that you are teaching mathematics they normally tell you I hate maths I'm not good but I think this should not be acceptable at the university and I think we should do everything we can in order to support our learners. My main class at the University of Derby is computational mathematics taught to year one students in computing we teach them basic mathematics to bring them up to speed in order to be able to complete their degree in computing. It's a pretty large class around 150 and 180 students and the National Student Survey feedback student satisfaction on the course with the course around 2012 was pretty low. I developed a wide range of resources including computer-based assessment a whole question bank and this seemed to have impact year on year so the student results improved year on year but then on the in the next slide you'll see that in 2018 I've met the gentleman on the left of the picture this is professor Graha Morput from from Canada and he was working with RETA and we've met and he told me my dream is a world where everyone enjoys maths I thought that's exactly my dream as well and then we started the very nice collaboration we started to organize workshops to raise awareness with regards to numeracy training and I start we work together to deploy some numeracy training courses in my class so that before students start to work on their more advanced maths I wanted to make sure that they have their gaps in the basic maths field and in the day-to-day maths which also helps them with their job and then we were able we found understanding and support from the University of Derby we were able to sign a partnership a three years partnership with RETA it's up now for for a renewal and we became their academic hub in the UK and we started to develop many courses to fit the needs of our learners we developed digital badges to improve the employability of our students and also we start they are now deployed these courses across the university through the virtual learning environment and also we are working on translations with partners abroad next slide please in the first phase of the project we deployed in my courses in our call in the College of Engineering and Technology in five modules we deployed one basic course advanced numeracy skills covering basic mathematics so again a reminder numeracy is the mathematics of day-to-day life so fractions percentages but this is the mathematics that you really need to get correctly this is the maths that nurses are using for dosage calculations you don't want nurse to have an incorrect calculation of a dose and we found we've received great feedback from students especially from the students who are coming back into education after long gaps we have a student who was coming back to the university after 25 years break and I remind you the structure of this resource so the idea is you first have a diagnostic you see where the student is then you have remedy lessons were appropriate and then you have summative assessment in order to confirm the mastery levels next slide please now the benefit for my course was that I was making sure that the students were going through this numeracy skills training they had their basic maths skills checked so I didn't have to repeat Pythagoras and other basic things in class I could see that the students were engaged by the resource and I could also measure improvement between the diagnostic and the summative it the improvement worked as expected smaller format students were already very familiar with the contents so their diagnostics were scores were very high but there was huge improvement in my cohort in computational maths in Greece because these students were also learning the maths formulations and the maths English formulations and what I could see you will see more data in a moment but normally the pass rate for my course was for 84 85 percent with computer based assessment and this went up to 90 percent now I'll make this story very short after the first year based on the success recorded with the three with the first 300 students in our college I worked with colleagues from across the university in the college of arts humanities finance business accounting and together we designed 12 numeracy training courses so this 10 plus two extra courses with to meet the needs of learners from across the university and this course is now worse than promptly deployed with support from our learning technicians across blackboard so we integrated with blackboard and they were deployed within a few some within a few months they are deployed in more than a hundred instances in blackboard then this also helped Vretta to upgrade the resource because we were able to build new courses we actually moved the technology to the next level because we redesigned actually the whole ethos around the the whole structure of elevate my maths so they adopted now the 2.0 designation so now instead of having a large diagnostic with 60 questions we divided the courses into small topics and each topic has this diagnostic remedy summative structure assessment for learning so the summatives could be repeated as many times as the students wanted and now the courses are also deployed through the university math skills guide and this led to different opportunities as well so we have interactive dashboards and also students could do placements as well these are the dashboards for the tutors so based on three bullets I can take I can tell exactly where my cohort is I can measure engagement using the this bullet R1 then as for as for if I look at the fourth summative I can measure exactly how many students engage properly with the whole resource so 128 and then I also have a proficiency assessment and then this proficiency following the completion of this proficiency assessment students can get a digital badge which is now offered through the open badge factory I'll show you just a snapshot of the impact emm elevate my maths head on my own course this is computational mathematics this on these are results starting from 2015 16 and then year on year you will see that the results the results without with which are not shaded are like results as they are taken from the great book while the results highlighted in different colors are there is are demonstrating the impact are highlighting the impact of different of different emm interventions I only focused on two statistics first one the first one is the first time pass rate so these are the students who score 40 percent or above in their in and pass the module and then on the good grades similar to what James was describing in Canada so 60 percent or above grade and this this is the number of active students so student number is the number of active students so the students who submitted something and then I take in 2015 17 18 I just displayed the pass rate which was 83 83 percent in the first in the first year so 83 percent 84 percent 86 percent and then the good grades the percentage was 59 percent save 61 percent and 58 percent so I think pretty comparable with minor adjustments leading to slightly improved grades then the first the pilot project was implemented in 2018 19 and here I'm plot here I'm showing you the data the data which is comparable so these are the students this is just the regular completion rate and then this is the impact of emm where I plot here I only take a restricted set a subset of the students only those who completed their emm course so 99 99 students completed so 107 students completed emm and then out of this 99 passed 69 scoring ab this means that the percentage of students who passed from the first attempt was increased from 89 percent to 93 percent and then the good grades increased from 58 percent to 64 percent then from the second year onwards I I deployed two courses d1 and the d2 and here I'm scoring the gradual improvement of this of this pass rate so the pass rate the overall pass rate was 89 89 percent students were also given extended deadlines so the pass rates were anyway higher however if you restrict our attention to the students who completed d1 the pass rate increases to 94 percent and if the students also complete d2 the student the number of students so if you pass your elevate my maths then the the pass rate for those students was 98 percent so now because we are in the covid era I can associate this with like getting your dose of numeracy training and then this helps you develop immunity to failing against course failure so I think this this figures show that this really worked in terms of pass rates and I think results were even more impressive in terms of the good grades because we can see that the good grades without emm were about 73 percent the year before they were even smaller about 60 percent and now if you restrict the students who completed their numeracy training this was 87 percent so I think this is like a pretty mathematical proof that this seems to have worked certainly actually increasing that bracket of good grades significantly is very interesting yes and then we also worked with the marketing team with employability with the digital skills team with we have we have awarded 400 plus badges for the completion of the course you can share the the achievement via social media you can check my own digital badge and we are in the top of the university badges for for completions of course we have this work attracted attention from various partners we are working with partners within the university with the math hub with the uo d applicants with the skills hub we have partners in the uk we have various educational charities darby opportunity area we are supporting academic partners increase as well and also in Turkey in Romania and we are also working with the number with an increasing number of SMEs we are going to deploy this as Joe as numeracy training for companies through the european social fund and also this the new platform the newly developed platform emm 2.0 is used now by more than eight eight thousand teachers from last bed so school-based teachers and of course we are looking for more collaborators we cannot demonstrate that this intervention works and can change student lives we had to leave much of the feedback provided by students outside this presentation because of the time constraints thank you very much for your attention and we are looking forward to your questions yes we have we have a question from Jim Turner about the emm being fully embedded in all areas so we are working with the university central services with the technology enhanced learning and because the courses are now integrated with with the blackboard it's very easy to deploy the course straight away in any module we also have a skills module which runs across university and users can also access the numerous training courses through that central module however if a lecturer let's say i'm teaching calculus module code for m a 501 if i want to manage the dashboard with my own students and track their progress then i might be interested to deploy this course in my in my own module as well thank you sorry i realized my mic was muted just now another question from Jim they're just following that up have you got any um qualitative data on the student you know student feedback do students like it have you had any of that sort of feedback from your yes students yes we are we are collecting and we are actually actively seeking feedback from the students and this is also a resource which was developed uh based together with the students at the university of darby we have the ethos of working with students as partners and we really care about what they have to say we did we deployed for example a computer based assessment in calculus which was developed and trialed and debugged with the help of the students first deployed as formative assessment and then after it became very reliable and the students even tweeted about this so we encourage them to go on social media if they like something and then just share it with the world and then we take that into account to improve the course i'm just gonna just gonna pick up actually just something you've mentioned earlier um i know darby had successfully run a separate attainment gap project um which trying to close the you know the achievements of the students of different educational demographic backgrounds um has this has this project touched on that at all and have you seen any improvements across different demographic demographics of students i think this is exactly the next step we on our uh on our road uh because through this project now that is deployed across the university we can actually add a mathematical dimension to every student profile so this means that what you can just if you have this extra math dimension to a student to every student then you can run you can trace uh the the evolution of the student in correlation with the math performance so i think this i think this collaboration may makes this project very feasible okay thank you um we're we're running out of time now um further questions can be um posed on the youtube recording of this in the chat area so um there's a couple of other questions we haven't got time for so you can you can respond there and obviously um anyone watching the recording can can respond there um so i just want to um say thank you um to uh video and james for uh presentation um sounds like an excellent project and product um and it's been very very effective so it'd be yeah it'd be interesting to see what what comes next as you said with it with the qualitative feedback and entertainment gap work as well so thank you very much both for your um uh presentation today thank you thank you very much thank you