 Good morning or good afternoon everybody. My name is Mathias Liffis. I'm from the Australian research data comments I'd like to begin this webinar by acknowledging the traditional owners of the lands on which we are all on today For me in Perth that is the Nunga Wanjuk people. Now, I'd like to pay my respects to their elders past and present today's the third in a webinar series that the ARDC has been presenting on the sudden shift or the sudden need to shift to online delivery of technical training and I'm very pleased to introduce Aidan Wilson and Anastasia Papa-Yuanu from Intersect in Sydney and they will be giving a presentation on how they're ensuring that the quality of their online training is Just as good as the quality of their face-to-face training If you have any questions during the presentation, please use the question module in a go-to webinar We'll have a facilitated Q&A session at the end. Over to you Aidan Thanks to Aidan. I hope I'm being heard Probably good. Okay. Thank you for the introduction Great. Thanks. So my name is Aidan Wilson and I'm an e-research analyst at Australian Catholic University for Intersect Australia. I'm also at the moment acting services manager for the Sydney metropolitan area as an aside and my colleague here Dr. Anastasia Papa-Yuanu is our lead research data scientist He's chuckling because we've just been practicing how to pronounce his surname We usually just call him Anastasia Papa-Yuanu Anyway, so this webinar is called Quality Technical Training in a post-face-to-face world Sorry about the slightly dystopian title, but it kind of I think captures what we were thinking in in March But it's a bit of a rosy route book as I'm sure we'll discover Before we start, we'd like to acknowledge the traditional owners of the land on which we are respectively located So for me, they are the Gadigal people of the Oranation here in the inner west of Sydney and from Asasios They are the Kamraigal people also of the Oranation in in the north north shore And we pay respects to the to their elders past present and emerging So just a little bit of a background of what Who intersect is we are a not-for-profit membership based e-research organization that was formed in 2008 by New South Wales universities and some state and federal seed money Today we're governed by a consortium of 13 Australian universities who are our members and we operate across five states and territories Here's a Here is our members They're mostly new South Wales universities, but we have later spread into ACT Victoria and most recently South Australia our model of Support really is is Centralized around our e-research analysts Who form the services team so may be one of them? So this team is really the primary interface between intersect and the respected member organizations the e-research analysts are typically based on campus at their member university and They work in conjunction with the existing support framework of research support networks within the member university. So They dovetail in with existing training Programs and other research support programs and other e-research governance frameworks and so forth Um They also facilitate cross institutional collaboration with the other e-research analysts helping each other out with shared problems So the this role combines it research needs business analysts Sorry business analysis and researcher engagement across a very broad range of research activities and disciplines The general responsibilities of this role are to provide advice to gather research specific it requirements And help guide the development and deployment of relevant e-research services And lastly to increase the visibility and acceptance of good e-research practice But today we're talking about training so training is one of our most popular services at our member universities And here are a couple of here a few Broad level numbers. So it's a date And to date means we've been doing training since about 2012 We have trained 15 000 researchers or more than in more than 1200 courses at 15 institutions across five states and territories um, and here's a little graph showing the trajectory of of Attendance numbers so you can see from a very humble beginning in 2012 with only 10 researchers Um, that uh, that graph is is continuing to steepen Which i'm i'm glad to say that It's good to see a graph that's steepening No, that's not what i mean. It's this is one graph i'm glad that is steeping. Um, and uh, yeah in 2019 we had Over three and a half thousand. I think attendees so it's getting This training activity is getting bigger every year and more popular This is a subset of our catalog our training catalog We try to cover the breadth of e-research tools So not just programming but also data management and collection tools like altrix and redcap Research computing so high performance computing and cloud computing And data analysis tools like excel sbss and open refine And this catalog is continually being updated and modified as we get more feedback and requests for different kinds of courses We have always hesitated hesitated to run online training Um, over the years, we've tried it maybe a handful of times and we've never really had a great deal of success Uh doing that. Um, the reasons for this hesitation are for example that Well, um, a big reason our training is so popular. We think is that we focus on the interactivity Uh, not just attendees following along and using the tool themselves, but also interaction with other attendees Um, also tech training. We thought surely needs in-person attention and assistance when things don't go exactly right Um, so if someone can't install python with if we're not in the room with them, it's going to be hopeless um, is our belief It's uh similarly too hard to help people with problems in an online setting And we also anticipated specific challenges for each particular course or course type So qualtrix and redcap, um These are web apps their point and click. Um, there's no installation. Um, and not a lot of keyboard shortcuts We assume that these courses would be much easier to deliver online than something like programming courses Where, uh, there tends to be a lot of individual help that we provide to attendees Typically installation provides lots of problems syntax errors spelling mistakes Uh directory paths, so we thought that programming would be very challenging Um excel i'll mention on its own. We uh, there's a bit of a special case um, we This tends to get a less technical audience and there's actually a lot of version issues particularly when people are working from home um, and the difference between Mac versions and windows versions. It has traditionally been quite quite The the gap has been quite wide although that's narrowing in in recent years that kind of bring the programs a bit more together. Um Uh But also there's a heavy use of keyboard shortcuts, which are quite hard to demonstrate online And this you've got a webcam on your keyboard So you actually find yourself saying things like the key above tab and next to one Whereas in the room people can literally see what you're what you're typing So we thought that the excel would be one of the most challenging courses for us to run online Um Another reason for the hesitation is that we have a very robust Workflow and system in place for face-to-face training and we weren't sure how this would translate to Or how we could extend this to to an online Online training offering so for example We do a lot of automated trainer assignments automated reporting And suddenly we've got exceptions in online training forcing us to radically change our systems to accommodate One example is that for reporting purposes Nowadays we would want to track whether a course was online or face-to-face But we've never had this concept in our systems Whether you know all courses have just been face-to-face and so we never never thought to include that category And adding that category in in our system is not not trivial And causes some downstream issues as well It it also means that trainers can now be So where trainers were usually Based at a campus They might they might have a range of universities or campuses that they could train at and the system would automatically invite them if If a course was in their local area But for online courses trainers can be invited No matter where the courses is notionally being held Which means that the system should automatically invite these trainers to courses Outside their local area, but only if those courses are online So that presents another challenge to kind of integrate these these categories into the system so So the covet 19 pandemic Which hit in the middle of march I mean earlier internationally but in australia when lockdowns really started was in the middle of march So this forced us to and everyone really to put aside those hesitations and jump straight into online training um So we started with a few pilot online courses using zoom uh in the late in late march And this was a monumental effort involving the entire services team in collaboration with members and other organizations We tried uh, we used these to try out different pieces of technology and methods So we tried shorter courses no more than half a day generally We used zoom With polls chat q and a breakout rooms or a combination thereof We really tried each of them in the beginning to find out what was going to be Uh most beneficial. We found that communication is key so sending out clear and concise instructions to attendees in the lead up to the course and Also being available in the half hour or so before the course started so that we could troubleshoot any Any issues that were preventing people from from uh commencing in the course Also during the course we had many breaks and opportunities for questions and discussion Um, and we enhanced the role of the assistant trainer in doing things like moderating the chat Answering questions and actively participating in the training This pilot process was extremely quick. Um, our last in-person training course was march 13 And our first online course was march 24. So there was only one week in which there was a complete suspension of training And then by april we were back to about half capacity um So the pilots really were only a couple of weeks before We were back to half capacity in april and i'd say we were already back at full capacity by By the start of may So our setup has quickly evolved into something that looks a little bit like this with a primary instructor who leads the course Focusing on the content and at least one assistant or you know, sometimes two assistants depending on the size of the course Managing the participants answering questions running polls assisting individual attendees with issues And even running breakout rooms if and when they're needed We also found it helpful to direct participants to a google document with Things like instructions links to further information And also larger portions of text that can't be nested can't be pasted into the zoom chat Like the whole like whole code blocks in programming courses, for example Um, so here's what it looks like from a user perspective. It's got a few screenshots and some NAF animations. I hope you you forgive me for that. Um, so this is what zoom looks like I'm sure everyone has seen it right now with the the the main trainer sharing their screen and in this case sharing iPython notebooks and running a python course alongside this there will be a zoom chat where the trainer can Provide links to information so links to the google doc for example and where attendees can Ask private or public questions to to the trainers and get individual assistance Occasionally we might throw up a a zoom poll To do things like gauge the experience that people already have So at the beginning of a course, we might have a poll that asks how experienced people are already with this tool That helps the trainer You can do this in person quite easily because you can you know People put up their hands and you can see if people are nodding But it's a bit harder online. So having polls to see how people are going along with it Another use of polls is if there's a couple of modules of the course that you know don't have time to cover both but You want to vote have people vote on which one they want to Go into so we've found polls to be quite useful as well As I mentioned earlier, we've got a google doc separately from the zoom meeting that people are encouraged occasionally to go into to copy out long pieces of code Or have for their own note taking if they want to share their notes with others And that stays online. So we don't we don't tend to move those off Unlike the zoom chat, which which unless someone remembers to save it is pretty much expired at the end of the At the end of the meeting so people can go back to that google doc later and get that code out if they want to The last line of defense really is Is the breakout room option? So we'll use this for more in-depth assistance So for example, if someone's just having a complete block up, they can't run the code There's something wrong They're not sure what it is and they don't know how to talk through it So it's easiest for us to go into breakout room for maybe one or two minutes See their screen catch them back up And then go back to the main session. We this is the last line of defense because Obviously while they're in the breakout room, they can't monitor the main session. So So they'll have to then be caught up to where the trainer is which again the assistant will do when they when they come back into the main session Okay, at this point, I'll hand over to one of such as who will start talking about the Numbers and the reporting that we've been getting out of online training for 2020 Okay, thanks Hayden and so we thought with Hayden that it's going to be also good to present you some numbers In terms of what we have done for online training and compare with the in-person training So here you see a summary of our total numbers in 2020 So you can see the in-person training and the online training So the courses we delivered how many unique courses we delivered based on our course catalog and the researchers and staff trained so for online training in these two three months like almost three months like 10 half that we delivered to around 50 courses to about 800 people Uh, one of the things that we would like to mention here is the registrations per course which has an asterisk for online training So we can see that the registration per course is 24.8 Or in-person training while it's 19.2 for online training and the attendant attendees per course is 19.6 for in person Whereas he's 16.1 Uh, so even though there is a so-up rate like higher in online training like the attendees per course and the registration courses are lower And this is not because like the demand is lower. Actually, it's higher It's because like we thought that in the beginning of the courses in the deliver of the first trials and early pilots Like we should limit the number of registrations because we didn't feel comfortable and confident about the quality So we wanted to ensure that the quality is really good We wanted to ensure that every person who comes to our courses is going to have the same interactivity and We're going to deliver the same quality of training. So In march where we had the full capacity of our training again going on online Uh, we thought that we can increase this number So from now on actually we start increasing slowly the numbers and see actually like up to what level we can manage So that's why these numbers are a bit lower so If you go to the next slide Aiden so all the numbers from now on uh refer to The online training in 2020 so all these numbers refer to the data we collect for online training and i'm going to try to give you some Um, uh discuss a bit about the comparison with historical data. We have for example here We see the number of attendees per course and the course course category. So Usually like this is a bit Probably misleading compared to our historical data because one of the most popular course categories we deliver is the programming one Of course, so last year programming was the most popular by far And 53 percent of the courses delivered were programming courses whereas now we see for online training more like data management courses, which are probably like, um if you Go to our website, you're gonna see that it's like quadrics red cup This kind of tools and this is because we thought in the beginning That early pilots are better to for like web app applications like red cup and quadrics Because it's point and click so we thought that it's going to be easy to start with them And then also programming courses required a lot of preparation. So we wanted to make sure that The installation the setup instructions all the things before we deliver a programming course were there And also be ready with some backup options in case like attendees have some problems For example with installations with jupiter in python or with our studio in our so that's why we see a bit of like We see programming being a bit lower Now since we deliver Our training in full capacity We have scheduled all the programming courses and like we're probably going to see later on this year that programming is going to go on the top So in the next slide, we can see the Attendees per course. So this is also consistent with what we see in our historical data So for online training, we can see that The main three categories like data management data analysis and programming have more than 15 Attendees per course. This is again based on the limitation of 20 Most of the time 20 parties by registrations per core per core. So it's it's really good numbers and we see a decrease in the Number of the percent of the percentage of no source. So this is also like Consistent with what we get and from last year to this year. Actually, we can see an increase in its course category So attendees per course in each category are increased And the largest increase can be found in data management where like last year We had around 20.8 And now we can see in online training and in-person training is around like 15 16 people So we also collect data about like the different roles and position and also the different network codes for the attendees so For the role and position we see that the biggest portion of Attendees is actually the biggest percentage is like PhD students following followed by professionals and then academics post-docs and A few master students and owners. So here I would like to mention that like In online training, we see around 38 percent of attendees are PhDs whereas like staff and academics are 22 and 17 Historically like PhD students actually are Almost 50 of the courses. So we see a drop in the PhD the number of PhD students attending our courses And we see a significant increase in the staff From 15 to 22 and we see also like another considerable increase in academics and post-docs So we see that professionals academics and post-docs are like willing to Go fast and learn more tools actually probably because they want to continue their research And they want to learn actually how to use these kind of research tools to keep going with their research And on the right side we see actually like the f4 codes. So one of the most popular ones historically is medical and health health sciences students and academics and stuff Followed by psychology biological science and all these so The actual data we have is more like again, like medical students are the most like popular FR code But followed by other ones. So this is mainly like shifted to be like and we see a few differences because we deliver more Qualtrics and survey tools like red cup Survey tools. So it means that we see more participation from this kind of Sciences So we thought that it would be interesting also to show you a bit of data about how many people return to a course So for online training into 2020 we see around like 19% of people returning to a course. This is based on the 50 courses we delivered and in different universities So this is quite low compared to our historical historical data because first of all, we limited the number Actually, what we call tourists. So like people who can jump from different institutions to the university so So this means that not many different courses delivered yet in Same university. So we see a bit of drop historically though, like the return to a course is around 20% in our courses. So 20% of people attended two or more of our courses And we do this analysis also per role and we do it per FR code. So I can just tell you that The highest percentage of return to a course is has been seen in PhD students with 27% Followed by postdocs around 23 academics around 22 Professionals 21 and the lowest by far percentage is in honor students, which is around 12% So a few things about course registrations. We try also to Find different ways and capture the different ways that work for the to raise awareness about like the course The courses we deliver and the services we deliver So we try to capture actually how did they like the registrar's registrants here about the course and we can see here all the different options we give So like most that on the top is faculty in school newsletter Then followed by research office and division university website Newsletters from the university supervisor and word of mouth as well is quite popular as well But we go one step further and we do the same and we check per role Just to check the different strategies for different people. So here we can see like somewhere in the middle column the PhD students where you can see that most of them Hear about the course from the faculty newsletter and then followed by university website and not research office and then university website Whereas for professionals, we can see that most of them here from their supervisor Or from the research office So we can see different patterns actually like based on the different roles and positions. So we try to capture these ones to Be better in the communication and the engagement part for our courses Okay, let's move on and now like talk a bit about the feedback so after each course we send a really short course evaluation survey And also we send this a week after we deliver a course like as an automatic email to us The attendees to fill in this sort serving. So here is just like Few information from our few questions from our survey and we focus on the delivery Absolutely, like I wanted to show you like the quality of the delivery and what we capture So here are some questions like did you feel that the training course atmosphere was welcoming Comfortable interacting with the instructors So if the instructors you feel that they were knowledgeable if they gave clear answers and if they were good communicators So this is our particularly useful for us because now especially like that we're trialing A new service like online training. We want to know that we deliver like similar or Same quality As in person. So you can see that all the average scores are more than nine out of 10 in a scale from zero to 10 Which is extremely good and make us feel more confident confident that we deliver really good quality training In this new setup, so we capture also some feedback in terms of Like good and bad aspects of training and also any suggestions So we take this feedback really seriously because it's one of the big things that we check when we for example test a new tool Or check a new setup or we would like for example to develop a new course So here we can see like a few comments talking about the pulse How good were actually like when like people were feeling lost or how we can capture like if they're on track A lot of people were talking about their interactivity of the course and that they set up with the Many instructors was really helpful and We were sure that everybody can get some help Some people also mentioned that this is a good opportunity to have this training online for later on after 2019 is not an issue anymore And keep it because like of the many people being in remote units or in like other And disability comes and they cannot attend these um face-to-face. So we take these feedbacks really seriously so The last part of the course evaluation survey survey is about the net promoter score So this is one way we check also the quality of the course So we asked them the question how likely is that you will recommend in a sub training courses to colleagues And for people who don't know what the net promoter score is Actually, we take the percentage of people who respond in nine or ten to this question is zero to ten question We take this percentage of from promoters or nine and tens we subtract the percentage of detractors So people who answered zero to six and then we have a number between minus hundred up to plus one hundred so Plus 50 is considered excellent and in person training this year based on the 355 responses Was around was 56 Uh surprisingly online training based on the 379 responses is plus 74. So which is Considerable high actually compared to in person training and we were trying to think actually what were the reasons to Like of of these chains and like that online training is much higher So we think that probably this is because like all the training all the online training Are I should say most of the online training because now we have also like casuals being involved Has been delivered by all our experienced research analysts So research analysts had like all of them delivered more than 20 30 courses in the past So they have the experience they they know the environment. They they know how to Manage actually difficulties and problems. They're more interactive They they they have the experience actually to make these more interactive and Make sure that everything is going well. So we think that this is probably one of the main reasons We see this difference But in order to compare them like in a fair way We should just do the two delivery methods actually after covid-19 is an issue and then we can see actually like Just to minimize the external factors So Final takeaways So online training as we said and essayed and said actually requires a few things before you deliver it so you need to carefully plan and you need to Do some logistics beforehand You need to communicate with all the attendees to be sure that everything is Ready before the course. This is a really important step Actually before you can lose a lot of time in the beginning of the course by dealing with all these problems and You like already the pace is a bit slower because it's online set up. So it takes a bit of time actually like to like To go ahead with all these problems. So make sure that everything is ready is really important Of course we to realize also on technological tools and that are used Probably like zoom and all the different features of zoom and all the other tools like backup options like art studio and all these things Like if you have a cloud version or if you have jupiter in the cloud version and goes on And of course like requires smaller class sizes and more trainers So more overhead to prepare things to prepare a course and then requires more assistance And smaller class sizes. So online training can work We found that online training can work as well as in-person training. So The hesitation we had in the beginning was actually proved that it shouldn't be that much and We proved that online training if you put quite a lot of effort can be as good as in-person training And sometimes may better suit the diverse needs of researchers Especially when they are in distributed comes just like adan is going to talk a bit more in the in the last slide and also like We need to have like careful attention to the planning and the delivery. So again, like it needs more overhead But if you manage to Take into account all these things Delivery and quality of training. It can be like similar to in-person training Over to you adan Yeah, so this is the final slide really This wouldn't be complete without talking about what what happens now in a post post face-to-face world and I think we It's worth pointing out that we we thought this was going to be We didn't know whether the COVID-19 pandemic was going to be Brief or very long. Um, it's been briefer or it looks to be It looks like it's going to be briefer than we that we had planned It wasn't feasible for us to Suspend training and wait until we could recommence face-to-face training Because if the pandemic had lasted for say the rest of 2020 then we would be in a very Bad situation with respect to our members and the value that they expect from us Of which training is a huge Part so our original plan was to run training online while the lockdowns were in place and then return to face-to-face training after the pandemic was over This is based on the, you know On the unfounded belief that face-to-face training was was just better for attendees and therefore better value for members But our success in online training and the numbers that we've seen and have shown today Lead us to Rethink this plan and so that when things return to normal Um, which thankfully is looking to be sooner rather than later um, we may very We're very likely to keep um an online training Uh platform as an alternative alongside our more our more traditional in-person training So this will allow us to work with our members More closely to to design tailored training programs at the university given their situation So regional universities for example with many remote campuses Uh, like acu or western sydney and others They may prefer to plan more online training to cater that to cater to their distributed Research community whereas the larger metro-based universities that are probably a bit more monolithic like usw and sydney They may prefer to return to face-to-face training More likely I think every university will want some kind of mixture. So there'll be some in-line in-person training And online training You know one question is if if the numbers are so hood, why don't we switch entirely to online training? And I think you know as Anastasia has covered on the previous slide It takes a lot of setup. Um, it takes a lot of things to get to get it right And also we can't we we don't want to fit the same number of people in the room because We're hesitant that we can't train as many people as we can when it's face-to-face So I think we'll always have a place for face-to-face training But our success is trying that, you know, we we should also make sure we've always got a place for online training in the future And on that note, I will say thank you very much Um and Hand back over to Matias All right. Thanks guys. That was fantastic We've had several questions come in So a reminder to all attendees you can post questions in the question module and I will read them out to Aiden and Anastasia So the first question we had Is a logistical question. How do you ensure that your registrants for online courses are from your member organizations? And not necessarily any member of the public Um, this is a this is a perennial problem that is not not just related to online training. Uh, we use a vent right for registrations and You can't limit people in a vent right from registering to To to courses based on their Based on their email address or or anything like that. It's not a f authenticated So we can't put it behind a an authentication layer Or anything like that. It's a it's just an issue we deal with we get we get if we get a few people per um I don't know what the rough numbers are but the very small numbers of people who aren't from any of our members turning up and We it's not enough for us to worry about so we'd rather not kick people off courses We have had in the past though And I mean before online training we've had we have had I was running a session for acu people for acu research specifically but that ad went out to Someone else who's from a different university and we had half of the registrants being from that other university so I had to step in then and politely tell them this was for an acu audience and I apologize, but I had to basically revoke their tickets um, but with online training we do get the The case where you know, it's much more possible obviously for people to enroll in courses that aren't at their home institution We call these tourists as as Anastasios has foreshadowed before We don't we don't generally One or two people per course we don't really care about but we have Put in place basically people when they sign up for our courses acknowledge that the course is for the home And that if they are not from the home institution then they risk their ticket being reallocated to someone on the waiting list for example Um, so that's been how we've we've done that just by warning people that we may do this and they have to click a box to say That they accept that Okay, great. Thank you Uh, okay next question. Um, this one's a little long So a wealth of literature has shown that instructor evaluations are heavily biased by the likeability of individual instructors Sorry versus the student learning outcomes Do you use any metrics that will evaluate whether students have acquired the skills that you are teaching? Or do you right rely solely on the likeability? Metrics the the net promoter score that you presented Yeah, good problem. The net promoter score is obviously not without its problems and um, I'm not, you know, personally across all the literature But I'll take I'll take the the question as um Uh word for it that um that uh, there are problems with with with doing evaluations like that We have tried in the past uh to To reach out to uh former attendees um Say a couple years after uh and try to um Ask them whether they have used the techniques that uh, they they learned in the courses Uh, it's pretty hard to run that. Uh, it's it's it's a pretty hard metric to kind of nail down really because um Someone might come along and learn some python and then they might never use it again But they might be a bit more knowledgeable about Things like programming or they might have learned a different programming language And so it's it's hard for us to causally relate those and those things um, so Sorry, just to add on the actually like our actual model is based on this further support So we know that the curve is steep for all these research tools So the main point actually of this training after we finish the training is actually to give them like, you know, conduct Of the era of the institution and then Keep in contact with the person like with the era and try to uh, help them actually like overcome this steep curve and then try to Assist up to the point where they feel more comfortable to just do the research themselves or use the tool So we know that this is quite hard and we try to do different like surveys to capture this But one of the main points of our Services is actually having new arrays actually to help during all these like steps of the learning process Yeah, that's a good point. So the um, speaking from personal experience in the last three weeks. I've had an explosion of red cap Uh, uh, consulting requests. Um, and half of those are a direct result of people attending at red cap boards so People who have seen what it can do in the training and need need to really tailor it to their Exact project. Um, so that this is where intersec kind of sees training as not It's an end within itself But it's also a pathway to to to further engagement and research support with not just individuals but teams So it fits into the whole uh whole workflow Okay, great. Thank you. Um, okay now something about uh teacher student ratio So it looks like you're currently working with about three per 16 participants But you mentioned that you were working to bring this ratio down. What do you think might be ideal? Hmm, I guess post-covid our ratio Um, I'm not so sure we we always have two trainers at every course Uh, actually, no, that's not not true. There's sometimes we um, we have the e-research analysts on their own delivering courses That they are very good at and and comfortably can deliver themselves It's up to them how much they try to limit those courses They're typically limited to around about 10 to 12 When we have casuals delivering courses, we want them to have no more than around about 10 to 12 per trainer We Are careful not to go too too far too large class sizes or too big a ratio difference between too too big a ratio of students to to instructors um, and so we've been carefully tracking the feedback and so on and uh, turns out that I think um Uh, pre coronavirus in 2020 the larger numbers of, you know, nearly 20 people per course um On average in some courses much higher. Um, we haven't seen a drop-off in in in evaluations Um, which is surprising actually so so we've had larger class sizes and and the the feedback is being better actually I think one of the one Instructor per train trainer trainees is the ratio we try to keep in person And but now with online training and all the overhead like it goes probably like one to six Which is which um like actually like it's quite low But we wanted to be sure that the quality is good But so now we keep um going up slowly and trying to test actually if we can increase these But we're not going to read something like more than one to 10 pressure Okay, great. Thank you. Uh, we've got a couple more questions. So, um, we were scheduled to finish Right now, but I think we can we can go for a couple more minutes So, uh, next question It's often a struggle to get participants to provide feedback and return evaluation surveys Did you achieve a higher return rate for surveys when participants were engaging online? Yes, yes, I can this one actually like because like based on the numbers we have Like we have like more people actually answering and actually doing the survey But this is also because we introduced a new system as well because uh, we automated the process We know that a lot of people when we do the training courses, they don't stay up to the end because we may go a bit Um, you know, we may be a bit late So it means that we may lose these people and they don't have the time or the the survey link So what we do actually like with um the training team is we send an email a week after to all the people who attended the course Like saying that you attended this course if you have like two minutes Please fill in the survey and we saw like a really big increase in the number of people who actually Do this survey actually later on so it's another reason Another reason we wanted to do that is our our feedback prior To doing this was possibly biased towards those people who stayed to the end and therefore got the Instruction to go to do the survey. So This way we're getting we we hope possibly more honest feedback and we are getting some, you know Of course, we do get bad feedback everyone gets bad feedback occasionally the training can't cater to to everybody Exactly but um to answer the question we I was just looking at the numbers We have 803 people trained post-covid and 825 trained pre-covid and Or pre online training I suppose and if we look at the Numbers that's 375 responses for online versus 355 for in person. So it's a slight bump but also 2020 overall is is much improved over over previous years because of the Follow-up a week later to ask people and and we send them a direct link To a half pre deal pre filled in form so they don't have to remember what day it was They don't have to remember which university it was and so on. So I think that's improved in the some courses in fact Are getting a hundred percent? Response rate which was unheard of before Okay, great. Thank you Right. So next question This one's a bit provocative So this person wonders if a factor on the net promoter score being higher with online training Is the participant's baseline expectations on the quality of online versus face-to-face training? So they're assuming that online training will not be as good. Do you have any thoughts on this? Yes, I think that is definitely an element. There was some There was some open-ended feedback in the first couple of courses that we ran Where people, you know, were quite glowing about the kind of like, oh, thanks. This was wonderful I didn't expect it to be good, right? So yes, it was probably Well, you know people said like better than expected So yes, there's probably is definitely an effect where people People's expectations were lower for online training Just impressionistically having led a few courses and assisted a few other courses um I think I think people's Interactivity in the course and and uh the the chat and so on. I think online training the way we're doing it is um You know pretty much working just as it as well as it did in person I think in person it's a little bit easier to maintain that social nature with with your attendees because they're right there in your face Whereas, you know, you only see a little icon for somebody and maybe a virtual hand going up or something um But yes, we definitely I have noticed that that effective Another factor is also probably that you see the fact that we still do online training and they have the opportunity to do online training So that's why I said like this is not probably a realistic and we cannot compare these two numbers because there are so many different Factors actually contributing to this increase. So we're not We didn't show actually to compare we show actually just to see the numbers and we know that this is probably Sorry because of different factors external factors that probably wouldn't be If you were delivering both at the same time without having any restrictions any problems or any issues. So yeah Okay, great. Thank you. Uh, so one last question. So, um Uh, this one is do you get proportionately more bad feedback for courses with prerequisites than beginner ones with no prerequisites? I think it's no. No, I don't think so I think we get really like some feedback of like intro to courses because I can tell you like my experience like based on the feedback I read all the time. So I feel like in intro courses is really hard to set expectations So it means that for example the programming course like we call it like let's say programming with our or python So it's really hard to set expectations who is supposed to come to this course So we thought that like this course is delivered only for people who don't have any prior knowledge But then you have people who have other knowledge in a bit of knowledge in other Programming languages coming to this one feel that oh, this is too slow So it's really hard to to get the right audience Whereas in more advanced Topics like people can see the topic and they know straight away like what this is about and they are like they know that they spend time More on the topics they want. So I think intro courses are a bit more like that we get a bit more Like not negative, but a bit more constructive. Let's say feedback like on like how to improve this thing When we read into the when we read between the lines with the negative feedback for the intro courses And so actually I should point out it's contrary to what the questioner Assumed that we that we'd have bad feedback for the advanced courses or courses with prerequisites but in fact the bad feedback that we get is for the introductory courses as i'm a start your state pointed out and It's when we read between the lines, it's it's mostly about expectation setting That people are expecting this to be a python course They know already they want to learn some python and what they get for half that day is fundamental programming Not python. So that's that's been the source of a lot of the bad feedback. So we're actually rethinking how we market the introductory courses and we're thinking of doing something like running a you know a few I don't want to give away too much, but having you know break not breaking it up, but having a couple of more introductory awareness level Courses that that lead into the introductory not courses having like awareness level webinars that lead into introductory Courses and making it clear for people that if they know any programming language That the introductory courses are probably too basic and they should go straight to the the second level intermediate level of So if they've done some python they can go straight into the intermediate are for example Okay, we're kind of working through that now on the basis of some of the feedback we've been getting from earlier this year Okay, that was it questions. So I'd like to thank you once again and Anastasia for presenting this insight for webinar And your email address is there on the screen if anybody would like to follow up with you directly Otherwise, I would like to wish everybody a fantastic day And we're getting some more question. Sorry more more feedback positive feedback about the webinar. So you're getting some great comments there Oh, great. Thank you very much. And thank you Matias for facilitating this and ARDC. So really great seminar webinar series you've been You've been putting together Thank you very much. Bye everyone. Okay. Thank you