 we'll do that when the recording starts. Okay the recording has started so welcome everybody to today's Eden NAP webinar about automatic transcriptions and very much about the EU accessibility directive and how we can comply with it in a good way. My name is Alistair Krillman from Linnaeus University in Sweden, welcome to everybody, to old friends and new friends, just a little bit of information before I hand over to the to the host and the guest for today that is that Eden will be issuing open badges to everyone participating in this webinar. If you have not registered you can actually register in retrospect I will put that in the chat there. So if you have come in without registering and you would like a badge just go in and register and we'll make sure you get one. Okay otherwise I would like to hand over to the host for today the moderator for today's session Wendy Chow and the floor is yours. Thank you very much Alistair and I also have to say a very formal thank you to Alistair for all his technical support because it's my first ever international webinar that I've delivered for Eden so welcome one and all good to see you there and it is great to see such a great high level of international participants here and also to see the support for Alistair and Forindu. My name is Wendy Chow I work for the London Institute of Banking and Finance and I am based here in London. We are an organisation that delivers education from schools right up to postgrad level and executive education and we are fairly well used to delivering e-learning but we can always learn more hence my involvement with Eden and the Nap Steering Committee. What I would like to do is very formally and with pleasure welcome Therindu Leanna Gungarwadina and I have probably mispronounced that so I do apologise and I did say to Therindu I would but I would like to extend an official welcome to Therindu and ask her just to tell us a little bit about herself please. Over to you. Thank you Wendy and you pronounce it perfectly. Hi all I'm Karindu Leanna Adhane I work for the University College of Estate Management here in Reading UK. I'm a learning technology researcher and also I chair the online learning research centre at UCM. My main research area of interest is technology and society so accessibility is basically in the middle. What we will be doing in this webinar is we'll be asking Therindu to present her research which won the Best Research Paper Award at the Eden Conference in Bruges this year and we would like to make it as interactive as possible so what you'll find is that there are two or three poll questions that we will be asking all participants to contribute to as we go through. So perhaps it's good to start with a poll and the question should appear in the left hand side of your screen and the question is do you or your institution use automatic transcription services and if so why? And I will give you a little bit of time to answer that. Oh no no no no no I know where that's going to go. I've got 26 participants so I can see them right so we've got one yes it's a time saver that's an interesting point. I'm a waitress but hopefully soon. Yeah absolutely makes our work easier. Okay there's some really good some noted advantages here. I'm interesting with the person who said we're hoping for a national solution could you explain a little bit more what you mean by that if that's possible that would be really interesting. I can see some people typing in the chat there. I'm also interested in knowing about who said that it's a time saver what they use it for whether it's webinar transcriptions. So Marcus thank you very much for responding to that so I can see that you said you've got a national infrastructure in the form of play service I understand so we do collaborate to get up. Okay and Stefan has responded to your query time time saver for captions and Jogun I apologize if I'm mispronouncing it but I can see that you're still typing so I will let you close out this poll. You tried to google sort of just yeah absolutely. I think Thirindu will be identifying with that as we go through. Okay thank you very much well without further ado I think what we need to do is hand over to Thirindu and let her tell us a little bit more about her research into automatic transcription services. So over to you Thirindu. Thank you Wendy. Good afternoon everyone so here I'm going to talk to you about and present the paper that I presented at Eden conference last June it was my first Eden conference and I'm really delighted that my work was recognized as the best winning the best paper award. So my plan for today is basically talk through these topics but before any of that I want to tell you a bit about where my interest in accessibility started. In October 2013 I was part of the University of Reading group who offered the begin programming build your first mobile game MOOC on the future learning platform. So it was a billiard like game and I was the lead facilitator with a team of facilitators facilitating the MOOC. One day there was a post from a lady who said that she could not see the ball on the screen and when she programmed the ball that she couldn't see she had to ask her partner to see whether it was moving properly the way she intended it to. In this instance it was a very easy solution for me to help her change the color of the ball and the background to make it accessible but actually the way we presented it was not accessible to her because she was colorblind. Later on I learned that green on red is a very bad combination for many colorblind people. So this is where my interest in accessibility started by meeting a virtual learner virtually who had an issue with the content that we presented made accessibility something real for me. So that's where my interest in accessibility started. Moving on here I have few statistics about the use of video in the contemporary world. You can see that the use of video has increased tremendously over the past couple of years with dramatic disc decrease in the production costs and also the ease of distribution. It is also the same when you consider the amount of video used in education. It has increased tremendously over the past couple of years. However using this multimedia presents a problem for learners with hearing disabilities unless we provide transcriptions or captions. However as you may already know creating captions or transcriptions manually is a very time consuming thing and it can it can it takes a lot of time as well as resources. Let me move on to the next slide. So what is accessibility? Accessibility can mean many different things in different contexts but here when I talk about accessibility what I mean is a person with a disability whether it's long term short term age related or any other can access the content what is being delivered. So as I said earlier when it is video and audio content captions and transcripts play a very big part in providing accessibility. However they are not cheap. On average my institution pay about 1 pound 30 per minute of transcriptions and that comes to about 78 pounds per hour and these are subjected to minimum fees as well as additional charges for more than one speaker in the video or audio. But at the same time we know that automatic speech recognition services such as Google Duplex, Apple Siri, Amazon Alexa has made tremendous strides in the past couple of years. So also both IBM and Microsoft have reached the point where their word error rate which is the mostly used measure of checking accuracy has reached almost the level of a professional transcriber. Let me move on to the next slide. There are accessibility laws such as the EU accessibility directive and the new UK regulations for a very good reason. We know that making things accessible is the right thing to do but then why don't we do it all the time? You will remember in March 2017 University of California Berkeley took down or removed over 20,000 videos and audio recordings which they have previously made available as open education resources or OERs when the Department of Justice ordered them to make these content accessible. The decision to take them down was based on the cost associated with making them accessible. But because of this none of the people who previously could access these resources were able to access it either. So it was completely lost to the public. But what if automatic transcription was an option? It would not have cost them so much to use automatic transcriptions. So this is where we need to look at new technology and see whether these could help making things available publicly and accessibly. Let me tell you a little bit about my institution. University College of Estate Management or UCEM is celebrating its centenary this year. It is a specialized institution offering supported online learning courses in the built environment sector. As many distance education providers UCEM courses also started as correspondence courses moving on to online recently. In late 2015 then College of Estate Management was recognized as a University College and we changed our branding. This time was taken as an exercise for looking at accessibility and you can see some of the work that we have done since then about making our content more accessible. At UCEM when we have recorded media audios and video we always provide transcriptions or captions. However we use lot of webinars. These are only transcribed only if they are easy student with hearing impairment registered in the course. But actually it is shown that captions help everybody not just the people having hearing impairments. So for example international learners we have almost quarter of our students as international learners. So we wanted to see whether there was a way of providing transcriptions for all our content including the webinars that we hold. So in my study I looked at six software which provide automatic transcriptions. I have given a detailed description of how I selected this software in my paper. I will not go into much detail because I don't have that much time. With these six transcripts I wanted to look at audio that is relevant to our subject area that is built environment education. So I asked our experts our tutors to provide me with subject specific text about 200 to 500 words. And out of the submissions I received I selected four topics consisting of 1000 words to create my text. This text consisted of four subject areas property management, construction management, property and contract law and building pathology. Then I invited a purposely selected sample of 14 participants with various accents native and non-native speakers to take part in this study. Out of these 14 10 people were interested in participating in the study and they were given access to this course that you can see on the screen transcription testing. Here I have set up a webinar setting where they could go and record their voices reading this experimental text. You can see here the data I have collected. I had seven recordings from both native and non-native speakers and you can see how they have described their accent. For the analysis I first got them transcribed using all these transcription software which meant that I had 42,000 words to be analyzed. First of all I checked it with the recording because sometimes when people read out something they can make mistakes and I didn't want those reading mistakes getting compounded. Then I compared the recorded transcripts with the original text using Microsoft Word Compare function. But finally I did a manual analysis of all the transcripts with marking substitutions, deletions and insertions with color codes. And then I calculated the word error rate. You can see the calculation how you calculate the word error rate. Let me show you how the word error rate is calculated. This is an example taken from Caption Accuracy Matrix project. So as you can see here each word is given the same importance in word error rate calculation. However in technical subjects keyword getting keywords correct may be more important than getting something like a or the wrong. However it is not considered in the word error rate. So ideally it is not the best way forward in comparing a transcript but this is the matrix that is used most in comparing transcripts. But there are other methods like weighted average word error rate and so on. But I used word error rate because it's simpler. You can see the results here in this graph. As you can see I had seven participants but I have eliminated participant one's recording. This was because only participant one recorded the voice recording not using the headset like this rather using the laptops built in microphone. So the quality of the recording was very low and I felt that I could not compare it against the other recordings which were done with a proper headset like this. And as you can see here this script the dark blue line shows the best word error rate that I have achieved in this experiment. However this is way below the 95% accuracy level that is normally required for a transcript. And as you can see here participant five participant five's recording had the least errors least number of errors and that participant is a non-native English speaker. Moving on I asked some of the subject experts to look at the descript transcript which was the one that had the best accuracy level and looking at the descript transcribed transcripts the experts said some of them were good enough and some of them were almost good enough. As you can see here even though participant five's recording received the had the fivest errors word error rate participant four who is a native English speaker's recording created the best or rather the good enough transcript transcription. Also another interesting point was that in property management I did not get any transcripts that were good enough as per the experts. This was because the property management talked about the paragraph talked about property and bonds and the word bond B O N D was transcribed as bombs, blonds, bumps, bones and even buttons. So basically it did not make any sense and it was not possible to make sense of what the conversation was about. So with all of this my little research experiment concluded that the quality of recording does matter and in a technical discipline word error rate may not be a sufficient predictor of the quality. As I said before I compared the participant four's recording and the participant five's recording but it's a good enough starting point for creating a transcript as an accessibility aid. I think I have described my research and I must say that there are a lot of limitations in this research and I have gone into great detail detailing them in my research paper so I welcome you to read the research paper about the limitations. Wendy do you think that's a good enough explanation? I think that's an excellent explanation thank you very much that was extremely thorough thank you for presenting that so well for us and making it so clear it's very hard to actually do this and not read from the slides that you've given us some great detail level of detail so thank you. At the moment it does seem that perhaps that the transcription services are not sufficiently robust for technical delivery so there's on a higher level of accuracy so what I think it would be good to do is to bring on the poll if we may and ask our participants so in the previous poll we saw that not many of them you use automatic transcription services currently but by virtue of the fact that you're here today that's suggested that actually you would like to but let's not let's actually challenge that assumption so could we please see your thoughts for the participants in which ways do you foresee a use for automatic transcription services in your delivery and I think it would be quite nice to broaden that out in particular in terms of what your technical expertise levels are thank you so and one way for me is that I currently work for higher education part of my organization and we talk about banking and finance but as part of that there's a lot of regulation involved in that and whenever we've used transcription services in the past it's come up with some a high word error rate so we have resisted for going that way so lovely actually so would you be happy to answer one of the questions Marcus for the moment while the poll is going on and please do still contribute to the poll but I see that Marcus is asking he's not sure how the different transcription services IBM zoom etc performed within your research is that something you're going to talk us through in a little while or can you answer that I can actually in my research experiment zoom came zoom had about 73 percent which I was going to talk about in a little bit but unfortunately IBM was the worst performing in my experiment so I must say that in this what I used was the free available service so if the service that's paid for is better than the free service that they are providing it would not be reflected in my research research results zoom had about 73 percent accuracy rate yes that's interesting some of my colleagues were saying that they would like to use zoom privately so we're looking into it from an institutional perspective and we're also looking at a different virtual learning environment provider and they're suggesting that their ATS would come up with a very low word error rate but we're yet to test it Marcus does that answer your your query hope it does is there anything else you'd like lovely thank you Marcus if we turn to the poll let's have a look at the reasons why people might be using ATS it's quite interesting because I'm glad to see that we're not just saying it's because we should be doing it we've seen to move beyond that um so level one adding a transcript level two creating closed captions okay and increased accessibility do we have any feeling in terms of our views on ATS of videos and movies serendu sorry um so on the poll people are talking about there's quite that to do the ATS for the lectures which is what you've been testing yes and coming up you might be talking about to have you investigated ATS in terms of videos and movies um movies and videos when we create videos yes we have done it um I have a slide do you want me to explain it with the slide I think maybe we'll go to that in a moment so if we can hold that thought would that be all right I'd like to have a look at the other the poll um okay sure yeah are there any points there that you would like to pick up on and I totally agree with as program director it's that's unnecessary for teachers and administrators I think it's something that is now going to be ingrained in our delivery and I found that some of my students really like to have the transcript just in terms of an opportunity to reflect on it I would like to pick up on this one we have tried both manual texting and automatic transcriptions but the latter one still requires a lot manual correction afterwards I agree with that if the there are so many parameters that makes a good automatic transcription the quality the the way the speaker speaks the accent there are a lot of variables so it's really difficult to get a very good transcription automatically I think we've come to the end of the poll so I'd like to say thank you very much for everybody for contributing there but I think if we can summarize that we see that there is a need for it and we're all interested in how it can be done better um but it's about giving people the opportunity to actually rely on an effective ATS without having to manually correct things as we're going along so I think this isn't now a good time for us to return to Thirindu's slides to see for um she can tell us a little bit more about further research that she's conducted in this area so thank you very much over to you Thirindu thank you Vandy so since my research UCM have now adopted Zoom as our video conferencing tool this research was done before we moved to Zoom so we were looking at Webinar software and Zoom scored highly because it had accessibility feature automatic captioning we have about 10 webinars in each module and as I said before whenever we create recorded videos we provide transcriptions and captions at UCM but for the webinars we did not provide them previously we introduced Zoom in spring 2019 that's this year the spring semester and actually it's not 100% accurate and my research has shown about 73.7% accuracy is there with the Zoom transcripts so we asked our students what they thought about their experience with Zoom and I put in put in some questions to get their feedback on the automatic transcripts so that we can see how they perceive the transcripts whether they are rubbish whether they are useful and to be honest I was quite surprised the question we asked was have you used the audio transcripts we had 283 students responding to the survey and out of these students 52.7% that was 144 students said that they have used the transcript then for the people who said they have used the transcript we asked a follow-up question to see whether it's useful so the question we asked as the follow-up question was did you find the transcript useful if so why we gave them the opportunity with an open-ended question so that they could say why they found it useful or why they found it not useful so this is what I found 104 students that is 80% of students agreed that the transcript was useful 14% said that it was not useful and where the student did not say yes or no and discussed both pros and cons of the automatic transcript I marked them as other that's the 6% so basically 80% of students found the transcripts which were about 73% accurate useful and then I wanted to look at the transcript usefulness of the transcripts and the student's demography we had 92 respondents from non-English speaking countries that is excluding respondents from UK Ireland Canada Australia New Zealand and so on out of these 92 respondents 40 responded to the free text question did you find the transcript useful and this is the result so 95% of our non-English speakers non-native English speakers find the transcript useful again this tells us by facilitating accessibility we also make it meet the experience better for all our students I'm going to share with you some of the different views that I saw in the questionnaire I'm still analyzing it there were some comments by native English speakers saying that the errors in transcripts will be detrimental for two non-native English speakers but from this big feedback we have received from the non-native speakers we can see that even with this level of accuracy it is useful to the students here I picked some comments from native non-native speakers a student declaring a disability to show they are different in views while for some students their transcript errors may make it feel like it's a useful useless thing for others the same transcript may be useful so it was also interesting to see how students use these transcripts overwhelmingly students use the automatic transcript as a search mechanism so in zoom if you search for a word that takes you to that point in the video so the students use the transcript mainly as a search mechanism they also use it as notes and there are students who said that when they are sitting in a webinar in their office where they can't use the audio they use the transcript and also as an aid for English as an additional language speakers and for learners with disabilities I think that's all from me Wendy lovely okay thank you very much and the rindu has been very modest there by not articulating it but there are the prizes and I was so honored because I was sitting next to her that night and when she was presented with the award for her research and you could not have had a more modest recipient and a delighted recipient that on the evening so well done to you the rindu for that and also for continuing the research it does look like there is a real scope for the research to continue and I personally would look forward to some further research publication what it would be perhaps lovely to do now is to ask the participants are any of you interested either in collaborative opportunities or in turn just are you inspired to actually explore new adoption of ATS so over to you for those opinions please thank you okay so switch to zoom says Marcus maybe is that something you'd recommend the rindu switch to zoom we are finding it very useful and as I told you earlier we had this workshop where David White was speaking and he wasn't wearing a mic or anything it just picked up from the laptop microphone and the session was recorded and I was amazed to see how good the quality of the transcript was so that sounds like a good route to go at least to try it out it very much depends on the speaker and the way it's picked up like I like Alastair's comment has been quite challenging there because I offered ATS we all have work to do I guess the thing to do is to actually make it as easy as possible so that it is literally automatic and so you don't actually have to to do any correction the point that I found most interesting is that for the native speakers we make this presumption that non-native speakers would find it problematic whereas you've got that 95 percent waiting in favor of it from non-native speakers I know my students really do appreciate the search function so they're actually so even if they're a native speaker they say well it's just so easy to use the search function I get exactly the right point that I want to so either for revision purposes or assignment preparation purposes can I interrupt you there Wendy another thing that I thought would be useful to mention here is that the fact that in my research I didn't have access to hearing impaired students so when you have when you have to when you have a disability or any impairment you would have you would come up with other ways of complimenting it so it could be that lip-reading that sort of things would also help students so maybe having a little bit of inaccuracy in the transcript may not be that much of a difficulty for them but unfortunately I could not go in that avenue explore that avenue more because I did not have access to students that's an interesting point because the participants won't be aware that Thuringdu is aware that I am hearing impaired and I think in something like this if you can have the little image of the lecturer actually that's great because the participants can have that opportunity to lip-read and that's far more effective in my opinion it's only limited but in my opinion that's far more effective than conducting signing because signing is obviously not an international language there is no international sign language capability whereas if you actually do the lip-reading you can lip-read in multiple languages and in some ways it's even better than when people have slightly different ways of pronouncing things when you can still pick it up and I'm just turning to the poll comments there yes I think that point about speaking slowly and more clearly to help is a key one particularly when we get quite passionate about our subjects sometimes it's very easy to start talking through okay I'm having the developments we'd like to see in this field but a good support for the small Swedish language so that's interesting so are we um saying that zoom perhaps wouldn't be appropriate in Sweden I think actually the uh it's Alistair here again uh noticing on the chat there I mean I think it's in the pipeline but of course uh everything gets done in English first and then the rest of the world's languages tend to come in a queue afterwards like Spanish Chinese etc they get done but then smaller languages uh but for us vital Swedish Finnish Norwegian Danish etc etc they are way down the priorities but I think there is work going on I think from Marcus he's mentioned those it's in the pipeline but that's the key really and uh all of these applications have to have to be flexible enough to to understand different accents and different intonations yeah absolutely and I also think here that the comment um the people that make it visible so that you've got the automatic subtitles during the dialogue that's a really useful thing for people who are hearing impaired um because if you can't quite catch up with the literally etc to have that that subtitle would be wonderful yes and uh Lotta I do agree with you it's um it's subtitles are also beneficial for those watching videos and for sound has to be off absolutely if we are turning around and saying that we can join courses and do our learnings on trains and buses and transport then that is exactly right all right I think we'll have a we've got multiple attendees typing so whilst we are very quiet we are definitely looking to see what has been mentioned and that's happening the Skype has for a long time been able to translate directly from Spanish and English um but again not all language combinations work but the obvious ones do to a certain amount of accuracy and when you get that and you can speak your native language and be translated but how do you know that you're being translated well yeah you could be yes you could be speaking very seriously and the audience starts laughing at you because the translation has made a real that reliability as well as accuracy and I would like to add something there I'm a native singular speaker and we normally have two languages like the the speaking language is very different to writing language it is the same language but it's very different styles so when I tried translating things it always translates the writing style so if you speak in the writing style people would laugh at you so yes I do agree I can remember being in France one year and listening to a an audio of an event and it was in received Queens or King's English and I think everybody in the English because they were the first to enter the University of Minnesota so I guess yeah not really there golfers can't today be a big analysis first uh we we wonder if um normally in these webinars we uh you are all very silent good to see you here Laura if you would like to uh wake up and uh get a microphone then it's perfectly okay what you do is if you want to if you want to contribute to the discussion orally we'll just put your hand up and we'll give you the mic and the audio if you and the video if you've got it it works it works perfectly well as long as people have got a microphone and it's set up but I know there we are there you've got a microphone you can even put your video on if you want but please speak hello do you hear me yep perfect hi okay thank you uh great um great listening to you uh I was about to comment the crowd edit api solutions comment I made in there I think it will take a while until we get a very good uh transformation from the Swedish language and also it has to be edited of different reasons not only to be correct but also to be um it's about to choose maybe another way of writing it sometimes and if we can hook up students uh or other groups in a crowdsource way to to help out I think it could be more financially possible earlier to solve some translations or transformations so and then it comes down to how are we going to solve that and you need some sort of edit function in a good way there are many ways to try to do it but it would be interesting to see how could we easily facilitate students to help out in this transformation work that's what I meant thank you for explaining that that is really an interesting thought surrender what do you think do you think that could work um if you know about side note uh it is something uh a crowd platform so for example uh it was uh it was created by the spin-off company from University of Southampton and in that one what they do is the the videos are transcribed automatically but then the students have the ability to change them and correct them if there are errors so as uh as uh Mark said right right now uh if people are contributing it is going to get better have the direct link that would be even better but it might take a little while for you to get it if you could just type in the chat anyone else with a question who would like your hands up can be a reflection let's go ahead and do um type the information no no okay thank you for that surrender I can see sign there so that's in the the box with the direct link that's brilliant okay maybe we if there are no so perhaps we can do some collaboration and I'll have some other people try it and see whether they have better luck with it they don't but I just want to see what A Williams is typing if we wait oh thank you A Williams I must admit I'm as guilty as many of the rest of us not doing this very much uh I need to do it more that'd be a while absolutely I think we certainly here um we're very interested because we work in schools um and also in executives and you find that they they find it to be a time saver that's very much what the participants were saying at the beginning is that it is a time saver for people um so thank you for your comment there um A Williams and uh look into fully consider and embed within UCEM so that's great to see as well right well it's without further ado what I need to do is to uh draw people's attention to the fact that we will be having a tweet chat um answer induced presentation later on which is at five o'clock GMT and six p.m tonight um central european time and obviously the topic is automatic transcription services that are they good enough for accessibility so it would be lovely um if yourselves or you can encourage colleagues as well to come and join us on that tweet chat and we look forward to seeing you there but before I go any further they're going to thank you so much for your time today it's been really great to see that and it's clearly a um a subject that has resonated I'd like to round off a little bit by pointing out that uh right down here you'll sign it uh you can just click on that link and thank you very much for giving me the opportunity welcome to do that um but if you go back to the recording you'll be able to click on the links anyway uh two web links at the bottom the eden events page to keep an eye on and on that you will find more information about next the next uh webinar which is called designing inclusive and engaging structures to facilitate creativity and learning that will be 11th of december more information on the eden web a reminder about the annual conference for eden is the 21st to 24th of june in timiswara in rumania uh information is not up yet but will be next week and we'll be opening for uh contributions then so uh get your uh get your pencils sharpened for your writing contributions for next year's conference and a way out there warning for next autumn october 2020 that sounds like science fiction but there's going to be another eden research workshop this time in lisbon portugal so i thank both of you uh thrindu wendy i think we are done for today and uh thank thank you all for being here uh you'll get the link you'll get your badges very soon if you haven't registered then maybe christina can put the link for the badges into the chat you get an automatic badge if you have registered but if you haven't registered then you have to register afterwards because that makes sense in that case we will say goodbye from here and turn off the recording