 So those of you who have read the program carefully might be surprised that there's only one person on the stage and not two My colleague Odin who was supposed to present with me has been prevented from attending today Which is unfortunate because he is actually the Programmer who made the tool that I'm presenting So yeah, I'm to the huggles, which is rather an awkward name to have when you work and live abroad as I've done extensively Actually the first time I used moodle was I believe in 2007 We had a private English school in Seoul South Korea teaching English to Korean middle schoolers any Koreans in the house No But now then I'm working for the Norwegian directorate of for education and training so essentially the Norwegian national government now we don't Do education directly that's a municipal Responsibility in our way, but we have a moodle site where we do some what you call Supplementary training in those rare instances where the municipal response of the Municipalities are not be able to sort of fulfill the responsibilities because Norway as Those who saw the presentation from Chef Fredx yesterday's heard is a very like Why long country few people sparsely populated? and the different municipalities Ability to fulfill the responsibilities in in education varies a lot from from Municipality to municipality So that's why we have this service another Quirk of Norway is we're we don't have many people, you know five and a half million. That's About the same as greater Barcelona But we have actually many languages we have to deal with There are two official languages in Norway Norwegian and Sami Of which there are several writings written standards. So Norwegian has two written standards Book mall and Nino's Sami comes in three variants South Sami Lule Sami and North Sami. So just in the official languages we have to deal with five different Languages so for us it's a bit weird that moodle like groups all of this into one Main language Norwegian and then the other language packs are sub packs to that main language And then we have three recognized minority languages Qvian, Romani and Romanez So the service that we run they call digila.no because of the Training we provide there we also have to support some more languages. So we support arabic Tigrinja and Somali and because of Ukrainian refugees from the war we also have a service so we support Ukrainian So our site our moodle sites has to support 12 different languages. So obviously for us translations and the quality of translations is very important Right, so the quality the challenge for us then is that The moodle language packs are crowdsourced in the AMOS. So anyone who wants to can just go in online and start translating Or suggest an improved translation or at least a translation Whether it's improved an improvement or not this might be up for debate And then there are these language pack maintainers that approve contributions Any who has contributed translations to moodle There are quite a few yes any language pack maintainers in the room Same right how about we give you guys This When we say that there is a challenge here that's in no way a disparagement on your work, which is very important But more our recognition that this is a huge task with thousands and thousands of language strings And that keep changing when functionality changes keeps you keep adding new strings with new plugins You know every time some new plugin comes out. It has to be translated into 55 or how many languages, right? So what we've seen is that the quality varies a lot with translations just looking at Our site we see things like typos. I've seen typos that actually break Placeholders in language strings. There are sometimes placeholders and other code So if you have a typo in there, it can actually break functionality in moodle We see lack of context sensitivity and now I'm in the sort of Norwegian context we see words that are actually valid Norwegian by like Dictionary definition, but they're not really used in that way in that context Incorrect Norwegian syntax, so we have a translation that where each individual word is valid Norwegian But the syntax is like in the original English syntax so There are some we have some issues with some of the translations and we actually also I should say I'm talking about Norwegian now, but we have funded Translations of the Sami languages because they are very small. They're actually in danger of being extinct So we've funded some translation of moodle into the Sami the three Sami languages say I mentioned earlier All right, so with this challenge You know what could be a solution So again leaving behind those 11 other languages we have to deal with for now and just focusing on on Norwegian We thought what about if we build some kind of Tool where we reduce the need for manual review all these like language string for some strings like 47,000 strings And we can pinpoint where which strings Need to be manually reviewed And also help translators and maintainers doing this manual review and using AI So that was the the idea so we decided to build proof of concept or pock of such a tool And this is an external tool as of now. It's not a moodle plugin or anything like that We're one of those many apps that was mentioned in the keynote on in the beginning of the conference where we are basically just One of many apps piggybacking on chat GPT So we did a test first using the chat GPT model text Da Vinci 003 be a command-lined interface And we only asked chat GPT to give a thumbs up thumbs down, you know are these is this string of other translation or not? true false So a very simple test really Ran all the language strings Through that So we got a Result of 10% being considered invalid by chat GPT so 10 thumbs down But that still leaves, you know for report maintainers for about 4200 strings to review and that's a lot of work So that's not very good Another problem was false positives so things that strings that were actually correct, but were flagged as incorrect or invalid so Da Vinci was not a good fit for us. So we went on to The next model the turbo 0301 model is still sort of these are all like submodels of the chat GPT 3.5 model Which has an API connection. So we hooked on to that And we made a chat where there would be three roles You have a system role that initiates the chat and defines its context sort of that the whole prompt escaping that you need to Give it get good answers as you know The quality of the answers you get from chat GPT is dependent on the quality of the questions you ask so we had the the system role set the The context, you know the user is working on a Norwegian Translation of an application used by middle school students in Norway i.e. our middle-sites And you are a helpful assistant who will for each other users messages respond with in this case then an integer between zero and a hundred Basically a quality rating of this translation. I won't read the whole prompt, but basically the idea is to Create a prompt for for chat GPT to give good answers And then the user asking the questions or in the in the process here like feeding chat GPT the language strings and Then the assistant i.e. chat GPT answering the questions In the way that it has been told to answer it by the prompt All right. So actually we did this in three separate processes So for each language each string and language pack we would then get three outputs Okay, so when we had the User ask or you know send in The current Norwegian translation in the Norwegian language pack We would ask okay question one is this a valid translation or not true false So basically the same as we did with Da Vinci In a separate process we would send the same thing Current Norwegian translation, but this time as you saw in the previous slide We would ask it ask a chat GPT to rate this translation How good is it on a scale from zero to a hundred or one to a hundred and Again in a third process We would send in the original English language string and ask chat GPT to come up with a Translation you know you suggest a translation of this into Norwegian. So what we'll get back would be true false rating from one to a hundred and Suggested Norwegian translation for each of the 40 some thousand strings in the middle database All right, so results of of this test done something weird is happening where The animation is showing on that screen but not on that screen That's very interesting, so I guess you just get all at once So as you see now, then you had about 25% were chat GPT Rated 100% and Suggested the same thing as is in the original translation. So then it's good to go About 50% was rated 100% but a different suggested translation still probably fine and Then you can go down to you know a rating of 80 to 100 that's 12% rating 30 to 80 that's 7% Rating less than 30. That's 4% and Rating less than 30 and Thumbs down this this is not considered a valid translation by chat GPT. That's only 1% So if you're gonna do manual review Start here Right, then you have it if you remember back to Da Vinci We got a 10% considered invalid So you would have to sift through about 42,000 sorry 4200 Strings now we're down to about 400 strings. So that's much more manageable And of course as you have researches you can go resources you can Start there go up to the less than 30% 30 points the 4% or and not going up the scale as needed So if you refer back to our goals with this proof of concept Can we reduce the need for manual review of strings check? You know you only have to review 1% of the strings Can we pinpoint where manual review is most needed? Yes, we get a list of this 1% so you can just use that list and go through it and we've looked at this list and It's a lot of strange text there and That brings us to the thing that the third one help translators and maintainers with manual review check because chat GPT has provided us with an Alternative translation which again we've reviewed some of these and we just copy and paste them right in this perfect translation So this is a very promising tool For and of course this is just for for Norwegian, which is kind of like in a in a sweet spot Where it's a small language. So you don't have that many translators But it's not so small that it's big enough that you have data online. You have a lot of Norwegian text online Right so going forward then This was just like a proof of concept a test will this work and how will the results be and in all honesty We haven't gotten to do as much with this as we had intended again. My colleague who couldn't be a today has For health reasons not been working for us for a while But some of the things we thought about when it comes to improving that tool, you know not product in it You know, it says product improvement, but it's not this is not something we're gonna sell or anything. You know, this is open source We could work on getting like automatic checks on new strings. So when there are new strings new plugins We can have the tool automatically check this We could define parameters where the The AI would automatically just fix it for us So we could have have sort of limits that okay these In these areas, we trust you and you know, you just make the changes But others will have set aside for manual reviews. So the maintainers still have control We also thought it could be a good idea to have the AI output its reasons for Rating, you know, it kind of like explainable AI. Why did you give this string an 80 or why did you give this string a 30? Which could help also improve the the tool going forward So those are some of the ideas you have and again, these are Ideas that anyone can just sort of grab and go with we can share. I don't know if our code is Open it's on github. I'm not sure if it's an open repository, but we will share it And we can use new AI models, you know, we did chat GPT 3.5 but four is already out. There are other models out there you can try And then ideas for how to sort of disseminate this tool The lowest sort of level in a way would be to create a moodle plug-in so that everyone just like installs the plug-in connects to their their AI of choice and Chucks their local install that would be kind of a very low-level use case of course the high-level use case would be to integrate with Amos so that Package maintainers or even translators could actually use this when they are working on Amos That way everyone Can benefit from their their work For someone has to foot the bill to connect to to the AI because that that costs money right to subscribe to chat GPT or what have you and Hopefully the result will be better translations easier and faster for all languages That's Thank you, that's really interesting. Do we have any questions we have some time we've just under 10 minutes So if we have any questions, please put your hand up and we can go around with the mic Okay I've got lots, but I'll let her let other people ask some questions first Thank you for presentation if I'm right this algorithm can be adjusted to other languages as well Yeah, we just connected it with the moodle language packs for English the original language packs and the Norwegian Translations in model so if you do the same and connected just to the original language packs and Slovenian or Mexican or what have you or sorry Spanish what have you then you do the same thing Great. Thank you. Oh This is one over there Yeah, I have no question But I have the idea for Dissemination that we should as a moodle open source community not rely on chat GPT but on That we should look at hugging face and find large language models that are free and open source and use them and Support this community as well. So we have we have to struggle with Salesforce in the in the Moodle partner community and I don't want to struggle with chat GPT in the whole moodle community Right, that's that's actually a very good idea. I hadn't thought of that but That seems to be the thing going forward that a lot of people are making their own language models Of course big companies are doing that corporations are doing that We are looking at Some other some are very interesting Establishing our own AI systems, but if you had like a sort of moodle AI they could do this for us. That would be great Thanks for presentation Tori. Is there a kind of consistency? Does it remember that it take one term to Translate a certain thing for example in Germany if you go to Simon to do a submission That can be an eichrein Upgabe it's it's and the thing with Human translated thing is I out there depending on what year the string was translated. It's sometimes not consistent Have you done any observation this is it consistent? Is it? We key Yeah, no, we've we've simply seen places where for example that the word sites In Norwegian at some point when it when someone started translating moodle into Norwegian They just made a choice of using the word portal the portal in Norwegian instead of sites and so Chat GPT keeps suggesting you know use the Norwegian word for site instead of put all which is But we could just sort of make a kind of like whitelist thing like you know Remember in this in this application when they use Put all as the translation. That's okay. You know don't don't fix that or but yeah, obviously you would have to work on the prompts to create that kind of sensitivity to the moodle specific terminology in a way I'm I'm curious about how you will continue forward with much smaller languages that perhaps chat GPT or some other large language model doesn't have much training data on And I have you any ideas on how Well, actually, I was I said before that that Norwegian is kind of like in the sweet spot where where it's big enough that there is Some or quite a lot of text online Whereas this three semi languages for example that we have in Norway are too small That's sort of the problem But the idea that our our colleague over there had about our own AI Maybe we could like train the AI on smaller languages to ourselves to address that in advocacy my tour my colleague One suggestion just for for this problem with the portal stuff is this solution is actually not marking any Translation as yes, that's what we want. So we just have we're just guessing on on the complete result list Seeing chat GPT is taking 50 percent or 75 percent. Yeah, that's a good translation But we're not training it So that would be the next step if we take our moodle translation And we every translation we take and then just saying okay This one is 100% cool and then we feed that stuff into chat GPT and then we would have We wouldn't have that issue with sites and portals because we define that's a good result So that would be we should train our language model to keep up with good translations Yeah, I know that the maintainer is mad at us. I was thinking about Consistency and we have a model.org in our Basque community at Glossary of those terms That we use translating moodle so you can train this AI with that kind of glossaries of good terms to translate To your language so you can have best translations in that way. Yeah, excellent Yeah, I didn't it was in the in the prompt on the screen, but I didn't read out loud But we told in our prompt we told them to use the Norwegian Dictionary, but we could also say use it in the Norwegian dictionary but also refer to the Glossary and moodle.org and you know, yeah Well, okay, we've just got a couple of minutes so if you could you go back one slide I think you were talking about the way forward. It doesn't matter Okay, and you talked of possibly having a moodle plug-in, but you thought that will be quite low-level Potentially having it in core. So are you in contact? Do you have accounts on langa.moodle.org? Which is the translation site for translators? Have you talked about this there? Have you are you in contact with you know? David Mudrach, who's sadly not here, but he he made Amos basically and Kun Rohamans who looks after all of the translators They're both teaching in high schools at the moment actually as we speak They'd be really interested in knowing more about this No, we haven't Yes, I said before we We had expected to come further in this both in terms of developing the tool itself and the dissemination part And actually we we thought that maybe someone else I thought of this too Well, I shall be speaking to them in our moodle chat very shortly. Okay