 a web developer at Required and a WordPress core developer with a strong passion for WordPress. Thank you for everything you do. He led the WordPress 4.6 release, is the technical leader of the Polyglot theme and lead developer at WordPress, a collaborative web-based software translation tool. He also takes part in the WordPress.org meta and security teams to make translate the WordPress.org more awesome and WordPress more secure. Dominic is mostly known as Ocean 90. Okay, please. Thank you. I think it's working. It's working. Hello? Hi, hi. Yeah. That's it. Okay. So yeah, good morning everyone. Thanks for coming. It's a bit early. Yeah, I know. So yeah, thanks for joining my session. Okay. And as mentioned before, I'm talking today about the topic about WordPress translation demystified. You probably don't know what it is. Me neither. So I try to explain it and I hope you are in the right session. So we will have a good time now. To make the topic a little bit more clear, I have changed the topic a little bit more. Our change for today is to sheet some light into the whole process of how WordPress has translation. And that's mostly important because at the moment we are really that far that we have reached more than half of the user base, which is not using American English as language developers. And to make it a little bit more interesting for today here in Tokyo, I checked the statistics of the usage of the languages. And there you can see that Japanese is really the second most used language after American English, even before German with 5.4% and Spanish with 4.9%. So yeah, I wasn't always working for the translation part. And there was a guy called Andrew Nathan, you may know him. He was the guy who was really working for me on all the translation stuff. He has managed a lot of related stuff to internationalization. And at some point he thought about working for the White House. So there needed to be another guy who takes over his job. And yeah, that was me. So it all started in 2015 where I got disposed from Andrew where he said that I cannot take over the system. And yeah, that was really much my face at that moment because you have to know just like Wordpress, the code on web.org is also about 10 years or so old. It's not really stable. It's not really nice. It's not really object oriented or what it is. It's not really designed to have at that point to really support languages or language packs. That was something I took over and over the last two or three years, I nearly rewrote all the parts of the station code related to translation or even some core stuff like a language user which we'll see later. So that's mostly what my talk is now about. What I have done the last three years. And yeah. So if you start with a new version installation, you pretty much have something like this. You have the Wordpress.dc file. And after unpacking, you will notice there are a lot of files with some English text, even the readme or the index PHP. And yeah, as I said, everything is in English. But we actually want to have it in another language. So yeah, that's the English dashboard. But what is, if I want to have my site in another language, for example, in this language, any idea which language it is? Sorry? German? Yeah, right. It's German. And so any idea which language this is? Southern. Yes, right? You're good. Let's do another one. Any idea? Yeah. Good. We have some really funny thoughts here. As you can see, this is also with the right and left support, which is also supported by 4. And last but not least, I also have this language. Any idea? Japanese? Yes, right. So that's Japanese. So as you can see, we have a lot of languages. Actually, there are a few hundreds more. But let's take a look at the string dashboard. There you can see all the translations which are currently available for a dashboard. And as you can see, it's not just one translation. So now you can say, let's translate all the dashboards included in the verb score files. And so the initial verb.zip file would include all the translations. But it may work for just one string. But currently, we have over 6,000 translatable strings. And yeah, that's a lot. And you have to know it's not just one language. We have over, I think, nearly 200 languages. So it's a lot. So what we really need is a way to maintain translation that first scales and also is accessible for translators, which are usually not developers. That's also always something you have to keep in mind when you work with translations. The part which is actually the translation part is not done by developers, but instead by novel users who just want to have their dashboard in their native language. So as you may know, a verb is written in PHP and there we have something called getTex. And getTex is a set of tools that allow us to make applications available in more than one language. For example, the main language might be English. With getTex, we can make the browser in much more languages. And in Rust core, we don't use the actual native getTex library, but we have a few functions which are custom-prepared for a custom library called PoMo. Po and more are two file types and I will say a little bit more on that later. In this example, you can see we have the underscore underscore function, which just includes the first argument, dashboard, which is the English string. And the idea of this getTex function is that it just registers the string dashboard and the result or the return value of the dashboard should be the translation of the dashboard in your current language. There are a few other getTex functions which I will mention because they are a little bit more important for the translations part. For example, context methods. That's really an important part in WordPress because, as you can see, we have the string about, which is in English the same for all the three contexts. So the second argument in this case is page title or theme starter content, which is the argument for the context. And that's important because some languages have different translations based on the context. Even in Japanese, we have two different translations, for example. The first one is about, which is used on the WordPress about page. So the session includes WordPress, which is fine, but it's probably not fine for the other two cases. It's just probably the translation of about. And beside the context, we also can define comments, which are also useful for translators. As you have heard, they are usually a developer, so they probably don't know what the person S means. In this case, you can add a simple translator's comment to explain what the string is about. For example, in this case, it's the user dashboard to be entitled, and you can explain the placeholder. In this case, it's the network name. Now we have the translations or the strings in core, but we still don't have them in a way how we can translate them. There's a tool called MakePod. MakePod is a custom tool by WordPress, which extracts the translatable strings from core, plug-in or C-file. It's custom because WordPress also uses the special syntax for the translators. And you also have a custom header, for example, for the scene description or plug-in name, which are also extracted by the MakePod script. And there's an important part about this MakePod script which you have to know as a developer that during this process the files are only parts and not executed, which means we shouldn't, for example, use any constants for the text domain part. So, as mentioned, MakePod is a custom CLI tool for PHP, which equates portfiles for the core files, plug-in themes, and also for some of our web support projects like the localized set-asides. It's an open source, of course. You can find it at this URL. And it's kind of similar to PoEdit, which is a Ruby tool for, I think for Windows and Mac and so on. But in this case, our script was called custom header feeds. So, how does a portfile look like or what does it mean? So, port stands for portable object template. So, this is basically the file which you get with the MakePod file. It first contains the reference which starts with the hashtag and the file path and the line number. And it also has the string of course, dashboard as the ID. And this translation is actually empty. So, message-swing is, in this case, empty because it's a template file which is used for creating the actual language files. Here's another example which also includes a singular and plural version. For example, the first one is a singular version and the second one is a plural version. And depending on the language or the local, we have different translations for the singular and plural version. And of course, the portfile also includes the translator comments which are helpful for the translators. And we also have the context in this case, the method context personal data could play with. This MakePod script is run by a grondrop and who knows what the grondrop is? So, it's basically a time-based job scheduler where you can define the specific tasks which should run at a specific time. And that's what we are using also on web.org. So, this MakePod commands are used in the custom web bar which is called generate-spot.sh and it also updates the SVN checkout and finally comments the portfiles to our SVN repository. So, now we have the portfile. What would be the next part? Why? How can we translate the portfile? There are a few tools like, as mentioned before, co-edited but there's also a web-based tool called TransEffects which you can use. But today I'm going to talk about a tool called ClubWaste. As mentioned in the introduction, it's a web-based software translation tool which we also use for translation which you can use for translation portfiles. ClubWaste was, or the initial comment of ClubWaste was done 90 years ago by Nikolai and it's an official assistive project just like B-Waste or ID-Waste. There was a time where ClubWaste was using the backpress library. BackpressLibrary was a project with the goal to have all the web-score files as in library, which you can use in another project. But because there wasn't really a maintainer or we didn't want to have the same work quite because web-score and backpress this project got abandoned and even after B-Waste and B-Waste also removed the B-Waste dependency it was time to let ClubWaste also follow this step. This step was done two years ago and since then we also had over I think over 20 releases as a web-based back-end. So everyone who wants to have a ClubWaste installed can just install the web-based back-end and have their own ClubWaste installed which is far easier than before. And we actually have over 35 contributors which are led by Greg Boss and me. So now of course everyone can install their own ClubWaste installed but that wouldn't be really a good idea because how do we get all the status into your own ClubWaste installed? So what we have is translate.webs.org it's a huge install of ClubWaste on the web-based back-end which is maintained by the meta team and it's really used to get the central platform to provide translations in a really simple way. The only thing you need is a web-based back-end account and after that you can submit the translations and either you are already an editor or not in case not your translations will get reviewed by an editor who will approve the translation. And it's not only used for four you can also translate themes, plugins, some of the meta projects or even the mobile apps. Everything on translate.webs.org you only have to have a web-based back-end account. So we have our profiles and we have translate.webs.org now it's time to import the profiles which is also done in the background it's also one on the corner of every hour it's used to ClubWaste default CLE importer and if you are on the web-based.org select you also get some notifications in case of new translations for core or meta projects. Now that the profile is imported a translator will see something like this for example we have our dashboard swing again that's how the current editor for translators looks like you will see them ok let's go we have the dashboard swing and in this case this is translation which is provided by the... can you hear me? yes ok perfect so we have the dashboard swing and this one is translation for the translators you can also add another translation in case this one was false and I mentioned before in the pod files we also have the references to the files which you can see here to get a little bit more context of the initial dashboard swing for example you also can depending on your role to reject or set the status to fuzzy which is usually used for automatic translations so now that we have the translations we still don't have them in your local WordPress install so we finally have to think about how can we send the translation back to WordPress in this case we do this in so called language packs which is just an export of translation in a zip file which is updated every 12 to 24 hours and it usually includes 8 files or 6 to 8 files which is done because of performance reasons for example W admin or WP admin and the front end have to get 4000 swings and each has 2000 swings so if you load the front end we don't need the admin swings so in this case you have to split the translation up into 6 to 8 files 6 to 8 because we also have locusts for British English or Australian English which are actually using the same city names or the same translation for the city names as American English so in this case they only have 6 files which is also continent cities file now back to the promo library as said before it's 2 file types which is PO which is a portable object which is just a textual editable file with the header and translations as you can see we again have our dashboard swing in this case the message swing part is filled with the translation not empty anymore and it also includes a date the revision date the fifth line which is used to update them for example if you provide a new translation you also need a way to know which version we have installed and which we can update and the other one is so this PO file is only really used for the date actually and webpress itself use a file which is called MO or machine object it only contains the translations in bytecode so it's just the same as before only a bytecode and this is actually used by the getTex function so now we have the language packs but still don't have to send to webpress itself for this we have something called web translation API which is a function in core with two arguments the first one is for the type you either want to update plugins, themes, or core languages or you want to get translation for core plugins or themes and those two are similar to themes API or plugins API which are just communicating with api.webpress.org so since we still want to have a translation for dashboard we would have to provide type with core and as a version we can for example parse 4.7.4 and the response would look like this and then you can also see the language packs at the package key so it's a pretty simple API which just returns all the language packs and translation now we have the API and now we can get back to the initial webpress installation which should look like this if you are using at least webpress 4.0 which you can finally use this language on the install which in the background loads translation files or the language packs and it's also the main reason why you don't find any translation anymore in the initial webpress.zip file which you are loading for on webpress.org and there's also a fun feature depending on the language it also updates the continue button so it's really awesome and really webpress asks language packs because of this reason to make it so easy to select another language than American English those language packs are stored in the WP content language directory which also makes it possible to share them across a site if you have for example a multi-site there's just one language directory and you can install as many languages as you want the language files are also updated automatically you don't have to press any buttons or I don't know, load them via FTP or so and it really shouldn't take longer than 24 hours to get new translations because we have a super easy process to update them which also includes the work by translators and editors and with the help of language packs you also can easily switch the language after for example if you start with English and now you want to have your site in Japanese you just can go to the settings page and interact without the language pack and activate the language and that's all you have to do these also includes the translations for plugins and themes which are also updated also updated during other updates and because I know some people don't like really an auto update so far this is probably even one more reason why shouldn't this evidence because otherwise you wouldn't get all the fancy stuff provided by the language packs now we have the language files on the server now we have to load them in this case we have the translation API we have for example the load default text domain function which simply loads the language file on the server depending on the context on the font and wp-admin or network admin and with the help of this function we are finally back to our getx function which is the underscore underscore with the dashboard swing and now it should return the translation of the swing dashboard in your selected language so the question might be are language packs also a thing for plugins and themes and the answer is of course yes and they work more or less like course it basically means that we started in December 2015 where we also allowed to have plugins and theme auto suits where we allowed plugins and theme auto suits to use our translation platform which means those autos don't have to update or create any pod files anymore or you don't have to ship translation files by translation files which may contain any spam or other swings which because you don't know the language actually and you don't have to do an update of 6 actually so the only thing we really have to do just define the request at least 4.6 then you don't have to define the text domain or load plugin text domain because our work score is so clever that it all does to the task for you yes of course the plugin and theme must be hosted in the official web directory and that's all you have to do to make language packs ready for your to make plugins and themes ready for your language packs if you want to know more about them we have a handle page where we have some more information on this some good to know things if you have your plugin or theme autos for the first time importing into translate.viz.org you have a custom or its own language called meta language packs which has some helpful information for example the first one was ok and the second one has some issues so the plugin autos now knows what they have to fix language packs for plugin and themes are updated every or updated 30 minutes after a change and they are usually generated when the project is at least 95% or more translated and there's also a special page on your web page on the state.viz.org where you can see the current language packs so there's a dot dom menu where you can select language packs and then you can get an overview about all the available language packs so you may have noticed that I have mentioned contract a few times today and it's probably one of the most important part of the whole system because without contracts we all had to do the task manually which was done, I don't know, five years ago or so and since we now wanted to support plugin and themes we really had to find a long-term solution for this and since you are already using WordPress for the plugin directory or now also CloudPress which is a WordPress plugin why not using WordPress for this and luckily WordPress has its own Chrome API with the function wp-scatter-single-event which we could use but there's one thing you have to know about wp-chrome it's not really a Chrome system like you know because it's kind of a sort of a Chrome system because every time a user visits your site it makes a ping to your or the site makes a ping to itself to get the actual jobs run so if you don't have any users you don't have any jobs executed and that's not really a good idea for using this for our language specs and so on but in this case we are just using another WordPress plugin which is called Cavalcade Cavalcade is a domain replacement for wp-chrome developed by Hugh Mayn and it basically looks like this we have our WordPress with the default wp-chrome API functions we have a custom table called wp-cabalcade-jobs which includes all the jobs to I don't know to import plugins import themes or update language language specs and it was really easy to integrate this setup into a WordPress.org we didn't need any code changes we just had to install the plugin, the top-end placement and custom runner which runs now on each server and has four default workers which can now run all the bone jobs some of the features are we have the guaranteed running so we don't have the loopback HTTP call anymore it's really run by a custom worker on the server it's also designed for multi-site which we are heavily using on WordPress.org it's scalable because you can run the worker or the runner on each server for example if you have just one server or one runner with four workers or if you have like ten servers you can run the runner with each can have each four workers so you can have as much one of the cavalcade to execute more bone jobs at the same time same time is the next point probably processing you can run depending on the actual workers for example if you have just one server you can run four tasks at the same time and because the contracts are all stored in a database table you also have some kind of status monitoring which you don't have for the default bone API as I said before it's a webstack in you only have to install the webstack in define or disable the default WPC bone API and you also have some WPCI commands to run commands in case of errors and the second part is the one which you install on your server which reads the jobs from the database table and runs each command and that's something we have now running since webcom puro I think it's 2060 or 2017 doing this it already has processed over 70 million tasks which is only really possible with the help of KVP and all the setup we now have so I really often get asked about some statistics about the translation platform so today I want to share some of them at the moment we have our 180 locals which means the only new for example the language user on the install could display 180 locals but there are only a few I think about 50 at the moment who are really fully translated but it's possible to translate in 180 locals we have a special role which is used to review and provide translations we actually have over 20.800 editors which can provide translation by 41.000 translators and those translators can actually translate over 100 projects which includes core all the versions down to 4.0 I think or 4.1 and also the plugins and themes and mobile ads which means we have a lot of translations here you can see a little graph for example in 2000 we just had in this year over 700.000 translations from 2015 where we started importing plugins and themes just in this year we added 21.000.000 translations just for plugins and themes and today we had over 54.000.000 current translations and these translations are also provided with the language text so each plugin and theme and core has its own language text and there we have actually over 100.000 active language text which are shipped from the API to any web installation which means we have a lot of downloads of these language text which is probably now a little bit more over 3.5 billion downloads now I am coming to the end but before that I want to talk a little bit about the future of each section which I have talked about today the first one is CloudPress we have some ideas for CloudPress the first one was to have cloud service for localization and also job localization there is something we have done last year some of the other topics are to have comments or notes on translations but also to have a way to reject translations with feedback there is also an idea to improve the API to use the WebPress infrastructure of the West API to improve the templating, the design but also thinking about introducing translation memory which is a way to provide such a distance similar to PoEdit which means we use all our translations which we currently have to have translations much faster to provide the translations or also a consistent translation for language pack we want to have faster updates for PoEdit as mentioned before they are currently only updated every 12 to 24 hours and for blocks it seems to have 30 minutes so we really want to have them also faster and we want to reduce the number of complex packages in this case, for example, Japanese also one of these we may know as WP multivide patch which is something we have talked about yesterday and we want to have release bit also more automatic which means move the readme HTML to consider the support and to stop the requirement to have SVN for some files and we also want to switch to some commands more to the WP CLI ecosystem there is a talk at 2pm from Pascal to know more on this part I really recommend to go to his talk and also some ideas for core of course we want to have some performance improvements because get access to one of the functions which is probably used the most in core even for American English and ideas for example to use the native getex library in PHP and also we have to find a good way to provide JavaScript localization in this case you have to take a look at Gutenberg because it's a JavaScript application and there is currently no I mean there isn't an API but not something I would like to see provided in core so we really have to work on this part and if you really interested in one of this part like Godwears or core you can get in perfect with me I'm happy to provide any help because I know it's probably one of the complex topics because it affects so many parts of what was itself and that was my talk