 Thank you for being on time. Have a great session. Thank you. So we were recording in your voice and your slides. Okay. And this video will go up onto the website. Today sometime, you have someone ask, is there a question if they need to go to that mic, or if they shout it out, you can just see the question over. Sure. We'll do it with the mic. Up to you. Okay. Yeah. It's on. If you can even take it off and pass it around. Okay. We'll begin in mic two. Yeah. Okay. Let's go. Ready? So I'm welcome to this Drupal.org conversation, because yeah, it's a session that we can converse and discuss about the things that I will present. And we will present with Gabbo. So my name is Seb Gorman. I will be assisted by Gabbo, which is the master of localization. First of all, this session is about localization, also known as L10N. Don't be mistaken for internalization, aka I18. This session will be about translating the interface of Drupal and its module. And we will present the current infrastructure and process of translating, where we are about the Drupal aid support and how to make the work of translators easier and more recognized. So if you want to know about internationalization, there will be a military world session with Gabbo, or also NVJ. And that will happen at 14 today, room 117. Okay. So localization, how does it work? It all started with the T function. This T function is in Drupal since for a long time. It can accept variables and contexts. And this T function is extracted, the strings used in T function are extracted from Drupal. And it's modules using products module. And they are extracted by the site localize.drupal.org. So this pure source files can be translated by many languages and by translation groups on localize and also originally on POA edit, which is a desktop app, or maybe on other sites like localize.biz, which is an online tool. And it's really nice. The interface is really nice, maybe I'll demo it. And also translated, it can be imported back in Drupal. So these strings are in your installation, in the two tables, local sources and local targets. And another option is, of course, to use the localization client module to directly edit the translation on your site. So these are apps to translate, but we are community, so we have localize.drupal.org. And this talk will be most about this site. This site is a combination of a few modules. There is a POTX module that extract the strings from the release of Drupal.org. They are automatically imported. And it offers a tool far more performant than desktop apps like your cross-release translation. So we have a pollination between project and releases. We have multiple groups for translating multiple languages. In this group, we have roles, and there are reviewers, contributors and admins. And then from that, we can build a search process that automatically updates Drupal translation and contribute inside your site. And it is now in Drupal 8 core, thanks to Gabor and his fellow core countries. So I'm going to present you the process, the whole process with the module used in localize.drupal.org. How many of you know how the inside of the site, localize.drupal.org? Okay, a few of you. The whole process begins with Drupal.org. From there, we pick up a release.tsv, so it's like four megabytes of all the releases that are on Drupal.org for all the projects. We pass it, and we take the latest releases through the localization server connector, which is a REST connector. And from that, we keep the files and we analyze them with Partix module. So Partix is used to extract the strings. And so the strings are stored in the huge database that is localized to the Drupal.org. And then through the localization community modules and localization groups, which is relying on OG, the community can translate all the strings. So the community can translate all the strings through the interface. The interface is provided by the localization community. So when you go on to the localize.drupal.org site, you can see an interface with a lot of rows for each string. And this is the localization community module that provides the interface. And then you have multiple ways to submit translation. You have localization remotes that is part of the localize. And then it communicates with the few localization clients that you can put on your site. So you are both translating your site and then submitting the translation to the localize server. So you give to the community without even knowing it. So it's really nice. And then once it gets submitted, the reviewers of your group, your translation group, can review your suggestions. And then we have a localization package that extracts for each language the PO files and makes it available on the ftp.drupal.org. And then from then you have the localization update module or Drupal 8 core that can fetch these translations back to your site. Okay, so this is the whole process of localizing. Any questions on this one? Is it clear? Okay, if it's clear, I think we will make it... we'll put the schema on localize. Because every now and then somebody asks, how does it work? And there are like products, a localization server, localization clients, localization update on our Drupal 8. It's four modules just for translating. Okay, and the great news is that in August, localize.drupal.org that was on Drupal 6 for longer. It was created in Drupal 6, I think. So the issue for porting localize.drupal.org was created far long ago. And then thanks a lot to these people. Many thanks to them. It is now D7 powered. So we can now accept feature requests and work on these features. And of course the second great news is Drupal 8 core now ships with automatic downloads of core and contrib modules even during install and this will hopefully attract more international users of Drupal. Okay, now some numbers, scary numbers. So you may not know that but localize is one of the biggest sites in terms of data because we are handling a lot of things, we are handling a lot of projects and for example we have a 112 translation group. It takes thousands of contributors and counting. One million translation recorded so they are kept back in the database as of now. And of course there are a lot of work to do as of translations like reviewing suggestions and so on. For the developer, well we have a huge database that is six gigabytes and about half of this database is localization related. So for example we have eight million lines just to match the file where the T function was analyzed, the line, the type I think it's a suggestion of translation and the project and the release. So just these columns are making eight million rows. Okay, so from that our current status for the whole suite is as is. So PODX module, the Drupal 8 compatibility that is the capacity of extracting strings from Drupal 8 is nearly done. There are three issues remaining, maybe a few now because yesterday we were sprinting on this. Two of them are criticals. That means strings won't get extracted if they are not resolved. Some strings won't get extracted. So the first one is support for Drupal 8 shipped configuration translable. So we have a few issues with our country modules that are depending on other configuration dependencies and external dependencies. And then we have recently renamed the translation wrapper class to translatable strings and the issue is quite resolved, I think. It actually got committed this morning. Nice. Not by me, by Haram. Cool. And then a minor one on the formats of YAML translation buttons. So we are pretty good on PODX. The next thing will be to port the PODX module so that Drupal 8 sites can extract themselves. Yeah, so right now the problem is if you want to check your source code if all the things you think are translatable are actually translatable on your Drupal 8 site. You now need to have a Drupal 7 drush installed so you can actually run PODX on your Drupal 8 code which is not very convenient. Especially since PODX is pretty version independent so we could make it work with Drupal 8 but we need to port some stuff to Drupal 8 so you can use PODX to verify that your code is ready for translation in Drupal 8. Okay, so now for the localization update parts, it's all good. We have now the capacity to update translation in Drupal 8 core. So meanwhile the interface of Drupal 8 core has been backbooked to a new Drupal 7.2 branch. And that's it. As for localization clients, we don't have a version for Drupal 8 yet. We have a new proof of concept made by Gabor and, sorry. And Kevin O'Leary. Kevin O'Leary. Okay, so it's based on the Toro module. I'm going to show you a quick demo. The main issues of this proof of concept are it's far off in terms of D8 API. I managed to install it on the last beta 15 but maybe some Dx things have changed. So the idea with the concept is people always wanted to do in-place translation of things on the page but Drupal outputs things at all kinds of places on the page that you cannot really in-place edit. So the idea with this proof of concept is to use something like the Toro module to point to places where you have things that you can translate and then you can in-place translate them there. And then the rest of the things that may contain dynamic information that we cannot find on the page could be shown at the end in a summary and then you can translate the rest at the end of the process. So you can kind of get an in-place translation experience on the page. It's not very scalable if you start off fresh translating a page because then you have hundreds of strings on the page and an input field jumping around on the page that you need to follow. But considering that the community should translate to several languages this is useful to touch up on things that are not yet done and using the remote submission functionality contribute to the community as well. Yeah. So about that, for example, strings that have variables in them can't be translated but we have kind of a fallback window for that, that is new too. And of course it doesn't send or save anything actually right now. So it's a bit of a shame but it's a good proof of concept and here is the demo. So I've installed only localization clients enabled one language and then you have a little button at the far top right on the nav bar and as you can see the tool module is displaying some pop-ups so some of them are a bit far outside of the page and it should replace the text that you are typing and then if you don't see some translation you have this fallback window at the end. So that's cool and you have a little percentage thing that we don't have in 777. Okay. So that's an idea. This is a video of actual code running on Drupal 8 but it doesn't actually save anything or it doesn't actually send it to the localization server so this is a proof of concept that we started as a hackathon project and then it's at the waiting for someone to help finish it up state. Looks really cool. So the question was is it possible to translate making on it if it was untranslated? With this proof of concept no but I think the tool module adds enough metadata on the elements that we could write some javascript to figure out if you clicked on something if there's an element in that area of the DOM that has the metadata and show the field. I think it's possible to do that. We need someone to help get this move forward. Wim Lear, if he has the time he's like triple our quadruple book or something on all kinds of things. Another question? Next up is the main part localization server that is powering localize. So now we have the Drupal 6 branch that is now discontinued except for security releases and back ports. But it has a lot of interesting remaining issues regarding usability and a whole lot of future requests that we can port now to Drupal 7 which is now powering localize.drupal.org So I'm going to review some of them because there are some ideas that are really nice and maybe at the end of the talk I will ask for a show of hands of the main ideas that you want to be put first. Okay, so on the Drupal side it's getting better with the localization updates and so on in Drupal 8 core but now we are facing a more problematic situation. The localization community won't get the recognition because people that will use Drupal 8 sites will just have a Drupal site that is localized and that just works. So we may need to provide some gamification logistics and badges for the community that is on localize and maybe feature some links to the localized sites during install and something like that to keep people coming to localize. The infrastructure is handling more and more strings because we regularly had languages. We have more releases, more projects and so on. So first we need to archive some old data that is Drupal 5. We have I think thousands of releases in Drupal 5. This will hopefully improve search performance and performance overall. Next up is a translation interface that needs a revamp. Right now it's 2600 or 7 hish. So we may need to think about the UX of the translation interface. So why not translate not exclusively but most of the time in localization clients or maybe in a whole other interface through REST APIs. That's IDs. The community aspect needs a push because even in the groups that are more and more active we have a loss of momentum due to now we have lost notifications through mail but we need to get it back maybe or maybe have activity streams like who translated what in yesterday or something like that. Maybe more discussion about the strings themselves like glossary handling and maybe report back to the maintain of a module that the string is not really easy to translate. Something like that. So we will discuss these problems through examples and IDs that I've picked up from localization server issue Q. First I would like to show you trans effects. How many of you know trans effects? For the record it's a site, an external site that is proprietary but a lot of open source projects are using it. For example, VLC, Django, own cloud and it's free for open source projects. So I'm going to demo it bit quickly. Just the image is blurry so it's not nice. The interface is that simple. It's fast visual search and the images are really, really nice and it's string-centric and we have a mass review search and replace and of course activity streams, announcements, statistics and so on. So I'm going to demo the interface. Not seeing your screen. So it's free for open source projects and you can have as many languages you like. You have some statistics on the front page of a project. Basically we have that. That's what's interesting, the transition interface. So you import some POT files that are templates of source strings and then you have a list on the left that you can bulk check and review or in review. You have even matching translates through Google translate engine or Microsoft but it's paid services. You have bulk find and replace that is really nice and some tag handling. So then when you click on only one string as I said it's very string so you have the untranslated string which on it you can add instructions about the strings like don't touch this this is frozen for example. You can add your own suggestion and save it and then you have a bunch of tabs that details about the string so you have the occurrence of the strings in the files of your project and you have all the suggestions that everyone has made not yet. History about the strings and a little of this little thing is really nice. The glossary I will talk about this later and commands. Okay. So this is a quick demo. So we could ask the question of porting the whole thing to trans effects but it has APIs and so on but the work that we've put to make localize and Drupal.org work together it would be a bit of a shame to not have everything in one place that we know we can handle. And then maybe later add badges on Drupal.org for translators and so on. So I'll report the question to the end of the session about that. So we just we just saw a glossary section on trans effects actually I am part of the French community and we have a French site it's a whole external site high and a colleague of mine manage. So it's an external Drupal 7 sites we have a content type that is glossary term. So as in localize we have source string which is legacy on this example and then anyone can put its suggestion and everyone can vote on this suggestion and then it get extracted to a JSON file we have some browser extensions that are available. So this is the site we have browser extensions for Firefox, Safari and Chrome. So we have like 400 strings I think and some of them have a status to be reviewed and some of them are frozen. So every time someone comes and want to translate we show them that there is a glossary to follow and it's really nice for the translation, French translation sprints. So this is really nice and we would like to put it directly in localize. Also I would like to point out that the interface of localize is much better to translate with these little little highlights. So these are available directly on localize and on localization client module in your site. So that's your browser extensions integrating the terminology information from the JSON that was exported from the French site. So we could put it on localize but we would lose the capacity of having your glossary on the localization client module. Maybe the browser extension should be kept. Okay. So now about performance. Is there any question about that? Yeah, absolutely. The question was was it some question but yeah. We could add some roles while some permission about that in the translation of teams to freeze the glossary and then everyone would be having good translation. Okay. So now to performance. So the third thing that we absolutely need to do right now we have Drupal 8 that is coming. Drupal 5 is really far. So we should archive strings and that should give us some space. Of course only the strings would be archived. The translation files would be kept on ftp.drupal.dog. What we could do then is improve the interface and performance interface by doing more Ajax request. Because right now if you select the filter of 50 strings there are rows of translations and their suggestions and it's really really slow. So we could add some kind of pair string loading and saving by Ajax. Then we have some SQL request made for statistics slow. So maybe if someone that has good SQL skill can come by and help us. Another idea is to index strings in another server, solo server or elastic search server for full text search and also for statistics it would be a lot faster. And anyone can add its ID. So remember it's a discussion so don't hesitate to bring up your ideas. About the UX improvements we have a few ideas about that. The main thing that I get when people are translating French is that we should focus only on the cases. In the search bar we should have automatically the latest table really selected for example so that people focus on the strings that matters. Another idea is to get search for example, reviewers would like to have automatically the filter on that would be great to review faster the strings of one project for example. So we have this idea of visual shots which is a little JavaScript plugin that even trans effects has and I didn't know that but I proposed this one and it's the same one. So there is an issue about that and I think it's 2.7 so it should be tested a bit more before getting into localize but I think it's a nice idea and it's a nice beginning for improving UX. Of course we have many other ideas like statistics on community activity improve new camera experience by using the back part or even the back part of the tour module or even a joy ride to welcome the new translator Of course comments on suggestions and translations would be a big plus. Translation memory which is auto suggestions from other strings it's known as fuzzy matching when you have translated for example one orange to an orange in French you have fuzzy matched with two oranges and it would be faster for translators. We have I don't think we have the same spam prevention on Drupal.org and it's getting a bit hard to fight spam so we could ask the DA to help us about that. Rest support would be nice for fighting source strings in external apps for example. And then we have some other problems like language inherent and some local language would like to inherit a localization string from their main languages like English from UK and English from US. We could have staging imports and bulk review in localize we could have localize.drupal.org that is localize so you could translate in your own language and then mark down it would be good. I have a few RTL languages that are complaining about the support of RTL languages too. Okay, thanks. We'd like to thank our respective company Acquia for Gabor and Machina Corpus for me. Any question or reactions about that? Feel free to use a mic. If you don't let's print all week on that especially on Friday. Yeah, sure. Use it. One question regarding trans effects so that is to have similar kind of features in localize.drupal.org or actually use trans effects because it's proprietary software. The thing is we would like to put some features trans effect into localize.drupal.org. Actually we are only missing like they have good statistics and and come on some strings and glossary of course. But glossary I think it would be really fast to integrate. So we have 70 or 80% of trans effects available so it's really nice. Thank you. Yeah, go into the mic Yeah, so you mentioned a whole slew of issues. What would you say the top three are in terms of priority? Which things would you like to focus on? Okay, the top priority is to archive all strings and then we have some work that is done. Like the visual search interface is done. The patch is ready. You just have to test it. The thing to test on localize is that you have drupal.org development sites to test on because you have to have real data and this is not done easily to work on your local development server. So we prefer to work on drupal.org servers. But it's really to test so we have archiving drupal 5 strings we have visual search and maybe the glossary would need to pick up the ID, maybe make it a feature and then adding for some permissions and the ID we just have to put it on the code. For performance I think there's probably some quicker wins with adding indexes to database tables and SQL type stuff. So I'd actually put that ahead of archiving which I imagine archiving would be a little bit harder of a task. But both are definitely good ideas. We already have some indexes but we should take every request made by the localization server module and explain my school. Definitely we could add some index especially for statistic that are run on cron runs and maybe take an hour to refresh or something like that. For some I think the main page the front page is doing is taking this time to process. There's probably a lot of opportunity there but we need someone with some time and some expertise. So you recognize some of that already in issues? Just one so far but I'm sure there's more. I think you found through database logs or something or how you recognize those issues? Yeah in general for that one I just got a query process list while I was waiting for the home page to load. Okay. But we could use better tooling for the Procona slow log monitor. You could probably install that in your home directory on the dev server and use that. That could be good. Cool. Yeah we should talk about that more. So I have a question about from a perspective of reviewers can we actually get out right now because I have no idea when translations come in? We have data about everything. Right. It's not available. Yes. We don't output the data because of our performance concerns. So we have data about the smallest detail of everything. So when somebody submits a suggestion we track when was it submitted, who submitted it which tool they used to submit it so we can tell if it was coming from localization client or the site or something. We track when it was approved, by whom it was approved if it was rejected, who rejected it, when it was rejected, if it was approved again or somebody submitted the same thing again who was that person. So we have a lot of data on everything. Which is part of the reason we have that huge database because we keep all that log information for each of the half a million strings and times the 12 translations. So we have all that data we need to figure out how to make that available we need to figure out what kind of data two people would like to see. So the idea that Sebastian explained about activity feeds is yes, now when people submit, if somebody translates views in your language, you have no idea they did it because you have no visibility to the data. So some teams worked around that by submitting a post say I just submitted the views translation, please someone look at it it's not very nice that you need to manually do that. I think we should have activity feeds to expose that kind of information and we actually had a Google summer of code student try to do that two years ago but he failed to find enough time in the summer to complete it. He would have been entirely capable technically but not capable in terms of time. So we had a chance there but unfortunately we haven't been able to get a result from there but it's a very good idea that we should explore. So we have an issue somewhere. We have an issue somewhere that's a short answer yes it's not a very comforting answer but yes. What could help would be what would you want to have displayed in terms of statistics. As for me I think that we should be focusing on the latest release so we should swap out all the releases that are in Ontario to this one. To the stable one. So if you have any ideas of how would you like to see data just based on the issue. It would be really nice. Okay thank you. So first of all thank you for caring about these because you rarely get probably a lot of thanks. But one practical question so obviously you have quite a bit of legacy in history with this. So I guess all of this is done using your own code and not symphony components or something because it's probably hard to switch but have you considered because at least in 2.7 there's a translator profiler or something like that. Have you considered for some parts looking from some shared bits and pieces or is it just maybe so different? Yeah so in 2.8 we use a lot of symphony components and one of our members in the multilingual team started working on integrating the symphony translator component Clemens Tolboom in fact and he found that there were at the time that was I think a year and a half ago or two years ago there was a lot of missing features in symphony that we already have in core particularly string context support for providing metadata in the code about your string and he started writing code for, no he started porting the Drupal code to symphony so that they have that feature that we need to have so that everybody is happy in symphony and have the feature but turns out you cannot port the feature from Drupal because Drupal's GPL and symphony the license is not compatible so you cannot just copy over the code you need to rewrite the code entirely for symphony so it's not copied over from Drupal's GPL code so license was not compatible and he spent a lot of time trying to convince people there to that they need that feature and he run out of time and he didn't have time to rewrite it from the ground up so it's licensed compatible so I think that was the biggest roadblock but we have a lot of history in trying to integrate that but then failing due to technicalities so we keep using our own transition code in Drupal 8 for mainly that reason yep okay thank you sorry for that what's the symphony license I don't know first of all thanks for doing a great job of this things really impressive stuff so I'm looking forward to things coming there my question was really just about what you mentioned there the contextual stuff my feeling is that many people try to translate strings that have different meanings in different languages stuff like the word post to post and up post and the people that at least from our experience are working with doing translations and stuff like that they're not really the techy codey people so the problem to like say hey this string needs context that information like never gets out there I feel so could you just say a bit about how are your thoughts regarding doing that process easier is there anything we could do to let everyone say in an easier way this means two things this is one of the things that keep us from porting to trans effects because we can add a button and that will hopefully maybe create an issue on drupal.org directly on the project on the projects that are using this string not that have no context or not enough context so we could add a button about this this would be really fast the main question would be automatic posting or just creating a new tab and you would do you would add your comment about this string needs that context for it to be translated but here it's one of the issues that is in localization server project so it's one of the things that we would like to add I think in terms of getting the community engaged in the translation and giving great feedback I think that's a great way to put in some effort making it easier for non techie guys to just say and we need platform support for that because in trans effects you can say this string means this and you can explain it in the description but in drupal what drupal is going to look at is the source string and the context and that localizeDrupal.org somebody wrote some message somewhere it doesn't care about that so if you use the same string for multiple meanings then we need the developer to add the context and the way we can get the developer is we have a button on localizeDrupal.org and we have single sign on drupal.org so we can get you you can then submit an issue on drupal.org or submit one for you because we know your user on drupal.org so we can get that done there but we needed to get back to the code somehow sooner or later from the developer because otherwise the drupal itself will not recognize that as a context I think it was one of the features that should be developed like fast and maybe tested and then get back on this if it's not not enough because right now we have a lot of issues but we are kind of scared of putting it to the site it's also because the localization server module could be used by others but we have really stable really done so we could we should let the master not the master the sieven branch dev branch always on localize and once every the whole localized community is happy with new features we should release it for all the other site that are using its module okay so if there's no more questions that our question for you is who wants to get involved in making these things happen because these are nice plans that we explain but we don't have enough people to make them happen so they will remain plans in the issue queue on this we have people step up to help anybody interested in joining us not yet thank you yeah I've seen your arm perfect okay let's get started and we'll see we'll get more people I'm sure so on Friday look for me if you want to sprint on that I'll be on the sprint thank you thank you thank you so so yeah yeah not that I know I wouldn't be surprised if there isn't a shield but yeah basically the Drupal.org sites they don't have any back doors to each other anymore so it's all public APIs it should be public anyway