 Our next speaker is coming from an exotic land of Korea. And he's going to talk about input methods in plasma. Obviously, Korean input methods and some surprises, I guess. Maybe. Thank you so much for coming. This is something, it's a topic that's very dear to me. And my ambition is that by the end, that may also be the case for you. We'll see. So I did write for 45 minutes and I have about 30, so I need to move at a bit of a fast clip, which means we'll skip over this part and get right to the meat of it. So what are input methods? And I'm going to sort of front-load you with some terminology. It's going to be a little bit abstract. And there will be demos later on that will make this sort of concrete invisible, so don't worry about getting lost. The input methods are ultimately about a conversion of input events into some sort of output text. And the most common input event is a key press, but it can also be a tap on screen, or it can be a more complex gesture like a swipe gesture. There could even be handwriting, drawing something, or even voice input. And it's ultimately about divorcing output from input. Like what you get out doesn't have to be directly mapped to what you put in. And that's usually stateful. That is, at some point, you start inputting and you do things, and then you get done. And this middle part is done by software. That's where the magic happens. And it can do that in a very rich and interactive way using various sorts of UI. And now it's about why would you even want that, who needs that, and what for. So there are two broad categories of use cases for input methods. There are use cases where an input method is essential to communication. That is about currently 20% of the world population natively speak a language that requires an input method to be written into a computer. And while it's 20% of the world population, it's currently not yet 20% of computer users, because they don't all have a computer yet, but we expect or hope that to change. So the number of computer users who require an input method to write their native language is going to rise over time. And furthermore, those users are currently underrepresented among users of open systems. That is, if you are someone who needs an input method to write your language, you will be more likely to use a proprietary system than an open system. And there are reasons for that, and we will get into those at length later. The other broad type of input method use is assistive. That is, you don't necessarily need it to type, but you want it to be faster or to type something that you rarely type. And we have demos of both of those. So examples of the latter would be word completions, word suggestions, even spell checking can fit into that scope. And that stuff is very common on mobile, but people don't use it enough on the desktop. And now it's a chicken and axing sort of on both accounts that is why people who need an input method are underrepresented among open source users is because our support for it is not good enough yet. And the reason why you don't use it on the desktop is also because our support is bad for it, because you don't know what exists, you don't know how to set it up. And that's also stuff we will touch on later. And now let's look at some demos. So you actually know very concretely what this is about. And a good example to sort of ease you into this is writing Korean. Korean is written using an alphabet. It's actually very similar to the Latin alphabet. It has about the same number of letters. It has consonants. It has vowels. It has a very simple keyboard layout. Every letter is mapped to a key. You press it, you get a letter. There's one twist to it. As letters form a syllable, they get grouped into a one character-wide block. That is, if I press this letter, if I press this key, I get one letter. That's the consonant that only matches H in English. Now, if I type another letter, oh, first off, you can see this is one character. If I use the arrow keys, it will skip over this. Now, if I do this again, but I press an additional key, now I have two letters in one character-wide block. You can see this when I use the arrow keys. The cursor will skip over this entire thing. Now, there's something really complicated going on here, which is there's already text in this text field, and as I press the second and third keys, that text gets yanked out and replaced with something else. That happens by way of communication between the input method and the application. There's a very rich protocol going on there to allow that to happen. In fact, you can see it's stateful because while I'm composing this block, the colors are inverted. That's, in fact, also the input method telling the application to invert the colors because this protocol has providing formatting hints within its scope. Another good example is writing Chinese. Many of you will be sort of aware that... Oops. Hang on. Now I made a mistake and pressed one letter too much. One click too much, actually. I will have to restart. Sorry about that. I wrote this in Qt Quick, so I could save awkward fumbling by switching to other applications to demo things, and now I'm awkwardly fumbling, so that worked out great. Now, many of you will be sort of dimly aware that Chinese is written using thousands of characters, and doing that on a computer is a problem that really only becomes tractable by using an input method because you need to somehow select among those characters. It's essentially a search operation. You input some sort of search query, and the input method will provide you with characters to choose from. So I typed one key, and it offers up a number of candidates of letters, and I can just accept the first one by pressing space and type another one, and that's hello in Chinese, and that only took two key presses even though there's thousands of characters, and what I did was type the first letter of the sound value of those characters in Latin, so I wrote Romanization, and it provided me with matching characters, and it was very smart about suggesting common ones, and that way I could type this very quickly. There are other ways of writing Chinese. You can, instead of using Latin, use a different writing system to write your search query. You might select among characters by their graphical features. You might draw one. So it's about... The point of this is that there's a pattern of providing a search term and being presented with candidates to choose from and picking one, and that UI that pop-up that appeared on screen that you selected the characters on is provided by the system to the application. It's not part of the application, it's a system service. Now, very similar to writing Chinese is typing emoji, except it's language agnostic because we all know that cat pictures are universal. Yeah, I think that was a joke. So if I type cat now, I will get numerous candidates, and as you see this, pop-up beautifully slides of screen, and there we have a cat. So that's nice. That's cute. And... Now we have another bug. Oh, my God. Whoa! All right, another restart of this. I'm really sorry. Anyways, the next demo is about word completion and spell checking, which you can also do within the scope of input methods. There's a cat to make us feel happier again. All right, so let's go right ahead. We can type something really complicated that's long and that we don't want to type in full, and again we get presented with a list of candidates, and we can press a number and complete something. And that's a really good example of something that's assistive, and in the case, for example, of mobile word completions about overcoming a handicap. The handicap being that typing on a touchscreen is really slow and we want to be faster, so we tap word completions on the bar there to be faster and have a convenient typing experience. We can do that on a desktop, but we usually don't, and let's get into that. Actually, first off, let's talk about input method technology, and I will sort of have to skip over that for time. So the way it usually works on a desktop is that we have a central demo running which hosts input method plugins, and there's usually one active, and you saw me switch between input methods that was telling the demo which plugin to load. And there's a lot of IPC going on. As your application gets a keyboard event, it will then forward it to the input method demon and the input method plugin will do its magic and send the result back. And the UI that gets invoked as part of that and that the user interacts with is another process which also talks via IPC to the input method demon. And what you get out of the central demo is that the input method plugins don't have to repeat all sorts of boilerplate to manage the configuration and stuff like that. The UI in this case was provided by Plasma and we'll get into that later. So on the desktop there's a bunch of players. First off, skim is sort of, it has fallen out of fashion so we can ignore it for the rest of this talk. The two big ones are iBus and FCITX. iBus is the default on many distributions. FCITX is the default of some of them. In some distributions, you will get one of those depending on which locale you install with because the respective user bases prefer one or the other based on their performance and features. Plasma being Plasma and KDE being KDE, we support all of them. And the way the application talks to the input methods, to the input method demon is usually done via the toolkit. In Qt it's done again via plugins. Qt has a framework of input context plugins which interface with the input method demon. Some of them do. Others implement an input method right in process in Qt. So sometimes a Qt application will send an input event that it received onto an input method demon. Sometimes the magic happens directly in the Qt process because the Qt plugin itself is an input method. Now some of those plugins are the Qt source tree, some are out of tree. Qt 5 bundles the iBus plugin. FCITX is out of tree. iBus used to be out of tree but is now in tree. And what in the end gets delivered to your Q rigid or to your Q quick item is a Q input method event and that essentially tries to wrap around all the things that the input method protocols can do. It will tell you the current states, it will tell you what text to replace, what text to insert, it will tell you the formatting hints and that gets interpreted and the text control will then manipulate the document and swap text out and reformat it depending on the events it gets. So those slides have desktop in parentheses and there's a reason for that because mobile is unfortunately not using any of the above and that's a problem in and of itself and that becomes apparent when we look at the input method community because input methods is one of those things where you can't divorce the technology from the social dimension because it's a really wide problem domain, there are many writing systems to cover, not all of them are currently supported or supported well on open systems nor are the existing input methods by any means done. There's a lot of work to do to make them faster and better and smarter and there's very few people working on solving those problems and there are writing systems where we rely on a single expert to provide support for a writing system to open systems and yet despite that we ask them often to do work redundantly because we have the fragmentation I talked about, we have multiple players on the desktop and mobile which in turn has multiple players which I will talk about later so if you're looking to bring support for your native writing system to open systems by way of an input method you probably will have to do much of that work multiple times to integrate in all of those systems and that's really a waste of their time and it means we're not exploiting that manpower efficiently and another problem there is that by the very nature of the problem domain we are dealing with language barrier and existing because input method users are underrepresented among users and makers of open systems people who make input methods often don't have the primary language of an open source community as their native language so you'll get language barrier effects and communities don't care enough about this topic yet to try to cater to that and make the first step and embrace those people so that's another impediment to solving all the problems that we will talk about KDE is part of the input method community there is overlap between Plasma developers and FCITX developers and we've also worked on the iBus support in Qt we go in there and fix bugs now specifically in Plasma desktop the thing that you saw me use down here in the panel is called the input method panel widget and that's a front end to all those different demons that I talked about iBus FCITX skim and through that widget you can access the configuration of the active input method or the demo itself it will display toggles that the active input method has and options for example the word completion one that I just demoed at the end has a privacy toggle because it can learn with use if you type words very often complete them very often it will offer them earlier and stuff again and you might not want to remember sensitive stuff so you can toggle privacy mode on and it will stop tracking what you do and the way to do that is via the panel widget and this technology also provides the popups that you saw me interact with and very briefly recapping our recent work in this area we finally moved the input method widget from the Plasma add-ons package to the core which was really the first step towards taking this stuff a little more seriously because now we can auto-add it to the panel when you first log on to Plasma in the Korean locale for example and we improved iBus support that's a little bit technical we can skip over that right now but very interesting is the Plasma team worked with the Neon team to sort of create a really intense integration test which is the KDE Neon Korean developer edition which is basically a customized variant of Neon that as it puts up into live CD mode or as you install it on a hard drive it will install in Korean and it will set up all of this stuff so there's something we can test to make sure it works and stays working and we made this originally for KDE's 20th birthday party in Seoul and we handed it out there for free for a lot of interest and we've gotten a fair bunch of patches out of that by the way this is the KDE Neon Korean edition and it's a little bit fuzzy but you can see some Korean writing down there and in the menu and that's that birthday party which was really fun now it gets interesting so what are the pain points that I sort of keep alluding to so first off setting this stuff up is difficult it requires expert knowledge if you are dealing with an English system and you want to teach it to be able to type Korean then you need to know what iBus is you need to tell your package manager to install it you need to know about the plugin you need to manually add the panel widget you need to configure it and configuring it is really hard because the configuration is not set to be it's not in system settings you have to go via the panel widget and in fact the moment that you start using an input method system a lot of the system settings become first of redundant but they also actually stop working because if you use iBus in many cases it will take over keyboard layout management so changing the keyboard layout and system settings will actually no longer work which is it's just a really big mess the problem there is that system settings only deals in keyboard layouts currently now the input method panel itself sort of competes with the system tray first of here is the current settings we have in system settings which are solely dedicated to keyboard layouts now the input method which it sort of competes with the system tray it has its own mechanism for hiding items you know it's its icons on the panel just like what the system tray does but it redundantly has its own system for showing and hiding items in there and that's a bit daring and not necessary and is another problem we need to solve now there are solutions and the most important of them is to make input methods always on rather than treat it as an optional thing and it doesn't matter much actually whether that's a build time dependency or runtime dependency what matters is that we force distributions to package it and install it along with plasma because if we don't do that we cannot we cannot users cannot rely on it being there but also we cannot write any settings UI that depends on it being there and we really need that we need to revamp the config UI we need to get away from managing keyboard languages, keyboard layouts and go towards managing input languages which by the way is exactly how it works on every competing system both open and proprietary the idea is if you want to type Korean you go to system settings and you add Korean and it will figure out the rest and plasma is very far away from that currently and it needs to catch up we have a keyboard layout indicator currently that's really smart when you configure more than one keyboard layout something will automatically appear in your tray and you can switch between keyboard layouts there and you can see that obviously is redundant with switching between input methods that needs to be unified and then you also no longer need to manually add the panel widget it should automatically appear and we need to integrate the input method panel with the system tray the input method panel should just register tray icons and the tray should sort them together then you can do show and hide via the tray settings now I kept mobile apart until now because it really is a part and that's a really big problem so very briefly the main difference is of course instead of a physical keyboard you can have a screen those are not mutually exclusive and that's part of the problem with a mobile device you may actually have more than one input device you have like an old slidey smart phone you have a screen and physical buttons but also we want to get to a place where you can connect the physical keyboard to a mobile device and we are in that place already in many cases and the very early screen keyboards you click the button and it would generate a key event and send it through the sort of conventional means of dealing with those which was clumsy but actually a little bit better than what we do now currently Plasma Mobile uses a keyboard virtual keyboard framework called Malit I'm not sure I'm pronouncing it correctly but it was conceived by Nokia together with Intel they did it for the Memo Amigo stuff back in the day and basically the reason we use it is because it's there and it's not very good and it's mostly unmaintained and it's horribly over designed and that's what it looks like it's not very pretty either and we want to get away from that we want to switch to Qt virtual keyboard which used to be a proprietary add-on to Qt which is now has recently become a proper open-source Qt module and it's implemented as an input context plugin in Qt now that might make you think it means it won't work in anything but Qt apps but where we want to use it is on the phone and the phone is Wayland which means we actually have a Qt app handling input events which is Quinn Wayland and so the virtual keyboard runs in process in the compositor and can then accept input from the virtual keyboard and forward it to the currently active application using the Wayland protocols and we do use it on the desktop as well we have it, for example, in the lock screen if you detach your keyboard from your desktop and you need to log in you can open the virtual keyboard from there so the reason why in mobile we have a valid and we have Qt virtual keyboard is first of we do need UI parts but the way they are implemented are for reasons that we as makers of open systems don't actually share which is mostly licensing a lot of the input method stuff is GPL device makers tend to be a bit scared of the GPL they don't want to have GPL stuff on there so they decided instead of reusing code we would create our own mullet and Qt virtual keyboard duplicate everything that IBUS and FCITX do if you want Korean it's implemented there redundantly so what I talked about earlier that we are spreading the community that works input method stuff then we are spreading it even thinner by the way we do things on mobile right now for dumb reasons and that means we are not achieving feature parity both ways Qt virtual keyboard and mullet leg stuff that IBUS and the desktop things do and vice versa and also not just feature but also behavior parity that is if you type things for example think of the Chinese demo you might come to expect a certain sorting of the characters that are offered to you for example because it learns with use if you select the character very often you want that to be offered very high in the list of choices but if you have a different stack on running the physical keyboard and a different stack running the virtual keyboard and they independently learn from each other you might actually get different suggestions which is bad it's a challenge to convergence where the idea is that you create your plasma mobile device and dock it to a screen and a keyboard and it should offer the same features and behave the same way and in fact it should even synchronize state if you think back to the Korean demo I typed three letters to form one block now imagine you do the first two key presses on your screen and then you want to do the last one on a physical keyboard it should work it doesn't right now because the code running behind the scenes is completely separate from each other you actually need to delete your text and start over it doesn't work and that sucks and there's very poor UI integration and no consistency if you use input method panel widget to select your input method that has no effect on the virtual keyboard and vice versa so currently the best idea we have to solve that is that Qt virtual keyboard itself is extensible you can teach Qt virtual keyboard a new writing system and what we would like to do is instead to teach it to outsource that job to something like FCITX FCITX can be used as a library so instead of talking to it via IPC you can also load it as a library and make calls into that so we want to extend Qt virtual keyboard with a plugin that calls into FCITX and then possibly use FCITX both for the physical keyboard and the virtual keyboard to solve those integration problems and we should reuse the UI we have for the input method panel widget in the keyboard tray so there's the same UI for switching between input languages and and finally I want to talk about why we should care about this and one reason is written down in our manifesto on manifesto.kd.org it says there we care about inclusivity and we have an end user focus and inclusivity obviously means that people should be able to write their native language on our systems if we want to be inclusive we need to cover all of those ways of writing and it should be efficient and it should be easy to set up that's the end user focus part and doing that will create sort of a feedback loop because as we enable our systems to be used by these kinds of people some of them will become our future developers and they will help make us that better and they will help us make many other things better right now somebody who would be a great kd developer might install our system and abandon it very quickly because they can't figure out how to message their mom and that sucks so getting those people allowing those people to use our systems will allow them to help us but there's also something more fundamental and that's why we actually do what we do and why we are here and I think it's because as software designers and as artists and translators and as programmers we get to sort of operate in a very exciting support role we get to help enable culture and build civilization and there's nothing more satisfying than writing software that allows someone to express themselves and to connect with others because that's when all that cool stuff happens and input methods are very directly about that because communication is essential to culture and language is essential to communication yeah, that's it how are we doing on time? for a really short few questions alright Jonathan it's essentially cutting into your break you said that we need to be forced as distributions need to be forced is there a release because plasma is packaged who would not package part of plasma? no, you need to be forced in the sense that we have the input method panel widget in plasma desktop but it's not a hard build dependency like the stuff that input method panel needs like for example iBus or fcitx is not a hard build time or run time dependency which means it allowed you guys the luxury of ignoring that problem and so what we need to do is say you know you can't package plasma without making sure that the foundational components are there and that sounds really scary but the thing is it's actually not that scary because you already have a package for iBus you just don't have it in the dependency list for plasma so if we say we need this you add one line and the entire situation changes have you discussed this with KDE packages on the list? not on the list I have discussed it with individual packages and for example there's a neon help by doing the neon Korean developer edition but also we worked with the fedora community to try out what I talked about that we now ought to add the panel widget if you install in a locale that is known to require an input method like Korean or Chinese and we sort of improved our first logon script and the fedora people worked with us to test that so you are right I should go to the list and spread that further but we've been talking to a number of different distros also OpenSuser they work on FCITX and they are pretty well on top of this so it's not hopeless we just need to do it so here as an application developer I don't really need to care much you only need to care if you implement your own custom text that needs to handle input method events and Kate is an example of that I fixed bugs in Kate for example the input method can provide formatting hints when I was typing Korean the colors were inverted some other input methods used color annotations and stuff like that that was broken in Kate the text turned invisible because foreground and background color became the same I fixed that so we have some cases in Kate where we needed to implement the handling for those input method events but typically if you use toolkit provided standard text widgets no you don't need to care with my Packager hat on can you teach me enough Korean or Chinese that I can at least test this sure find me I will excellent I have done with many other people more questions Jonathan you had one why is it so hard to set up from the Katie Neon Korean edition that needed a whole separate edition where it pre-installed some stuff and then you have to set some sim links and some settings and then reboot so there's a number of reasons of that first of you need a couple of pieces for example for Korean you need the iBusDemon so that's one package but then you need iBus-Hungle which is the Korean input method so now from a distribution perspective you get the problem of at what time should I actually pull in that iBusHungle package like how do I know it's a Korean system effectively what we will need to do is something a little bit like language pack install or codec install where if you go into system settings and you add Korean as an input language system settings actually needs to tell your system package manager that you want to install iBusHungle so what distributions need to do is to provide an easy interface for us to say we want Korean support and you install everything that means so you need to have sort of bundle distributions already have that as dependencies of the language packs and so on so like a lot of the infrastructures already there there's integration themes and the other stuff that's difficult is mostly on the plasma side like even if you have all the packages installed you still need to do the manual step if you start with an English system of adding the panel widget and then you need to go through the panel widget to add the Korean input method to your list of active input methods in iBus so from the user perspective there's install packages panel widget configure input method demo start using and you need expert knowledge to do each of those steps and so on the plasma side we need to fix system settings we only need to do one thing and system settings needs to interface with the distribution part to get the stuff installed but actually like why are you saying here I'm here why are you saying that it's actually the distro's problem right like if plasma knows that it's in Korean plasma should install whatever it takes and make sure that it shows in Korean if when you install the distro you select Korean plasma should be installing whatever dependencies are done so it's your job not other people's you're right so that's interesting reflection of what the landscape currently is so we have fragmentation right we have iBus and FCITX now on OpenSuser depending on which locale you install you will actually get one or the other because what the users of OpenSuser said to OpenSuser is we are Chinese we prefer FCITX because it works better and some other language users told OpenSuser we prefer iBus because it works better for our language so OpenSuser currently has the intelligence of installing the right stuff and I agree that we need to uplift this into plasma and it needs to be plasma calling the shots and we need we may have to make a hard decision of which input method we want to support which is a really political thing and KDE likes to be conflict the worst and so we just support everything and then you get multiple multiplying of integration seams so we need to realize that it's our responsibility and that we may need to ruffle some feathers and just to point out Nome Nome has the setting stuff pretty well done and one of the reasons is that they decided to only support iBus but if we decide to only support iBus we already piss off several plasma developers who also hack on FCITX it's a hard situation okay this is all the time that we have so let's thank Ike