 Okay, so this is going to be a talk that I actually gave last year. There is a bit of update, but I think it's a it's quite interesting and important. That's worth going over again a little bit about myself. I've been contributing to liberal office for around six years now. The last five years have been focused on online, and I really enjoy hacking away and adding features and fixing bugs, so that's that's that's me. What I'm going to do is I'm going to walk through the sidebar and talk about the little bit about the challenges and the implementation of how we went about trying to bring it to the browser, which actually turned out to be not as straightforward as one would have liked. I don't tend to read my slides. So if you try to read the bullet points while you hear me speak, I think you're going to benefit the most. And at the end, I will leave a few minutes for questions. So what is the sidebar just to make sure that we're all on the same page? If we look at the desktop window of LibreOffice, we're going to immediately see that we have the sidebar hidden that we can pop up to edit the properties of whatever context that we're in. And here I'm showing the club office build, and you can see for the different document types, we have a slightly different properties and slightly different set of options and settings that we can tune. So what we really would love to have is the same kind of power and flexibility in the browser. And that is it's much more difficult to bring all these settings without really doing the the core implementation of all of that all over again. So the idea here is to find an easier way to make sure that all of this is available without necessarily rebuilding it from scratch. Here I'm showing the latest screenshots from the browser side, and you can see that it immediately looks very, very similar to the desktop. The right two screenshots, they have scroll bars, that's because the sidebar is actually longer than the actual browser window size. And that works seamlessly in the browser without any performance delays or flicker, as we're going to talk a little bit more about the challenges for doing that. On the left, I want to bring your attention to the pop up color swatch pop up window. And again, that is all done in HTML and it's it works seamlessly as well. So before I move on, I really should take a moment to thank our partners. We can't do most of the main features that we would love to implement and the bug fixes without their support and without their financing. So this is a big thank you to all of our partners who help. So the sidebar, how hard can it be to bring it to the browser? I mean, after all, it's just a window and it should be fairly straightforward to make sure that that is working just as the other windows and dialogues. So that's the first intuition that the sidebar is a type of dialogue. But immediately, we see that it's actually a special kind of dialogue. It's a kind of dialogue that isn't floating unless we make it. So it can actually be docked on one part of the screen, or it can be its own window. And we can close it and open, but it doesn't lose its state. Indeed, its state is dependent on the context. So if I'm editing text, then the sidebar is showing properties of text and paragraphs and characters. But if it's an image or it's a graphic shape, then it's the properties that I would expect to see for images and graphic objects. Similarly, it's flexible in that it has collapsing pains in it. And I can control its size dynamically, and I expect its content to also become very flexible. So all of these things make it different from just a regular dialogue. So if we take the idea of pretending that the sidebar is just a dialogue and we build on top of the dialogue tunneling that we already have, then that is obviously, you know, it brings us closer to realizing the goal of having a sidebar working in the browser. So let's quickly look at how the dialogue tunneling works so we can understand how we can get the sidebar to work. The dialogue tunneling idea is basically to render any individual dialogue that we have in core as an image and separately handle all of its messages that are being handled in core to be sent back to the browser. So that the browser would know, for example, that a dialogue has been created and would give the dialogue a unique ID so that whenever we have any event that happens on the dialogue, like resizing or indeed even just updating the refresh, then we would be able to uniquely identify it and make sure that all the inputs as well are sent correctly to the correct dialogue. So to do that, we have a list of messages and message types and we have the IDs to use for communicating all of these messages, including the resizing message, the creation and destruction and rendering, which is the window paint. So with all these machinery, we can communicate with the browser and let the JavaScript browser track the visible dialogues and update its UI accordingly. So we built on top of this infrastructure and we've added a new kind of window, again because the sidebar is special and a different kind of dialogue, we had to make sure that the browser code can identify the sidebar as a special dialogue and not just a generic one, primarily because a generic dialogue has certain features that the sidebar does not, including closing it whenever you want, the size and other attributes that we've already mentioned. So when we've added special handling for the sidebar, we also made sure that the JavaScript is able to have its own input as in when should the sidebar be visible is no longer something that is controlled by the core or the back end, but the user is also able to show it or hide it at will. And indeed, very recently we've also been adding support for remembering the user's preference so that we load the document of the same type. If you've had the sidebar hidden, then it would remain hidden by default unless you make it visible again and vice versa. So all of these special cases were important to make the sidebar behave as we expected to. So a little bit about the relationship between the code and the visuals, and I don't expect most people to be familiar with this, but it's important for the remaining of the talk to make sure we have a mental image of things. The sidebar itself is a multi-layered component. On the very top level, we have the sidebar docking window, which is responsible for docking and undocking sidebar. In this case, you can see a habit floating. So it is as if it's a dialogue, except it's a resizable dialogue. The sidebar also has embedded in it the tab bar. This is hidden online. So on the browser, we don't have a need for it. And instead, we have all of the top bar buttons as part of the toolbar itself on the top, or indeed the menu. Within the sidebar itself, we have panels. And you can see the panels here. They have a hat icon next to them, which means that they're collapsible. So here, the style, character, and paragraph would be panels. But the panels themselves are within decks. And the decks are the different sidebar views that you can have from the tab bar. So the properties would be one deck. And you would have the styles or the gallery and so on. Again, one of the oddities of working with the sidebar is that it has some counterintuitive features, like the fact that the sidebar child window is in fact the parent of the docking window. This is an unfortunate consequence of the usage of the word child in the hierarchy of objects and clothing. So sometimes, some things can be quite counterintuitive, but that's life. So when we are trying to tunnel any dialogue in general, you essentially are trying to represent the top level dialogue as your main component that you're trying to tunnel. Whenever the top level dialogue is being resized or rendered, you want to convey all of these messages to the browser and expect everything to work just fine. But as we've seen, the sidebar is a multilay component. And it's not obvious what you want to, which level you want to tunnel. So the most obvious level would be the deck, because the deck is the one that has all the panels and that's the main body of the sidebar. You don't really care about the tab bar on the side or indeed any of the other borders. But that unfortunately didn't pan out, and I won't go into the details. There are multiple problems there, and the main one is that the decks are interchangeable and you do want to show the deck that is visible. So tunneling each deck separately doesn't make much sense. So we tunnel the sidebar docking window instead, and that seems to be working just perfectly. So the logic to having the sidebar implementation is to have a special casing for the sidebar docking window. What we do is we handle the creation logic within the resize notification. Again, that's a technical detail, and the reason for that is because at the time of creating the sidebar we don't have specific dimensions to it, and it doesn't make much sense to tell the browser to create a window that doesn't have dimensions yet. So there are a little bit of oddities like that in the implementation. We also need to make sure that the sidebar is being loaded from the browser, then we need to hide the tab bar and we need to account for the dimensions of the tab bar that we've hidden, and we need to make sure that scrolling works fine. I'm going to get back to the difficulty of scrolling the sidebar in a second. So another challenge is that we're not just tunneling the sidebar itself, but we also want to tunnel any pop-up windows that it contains, and the most obvious one is the color swatch, and that is not the only one, but we also have drop-down menus and have other buttons that interact on multiple levels. So what we want to do is we want to make sure that these child windows are created within the scope of the parent sidebar, but at the same time there has to be some ownership relationship, because if the user closes the sidebar, you would expect that the pop-ups would also disappear. Again, remember that all of this is happening in the HTML world and over there, the relationships have to be preserved for things to work correctly. If you create independent HTML elements, they respect the parent-child relationship when you close the parent or dismiss the parent. So there are minor details like that to be mindful of. So I'm going to talk a little bit about the challenges that we had in a little bit more detail, because they're fairly instructive and it helps to understand the kind of effort goes into building out even small features like bringing the sidebar to the browser. So some of the unexpected things that we've had is that, as I said, creation is not necessarily the point where you want to make sure that you pass the creation messages to the browser, but you want to make sure that you notify that you've created a window when you have everything necessary to realize the creation that includes the dimensions and so on. Another thing is that you want to make sure that when you're creating a window or indeed whenever in the background, that window is interacting with the document, that it doesn't interfere with the user's interaction on the browser. Again, this might not be obvious, but when the user changes the context, desktop, it might be reasonable for the input focus to move to the sidebar where the sidebar updates its inputs, but those events trickle back to the browser and the browser cannot differentiate a focus move that was initiated by the user, meaning the user is clicking inside the sidebar to add or click a button or do something. And the sidebar just updating itself as a result of the user maybe typing in the document. So these things can get a little bit modeled and unclear and we need to make sure that the user experience is as smooth as natural as possible. So we have to keep track of really what's happening and make sure that the sidebar does not adversely affect the user's experience and interaction with the document. Minor things like the difference between the different documents sidebars again can play a big role. Historically Impress had a slightly different workflow than Writer and Calc and those were surprising indeed because we discovered that there were certain rough edges Impress that Writer and Calc did not suffer. On a more detailed level, the relationship between the internal structure of the document window, which is the UI and how it interacts with the view shell and how it interacts with essentially the document updates is different said between Impress and Calc and Writer and we had to account for that. We had to make sure that the relationship is preserved even though in Impress, we do have multiple views and we need to make sure that the correct view owns the sidebar. And you would imagine why that is it's because every slide comes with its own logic and that affects the sidebar as in Writer and Calc, you don't have that distinction really. So minor things like that can become really big time drain and resource drain to figure out what's the best way of solving the problem without having a lot of temporary or workaround fixes. So let's talk a little bit about the score because I think that is quite an interesting problem or oddity of the sidebar. So if you go back to the sidebar on the desktop, you will remember that the sidebar, if it is docked, it has the height of the available space in your vertical dimension on the screen. So if your screen is a little bit shorter in height, the sidebar is gonna be shorter as well, but that is fine because the contents can be scrolled. If you have undocked the sidebar and make it floating, then you can decide how high or how wide you want the sidebar to be. And in that case, it behaves just like any other dialogue except that it would still scroll its contents if you overflow. Oh, this is really fine, except in the browser we do not want to have a scroll bar. And if you click on the scroll bar, we have to render the complete sidebar over again. You can imagine, I mean, you can do that for just one click that would get back to you within a matter of some milliseconds, but if you try to continuously scroll, it just would not scale, right? It would be very slow and it would potentially flicker or cause other visual disturbances. So what we really want is we don't want any scrolling to happen in the sidebar itself. We want the sidebar to as large as it needs to be to contain all of its panes and panels internally. And we wanna render it as one big image. And then that big image can get embedded in the browser. And if it is larger than the available space, then the browser can allow us to scroll the image, right? And scrolling an image in the browser is as fast as it can probably be, right? So that's the intuition or that's the basic idea, except there are some panels that are greedy, meaning the more space they have, the more space they consume. It's just because they take up whatever space is necessary and they divide it amongst its elements. And so you can't ask the sidebar, what's your optimum height? That there is no optimum height. There is a minimum height potentially, but there is no maximum height in some cases. So again, you're back at the point where you don't have a clear answer but what you need to do is you need to come up with some engineering compromises where you try to avoid the worst case scenario, which is you're cramming all of your sidebar elements in a small space or you have a very large sidebar that just looks unfriendly to the user. So finding a middle ground was one of the challenges in making sure that the scroll bar, the vertical scroll bar doesn't show up in the sidebar itself was the main challenge and that remained actually a minor problem for some time. Here you can see, for example, that we were able to reach the sidebar internally to some acceptable height, but still unfortunately we have the scroll bar show up that actually scrolls one pixel. That's because there is nothing more to show, but because of the math and how it works, it still thinks that it needs a sidebar to do some scrolling. So getting rid of that annoying sidebar that's completely useless was important. So we did fix it and we were able to do it. So as I said, this is a talk that I gave last year and within the past year, there have been quite a number of improvements and advances. So it's not completely the case that, there isn't anything new to say about the sidebar. The big change was that we've made sure that the sidebar works not just on the desktop, but also in mobile and it works across multiple devices and it works in a very seamless way so that the experience is really good. But obviously we've been able to fix a number of corner cases, bugs, really made sure that the user experience was as best as it can be with a smooth freshing and making sure that it does not interfere with the user's input or interaction with the document. So that has been the general progress. As I said, we've got rid of the double scroll bar. We no longer have that annoying internal scroll bar. And for the mobile part, I encourage everyone who is interested to see the reusing the sidebar on phones by Shimwon, which is on the last day on Saturday at 12.25. PM. And with that, I wanna thank you and I am happy to answer any questions or comments that you might have. If there is anybody who is still around and can hear me, assuming I haven't been talking to myself for half an hour. I can hear you. That's good, thank you. You can be here. There are some questions in the chat. Please. Maybe let me start with a question to you. So you can read in parallel is a message in the chat. One is we have made some effort to make the sidebar accessible by using a function key which steps through the various UIs in the sidebar and then between the controls and panels and so on. Have you considered accessibility? Yes, there are some serious challenges there but it is doable. The main challenge is that the relationship between the different elements in the browser and the actual UI components in the sidebar and in the core is not one to one. As I said for the sidebar on the desktop, the sidebar is rendered as an image. It's really a single solid image and that is visible as a single HTML image node in the browser. Of course on the desktop, proper, you have all the different combo boxes, edit boxes and buttons as individual elements and you can tap through them and you can get a disability on that level. So to be able to have accessibility on the individual UI elements for the sidebar at least, you would need to have something much more native to the browser. So for example, we've been talking about the possibility of porting the sidebar to be native HTML components and one idea is to tunnel the UI as an XML and then we convert that XML into HTML elements and every element would interact and would render individually with the user. Once we have something like that, I think it would become much easier and more realistic to have accessibility on that level. I would only add to that, like it is not actually that far because like we are doing before the node book bar via JSON, not XML in this case, but the idea is that hopefully we will get that for the sidebar as well at some stage. Yes, indeed. Can I also add something? You said, hello, this tour. You said native HTML, but if one considers the iOS and Android apps and same thing is also used there, then there is really nothing native as such about the HTML but what would be native would of course be the real platform controls but that's a completely different thing. Yes indeed, that's a really important point and that is that we do support multiple devices and platforms and we have to be very mindful that whatever solution we come up with doesn't require a lot of special casing and to Candice's point, there has been a lot of progress on the mobile side with JSON and we can leverage that same technology and approach in other places as well. But as one would imagine, the way complex things like this progress is when we have interest and we have funding so that we can implement it on one platform and then leverage that in other places and that doesn't necessarily always happen in a coordinated fashion, but slowly we're really improving things as one can see. I have another question if we have time, drop down menus are really slow if they are very long like font selection. Is there something planned to improve it? For example, for font selection, it should not all installed the drop down. Okay, so the fonts is special case and it's a special case for two reasons. One is that it doesn't have an upper limit. The size of the combo box or drop down is really as large as the number of fonts installed on the system and that makes it really, at least linear in the number of fonts that you have. So it's unlike the other ones that like the font size, for example, which you know how many elements you have and every time you're rendering the same thing. That's one challenge. The other challenge is that there is a complex font caching in the background that we need to query and we need to, there is a bit of code that needs to execute. That happens on the desktop as well, but remember, all of that is really rendered on demand on the browser. So I agree with you, it is slower than it needs to be, but there aren't obvious ways of making that faster, especially on a system that has thousands of fonts, for example, I'm open to ideas and suggestions on how to tune that. But as I said, these are the challenges, these are the realities of the font menu. Any other questions if we have time? Thank you, Heiko. These virtual friends is lexic interaction. Yes, but I think this is a good time to end the talk. Thanks everyone for your attention. And if you have any questions or ideas or suggestions, I'm happy to take them online on IRC or indeed by email. Thank you so much.