 So, hello again. The joy of having multiple Talbot scenarios is that hopefully I can skip most of this introduction this time. And I can skip to how collaborative editing is done in LibreOffice Online using LibreOfficeKit. LibreOfficeKit is just an API to be used by programmers. But the end result can be visible in the LibreOffice Online, LibreOffice Android port, the app that's a variable from the Play Store that is using LibreOfficeKit. Also, LibreOffice Online uses LibreOfficeKit. And lately, all the features that has been added to the LibreOfficeKit API was for the LibreOffice Online purposes. First, I would like to show you a few results, how it works and how you see it as a user. And then we will dive into the more technical details. So, just to recap, I think what Tyrandeering is was already explained at least two times in different talks. But basically, these are just these 256 pixel-sized pixel buffers, these tiles. In this picture, do I have a laser pointer? Sorry, so hopefully you can spot those small red rectangles that those are drawn in this debug mode at the top left corner of each tile so you can guess how large a tile is. And if everything goes correctly, then you don't see any artifacts of the tile joints. So, it's completely transparent. As far as I know, this was mainly invented by the Mozilla Firefox Android trim, and we took it over. And now we still use the same tile size for online, and I'm not sure if that's a good or a bad idea, but that's how it is. So, first there was this Tyrandeering. And then last year, there was a project ending in May for tile editing, or as we call it, the Android editing. The idea is that once we are able to render parts of the document to these tile buffers and then have some custom platform-specific UI stitching them together and presenting you a document, wouldn't it be nice if you could actually modify the document? That needed a way in the API so that you can transfer touch-events or mouse-events and keyboard-events and so on. And also, the other direction so that when some part of the document is changed, then the client can re-render the changed part. Also, native selections were added. On Android, you can long-push on a word and you get this section with the selection handles that's hopefully familiar from other applications. So, this is Tyrandeering and Tide. Other thing, this is already done. This talk is about collaborative editing, but it's just editing except that you have multiple users editing at the same time. So, you get to get conflict. And the question is how to do conflict handling. In this API, we decided to go for an optimistic approach. So, we do our best effort conflict resolution and then the result is presented to the users immediately. In case the result is not satisfying, then the user can quickly and easily change the result. One example here that you can see on the screen is two users. On the left, the user selects the word document. And on the right, the cursor is in the document word. So, in case the left user would decide to delete the word, then we just move the cursor out from the word and then do the deletion. And in case this is not what they wanted, they immediately see the result and they can press undo or something. This is the most major problem to handle. There are multiple inputs and they are conflicting and something has to be done. Once this is working, then if you ever saw collaborative editing in, for example, Google Docs, you saw that there are these collaborative text cursors where you have your own black blinking cursor, but you still see where the others are. And we implemented the same in this API and then it's visible in LibreOffice Online as well. So, we have these collaborative cursors or as we call it in the API, the view cursors are not blinking, they are colored, and they have visibility, so possibly they are not visible because shape, for example, is selected and then the text cursor is not visible. They have a position on the side. These collaborative text cursors are used for right or text. They are used for anti-tangent text that includes the calc cell text and also shape text in any application. Then, basically, the very same overlay that's used for normal cursors can be used for these collaborative cursors as well. These screenshots are from a text tool called GTK-Tagview. I will mention it at the end of the talk how we do testing, but the point here is that all these screenshots are from the test tool. So, it's not exactly how it looks in the browser. In the browser, you also have these cursors have hats, and you can see the name of the user at the top of the cursor. And if they don't move for a long time, then they are automatically hidden and so on. Once text cursors are handled, we can think about collaborative text selections. That means that... It basically means two things. One thing is that you have this selection rectangle, and, again, your one is some fixed color. You can recognize, like, blue or something. And everyone else's selection has the matching cursor to the collaborative text cursor, so you can easily match them. And also, again, there are some names on top of the selection and so on. For auditing the shape text, only one user can edit the text of a shape at the same time. So, what we do is that this is an impress example document that you can see here. Most of the picture is from the left side. We are auditing there, and on the right side you can... The small picture shows a small log icon. So, we see now the other views that this shape text is being edited, and if they click there, then the editing won't start, and they will understand why the editing doesn't start. One thing that was implemented in Core to help this situation is that, by default, when you started editing the shape text and the other views disappeared, if you ever dived into this shape text rendering, then you know that there are different code passes for viewing the shape text that's handled with drawing layer primitives, and when you are editing, it's handled directly in the editing gym. So, the problem here was that when you start editing, then we switch to this editing gym-based rendering, and that only paints in the current view. So, I extended this so that as you type in the shape text, as your own view is updated, every other view is updated, and there would have been nothing LibroFisk specific about this code. The only problem is that no other views also get the blinking cursor, and that's really confusing because you get the cursor, but if you click there, then nothing happens, and you guess that you could edit the shape text, but editing is still limited to a single view. So, at the moment, I made it conditional for the case when the LibroFisk code is running in headless mode using this LibroFisk API. So, at the moment, you still see the old behavior on the desktop. Once we have the text editing and shape text editing sorted out, with the selections and the cursors, we can move on to graphic selection. But you can see here, again, the normal graphic selection that was there already since the Android editing was implemented, and the other views get notified and the other views get these color versions of the very same selection. So, once the text cursor and the selection is orange, then the graphical selection will be orange as well. Luckily, unlike many other cases, these handles are implemented in a single place, and there are no duplicate implementations in the code base. So, this is behaving consistently in every application. So, that covers text cursor selections, text selection and graphic cursors. The next thing is color specific, the cell cursors. Because you are not yet editing text because you didn't start cell text editing, but you are not selecting a shape editor, so that's a specific cell cursor. It's, again, the very same. On the left, you can see the upper cursor is your own cursor. It's a black rectangle or a black frame, and the lower one is a blue. That's a view or a collaborative cell cursor. And on the right, you can see that the collaborative cursor is orange, and the lower one is black. So, whenever you are not selecting a shape in a caulk, or you are not editing a cell text, you can see where the cell cursor is. So, that's about the cursors and selections and collaborative feedback like this. The next problem is that, in case you edit a longer document and an independent part of the document are edited, then it can happen very easily that you type in your area and somebody else types in another area and something unexpected happens. Like, I don't know the other user accidentally selects the whole text and deletes it, and you don't understand what happened. So, for that, we came up with a document 3-par dialog, which is basically just listing the undo, reduce stack that we have on the desktop as one. The only extension is that I annotated these undo objects on the undo stack with the creator ID or the view ID. So, in case there are two views, two users editing, and a word is deleted, then later we can find out who was the user who deleted this. Presenting this in a table hopefully helps the user to understand what happened. And as good or bad the undo, redo implementation is, you can go back and forth in time and see how the document looked like before or after some editing. This complements the feature of, for example, with the on-cloud integration in LibreOffice Online, you also get versioning. And in different situations, the different versioning or different time machine-like feature can help you the more. Another related feature is that next to just being able to undo and redo, we also have on the desktop version the change tracking, which works together like you can reject a change and undo the rejection and so on. The LibreOffice Kit API also exposes these track changes. This is currently implemented only for writer, but clients can already work with this API and when the cloud part will be ready, that will be completely transparent. One challenge there is that LibreOffice exposes that the user name is part of the user profile, so the user name rarely changes. On the other hand, with the collaborative editing, multiple users are constantly typing and the current user name is constantly changing. So instead of hard-coding this change user name all the time in the configuration manager, no writer can store the user name in the view that belongs to the editing user. And in case there is a user or outer name SADAR, then it will use that and not query the configuration. And based on that, you can again come up with a table where you can see the straight-lining items with correct outer names and descriptions and so on. It's a bit confusing. There is a track change. You can have a command and a description as well and they are not the same. A command is an anti-by default and you can manually add some command there. The description is something auto-generated based on what you did, like inserting or deleting sometimes. So that's pretty much what you can see as a user and as long as it works, then you are happy and we can go home. But in case something doesn't work as expected, we are interested in how this is implemented. And the main trick is basically each user of the API or each editing user is mapped to the feature of multiple windows on the desktop. That's how we can get lots of conflict resolution for free, like the example I mentioned with the cursor being in the world and the other window deleting that very same one. Although this is not too frequently used feature, so there are a number of common cases where something strange happens when you have multiple windows because most of the time users have only a single window. So for example, in some situation it was enough to just type some new text to the document and undo it and only one view was repainted and a box like this. The good news is that in writer and in press the windows are mostly independent, so they already work mostly the way we want for the collaborative editing purposes. In writer, you can easily type in parallel in multiple windows and so on. In cargo, the situation is more complicated. You can really type in parallel in two styles, for example. So that means that at an API level we have this log document class and that has a number of new methods. The most important being this set view that's called before any other methods, so that in case two users are typing in parallel, that means the LibreOffice WebSocket demo that I should be presenting later sets the view to one window and then posts the keyboard event in processes synchronously, and once it's finished, it can release the lock and then the second user can get the lock, set the view to the window it wanted, it can process their mouse event or keyboard event and so on. This way, in case somebody is not interested in collaborative editing, like the Android client, it doesn't have to do anything. It's backwards compatible. You just don't use any of these view functions. The creator is basically equivalent to this new window feature on the desktop UI. You can close the view. You can set which one is the active. You can get an identifier for the view, which is just a CRL incremented integer. You can get the number of views and so on. One problematic part previously was that in case LibreOffice Core wanted to notify the client about something, like you type the character and you need to repaint part of the document, so you need to paint new ties. Then this callback was assuming that there is only one view, so that the callback sometimes was fired from the document model, sometimes from the view, and it didn't matter because there was only one view all the time. Now this has changed to be a view callback. That means that in most of the cases, the view is that fired the callback. For example, you type a key, and in the writer case there are two edit windows for the two windows, and both of them get the invalidation, and then we fired LibreOffice Kit invalidation even in both SW edit windows. This way we don't have to do any multiplying of these evens at the core level. There are a few cases where we still do this, like in Impress if you insert a new slide, then the easy assist to fire this event in the document model, but then we iterate over all the views so that everyone knows that there is a new slide. Again, from the client point of view, you register the callback after setting the given view, so that in case you are not interested in this model callback and view callback and so on, then you just do what you did before. You just register the callback and you will get the same events that you wrote before. As mentioned previously, one feature that required quite some work was to tag each and every undo item with these UIDs so that we know who did what when you list the undo stack. That meant there is some common place casting writer and then you can get who is the... What's the active window? It was the same. There was a similar base class in Calc and Impress, and there is the shared module for shape handling. There you had to somehow access which one is the currently active window, and also the same was needed in Aditangin. The problem with Aditangin is that it's a lower layer than all these... It's a lower layer than the SFX2 where all these ViewShall and ObjectShall is where you could query the ID of the view. So now there is an interface class in Aditangin that describes just the things from the ViewShall that's needed for these purposes, and then the SFX ViewShall derives or implements this interface. So that was a bit complex. Complex2 first came up with an idea how to invert this dependency. So once we know who did what, we could do more interesting tricks with this information, like the undo manager is specific to the document so that in case you are typing something here and something has there, then the first view press the undo, then it will undo the changes of the second view, and that's quite confusing. That's expected if you have two windows on the desktop and you sell both windows in parallel. But with the collaborative editing, that's a bit unexpected. So at least in writer, it already works that the collaborative editing case, the undo is limited so that in case you are not owning the top of the undo stack, then you can't undo... So you can't accidentally undo the work of somebody else. But this is not a completely dead yard. In other modules there are cases when this limiting is not there. Regarding the track changes, actually everything was there at the core level what we needed. There is a generic method called getCommandValues that gives you all the values for a given command that's valid. So this way you can get the different values, valid values for styles or font names. And also the track changes, the initial list of existing changes is queried this way. Then as part of the initializing the view for title rendering, you can provide an outdoor name. And this way whenever you type here and there in multiple views, the red line and the inserted command is always created with the correct outdoor name. And then it's up to the user of the interface to provide these names, so that in case the WebSocketDemon talks to onCloud, onCloud is the ultimate source of the username, and it goes through the various layers at the core, knows what's the correct username for the given view. And also there are various callbacks, so that once you have an initial list of existing red lines, you know you got the updates and you don't have to query the full table again. So you got the insertions, modifications, and remove us from the table. So basically what remains is how we test all these features. First, during development, I usually use the GTK type viewer. That's an executable. That's not part of the installation side, but if you build a code, then you can run it. And it's a very simple client that tries to still use all of the API, so that I can interactively test what I implemented is working correctly or not. And most of the time it works correctly. That's how you could see all those screenshots from GTK type viewer. And of course, we also need automated testing so that fixing new kernel cases don't break other ones. There are two layers. There is a more high level in the desktop module that is basically using the API as it's exposed to the external clients. And there are dedicated test suits in the writer, calc, and impress module where we are loading the document ourselves, just as for filter testing. And then we can use the module internal API for asserting what happened. Or bringing the document to a state that's interesting from the point of testing. So today we have almost 100 test cases, and there are many cases when it wasn't needed to add a new test. A new test case for a given problem it was just enough to extend it. So only 100 test cases, but about 400 assorts in the core side. Hopefully this way the way the API is accessed from the outside is working mostly as expected. And then the ones who work on the Android user interface or the website with them can focus on their work and not on the core bugs. At least that's the hope. So that basically concludes my talk. Thanks for listening.