 So I'm here to talk a bit about the leap profess update problem. I will first motivate where I'm coming from, how I decided to look into the problem. I gave last year presentation about the leap profess crash reporter. I've implemented that for leap profess 5.2. We are using that in production in the TDF builds already. So, how does the leap profess crash reporter work? It's based on a breakpad, a library developed by Google for Google Chrome, used by Mozilla for Firefox and Thunderbird. On the server side, we have a simple, jungle-based server that accepts crash reports, possesses them, uses the symbols that we generate during the build to symbolify the crash reports. On the leap profess side, a signal handler intercepts the crash and writes out a mini-dump file and during the next startup, we upload the file plus some metadata to the server. So metadata is, for example, the OpenGL driver, the device, the GDI count recently, if we are in shutdown, because we have quite a number of shutdown related crashes. So, how does it look? So, that's a chart from the crash reporter website and it lists the number of crashes per day. You can see here, that's 5.2.4, and we managed to squeeze in actually two nasty issues that we managed to fix for 5.2.5, where we had some shutdown crashes if there was an object in the clipboard and some font stuff. If we did not find a font and we did not find a font because we only discovered new fonts and all the printer related fonts were not discovered, we uploaded it and created a crash report. So, the crash reporter has some issues. The 5.2.4 build, we could have discovered the issue quite early if we would have known that there's a crash that affects a lot of users, but we only have absolute numbers. So, we get only the information, okay, we have 50 crashes per day, but we don't know if these 50 crashes are from 500 users or from 5 million users. We have the issue that our RC builds are not used that much. I see it a lot in the crash reporter. We have maybe 20 to 50 crashes a day for the RC and beta builds and then we have 500 or 5000 for a final build. And our second issue that is quite bad, if you look here, we still have 250 crashes a day for 5.2.0, which was released I think in August last year and 5.2.1 was released in September last year. So, 5 and 6 months ago. And still a lot of users are using that and are reporting crashes. So, these numbers show that users are not updating regularly or not everybody is updating. We released about once a month a new final build, but apparently quite a number of our users stay with these old builds and don't use a new build. So, the solution is to, instead of reporting absolute numbers, divide the number of daily crashes by the number of active daily users or active daily installs. That's what Mozilla does, that's what we should do. So, our problem is we don't have any idea how many daily users we have. We have some ideas how many people have downloaded LibreOffice, but not how many people are actually using LibreOffice. So, I started, therefore, to look into how we can solve this update problem. We have some code already in this direction. We have the update checker. It checks if a new version is available and notifies the user. The user still needs to install manually, and apparently most users don't do that. And we had a Google Summer of Code project 2015 that already worked on the code for Mozilla, porting it to LibreOffice, integrating it into our build. But it's currently only enabled if you use this special configure flag here. So, I did some work there recently. I started with the Google Summer of Code code because it was already integrated. The Mozilla concept looks good. I've enabled the signature tracking. I will talk about signatures later a bit more. They are important to ensure that we are only installing what we deliver and not some other stuff. I fixed the Windows build system and the update service. I will talk about the update service on Windows in a bit. I updated the version of the updated code that was brought into the LibreOffice code base as part of the Google Summer of Code project. Actually, that was quite a lot of work because the code has changed a lot. The code comes from Mozilla. They have a lot of specific needs. It had some code for Firefox OS. Obviously, we don't need that. There's a lot of hard-coded Mozilla part in there. It's now integrated into the build system, the generation of the update files, the libraries, the services, the executables that we build. So, how does updating work? Because it's not that easy. We have an old LibreOffice that we want to update. It asks the update server and the update server tells it, okay, here I have a new update. LibreOffice instance downloads the update. But now we have a new executable, the update that comes. We really need that because on Windows, we can't change a file that's currently open by another process. So, we can't just change ourselves, which we can on Linux. The problem is that this process is still a normal user process, and the LibreOffice instance is normally stored in a system folder. So, we can't change that LibreOffice process. The solution that Mozilla has developed, and I think a few other update services use, is to have Windows service that is installed. During the installation, and that's a system user. So, you start the system service, and that system service can now start an update process, more or less the same executable, but this time as a system user, which can update a file in a system folder. So, actually, we run here the real update code that generates a new LibreOffice instance, overwrites the old code, and then starts LibreOffice again with the old parameters. And we have then a new LibreOffice running. So, the fun parts. I already mentioned there are a few problems. The first one is you can't change a file that's open. We have this update service and the update executable for that. And the second one that you realize quite quickly if you read Mozilla about the update code, is there are a lot of things that can go wrong and that screw up your security badly. Like, we are running executable automatically that runs as a system user and install new stuff in a system folder. We could more or less do anything if we screw up there. So, to avoid that anyone ships a code and reinstall, we need to make sure that at every step of here, the update file that we transport is signed that the executable that has called us is signed by us and that the executable that we will execute is signed by us. So, make sure that we can trust every part of this process. And a huge problem that we have is multi-user setups. We don't store any installation information in a location that a user can change. So, we can't store the information that we currently have started an update process until we reach the stage that we have the update service running. So, there's potentially an issue with the race condition and I haven't looked into that yet. And I suppose there's no easy solution on the leap-off side. Yet, I think Mozilla users writes into the registry and checks the registry to detect that issue. The leap-off does not use the registry on Windows at all. So, how do the update files look? So, they are MA files. It's just something that Mozilla guys have come up with. It's bzip-based. We just take an archive. But it has the ability to generate partition updates. So, it can contain information that some of the files have changed and just write the diff of these two files. We have a tool to generate these partition update files. It's integrated into the build. My solution is I have the final, the full build files that I create. I download them when I want to generate a partition file. I diff the two versions and generate then the new partition version. An important part is the file can be signed after the whole archive has been created. That's quite important for the release engineering. So, we can sign files in a central location. You can even store some release channel information and you can check them in the update if you want to ensure that we are not going back or not installing updates from a different update channel. So, that the user can't trick us into installing something that we have signed but that we actually don't want to use. Like in a company, you might want to use an update channel that only provides the stable releases of our still build. There's an executable. It's called MAR. It's quite easy to remember. You work with more files. The executable is called the same and you can use it to check what's in the file. You can uncompress them with that. You can check the signatures and so on. So, the server. Mozilla uses Birog. It's tightly integrated into the Mozilla infrastructure. I tried to reuse it but similar to the crash report, it makes a lot of assumptions about the infrastructure that we at TDF just don't have. So, I looked at what we actually need and we just need to return a few information. We will send to the server the information about our build and we get back a JSON file with the information about the update that's available or if there's no update available at all. And the prof is just then downloads from the location that we provide the update. We have some unique problems that are not available for Mozilla. We have language packs. We have health packs. And I mentioned earlier we want to collect user information. We want to know how many users we actually have. It's not really user information. Trust how many people are checking for updates so that we have an estimation about how many users are currently using which version and integrate that back into the crash report. I have a simple jungle-based server running at the moment. It's running on one of my servers. And I've used that to test the infrastructure. So I have a prototype that works with Linux already. So what's done? And what... As I mentioned, it works on Linux. But on Linux, we can't update a reprocess installation that's installed into a secure location yet. I'm not sure if we will ever be. I'm not sure if we want to. It compiles on Windows. So every part now compiles even the update service. We can generate the more files, complete and partitioned. It's part of the build system, at least in my branch. You can tell it, okay, generate these more files and they end up in work here. And the initial server. So I think about 70% is done. But we still have a few things on Windows installation. How does MSI play into the role? We install independently suddenly from MSI stuff. How does the installing work with MSI again? The update service is not integrated into the MSI yet. So it's not installed at all. I haven't tested that it works on Windows because the update service is not installed yet. We need automated tests because if you have these partition update files, you have the problem, you have so many different configurations suddenly. You have the build that you have done, a complete build, and you have the situation, you have an old build that you have updated but still need to make sure that the updated build is more or less the same as the old one and that it works and that the installation works. So we need tests there. It needs to be integrated into release engineering so I still need to talk to our release engineers and see how that can be done. Thanks for listening. Do you have any questions? Yes. I will come to you and see how we can make at least an update notification for the OSI. I want to know as well how many people are using your versions of the new version or just notify them. It's running for a year without problems for people to do an update. I just want to give them the idea to update. Yes. So we have that with the update check a bit. But that's about for TDF builds to distribute then the complete update information. There's a difference. We, for OpenGana and so on, we don't need a distribution just to trigger the users to update. Okay. Can we? Is it possible to update the updater? Yes. So this updater executable is part of the normally profile. That's why we copy it. We copy it to a different location in a temporary directory so that we don't change so that we don't change ourselves. If you want to run this service, how are you going to do that? What? If you run your updater as a service in Windows. Yes. How are you going to reinstall the thing? That's the more... That's the more complicated part. Mozilla does that. I still need to look into how they solve it. But they have solved it so it's not impossible. We're getting on the moon so no problem. We'll get there again. The bigger problem is because the update service is independent. It's independently installed as a different executable or even a different application. You don't update that with a normal update. You update your LibreOffice that contains that code but not the version that you're actually running. Are there other files that contains the full LibreOffice or just the diff between the two diffs? You can have both. So the full MozillaOffice contains a full LibreOffice and the partition one contains a diff between basically two directories. Or if it detects that the diff is too big it just takes the full file. Okay, thanks.