 All right, should we start? All right, welcome everyone. Thanks for coming. This will be talk about packaging Chromium with some focus on Fedora. If you have any questions for this talk, there will be Q&A at the end. But also feel free to just interrupt and ask during the talk. And my name is Pavel Haydn. Let's see what's with this area. So Fedora was the first to not exist so I installed. So that kind of motivated me to come to this conference, present the talk about Chromium, where I'm a committer. And so I'm trying to make things better in this area. OK. So this is fresh news. Chromium has been accepted into Fedora just a couple of days ago. And so I thought I would start with this. Because this is almost going to be like, why is Chromium still not in Fedora? But now things have actually changed. And that's very nice. So I'm going to talk later a little bit more about why is it a non-trigger package to include? Why it took a bit long? And it's been a lot of work. So thanks for that. I'm actually really glad this happened. And actually, it might not be that easy to keep that package up to date. So let's see how it works going forward. So this is going to be very similar to my Linux contact three years ago. And one of the goals, it's not so much about any information or kind of things. They change very quickly. And it will see that later. My main goal here is actually get people to work together. Get these communities. There is Chromium is mostly developed by Google. And there are other companies contributing as well. There are individual contributors. But largely it's been started by Google. And so then there is the open source community, the wider one, Fedora, Linux, all of that. And I would like these two communities to work together. Like ideally, they should be kind of the same community. What we are trying to do, we're using Chromium.org addresses for day-to-day development. Like we see ourselves more as Chromium developers, basically. And the idea is to create a very inclusive group. And we also often need some changes in other open source packages, say libUSB or Huntsville. And this is not always so easy. So the basic idea is we just need to work together. We cannot say, oh, these Chromium guys produce something which is hard to package. Or these Fedora guys have some really hard guidelines. And why do they have these restrictions about bounded libraries? No, we need to talk, understand, and figure out solutions. And so far, this has been fairly successful. But I'll go on more details later. Also, to start, well, do you know where this is from? That's the comic that came out in 2008, first public Google Chrome release. And I really like reminding people about this. Like our rules are open source. People who created Chrome had a strong Mozilla background. And they were contributing to Mozilla. And so it all started with the thought that, OK, all of this technology is going to be open source. We're going to be a good citizen. We'll actually help other people develop better browsers, use this really help us change. To a large degree, this is happening. And I think I would like to encourage even more of that. I actually created this site today because I was talking to people on this conference and realized that the scale of Chrome is not so obvious to everyone. I actually work on it day to day. To give you a sense of scale, I counted over 3,000 contributors. The compressed sources, the tarball, is over 400 megabytes. I won't read all of these numbers, but just some highlights. We have several millions, lines of code. Now there is this. We use over 200 third-party packages. And we check them in because on Windows, where we also ship, there is no concept of system dependencies. So we have to bundle them. And there is over 200 of these packages. Think about that. So that means on this scale, we're kind of like a mini distro, some degree even, which creates interesting challenges for us. And then on all that scale, we do a stable release every six weeks. And we do, we see over 200 comments every day. I'm actually working on a team which creates infrastructure for this and kind of keep it running. When we have an outage, we kind of lose productivity 200 parts a day. It's really challenging. All right. I put the slide at the beginning when people still pay attention. But here are some suggestions. I don't like and I don't intend too much to tell people what to do. But I was watching how the Chrome Fedora package was kind of developed and all of the difficulties with that. And I also saw successes. I also were like, what works, what doesn't work. These are the suggestions I came up with. And the first one is, have a team. OK, so it's not like a simple package you're going to compile in five minutes on your laptop. I mostly package Chromium for Gen2. This is actually a Gen2 laptop, by the way. But we have a team in Gen2 of about five people. Just for that one package. The way we work, I usually focus on dev channel releases, which come out roughly weekly. And almost every week, there is some breakage. Like I need to make non-trivial changes to the package to make it compile, to make it work. Actually, it's mostly about compile. So it's easy, right? If it compiles, it's perfect. But then other team members help with things like beta channel bumps, stable channel bumps, when there is a security update. I don't need to worry about that. I can focus on the hard parts. They help out with bug trash, do the bugs reproduce. It's really, really, and it's the best part. So when I go on vacation or something happens, we just have other people. We can discuss our ideas. It's really, really helpful to know that it's not just me. That's everything, right? And then, and that's also encouraging. Every distro has a room for this. There is a need. If you're looking for ways to contribute to do something and encourage you to join a Chrome team or whatever, I don't know how it's going to work for the order, but I wish there was a Chrome team eventually. Because every small task, I'm very sure it's going to be helpful. I saw a bug queue of, say, Ubuntu Chromium bugs. They have a lot of bugs reported against Chromium. Same for Debian. I actually didn't see Fedora, but the official Chromium package is only a couple days old. But over time, the bug queue itself might need help managing. And then the next thing is, well, limit amount of custom patches you apply. The package I maintain has about five custom patches. And it's going to be down to four next week, I hope. I just need to upstream one of them. And I think the current Fedora package has around 15. I'm not sure about the exact number. But at some point, it becomes really unmanageable. And I'll mix these points a little bit. But there is this thing. I have a really fast machine. Which compiles Chromium in 30 minutes. But, say, on this laptop, it takes five to seven hours. I should actually have measured it, but a couple hours, basically. And so when you're iterating on packaging, I mean, I obviously recommend when you're iterating, have just a checkout. And use given release working that checkout and do incremental builds. But when you do packaging, you probably want to do a clobber build, eventually. So that's going to take a long time to make sure it all works. And in general, when you have to rebase your patches and recompile, it's a really bad cycle. And basically, when you saw we had 200 patches every day, that also means upstream is moving very fast. So if the more non-trivial patches you have, it might be a nightmare to try to rebase them. And there is also a culture of things about the Chromium project that we move things around pretty freely. Like it's not like, oh, this code is here, so it stays here. Now, we have two or three big refactoring efforts going on where they're actually kind of, when you see a stack of components, they're actually kind of wrapping around the other way to make them more flexible. And it's kind of scheduled for like two or three years. And so things can change in the code base quite rapidly. And it's really best if people can upstream that these two patches, it's just like the Linux kernel. Once you get your patch upstream, you don't need to rebase it. It's pretty much there. And in practice, it works for me. My packaging experience was that generally these patches just stay upstream. Occasionally something new comes up, so there is temporarily new patch in my package. But then I go upstream that patch. It works very well in practice. I already mentioned packaging dev channel releases, not just stable releases. And again, the idea is you deal with a smaller number of breakages at given time. One week, that's about, say, 1,000 comments. So that's manageable. And usually you have one or two breakages every week. You can deal with that kind of incrementally. But imagine now stable release, which is six times that. And now you get a lot of more issues. There is usually more pressure because there is a security update. And compile takes you a couple of hours. People might just burn out if they try to do that for too long. So that's another suggestion. Another good thing is we've created a mailing list called Chromium Packagers at Chromium.org. And the idea is to have a place for Chromium Packagers from various distros and also Chromium community members who are interested in helping out in having a high-quality Linux packages to discuss issues. Like if someone found, oh, build with this system library growth, and here is the solution, other people can just reuse that solution versus repeating the same debugging process every time. And you can look at the archives. Historically, a lot of issues are very common. So the idea is, I don't want to get into Dev versus RPM or anything like that. That's going to stay probably, right? But what I mean is at a higher level, there are things that we can hopefully only solve once. And then packages can adapt. OK, this is the way in our package, that's how we apply that fix. But at least, you know what to do. So I encourage that it's not strictly for packages. Anyone can join, anyone who's interested, or you can just look what's being discussed. Then there is this. Actually, that's another very exciting thing. I tested this yesterday. Just yesterday, now there are no SETUID bits required for Chrome sandboxing to work. This has been tested with M54 currently in a Dev channel. So in about 12 weeks, this should make it to stable. And there are a couple of kernel config options needed to get this to work. Mostly namespace support. Security Yama is recommended. I think it's not strictly needed. I just put it here for completeness. The idea is if Chromium will refuse to start if the sandbox doesn't work for some reason. So it's really easy to test. And there is one caveat that this won't work inside Chirrut. So I actually had to use my laptop to test this. Because in Chirrut, the only way to sandbox is still SETUID binary. Then I'm going to talk a little bit about the solution that made unbundling libraries easier. And I'm really glad Fedora started using this. I just believe that solutions are only good when people start using them. That means they're useful. It was created in 2014 after various discussions with the Chromium community. Basically, the idea is we need some switches or changes to the build system to enable build with system libraries. And the Chromium community doesn't like these conditionals. There was a different cold path for various things. And they didn't know what's going on. Typical Chromium developer doesn't necessarily know about, say, PKG config or other things that are just daily things for Linux developers. So it was kind of confusing for them. The idea was, OK, let's create another directory, put everything there so that they don't need to look at those. But there would be very useful bits shared by the Chromium packages. And that's another thing. Like contributions to that are welcome. Files are very small. And I will shortly describe what's inside. Well, first, it's a little script called remove bundled libraries, which removes bundled libraries, obviously. Tricky part is the list you see is not the list of things to remove. It's the list of things to keep. And removes everything else, which has third party in the name. Well, actual list wouldn't fit on the slide. It's over 100 lines of virus exclusions. Something to note is that in that source tree, there is up to three levels deep nesting of third party directory. So in the last example in red, you can see it's like third party, trace viewer, third party, TPCM, third party, Polymer. So there is this weird nesting. And the script actually removes everything not explicitly why to list it. So it's not enough to have an exclusion just for the top level, third party, trace viewer. It will still remove third party directories down there. So the idea is it guarantees you have a consistent list of third party code that's still kept bundled. And actually, if any of these entries is stale, if some of these directories have been deleted from the source code, it will also fail. And so the idea is your package will create very accurate lists so that just looking at your package, you can see all of the bundled code. OK. And another tricky thing with Chromium. I'm not sure if any other project is doing those. But again, this is somewhat special. Shim headers. The Chromium style of including headers, that's actually from one of the CC files. It has those double quotes and refers to a PNG header with third party prefix. And the idea is we actually can't change that line in the CC file because the Chromium community is opposed to that. It will need to have conditionals, or it will need when you build Chromium bundled, you want to use the same headers and the same sources. When you mix headers and the library you link to, bad things happen. So they don't want that either. And it's very obvious when you use double quotes that you're using the bundled version. But for the distro build, we want to do the opposite. And so what can we do about this? Long, long time ago distros used to have these giant patches just changing all of the occurrences of includes to use the system header. But it's not sustainable. These patches are huge. And they're really hard to rebase. So instead, after some discussions, and again, this was actually a suggestion from one of the Chromium people with Mozilla background, we found a solution that's acceptable to both packages and Chromium community. And so what we do, we generate shim headers. We create the expected third party directory structure inside the build output directory. We put the png.h header there. And what it does, it includes the real png.h. It works. And again, the idea is it just doesn't require any changes to the mainline Chromium files. It's totally separate and just works. So I thought this is a nice clever solution that normally would be overly complex. But we have various requirements here. That's why we do this. Well, then there is less. And if what you see on the screen is cryptic and unreadable, it really is. That's the point. JIP was the previous build system used by Chromium. It actually replaced visual studios, cons, and x-code files checked into the sources. So you can imagine, especially with x-code, it was very difficult doing cross-platform development when platform-specific IDE config files were checked into the repo. So when you are adding a file, it's relatively easy to modify scons and visual studio. That's just an XML file, so you could figure it out. But x-code was particularly hard to just add a file to the build system. So the idea with JIP was people created something that's just like a Python directory-less syntax. And they came up with this kind of least common denominator for all of the free big IDEs. And then JIP generates dynamically the files for your build system. But as you see here, it's kind of cryptic. There's also a poor documentation for JIP. As you think of those, when you got to fifth or seventh level of nesting, it was really hard. It was actually hard for Chromium people working on those day to day. And especially so, if you were a Chromium packageer, mostly familiar with AutoTools, for example, right? I mean, AutoTools are not pretty either. They're actually very ugly. But at least people are used to them, right? And they're documented. But this thing is very Chromium-specific. So a very common complaint I've heard from packages was that, hey, look, I don't know how to work with this, even at the compile flag, at the file. These are the things to actually do when packaging. This is being now replaced with GN. That's a new system. GN stands for Generate Ninja. And Ninja is a make replacement that the Chromium project uses. It's faster. By the way, Ninja is really cool. If you want to make notes, check it out. And as you see, this syntax is actually cleaner and nicer. It might still be different, but there is extensive documentation. I'm actually going to mention on the next slide. There is documentation for it. It's generally simpler. You can understand more just by looking at the examples. There is one thing. GIP used to be in Python. GN is in C++. So when you package Chromium to use GN, you need, first, to bootstrap GN. And there is a Python script for that. Sometimes bootstrapping breaks, and the script needs to be up-labeled. And then, well, it still doesn't support CFLAX and the other variables like CC or CXX. I'm working on that. And so we should have things working in the next few weeks. Meanwhile, it might require either patches or something. And then, what have we unbundled? This is mainly a list I talked. These are kind of like non-trivial libraries that require changes to the build system. The libpng, libxml, zlib, not only ffmpeg, it's been pretty successful. I think that ffmpeg.lib.v4 kind of calmed down. It's all ffmpeg back, so it also helps simplify things. There is one custom pass you need to apply to use system ffmpeg. And I think it should be possible to work with ffmpeg community not to require that pass. Basically, they're keeping some. Chromium is using some internal ffmpeg data structures in one place. And system ffmpeg doesn't expose these headers. So it could work with ffmpeg to just expose the functionality that Chromium needs. Some interesting ideas like what we might do in the future. And the actual list is much longer and probably includes things I didn't think about. But these are something I came up with. What could we do? Beyond just compiling something. Well, we could unfork Hanspahl. Chromium has some changes to Hanspahl. Last time I gave the stock, I said, hey, look, Hanspahl is really terrible. They still use CVS. And so no wonder people find it hard to contribute to Hanspahl. It's now on GitHub. Public requests are getting accepted. And so, hey, if anyone is interested, I think it's going to be much easier to get any patches to Hanspahl. Chromium people actually won't go out of their way to get these patches. Like, a bundled copy of Hanspahl works in Chromium. And so then there is, like, Mesa Chronos headers. We could get them more in sync. Last time the issue was mainly some discrepancy on 32-bit and also handling extensions with G. So we needed some changes to the build process. It actually worked for a while, but it was fairly complex. So we might give it another try with GN. Should be simpler now. Also, I think there are new iterations on OpenGL. So anyway, I'm mostly highlighting things here. Reenabling system SQLite. That's another one. SQLite doesn't want to make all the changes that we would like them to make to work inside the sandbox. So it's pretty challenging. Then there is, you know, system LibreSRTP doesn't work inside the Chromium sandbox. It requires on, it depends on boring SSL modifications. That's like an open SSL fork. And so again, this is like, you know, we have a fork in the community. There's open SSL. There's boring SSL. There's Libre SSL. And like, well, OK, what do we do, right? How do we keep this all compatible? That's a challenge to the wider open source community that Chromium project itself isn't going to solve. And if you solve for those, I guess it would be much easier, you know, to make this all work inside Chromium sandbox. Fixing build with system ICU. It work at some point and just needs work. PDE fume bundles. Oh, here's the thing. How many of you know what PDE fume is? No one. All right. How many of you know about the PDF viewer inside Google Chrome? All right. So it used to be closed source for at least a couple of months. It's fully open source. And it's called PDF. So it's based on Foxit code base. And it's available as open source part of Chromium. So generally, our Chromium packages now should have PDF viewer out of the box. It just has some bundled libraries that don't have to be bundled. Renabling system protobuf for Chromium. And then, you know, crush reporting. I saw distros have some kind of crush reporting mechanisms. And Chrome also crushes. I don't know how often it crushes for you. But it certainly does crush for distros. And some of these crushes might be distro specific. So a possible idea for the distant future might be, OK, let's figure out how to connect the distro crush reporting mechanism to the Chromium crush reporting thing. Because Chromium uses the multiprocess architecture. So a renderer might crush or some other background process that it uses. And I'm not sure if the distro crush reporting catches that, if it properly symbolizes them, especially that these processes are often running inside the sandbox. So that's mostly throwing the idea out there. It's certainly a huge non-trivial project. And then we're getting close to the end of the talk and to the Q&A. I would like to reiterate that contributions to the Chromium project are very welcome. These are the top domains of contributors. I took these stats yesterday. So they're very up to date. Most of them you see are Googlers. And then there is a lot of Gmail addresses, which are probably individuals. And this is great. And then there is, actually, when you look at the number of people contributing from each of the other companies, like, hey, this is amazing to me. I'm actually very happy to see that people are kind of investing. So, so happily into those. And this includes Blink, which is like Fork of WebKit. So there are some people and companies that focus on, say, the HTML engine. And that's also great. That's all fine. That's all part of Chromium. And Chromium is also included in QT, or Qt, whatever you pronounce it. So it ends up, also, on Linux, you know, systems as a library. And contributing to Chromium might be a way to improve things there. All right, another thing. And I hope it will be very interesting. One of the concerns I heard about contributing to Chromium was, oh, you need to sign the CLA. And yes, wherever it's strict about this, we do have automated checks that you've signed, a CLA, either individual or corporate. Actually, here is a thing. I'll mention it later. Anyway, so it's not a corporate assignment. And as far as I can tell, it's very similar to Fedora Project Contributor Agreement. So if you're OK with the Fedora Agreement, I hope you'll be OK with the Google CLA. If anyone isn't, I'm actually very interested in talking to people about what their concerns are about contributing to Chromium. What are the barriers? How can we make that process easier, basically? Because I know we have a lot of people who only contribute one patch, for example, and that's great. I wish it was even easier to contribute to Chromium. Like there is a huge code base to download. It's several gigabytes of code. It takes a lot of time to compile. So by itself, that's a barrier. But if there are any other things we could remove, that's great. And by the way, sometimes people might think, oh, my use case is too weird. Or why would the project accept my changes? I actually went through and I looked for these kind of weird things that got upstreamed. And I've noticed some support for FreeBSD, OpenBSD, Solarix, AX, QNX, actually, MIPS, Spark C, S390. Many of these things are not for Chromium itself, unlike entire Chromium, like hold, run browser on the mainframe. Probably doesn't make sense. But in V8, which is the JavaScript engine, they actually do have support for some of these architectures. And it's not developed by Google. It's been contributed by people who actually use this code in these environments, and it got accepted. So it's possible. There is also Wayland, also on. They have some parts in the Chromium code base itself, and they have their own repo, and they wait to combine the tool. It can work. And a lot of times, the barrier is, you need someone to review your changes. You need someone to help you make your changes in a way that's suitable for inclusion in upstream. You need a reputation that you will be around to fix things if needed, especially for larger changes, like architecture support. People do not want to have something that will be that cold too soon, I guess. But it's manageable. And I think for small changes, like fixing a bug, fixing something there, and nobody is going to ask you to stay around or maintain your line of code, it's mostly for larger changes. When people come to the project and they say, I want to make large changes, I think it's reasonable for that project to also say, OK, but we'll support this. I think that's been the biggest barrier for any larger. Unbundling was one of these changes, and the solution we found was actually, OK, there will be nothing to support because we're going to live in a completely different place. And that said, again, I think it's mostly about saying, OK, Chromium is open to contributions working with Fedora community, any other community, and encouraging people to ask, participate in that process, and help us welcome, et cetera. And I think we're ready for questions. No questions. Yes? So do you do some performance benchmarks from you? For example, how we transfer Fedora into? Oh, you mean a distro-specific benchmark or part of the upstream development process? Yeah, I don't know. I mean both. OK, so as far as I know, individual packages are not performance-test-led, but upstream has extensive infrastructure for performance testing every comment. We actually revert patches, which introduce severe repressions. And it's all part of the continuous integration that upstream does. So yes, it's performance-test-led. It's not always perfect, but we're improving it. And I guess, again, it's also something which is open source. And if people want to contribute to that, it's possible. You'll probably need to ask around where all of the pieces are, because some of them are server-side. Not everything lives in the browser itself. There is also surrounding infrastructure, which is a different repo. Yes? Is there a place where I can add some code to your work in progress programming package for Fedora? Because the only Chromium code I found is two months ago. So don't you want to answer that? Sorry. Chromium went into Fedora. You don't need to go to Opera anymore to get it. It's in there, and it's current. That's cool. All right. Next question. How do you support programming in Fedora? Right now, i386, x8664, I'm planning on doing the changes to enable ARM shortly. Is it a programming or just a program? It shouldn't be a program either. And is it the bundle? V8 is bundled, because Google often doesn't have a mechanic yet. Yes, maybe a couple of comments. On the V8 case, I used to actually package Chromium with V8 as a shared library for gentlemen. And I came to a conclusion. I could probably argue with myself one year earlier about this. But I actually changed my mind and came to a conclusion that it doesn't make sense. And the reason is both projects move at a very fast pace. And so remember, the release cycle is six weeks. And so there is a very fast pace of development. And what they do, what Chromium is their main client. So V8 is actually evolved into Chromium a couple of times per day. So the cycle is not the same type of cycle as you have with GDPC, which is very stable for this reason. Or even GTK, AGTK have a lot of controversies with the recent things. APIs are hard. That's the point. And the things they do in V8, they often expose new APIs because they add new functionality to the language. There are things like that. And so they move very fast. And they can keep Chromium up to date as well very easily. The problem is there are other projects, say Node.js, which do not move as fast. And so you have something moving very fast. And then something that moves very slow. And they're not willing to have big compatibility either because that takes effort. And here's my point again. If it was important enough for someone to do that work, it's not a question that someone wants to break you. It's a question that people are not willing to do work. They don't need to, right? And so if someone was willing to create the compatibility layer of something, and that's a non-trivial amount of work, then we'll go for it, right? But someone needs to do it. And how do you produce the other bundle of pieces that are updated? There are updates to wherever the upstream makes new release, or they are just sitting there. Oh, you mean in the Chromium called base? There's a couple exceptions. Most of them are just sitting there. But things like I see you are getting updated pretty regularly. So it really depends on the component, the people that are working on the component. I always remember that it's not three people that are working on Chromium. And so there's oftentimes three to five people that are working on each component inside of there, each bundle component. And so it really varies on what they need, but there's a feature change. And sometimes what happens is what usually happened with ICU where they didn't want to go above 52, but there were stuff from 53 that they needed. And so they were backported stuff from 53 into the bundle ICU rather than pushed to 53. So how much are you feeling today with which component we can pull out? Yeah, and in practice, I wish there was. And I'm also telling those, this is the idea, to have this two-way communication, because I also have similar talks at Google. And so the whole idea is to actually bridge this two worlds, because who's an adult? I can tell you that from the other side, people are surprised. It's like, can it require that your package depends on a specific version of some other package? Like require a minimum. It's like, yes, dependencies work. And people are surprised by those. So the idea is there is a lot of just understanding of these two worlds. And in practice, what happens with the bundled libraries, they get updated where they need an update. So when there is like a security bug, and when there is, otherwise, I think nobody is motivated to update them. And I wish we had a more distraught-like process, which is like, OK, let's test this new version of the library, because you know. I think that was more like someone checking for the security stuff and stinking and raving and looping into it. They are. But some of it is because the bounty on security bugs in Chromium is not a small amount of money if you find a security bug in Chromium Chrome OS. So some of that is preemptive. We don't necessarily want to have to pay off this bounty that's aggressive in there. And also, because the bounty is so much, there's so many eyes on that code stack. So even if Google wasn't being super aggressive looking at that one old stack, there's plenty of other people that are looking for each one of those bundled codes to directly find a vulnerability that I can use to chain in and get from Chromium Chrome. All right. Any other questions? Yes. I wanted to ask, was there an opinion on the emerging bundling systems, like BlackBank, if they need something that can simplify distribution of Chromium to Linux users? I don't know yet. There are certainly several alternative solutions. I don't remember the Ubuntu one now, but there is also something else, which are kind of trying to do the same thing. Oh, yeah. Well, so when it comes to packages developed by companies, say Google Chrome, it could improve things, because currently we provide devs and RPMs, and they happen to work. I mean, we officially support Ubuntu Fedora, maybe something else, but say there is Arch, there are other distros, which are currently, you can't run Google Chrome there, which is also one of the reasons to get Chromium packages. And so FlatBug could ideally work in all of these cases, and this is something companies like Google could use, to ship Chrome. For packages, like native packages, there still may be a room for that, like unbundling libraries, codec handling, I guess we'll be figuring out that the idea is we don't want to create packages that people wouldn't use either. So the idea is as long as there is a need for this packaging work and people find it useful, we'll provide those. And a lot of that, interacting with communities, I also encourage Chromium people, when they modify libraries, like GDB, USB, Huntsville, and so on, to contribute these packages upstream, because then they'll be easier to update in Chromium. And that will stay, no matter how we package Chromium, this work of connecting these communities, I think will stay, because we use 200 packages, and I'm sure there's a lot of interesting things going on with the combination. I guess I'll ask my question. Is there any likelihood that we're ever going to see the source code open for Pepper Clash? That's the number one most common request for Chromium packages, literally, is Pepper Clash, because everyone just wants that. Well, I'm not sure. That's a question for Adobe. And I've heard, given the amount of security bugs, they might not want people to look up the source code. But there is a good news that Adobe now provides a separate download from their website to get the PPAPI Flash. You get a tarball. That wasn't the case just a month before. So it's like a new thing, and it's the best we have. Yeah, no, I definitely want that one. Right now my script is just going out of it. Oh, yeah, yeah, yeah, it's terrible. So that should make it a bit nicer. Yeah. Do you compare performance between the bundles and the bundles version of Chromium? Is there any difference in performance, or? I didn't make extensive things. I wouldn't expect the difference to be that way, because we mostly have changes for sandbox and not for speed. There might be differences between Chrome and Chromium, because Chrome is investing more and more into link time optimization, profile guard belt optimization, where Chrome can take several hours. And so Chrome might be faster, because it's more optimized, possibly, than Chrome. I don't think it's on the Chromium, is there a difference? I didn't do measurements, but I don't think there should be any. I didn't just check the program with music. So I did benchmarks three years ago to see whether there was anything conclusive. And one of the big changes between Chrome and Chromium when you try to debond world names is that you're using shared libraries. And so when you start using shared libraries, you get better runtime performance, but far worse optimization for launch times. At the time, three years ago, the performance focus for Chrome was, how quickly can we launch the browser? And how much memory does it use when it's running, and not anything else? Those were the two main performance benchmarks they were tracking, whereas when you put shared libraries in, it takes a lot longer to launch, because it's loading a ton of shared libraries in to run. But once you run, you get better performance as a result of it. Yeah, so it wasn't my question if you saw one of them. But again, that's less about the debond link, and more about floating the shared library flag. You can get the same result without debonding it all, by just telling it to debond all the shared libraries, which is what we do with Fedora, even though it's not supported. But I think this is something that community could just say, write a blog post. Like, I tried those configuration books, and these are the results. Say, I wish Chromium people blogged more. One thing I like about, say, Planet Fedora, I actually read Planet Fedora regularly. Hey, it's interesting, is that people just blog about their work, and not many Chromium people do. And I sometimes go and encounter blog posts, which compare commit rate in Firefox, Chromium, WebKit, Rust. And they did interesting analysis. Inside Chromium, we had a lot of discussions about that post. So when people just say, it doesn't need to be a big part, so you don't need to come. It's like, I think it's a contribution, then, of some kind to have these measurements. If anybody wants to help out with the Fedora Chromium package, they're welcome to come talk to me. I'm happy to have help. I just reported the blog. Fantastic. It's a precious one, though. We wish for at least. Prove it. Yeah, we know about that. There was no patch on nowadays. All right. Are there any more questions? Like, I will still be at the conference today and tomorrow. And if anyone has questions, I would actually be very interested. I'm mostly here for Chromium, not so much Fedora. Sorry. But if anyone is interested, feel free to chat. We should sit down and look through the patches that are in the Fedora package and see if any of them can get up to you. Oh, yeah, exactly. If you have time, I would be very interested. Anyway, no more questions for now. I think that's the end. Thanks very much for coming. Thank you.