 Welcome back. I hope you enjoyed your lunch. The last presentation is going to be very, very interesting. It's my pleasure to introduce Niklas Piron, who is another long-time contributor of Niksauce. I think it's also been working on a project for at least seven or eight years. Niklas is a professional compiler engineer, and the most significant contribution to Niksauce is probably the module system that we all enjoy these days. So when we write a configuration.nix, the mechanism that actually turns this into something that runs is his invention and original design, and today he's going to be talking about a novel way of shipping security updates. So enjoy your presentation. Thanks, Peter. Okay. So I have a bit of a problem with my presentation. I realize that after repeating too many times, I kind of shorten it and shorten it and shorten it. So I found a solution, which is I will do something during the introduction to speak how I accidentally shortened my presentation by repeating it. So we are going to talk about shipping security updates. One of the first things I want to show you is security update matters. And I have a few links. So unfortunately, the ones that you can find in my slides that I just mentioned on Twitter are kind of a bit off because they decided to put the page to a 40 or four just today. Who knows? So they have a new presentation, but I will probably show you the old one because the old one is sorted by dates, which is not the case of this one. So let me bring the old one. So I looked for Debian not because Debian is known for having lots and caring about security issues more than anybody else. So I looked for Debian. And I found some few interesting things. So we use unzip, okay? Separated four, okay? We use WordPress, okay? That's probably the same class of bugs. Unzip, unzip, WordPress, next page. So PHP, we use PHP. I hear some people complaining about using it. So, okay, PHP, JDK, PigsBuff. Oh, this one, we use it in Firefox, if I remember. Oh, mini-upnc, okay? So MariaDB, open JDK, virtualbox. We use virtualbox. And, okay. And if I remember correctly, on other pages, you have like Firefox and others because we're making releases and stuff like that. So the user link is a link to the US website with the CV enumerations for enumerating bugs which are reported and with security flows. Oh, here it is, Firefox. So if we look at the bugs from last week or, yeah, last week, so we use an acrobat reader, okay? I won't ask about Cisco stuff. We use Android and Firefox, okay? So that's my point. We can get back to the presentation, okay? So basically, there are lots of security issues in the wild and you're probably using software which has security issues. So can we trust this software? Depends what we define by trust. I cannot trust myself for not making any bugs. That's a problem because I'm also developing in Firefox and I kind of add lots of bugs. Hopefully we have a good security team which uses fuzzing on Firefox to make sure that we don't ship any of these bugs to the users. Sometimes it happens, but hopefully not too frequently. So the question is, how do we define trust knowing that all software has bugs? Mostly. So in terms of shipping security updates, we can view the trust relation as do the software developers and the person who will release it care about you. Do they fix it? Do they ship it? Do they ship the security updates? And do you get it? So do they care about you, basically? And to show that, I will take a few examples that we saw in the list previously. Not next to us, not yet. We will see Android, which is well-known and developed and released by different people. And we will see Firefox also, which is developed by Modzilla and released by Modzilla as well, but also by Linux distribution, such as NixOS. So first, I will talk about this graph. This graph is how to explain it clearly. This is how the API, the versions of the API of Android in the wild. So as of today, we have less than 10% of 5.1. And this version of API are mapped to the version numbers of Android. What this graph does not show are the dot releases, which are corresponding to the security fixes, which are applied. So first, we will not look at this graph, but only at the 0% line. The 0% line is basically what Google developers are doing, or Android developers, to be more general, which is like, you see that they are making releases at quite a nice space, which is kind of good. The other interesting point is that they are not making dot releases after they made a new minor release. So as soon as they made 4.4s, they are not making any dot releases for 4.3, which is kind of a problem if you don't update your phone. So just looking at the bottom of the graph, I guess we can trust Android developers for making frequently releases to users. But there is another part of this graph which shows us what people are using effectively. And this is not Google. This is like your phone carrier, or a phone manufacturer, or a network operator. They are shipping. They are selling you the phones. And the problem is that it costs them money to get a phone validated to work on the network, and they don't care enough to pay for getting new updates to the users. So basically, as it costs money, they don't want to ship new version. And that's why you end up with a bit near 80% of users which are running versions which are unlikely to be updated anymore. Hey, who are the Android users? Yeah, so this diagram is updated as probably every, or, OK. I don't know all the details about Android. And I took this diagram from the Wikipedia page about the history of Android versions. So this diagram shows us two things. You can have developers who cares about users, but if it's not released, that's a big problem. So let's look at Firefox. Whoa, that's a lot of releases. And by the way, if you see the 18.0.1, hey, I've made a patch over there. Kind of my fault. Sometimes things go in releases and we know it. So one of the interesting thing about Firefox is not that they're making a lot of releases. Oh, it is. OK, rewind. Firefox is making a lot of releases. We can see that it seems to care a lot about their users. That sounds like you're good news. And one of the questions we might ask is how do they do that? How do they ship that many releases? Like if we look between 2013 and 2014, we see that they have roughly 10 major releases and about 20 patch releases. That's kind of a lot of releases, like more than months. So we might wonder how they do that. And one of the things that you have to remember when you're making such updates is that you don't want something which involves stress. Because when this involves stress, you are making mistakes, like typos and copy and paste errors, which apparently somebody caught in my slides. So you want a process for making releases. You want to make sure that if you have to ship a security issue, then it will follow some things that you are sure to not make any mistakes because that's something that's written down, step one, step two, step three. So for Firefox, we have a process which we call canspell for chemical spillage, which is kind of the vocabulary that we have in signed Mozilla. And the goal is to make sure that we do the things right. So if we look at what this looks like from the time. So before this graph, we have some developers making sure that they have the right fixes in time and checking that these are working. And then this goes into the build farms that we have, which ensures that as soon as we build, we know the time frame and we know what to do. So if we look at this graph, we have a few things. At the beginning, OK, we receive the information that we want to build it. And we tagged it. And then we do the Linux build. We pack it with the language pack because we need internationalization, as opposed to NixFest so far. And then a few hours after, we are able to ship Android and Linux versions to our users. So as a distribution, that's where you want to start shipping versions of Firefox. But Firefox is only on Linux. So we have a long process which goes and goes and goes, where we have QA to make sure that this works on Windows and which works correctly. And then we ship it to all our users, which are mostly Windows users so far. So that's about the day. But for us, it's about four hours after as the Linux distribution. So now let's look at how we do that in NixFest or Nix packages. So we have multiple ways of doing it. We have the solution of, you can wait until everything got rebuilt, in which case you can wait a long time, depending on what's not building anymore. Or you can have a solution of taking a small channel of a few packages. But then in the case of Firefox, you probably have to recompile it yourself. And if there is something broken in the middle, you will probably not be able to recompile it yourself. And you will end up with still the old version of Firefox, which has a security issue. So to fix that, we kind of introduced like, OK, is that mostly for libraries? If you have a library which does not apply for Firefox, but mostly for libraries. If you have libraries which is wrong as an issue, then we have a function in packages replace dependencies, which has an NixFest option equivalent for doing that, which goes through, OK, you have a closure of, or you have a package, a user environment, or your Linux distribution derivation. And you want to go through and replace one child by another and say, oh, it's fixed. So let's take an example. Imagine you have your boss in Asia for some reason, and it calls you at 3 AM in the morning and tells you, OK, I heard that we have an open SSL issue. Can is our system safe? And then you go on the computer, you check, OK, NixOS wiki. And then you read, OK, dear user, please proceed as follow. So you create a module, and you put this stuff, and you include that in all your configuration. What can be wrong? I'm asking you. Any guesses? Only errors? Yeah, with multiple outputs, that can be a solution. That's another solution. The wiki was updated by the hacker. Yeah, this solution does not solve the problem of setting clinking, but I have another one for you. It's 3 AM in the morning, and you just copy and paste this stuff and didn't realize that as a Vim user, you confuse the L and the H. And while you were typing the original equal open SSH, you made a mistake of looking yourself out of all the computers that you are updating while keeping a security issue in open SSL. Yeah, Vim users. So you don't want users to endow themselves security issues. So the current model that we have requires user awareness, like update this channel or do this crazy manipulation. And there are some tricky things about ABI compatibilities that the user is not necessarily aware of. And that's still an exception. That's not part of the normal deployment. So what are the solutions? You can wait. What can happen after a month? You will get an update. Maybe not. You probably know this logo. This logo is the early click logo from the open SSL issue. And what surprised me about this logo is that they got a logo. It took us a year for the next, a bit more than a year and two or three, I can't remember, to get the first logo that we recently replaced. And in a few days, they got a logo. What is this world in which we're living where security issues are getting a logo? So let's see how we can fix that. So currently, replace dependency is one of the nicest ways that we have to update for security issues, because it does not involve recompiling everything. It only involves recompiling the ones that you have a fix for. So the problem that we have today is that it's maintained by an we don't want any user interaction. One of the other problems is that the way it does today is like it makes sure that you have all the packages and look for the runtime dependencies in, hey, you have to compile them. And look for the runtime dependencies and then check, OK, is there is this runtime dependency? OK, replace it by this one. So this means that you have to compile everything before knowing that you want to update it, which can be kind of problematic if you're just wondering if you have a different one. You just want to check, OK, do I get the same derivation? Oh, no, I get a new system to compile. So and there is also the problem that Elko mentioned about the fact that we still have issue about static linking. But that's something that we can handle in some way. And that's the line that if I do that, it disappear. So we will address in the rest of this presentation a way that I started to make in a pull request to make sure that security updates are transparent for the users. That they are fast, which means that IDRA only has to build one thing and one and make it the first thing that it has to build. And also, it's easy for you, we're packaging software to make sure that, OK, I know what is an ABI. I know that this is an ABI compatible change. So let's update it this way. And saying in the sense that we don't look at front time dependencies to know if we should update or not. So by transparent, what I mean is that if you were to do the first two lines, then you should do the next two lines. Yeah, it's the same. Ideally, you should not do anything. So that's how Firefox is working. You don't, hey, update it. You got a new version. But no user action at all. But that's the ideal world. And we can leave that for later. So what does that imply? That implies that we have next packages as of today, which is a fixed point with a list of package where we have a package A, a package B, which depends on A, and a package C, which depends on B. So if we want C, we have to pull B and A. And so in terms of implications, this implies that we have to have the same layout as next packages. So as a start, we will take next packages as our solution. That's transparent. But we want to have it fast and to avoid recompilation. So that's one of the solution, which is a bit more complex. So let's look a bit at it. We have A and C, which are a fixed version. So I have a fix for A. So I can recompile A. Oh, I don't need the arrow from A to A. That's a detail. No, I need an arrow from the dependencies of A to the blue A, whatever. You can do this transformation in your A then it will rip. OK. So we have a fixed version of A, which is recompiled not against the fixed packages, but against the old packages, the brown one. Same thing for C. So what is this B? That's a green B. This B is a version of B where we compile it not, sorry, we don't compile it. We take the one that we had before in next packages, the brown one. And instead of recompiling it, we apply the change that we changed the old A, the brown A by the blue A. So the green B is replacing the brown A by the blue A inside B. We don't know if it's used. We know that it's a compile time dependency and we know that we might do that. So this is how it looks like. You have a first version that you already know that's your stable channel where you have everything brown, you don't know if it's stable or not. Then you have another version where you have fixes for each of the packages which had issues. And then what you do is basically the same idea as replace dependencies which is fix everything which is done the same with patching and by replacing one dependency by the other. OK, keep that in mind. Or if you don't want to do it, I can do that for you. And we are back to the presentation. We're back to the presentation. So you saw the fix point before, right? You all understand how it works. So a fix point is that if I get these next packages, basically it loop into itself until I get one thing. So you can see it as a for loop which only exists as soon as you are able to see the same result twice. So let's go a bit deeper. So we have a fix point. If I call F or if I call it twice, I will still get a fix point because I will get the same result. So one more step. If I have a fix point on next packages, I will get a full set of next packages. And if I apply next packages a second time after, I still get the same set of next packages because applying next packages once until I get a fix point means that I don't have any more changes. And if I apply it one more time, I don't get any more changes because I had a fix point. So the trick is if you want to recompile only the software which got patched with fixes for security issues or others, you just take another next packages with the fixes and you apply it after. So hey, you only recompile the packages that you have fixes for. And all the others will recompile the same way. So that does not give us the green one. We have all the blue one, but we don't have the green one. If you apply it one more time, then you get in a tricky position where you have some which are recompiled once with the fixes. You have some which are recompiled with the fix version. But at the end, you still recompile, which is what? Oh, gradient, which is kind of bad when our goal is to be fast. So we don't want recompilations. So we don't want this red one, which implies that we will recompile with everything. But this red one has an information that we want. The information that we want is that, hey, something changed. So here is the tricky part. What we are doing to get the fixes right, to make sure that we are patching, is we ask, hey, what happens if you do one more step? Do you recompile or you don't? Basically, packages that we have today are like STDN blob. OK, I have no idea what's in this blob. I have no idea if that's an input, if that's something which is used for fetching the sources. No clues. So the only way we have to make sure that we know if something has to recompile or not is to, hey, is the output different or not? So basically, what this one is doing, it's doing, hey, I have the quick fixes in one side. The one which are doing one more step, do I get the same output? If you don't get the same output, then means that the one which has one more step is being recompiled with one of its input. So if you take the list of inputs of both packages and you look for the differences, you will surely find one difference. And then you apply the difference to your result, to the result of the ones that are in the blue part. And you know what? You are fixing. You are basically replacing something with, you are replacing the brown one with the blue one and the green one. OK. People might be lost. So what this is doing is basically asking, hey, do you recompile? Yes. Check the input. You check the input. You map them one to one. And you know which dependency insights inputs that you give to your expression, you have to replace inside the current input that you get. So basically, you get the same as replace dependency, except that you have it at compile time instead of with the runtime dependency, which makes it sane and fast because you only recompile the quick fixes and you get to patch all others, which are depending on them. Everybody's OK? OK, so now let's look how easy it is. That simple as the first two. You get the old one, hey, stable branch. You get stable packages in one place and you get a few fixes over there. So instead of shipping one version, which is a stable branch, you just ship two. Two versions of the next packages. Not a big deal. We can do that and call that the security channel and that's all. We're good. So what happens in terms of release management? So as a package, you want to know, OK, what do I do as a package manager knows that we solve the user case? That's quite simple. You know when something is compatible in terms of ABI? It's either in the version number or either it's contradicted by the change log when it's not in the version number. And basically, you know, OK, this is a fix which might be a security fix. So let's say in GitHub, you annotate your pushes to master and say, oh, security fix. Then somebody will run your patch and say, OK, this is a security fix. I need to go to the ABI compatible branch and cherry pick this one. So you have one model, which is like you have a branch, a master branch, and you have an ABI compatible branch, which starts every time we have a new channel, and which cherry pick every changes that you have. So then you can wonder, OK, how do we handle this ABI branch, this ABI compatible branch? You have multiple solution. Also, you can either merge master into it, which has some nice property of keeping the ones which are not there already, or you can reset the branch and start again. That's something which is up to debate, and I am not in the right position to ask that. Maybe domain. So if you have other ideas, feel free to contribute. And unfortunately, this model does not solve all the problem. It solves like a big chunk of sanity and making sure that you get transparent updates for users, which is like, hey, open SSH, oops. And this requires also some instrumentation if we want to get it optimal. OK, we're doing that with the compile time dependencies, but if there is a GCC update, OK, you probably want to have the compile time dependencies there. Let's take another example. If there is a bash update, OK, if there is a bash update, and your program does not necessarily depend on bash, and if it does, you want to update it, if it doesn't, then you're going to patch something with an expression which will say, OK, it takes the old hash of bash, takes a new hash of bash, and replace it in something where there is no occurrences of it. So that's kind of pointless. So one way of optimizing that, and I'm saying optimizing because this is the same version, and we don't necessarily want to patch things with no op. So a way of optimizing that is when Hydra builds Nix packages, we can say, hey, I scanned the build after building it, which is, by the way, already done for the garbage collector. So you can get all the dependencies that you have at runtime and say, oh, you had a list of inputs. Among the list of inputs that you have, here is how they are mapped in the binary. So basically, you can change that to the list of runtime dependencies. And then you get an optimization, which is, oh, don't have to look at every argument of my Nix expression for my package. But this has a downside of not giving you the same shot, because if you were to patch once, you might get the same content, but you had one more derivation to do. One more thing for Nix 3.0. So and, of course, you don't have static library support, which is another issue in terms of security, which is annotating, hey, you depend on this statically. But that's another issue. And that's the end of my talk. I have to admit, I didn't quite understand the whole fixed point stuff. Can you sort of explain, for instance, in terms of the open SSL example, what would this mean concretely for the package maintainer? So what does he or she commit to, say, the security branch or the master branch? And what does Hydra do? OK. So how is it magically going to prevent recompilation? So I won't take it as like, there is a package that got the information that there is a new open SSL fix in addition to the fact that we heard about the issue before. And so you have a patch. You have a patch which switch open SSL and to version L. You make a pull request. You apply it to master. And you go to the ABA branch, and you cherry pick this change on top of the ABA branch. What this will do is that here we have master. Here we have the ABA branch, which are emerging to one channel that the user will receive. But they're basically identical because you, well, I mean, you cherry pick. So not exactly. You have the latest master which compiled, which had, we still have the inversion of open SSL. You have the ABA branch, which has only the fixes since the last master which compiled. So here you have basically the same version with the fixes. So what happens here? If you have a fixed point of next packages here, you will basically do the same compilation, except for the one which have fixes because they will have a different derivation. So you will get a different result than the fixed point. So Hydra will see, oh, there is a new derivation that I don't know about because it's not in the stable channel. And you will recompile this one. So won't there be many changes if we update open SSL? So yes, there will be many changes here. But what Hydra do stops here. So as a user, you will repatch every package which depend on the SSL in your build. But Hydra only recompiles open SSL once and does not recompile any of the dependencies after. When you say it doesn't recompile any of the dependencies, you mean the reverse dependencies. I mean Python. It's not going to rebuild anything that depends on open SSL. Yes, it does not recompile anything which depend on the SSL. Yes. OK, and then what is the output? Is it going to be basically a hash rewriting thing? Is the output of the patching phase? Yes, that's the same as replace dependencies, which is rewriting the old hash with the new hash. And in what kind of format? I mean, ultimately, a channel needs to come out of this. And what will that look like? So currently, I think this is up to discussion. The way I made it in the branch that I have is basically having a quick fix folder, which is just mirror of the ABI branch. OK, I think I understand now. So you call the ABI NICS packages and you pass it the master NICS packages. So it uses the other way around. It's the other way around, right? So in terms of flow, as a current stable one pass into the ABI fixes, which will give you the same one if you have no changes and give you the fixes if you have new ABI fixes. OK, thank you. Thank you very much. I'm not, maybe I just asked my question whether I understood it right. So in the brown box, it's compiled in a way that all dependencies are just compiled through. So if you would change, the problem now is that if you would change somewhere down in the graph, it would trigger a recompilation of everything which depends on it. And this is what you try to avoid. So what you do instead is you say, you have a change and now normally in the normal build process, everything that depends on it would have to be rebuilt. And now you say, you have the additional information that the ABI is compatible and then you stop this recompilation process of the whole tree. And then just take this and overlay this new thing that you stop by knowing that the ABI is compatible with the old version. And this is how you fix it. Do I get this right? So I don't stop. I just do it once. Well, yes, but you stop in terms of you don't recompile all the dependencies that you normally would compile in building the fixed point in the brown. Box. Yes. So it comes down to the information that you have to give the system that the ABI is compatible for this particular change, which is the developer responsibility. Exactly. And so to apply this kind of mechanism, what we need is a way to actually put this information into the system so that the system knows, right, this is something where I can have an early stop because the ABI is compatible. And so I can just overlay this onto the current branch or the current tree that has been compiled. And no, because what you're suggesting, at least from what I understand, what you're suggesting by overlay is to move the changes here. No, build a fixed point in the end. Not move the change on it, but have a, how to say, you have the brown thing, where the fixed point has been fully compiled, and every dependency is compiled. And in the other one, if you would make a change, normally, the dependency would propagate by the derivation that you build and not by the API. So to stop the rebuilding of all the dependencies, you say I have this additional information that the API is compatible? I don't stop. How do we make it faster then? I don't stop dependencies. Next packages is one big function which takes itself as an input. So these functions are like, hey, I have one package which depends on the input of next packages, and for each attribute has one function which we applied once. The fixed point is here to make sure that you go through all the inputs deeply at multiple levels. By just applying it once, you just do one extra step. So if this is a function. So maybe another way to explain it. So in all packages.nix, the way we pass dependencies is not by passing, for instance, open SSL, but by passing packages.openSSL. And packages.openSSL in this case would refer to the old open SSL. So everything that depends on open SSL would still be referring to the old open SSL, so it wouldn't get recompiled. It's only open SSL itself that would be recompiled. So if you look at this as a function, I'm just doing F plus 1. I'm just calling it n time plus 1. And we're just compiling open SSL, but not any of its dependencies, because all its dependencies are seeing the old version from the stable branch. While open SSL is still seeing the old version, but we have a different derivation for it. Maybe we can continue this discussion after if you want. So well, I'll just try it again. So if this is a method to make things faster, why don't you use it as a default? That's what I'm suggesting. Oh, OK, so I didn't understand. No, he is asking why don't we always? Exactly, yeah. Any? Oh. But that's, I mean, Hydra would be too. So it doesn't work for static builds, but basically that's what Hydra is doing. It says that Hydra doesn't go beyond the project that it already has compiled. But that's another detail. OK, I've got a remark. I would question if this is really a sound approach to doing security updates, because the first thing is not all security patches are necessarily binary compatible. And second, we still have the problem with static Kali linked and stuff. So wouldn't it, in contrast, not be better to somehow speed the whole process up so that we can just get a soundly recompiled whole new tree? So as I mentioned, static binaries, yes. Static compilation is not addressed by this solution. I agree. We will have to address it in probably a different way, which might rely on that to make sure that we only recompile the one which depends statically on other programs. So then updates which are not binary compatible complain to the one who made the package. I can't fix all the problems. Yeah, I mean, we don't have many statically compiled things. And we want to avoid that. So my question is, so the patching would happen on the user machine, right? So the patching will happen on the user machine, yes. And from what I've heard from the Gates guys, it's a matter of minutes, less than 10, from what they told me. So basically every time the security channel is updated on, you know, every time the master branch finishes with the whole rebuild, the security branch will reset because you don't need the fast branch anymore because you have the binaries. So this will always reset. And we just cherry pick in the updates. And we pull both channels down. We apply the differences. And that's how it works, right? Yes. It says that I was thinking instead of pulling both channels, we pull the two branches into one channel where you have a quick fix directory which corresponds to the ABI fix phase. OK. That's about the same idea. OK, yeah, cool. Thanks. As a note, the problem with static linking is more than just linking in a static library. It's also when a package takes the actual source code for a Zlib whatever includes it with them. So the solution to that one is actually doing a binary scan for signatures of the vulnerable functions. So yeah, that's the problems that we have in Firefox, for example. And one of the examples I have in mind is, for example, the Zlib. We include the Zlib inside Firefox. And I can't remember when exactly. We had a security issue in Zlib. And some of the developer looked, OK, what is the patch? What does it affect? And that's the responsibility of the person who were embedding these packages directly in their source code to make sure that they also update when there is an issue in the packages that they embed. So in the case that I'm thinking about, we didn't make a security update for Firefox because we were not affected by the Zlib problem. So that's no longer oracle. Except if we do recompile Firefox with an external Zlib instead of the one which is inside your tree. As a maintainer, how do I keep up with the security updates? Follows the mailing list. So if I'm a maintainer on the next packages, is it my responsibility to do that? Can I get notified if one of my packages is? If you're following the mailing list and you're probably going to be notified. Otherwise, I hope a few people will keep looking at the CVE and others to make sure that, hey, think the right person or gives a patch for a patch or an SSL or whatever to make sure that it is applied. Even if you're a maintainer, you're proud. You might be in vacation or something else. I don't know. So that's something that we have to discuss as a community. What is the responsibility of who? I was just going to say, we've also got the Nixos packages monitor, which is I think monitor.nixos.org, which goes through the CVEs and tries to compare them against packages we have in Nix packages. So you could at least look at that and then see if you're a maintainer for anything that has known CVEs. Can I get an email? So somebody mentioned monitor.nixos.org to give an answer to that. OK. So if there are any other questions, Nicholas is still around, right, for the Marine Doctor Conference. Yes. So thank you very much for the presentation.