 Okay. Let's start. As already introduced, I'm a former Debian maintainer and I've been contributing to NixOS since 2015 after I got fed up with normal distributions or doing package management and how configuration management is built on top of package management but you're always on end to trouble if you use tools like Puppet, Chef, or Ansible and since then I started like looking into new things and one of the things that I really like is NixOS. Since then I've been a release manager and I'm on the security team and on my day job I'm an infrastructure engineer at Mayflower. Also I've been able to work on company time for NixOS at Mayflower and we are providing consultancy services for the NixOS ecosystem. So about this talk, I've done a few Nix introduction talks. This will be no real Nix introduction talk. I want to showcase some Nix concepts and the value to the greater Linux distribution community. So it will be on a technical level and I assume that we have some new Nix knowledge but please do ask questions during the talk if something is not clear. Okay. So about the terms I'm going to use. So Nix is a package manager that was developed around 2005 as part of a PhD thesis and another lens by Alcold Olstra. Nix is also used as a term for the Nix expression language. That's a purely functional programming language that was tailored to defining package builds. And there's Nix PKGs or Nix packages. That's a package set containing package build definitions for like 70,000 packages. For instance, we automatically import the Haskell package set. So all Haskell packages are by default available and tested for our environment. And it's just a GitHub repository. A big GitHub repository with over like 1000 contributors. And there's NixOS which is built on top of Nix and on top of Nix PKGs, the package set. That's building a whole Linux distribution with the kernel, with the bootloader. There are service modules that basically replace configuration management and a few other goodies. There's also Nix Ops which is a deployment tool that fits into the whole ecosystem. With that you can deploy NixOS systems to like EC2 to a local virtual box for testing or to just a host that has NixOS installed via SSH. Okay, let's get started. The typical procedural approach to package management roughly works like this. You have some packaging instructions and metadata and you have the state of the file system which includes headers, libraries and all the dependencies you want to have. You take those who invoke the build system of the software you want to package which then builds the software and the files are then placed on the file system at a specific prefix, normally slash USR and that in turn modifies the state of the file system. And after that you can build other software that depends on that new software that you just built. So what happens here? The build results depend on an inherent state by previous builds or by previous package installs and installing packages modifies the global file system state. Okay, so far so good. Now if you go to the functional approach you have to understand one thing. What is purely functional in that context? You may be no functional programming languages like Haskell, Scheme, Lisp but purely functional is quite a very strict definition of a functional language that is functions are pure if and only if they always evaluate to the same results given the same arguments. For instance, this might not be true for languages like Scheme or Lisp because they are not purely functional but languages like Haskell or Akta are purely functional. They have no side effects and the evaluation of a function causes no semantical observable side effects either in the system or the program itself. That means purely functional. So how do we like do that in a package management environment? The functional approach also has packaging instructions and metadata as the input and that it will invoke the build system of the software package but in these packaging instructions and metadata all the dependencies are also included. The output will then be stored in some unique prefix so not slash USR but some other prefix and that will later be used in the packaging instructions to refer to that exact version of the program. Okay, but what do we gain from that? So at one thing we gain is we have a conflict free system. We can have multiple versions of different programs installed or the same program installed because they are installed in a completely different prefix. By definition it has to be immutable because you always want to have the same software. It's also reproducible because you always define what exact dependencies you want to have. So not only the version but all the transitive dependencies along the way. And you can do atomic operations on them because you can assume that you have no conflicts, everything is immutable and everything is reproducible. So how does this look in practice? In practice, Nix has a concept called Nix Store. So that is that custom prefix where all programs will be installed. In this example, I have crypt setup which you probably all know. It will be installed in slash Nix slash Store. Then hash sum and that hash sum is built over all inputs so all dependencies and the package build definition. So if you alter the package build definition like increase the version or add a new command that does some fixing up for example, you will get a new hash and a new store path. That of course means that if you change something, you need to rebuild that package and all packages depending on that package. So for instance, if you update Chilip C, you have to rebuild everything. You might say now that that's a bad thing but actually it's a good thing because we always, if you like, change some core library, we always bootstrap everything and we make sure that it's buildable every time. For instance in Debian, you have situations where some library bump, after some library bump, you have to recompile software and in Debian it's a very concerted effort to build the libraries or the programs that depending on the library and then upload them. We don't do that, we always like rebuild everything. Okay, we also have a notion of multiple outputs. So that crypt setup you saw has like all the binaries, you can execute the libraries and some stuff in Chilip local. We also have a development package, we call that output into the store path, that's the dev output and the dev output, we have stuff like the headers or the package config file and some special NIC support files. Okay, so how do we do that? We lock every single dependency down to the NIC store path. So if you look at the libraries used by crypt setup, the crypt setup binary, you see that every single library has a fixed R path to the exact version of the library that it was compiled with. So if I oops, try to scroll, it doesn't work right now. As you can see it's just the whole store path and after that comes the library from that exact store path. So far so good, this is for like normal ELF executables, but what about other stuff? How do we lock down the dependencies there? So for instance, this is a Borg or a Borg backup, it's a Python program for doing backups and not only does Borg require some Python libraries, it also requires OpenMSSH to SSH into machines and copy backups there. So what we are doing is this is the Borg executable, it has a shebang that is fixed to a specific version of bash, this is just a wrapper script that appends the path to the OpenMSSH bin folder into the path and then executes the actual Python executable or the Python script of Borg. If you look at the wrapped Python script of Borg, we see that the Python interpreter on the top, the shebang here is also fixed to a specific Python version and there's some special stuff inserted that adds the needed Python libraries to the Python side path. So for instance, Borg needs Siphon, LLFuse and Contextlib 2. They are added and explicitly defined and only that exact version will be used. We do that not only for Python, not only for shell scripts, but for Haskell, for Ruby, for whatever language you can think of, even Node.js. And MPM. There's another concept called profiles. These are basically version symbolings where you can have a set of packages or a set of store profiles that you can put together and put into an environment and then like version it and do like updates or changes and then roll back. What you see here is that there's a main sim link, in this case it's the system profile and it links to a version of the system profile which then links to the specific system profile version. This works for like the system profile in that case, but we also have user profiles. And if you for instance look at the system profile, this is a mix OS operating system profile which includes stuff like the activation script to activate the system, slash ETC, which we also statically generate, some firmware, the init, RD, the kernel, the kernel modules, and as we use the actual software available on the system, which is also like a sub environment. And we also have like system D on there, which is kind of special because it's the init system. We also have user profiles because we have multi-user package management. So every user can actually use nix if nix is installed on a system in multi-user mode and can build their own package that is only available for them. For instance, and the user profile, user profiles are also versioned and user environments include also their own prefix where every software that is defined in environment is linked together. And it's made sure that all the software finds the exact versions of the dependencies that are needed. So for instance, if I look at my path variable when I log in, I see that nix-one-nix profiles per user or there's the user name missing fblads profile slash bin is in my local path variable so I can actually could everything that I installed in my user profile by default. Okay, so how do we achieve piratnix? So on the concept level, a package or how we call it a store path in the next store is a result of a pure function that is written in nix and has, as said before, no side effects. And the result on it depends on the functional arguments without the side effects. Okay, so if we are in a nix build environment, we have a sandbox in place. So on the one hand we have that functional definition which by itself is like functional. We use of course nix namespaces and z-groups just as any container technology now that they would use. You can think of it like a lightweight container but we don't use every namespace out there. Just the ones that are useful to achieve the level of isolation we want in the build environment. We set a fixed time. So every build will be executed with a fixed timestamp. All created files will have a fixed timestamp so that there's no deviation between when you build it or which machine you've built it. There is no networking available so you can't just do like npm install to install dependencies that you don't know like the real contents and the version of because like other package managers just don't know stuff depending on the branch or the version. And we also do stuff like patching the she bangs of shell scripts or like some build scripts automatically that are in the source table. And we have some patches to tooling to ensure reproducibility. We don't have much patches left because we're also part of the greater reproducible build effort and a lot of stuff has already been upstreamed. So not much patches are left. So at some point we have to introduce impurity and one of the impurities we have is patching source turbos. So what we do is if we have impurity we always guard the package definition by a checksum. In particular the cryptographic checksum of the contents. So if we for instance define the package crypt setup and we fetch the sources from a kernel mirror we always include the checksum of the content so that we can be sure that we always get the same source table. Oops. There's I think I was missing but what this is called is it's a fixed output derivation where you already know beforehand what the output of that build is going to be like. So you can think of this fetch URL call as another package that will be installed but you already know what contents it will be like. So in that case of a fixed output derivation we enable networking and downloading is actually possible. This is the only way you can do impure stuff. You could for instance do like npm install in a fixed output derivation but then the output has always be the same and it won't be. Okay so how is the NICS package set structured? In NICS there are the set of the data type called attribute set which is basically a key value pair or like a key value map of key and n values and here's a small snippet of some examples. For instance we have crypt setup that calls the function call package with the definition of crypt setup which is in some other folder. For instance if we look at other cap this is special because other cap doesn't compile with a recent version of OpenMSSL which we have by default. So we have to pin OpenMSSL to an older version to make that package compile. So in this case OpenMSSL 102. And if you look at OpenMSSL we do something similar for PUM because the NICS package set can also be used on Darwin systems where no PUM is available. So if we are on Linux then PUM is available, else PUM will be null and PUM will not be used for package builds because on Darwin it would fail. We can do similar stuff and like more complicated code for like every package out there. So we get exactly the variation of dependencies that will build for that system that you're currently on or like for which you define the build. Okay so on to defining a package build. This is actually what's in this directory where call package will go look for the source. And on the first line you see basically a function definition with some arguments called start n fetch URL LVM2. These are the same attribute names that are present in the top level attribute set. So call package just passes the right attribute set names with the values to the package build. And after that we have the first line the actual definition which says we want to make a derivation which is like the internal name NICS for intermediate format for like defining package builds. And we use STDN here which is the standard environment which includes a lot of like convenience wrappers for build tools. So for instance if you have an auto tools project or a CMake project we already have code in there that detects that and automatically call the right build instructions for your build system. Of course not everything is supported but most of it is. So if we continue we define a name. We already looked at source. It's just a fetch URL call. We can add patches like define custom configure flex because in this case we are dealing with an auto tools project so we have configure flex. And we even can do stuff like if Python is enabled by that flag we also add the enable Python flag to configure. Then we have like the definition of the dependencies. We have one hand native build inputs which are dependencies that are to be run on the system where the software will be compiled and build inputs are the dependencies for the system on which the package is to be run. Because we have native support for cross compiling like for instance I can cross compile on my machine, my x86 machine, a package for AR64. But for that to work I need the package config for my system because it has to run on my system but I need the LVM2 for instance of AR64. So that is there are even like more options for that because there are some more cases. But this is the normal easy case. And we can define the outputs which in this case is just a default output, the dev output and the main output. And the cool thing is these outputs are also like automatically generated. So if I define a dev output the standard wrapper will automatically add a configure flex so that the include files will only be installed to the dev output and the main pages will all be installed into the main output. You can also do that manually and the whole package build is divided into phases. You can add hooks to like the install phase, the build phase, the configure phase and like do your own thing if it's a non standard package package build which is mostly the case. Okay so on to next packages. You can overwrite packages so if you're a user of next packages you can say well there's this this kause package but I want to like execute as muse. So I can add a post install hook that just adds a link to kause called muse in the output. And if I then use like anything that installs kause for my local user I will get that version of kause that has the link to muse. You can also do that in the next configuration but we get into that later. And so next you don't have to understand anything that's on the slide. Package overwrites are an old concept because you can only refer to packages after they've been fully evaluated. So next packages itself evaluates the whole next package set to a fixed point which basically means that the dependency resolution will be taken care of like the fixed point combinator. And the cool thing about that is that you can hook into it and overwrite packages but without like having like an infinite recursion if you like overwrite a package and you access something in that package again. So that's basically everything that happens behind that with some example because like if everything is a function like in that case that always takes itself as an argument. You can refer to attributes like foo and bar in the package set itself because it will be recursively evaluated until there's no nothing left that has to be evaluated. So this is implement in pure next. This is the fix function. There's nothing magic going on. It's available in every functional language and if you evaluate that this function we get what we expect. So the concatenation of foo and bar as a string. And if you go one step further and create extensible attributes that can be like where you can define overwrites. You can define the extents function that you don't have to understand. But the cool thing is now we can define a new function called g that takes self and another super argument and then you can redefine foo by the previous version of foo and you can change it for instance. So in a normal case when you define it like that above with the fix function it would result in an infinite recursion. But in that case if we use fix and extents with those two functions you get what you expect. We overwrite foo with foo plus bar. And if we evaluate foo bar again which that function will do we get foo plus bar instead of foo bar. So that is what's happening under the hood for every package to define. And with that we can define much more powerful overwrites. We call it overlays. You see the same here. We are using self and super as arguments. And for instance if we want to overwrite boost we just say we want to use the previous version of boost like the version of boost before our overlay was introduced and evaluated. And we overwrite Python with Python 3 for instance because we want to have like the boost Python libraries built with Python 3. And now we have a boost built with Python 3. Also you can do that with like your own packages. You can just call package like in the top level package definition. Have your local package definition here. And for instance you want to compare it for 32 bit. No problem. There's a standard for 32 bit. And you have on your x86 machine x86 64 bit machine 32 bit executable. And to use it when you like instantiate next packages you can just add it as a parameter. Both in the next configuration by environment variables or like at import time. The next concept are development environments. The tool we use is called nix shell. If you append a dash A you go into the build environment of that specific package. So if I do nix shell dash A script setup I'm in the build environment of script setup and all dependencies for script setup are available. If I would be in a git check out of script setup I could just call configure and everything it needs is available. If I exit the shell everything is gone again. And this works for like C libraries, Python packages, Ruby packages, Haskell, whatever. But not only that also all the phases are available. So if I call the special generic build function the whole package build will be called or just like executed. And also you can invoke specific phases. So you could say I want just to invoke the configure phase. Only configure with the specific configure flex defined in your package definition will be executed or I just want to like call the build phase. And so you can do like incremental builds, changing stuff, patching stuff. Oh, yeah, now it works. Create the patch, put the next big edge is, yeah, it works. You can also do that for like your development projects. You can do on the one hand imperative environments. So in that example I want to execute Python 3. Python 3 is not available in my normal environment. So I open a next shell with dash p to have the Python 3 package available. After that I'm in a next shell denoted by that special prompt. And if I then call Python 3 it's available. But not only that, if I want to use the request's Python package which is not available because it's like a regular Python. I exit the shell. I use Python 3 and the Python 3 packages requests. And after that request is available. If you want to use declarative environments you can create a next file for instance in the project route of your software where you can define which packages are needed for like building, developing, building some front end stuff. And in this case it's basically also a make derivation with built inputs. And this is just another way to denote which Python package you want. After that you can just execute next shell because next shell uses the default next in the local directory by default. And you're in next shell and you can do like development. Okay. Next was demo time. I don't have that much time left. So I skip it for now because I was light prepared for that. So key features of next was we have atomic upgrades and rollbacks because everything is just one sim link. If I want to switch from one version to the next to a package upgrade to a configuration change or whatever it's just one sim link I have to change and the shell script to execute of course. But one sim link and the new system is active. Multi was a package meant or I told about that. You just have to write one configuration file and you can build it for like a multitude of targets. You can build Docker images, ISOs, net boot images, container tarpaul, whatever. We have about 800 service modules available. So like for engine X Apache, index cloud, GitLab, I don't know, so much software, so much modules which make it very easy to set up for instance Grafana which I had prepared as a demo which is just a few lines of code. Complete with let's encrypt certificate and everything. And we also have containers for system dn spawn and testing framework using four m's. We have some features coming up. For instance to use like nix for nix expressions as a package manager itself. It's called nix flakes or like you want to use system dnetwork by default. We have to bump the gcc version and some more. Oops. And we have some challenges. For instance we have world readable nix store. That means like you don't have to, you shouldn't put secrets into the nix store. This is a thing that we can't easily fix for now. Also we don't match application state. That's for the administrator to do. And our declarative containers don't support user namespaces because system d doesn't support dynamic user in system dn spawn containers with user namespacing enabled. That's an issue we still have. Okay. And that's it for now. If you don't have questions, I have the demo for you. Every one wants to see the demo. No, it's a question. I think I know what a bunch of the answer is when I ask it of you. What's the current state of arm64 support on NixOS? Good question. We have first class arm64 support in NixOS and actually the arm64 builds are much quicker than our x86 builds because we packet which is also a sponsor of this conference donated a lot of build machines to us because we have to build, as we've said, we have to rebuild a lot of stuff. So we have a lot of powerful systems. And thanks to packet net we can do that for at least two architectures, two architectures which are main release architectures which are x86 64 and arm64. So I wanted to ask about the language used in the configuration files, how long it's going to take to learn it or why not use like an existing function. I don't know what it is actually. Okay. I didn't introduce Nix itself because of the language because it would take too much time but you saw a little bit of that. So it takes a little bit getting used to because some of the syntax elements you see you maybe know from other languages but they are not. So if you already like did some functional programming with any functional language you might be quicker to learn it but if you have not yet before used the functional language it might take a while. But the problem itself is not the Nix language but all the concepts and all the abstractions we have for package builds. So you have to know the specifics. We have a lot of documentation by now but most of the times you will have to read source code and for beginners you have, you should read the source code of for example Nix bigages to like understand what Nix language constructs are useful and you should actually use. So if you want to use it there's a very steep language learning curve ahead of you but in my opinion it's worth it. Okay. How much time do I have left? Five minutes. Then we can do a demo. Okay. So I have a machine on the internet you see on the right and an SSH on the left. You can see that not much is running on there. I have a NixOS configuration which is by default in slash NIC ATC slash NixOS configuration and this is basically what it looks like. So there's some boot loader configuration, there's some networking configuration for peer addresses, default routes, DNS server, I enable bash completion, I make some tools available, I add some public SSH keys, I enable open SSH, I open the firewall and if you now look here that's the domain, nothing is available, the port 80 or 443 are closed and I now go back to the configuration and I enable these two services. So I enable NGNX, I enable a virtual host and enable ECME which automatically fetches and that's the crypt certificate for me. I use 4SSL to add an HTTPS redirect and proxy to localhost 3000 where Grafana is running if I enable it and don't configure it anything by default and if I want to apply that I just call NixOS rebuild switch which will build a configuration and download the software package as needed which it's doing now and rebuild some stuff like our users and groups, some system D units, the whole user environment like system environment, all the software is installed and if it's finished it should start the NGNX service, the Grafana service, the services to fetch the ECME certificates and that should be it. Let's see, it's a VM running on a virtual host somewhere so it's not that fast unfortunately and I think my time is up. Okay, now you see system D service were restarted or started and if I go into the browser and refresh, we see bad gateway because Grafana is still starting up, the fafficon is already there because I was on that page before but no, Grafana, come on, there it is.