 Yeah, so basically, I want to talk about a very hot topic at the moment and we have only 40 minutes, so better start. I'm going to talk about a tool set that I've been using for the last year and a half called Nix and how it applies to the whole stack of packaging development and deployment. Basically packaging in Python, we all know it's a quite a depressing topic, but it's getting better. And one of the main things that I really, really, really hate about it is that we have the setup.py that is dynamic and whatever you do, you have to run this dynamic script, at least run the egg info command to get something out of it. And that's why, for example, we don't have dependencies in the PyPy and so on and so on. And for example, Node.js community has this simple JSON file that is static, you write it down and you can easily parse packages and stuff with it. But there is hope. There is the PEP426 meta data 2.0, which specifies basically the JSON meta data for packaging and hopefully people will then generate this file, put it into a distribution with the Python source and we will have static meta data available. It's in the draft mode, so who knows when this will get upstream. But yeah, there is hope. And the second point is we have a lot of legacy infrastructure, which is kind of connected to the setup.py, but there is now Python packaging authority group that's working on this and all the contributors. And I think they really deserve an applause for their work. And then there is this scary third part of the problem, there's our non-Python dependencies. And this is the problem that every community tries to solve by, you know, building an infrastructure to package all the other stuff, not just Python. And we all share this goal. But maybe it's time, you know, to look out and take something else, you know. Nowadays JavaScript stack is basically inevitable. You have always JavaScript stack in your tool set. And you know, we can either build all the tools in Python to process JavaScript dependencies and so on and so on. Or we can take the JavaScript tool set that they use. But then we need a tool that will actually package Python and JavaScript, right, for our application. And there might be even other things. So Nick's project was basically started 11 years ago. And it was developed by Elko Dostra as part of his PhD. And the PhD talks about dependency hell and how to approach that. And it was done in a university in Holland in a functional language department. So the idea is to take functional model thinking and apply it to the packaging problem. And it turns out it's a really, really, really fits the problem. So basically Nick's is two things. It's a package manager and it's also the language, which we also call Nick's expressions. It's a very minimal language. It's basically a configuration files plus lambda functions and a little bit of other stuff. It's lazy evaluated. That's something that we're kind of not used in the Python community that much. And you have to get used to the whole thing that when you actually something is actually touched, then it's actually evaluated. And that gives it a really a lot of power for configuration, which I will show a little bit later. It's a standalone package manager. You can install it on any POSIX system. The official support we have for Linux, Mac and 3BSD. There could be, it could work on Windows if a company will sponsor that work. But currently the support is basically discontinued. Yeah. So what is a purely functional language? I will give a very vague description in the sense of the software package. So basically the software package should be output of a function that is deterministic. And it only depends on the functional inputs without any side effects. So when we describe the packages in Nick's, the metadata that we put on those files, that's the only thing that should affect the package and nothing else. And we call this purity in Nick's. So there you see an example package. This is the prefix where the package would get installed. So slash Nick's story is like a flat repository for the packages. And then you see the cryptographic hash as how on and then the name of the package and the version. So every package is stored in this separate folder. And because we want these results, the outputs to be deterministic, we want to make it immutable so the whole Nick's story is mounted as read only just to be sure that nobody will touch it. And all the timestamps are set to Nick's daytime plus one and so on and so on. And this hash that you see there is basically the hash of all the inputs. So if this theory works, the result should depend on the inputs. Then if we hash it, then we can uniquely identify a package. So this is, can you actually see this? It's a little bit, I hope you can see it. But this is like an example engine package, how we would package engine X. It's a little bit simplified at what we currently have in the repositories. So at the top you see this is anonymous lambda function. It gets high order functions. So those are just another functions passed in. And then we call the standard environment make duration, which is the main function that does all the heavy lifting. And in there, you basically see it's like we call it attribute sets in X, but this is basically a dictionary. And we pass it name, version, and then we tell it where to go to download the sources. We tell the dependencies, which is called build inputs, some configure flags, and then some just description about the package. And then all this is basically passed to a bash script that goes to different phases and it knows what to do with this metadata. And what you see here, this is basically what gets hashed. These are all the inputs to build engine X. This is all the information we need. And of course, there is a dependency graph of the packages, so OpenSSL, Zlib, and so on and so on, are also written in this index. So this is like a quick example how powerful Nix is. If you look, if you go a little bit back, if you look at this file, we want to override two things, basically the lambda function at the top and the metadata by the package. So we want to give a user of this distribution or package repository the power to change anything. So the top line basically overrides the lambda function with something new. But it's probably fell asleep. So basically it overrides the lambda function and we can say, okay, let's take another OpenSSL and feed it into and then we get the new engine X package with a different OpenSSL version. And we can override the derivation itself and, for example, at the bottom example, I override the source and we can, for example, take engine X from Git. And this is what you can do in the user space, right? Because sometimes you have to change what the upstream does. So to install Nix on your distribution, this is like, I mean, from a security point of view people will go crazy, but you can download the script and see it's not doing that much and you can run it yourself. But basically this is the easiest way to install it. And because everything is stored under slash Nix, you can just remove slash Nix and you don't have your package manager anymore. And you also have to remove the profile in your free user. So this is basically where everything comes together and there's a lot of things to explain around how the Nix works. So okay, we said that we have slash Nix, slash store and inside there are the packages and you can see version and Firefox. So somehow we need to get this file system here that we're used to nowadays, right? That basically is then joined together into a user environment which you can see on the right. And this is basically your environment where all the binaries are, libraries are stored and under slash bin, slash label and so on. And because we have this set of packages, wouldn't it be cool that we can have multiple of those, not just one on the system? So this is what we have so-called profile synics. We have then, I will talk later about Nix size, which is distribution builds on top of the package manager. And we have a system profile there, which is basically your distribution, but then each user gets a profile and you can create profiles on the Plyper project. And each profile has its own life cycle of how you install packages inside and upgrade and delete them and uninstall them. And basically then the profile also has a whole history of what's changed. So basically, when you install a package, you would get a new user environment with that binary inside and the profile would get like another version number in the history. And basically the last thing that you actually do something to the package manager is that sim link, Nix profile default will then at the end of all operations change to the new user environment. And because sim links are atomic in Linux, that means that we have atomic operations of installing, upgrading and uninstalling packages. And then on the left you see that each user then has its own profile and that means that users without root access they can uninstall packages for themselves. Of course you can disable this if you want and so on. So one of the really cool features is that Nix is source and binary based collection of packages. This is very unique and the way that this actually works is that we have a so-called build farm called Hydra and we build all the packages there. And basically because the hash basically provides the, basically uniquely identifies the package you can go ask the Hydra server, oh do you have a package with this hash and if Hydra has this package it will fetch the binary and if not it will go and compile it. And this is something that companies then use to set up Hydra at their own servers and have their own like basically continuous integration tool for building the packages. And there is now in the Nix 1.7 I think there is also support for SSH so you can do the same thing through the SSH protocol not only HTTP. So just, I don't know if this is going to work but let's try it out. It's a little bit, so basically like this is Nix so it's a little bit different than if you only use Nix but like my VIM binary then points to the VIM binary that is stored inside the Nix store. Let me show you Nix store just so that you see that there is a bunch of stuff in there. So yeah this is the whole thing. And for example if we look at the linker of VIM you will see that all the dynamic libraries point to precisely one package in the Nix store. And that makes it very deterministic that you know for sure that if you build this on two different machines you will get the same result if you use the same source of Nix packages. So right let's talk about Python. Of course we have also a collection of Python packages and we have this function called build Python package which is basically a wrapper around make derivation that knows about these tutorials and setup tools and it works a little bit quite the same than the make derivation and this is like for example how you would package pillow, provide the dependencies, source, metadata and build Python package will know how to run the setup pie, build and then setup, install at the right phases. You can check inside the Nix packages repository how it works it's just like 200 lines for the whole implementation. So when you have a lot of packages for example I also do a clone development and we have like 250 packages. You don't want to do that by your hand. So there are basically two tools for this. Python to Nix is basically just goes there grabs the table gets the hash and splits out a template that is very generic and if there is something really like non-Python dependencies then you will have to fix that on your own and there is this cool tool called PyPy to Nix which we will also be working on during the sprints. There's quite a lot of Nix developers here that tries to handle all the edge cases and automatically basically fetch packages from PyPy and then generate these Nix packages for you and we have these tools for Node, we have Node to Nix and we have R and so on and so on. Right so that's packaging, let's move to development, right? So wouldn't it be cool if we had a tool like virtual environment but on the layer of the package manager not just for Python software so that you would activate environment and you would get like geeks and all the other non-Python basically dependencies and tools available and that's what Nix shall do. So basically how Nix shall works is that it will build all the dependencies of your package, it will source all the information it has about those and instead of actually go and building this package it will get you into the shell that it would actually build the package so you have everything there available. And there is a cool, well not really a hack because it's also meant to use this way but there is a cool feature that you can say that you're not building any package you set source to null and then you just provide the build inputs and you say Nix shall and you only get these dependencies available in your shell for example and in this right this works on on a POSIX system so this POSIX system so you can give this to developers and they will get always the same environment with the same geeks and so on and so on. There is also a flag called pure so by default Nix shall will inherit your current environment and you will have all the tools available and pure basically means that it will not do that it you will have only the tools available that you list in the build inputs. So let's I'm sorry for the I hope you still see something so this is basically activated Nix shall I mean I did this before on my laptop otherwise it would go and download those packages from from Hydra but the network here is a bit flaking so basically now I have git available well let me just see that file yeah so now I have git available and if I do pure here and then git it will say it's not available because it will not inherit it and this is one way to to make sure that you have all the tools in your Nix shall so the same thing goes this is I use this this trick to actually install a media core on CentOS because I just didn't want to bother with Python there so I just use the whole Nix stack stack of packages and to use Nix shall and then I have everything available and run virtual install and that's it um so the same goes if you have a python package basically so this is for example a python package of this is one project that I did like I have gstreamer in there debas and all kinds of things that are hard to package normally with python and and there is a this cool trick we call we have a like we have a variable called Nix shall so when you actually run the shell this will be true and we can act the extra dependencies in this case and if you only build a package those dependencies will not get into the derivation so so then okay then okay we have this set of packages how could we extend this idea of a functional language to to the whole operating system and build a distribution on top of it and it turns out yeah that yes this works really nicely and when you think of it like a configuration files are basically just one file and software package is a bunch of files the only difference is that your linux distribution will package software for you and the the configurations files is what you will normally write yourself or change some default but nix nix is basically the language that we have now so you can use this language on both sites so nix says basically is a stateless this uses a stateless approach to configuration so for example in puppet and and chef they they have declarative configuration in front but at the back basically they still execute a step-by-step imperative they check if engine x is up and if not I mean then it will start it and so on and there are a lot of edge cases to cover here um so a lot of errors that you can hit onto and here basically the way it works is if something changes I will show later an example of how how to do a system d process and if any parameter to that system d process changes then it will know that it has to restart or reload that process so it all boils down to this data going through these functions in x and and when something changes it will do an update so here you can see like a minimum configuration and I just like a configure a morning um and uh you would then say nix OS rebuild switch and it would activate and get the machine into this state so so one of the things that's also good to mention here is that nix OS is basically DevOps from the beginning you won't go and change some configuration files by default you have one file you specify it what do you want your state machine state to be and you execute it and then we have a tool that basically does provisioning of cloud servers and so on on top of that um so for example if we wanted to use pyramids which I'm using my day job basically we would import that to default that nix file that we were using before for development so the project is already packaged but then we would say package write tests and we would write the production any uh file to nix store that's write x is basically a function that will write a configuration files to the nix store and then we will do a declaratively uh specific file okay we have a process a system d process uh that should start with pyramid being served and pass the production any file and for example if the production any file changes here then uh this hash of will change of this uh service and it will know it has to reload or restart it and so on um and then of course on top of that we want to use a provisioning tool right um and this is like the minimum example how to how to then provision nix OS machines uh you install nix ops you specify uh for example this is a trivial a trivial machine so we have like a web server uh running Apache serving some static files uh this is like the the physical configuration and then we have the logical which is basically where do we want to deploy it uh we say okay the back end is virtual bugs giving me one gig i'm a bite of memory and we then i have like a trivial hat snare um because nixop supports like amazon uh hat snare uh and now also google compute engine it's experimental bits and docker yeah and then you would say create the create this uh configuration and then deploy it and it would actually provision the virtual box and and um you would have Apache running in your virtual box so uh i don't really have a demo for this um because it's going to take a while to actually show it but just to just to show you the whole stack so when i would actually deploy my projects then i would have three files one in default dot nix which is for the development and the building of of this project i'm developing example machine which defines the physical and then the physical state of machine and then the virtual box okay and then also the hit snare so it's four um and if we look at all those files first default um it's a little bit stripped but basically there you have just like build package my name sources current directory and dependencies pyramid and this is this is for example uh configuration of machine that would launch a chromium full screen in kiosk mode and serve the pyramid so at the top you say we say okay import the package uh then enable x server enable display manager window manager and et cetera et cetera and the desktop manager we provide our command which is basically waits three seconds and then run chromium at a local host 8080 uh and then uh lower you see again the configuration of the system d service uh for pyramids very simple example and at the bottom you see how we define declaratively uh a user called guest that we use for the for this um for the chromium um graphical interface uh and it's this is basically then the whole configuration of the machine um so you can see like your package the package was about 10 lines this is about i don't know 100 lines or something 150 and the the virtual box uh is is like a few lines also and this is and this is the whole packaging development and deployment stack that you then use to to actually and let's let's still try to so now it tries to download a basic image and it's yeah we can wait for a little while but uh i guess there is no point so basically now it would download first the image and then uh all the dependencies uh configure them to launch the virtual box copy all those sneaks uh store packages inside and then activate and you would get the the full screen chrome with with the pyramid application running we are having the second nix i sprint in in Slovenia and Ljubljana where i'm from in from the 23rd to 22nd of august so this is a great opportunity to me the developers um talk to them the the core the L code also will probably also be there the core developer and this is this is basically the image from the last year um just two two shameless plugs i wrote a blog post a little bit more into detail why why my nixis tries to solve this problem in a little better way than other other solutions and i had a talk at four's them about uh nixis it was more focused on nixis so there is a video on youtube if you want to watch it and of course well check it out on nixis.org and you're welcome on free note to stop by and say hi and meet the community so questions yeah um so it looks quite interesting and i didn't know nixis as before so it looks like my puppet my vagrant and everything i can throw away and then get nix and it all runs out of the box why isn't it um like widely adopted so um so far um what do you can you mark out uh what are the differences what are pros and cons and so on so the question is if why do why is it not that's popular yet um yeah a comparison to like vagrant and all the established tools uh which are of course very different have different dsl's and so on so of course it would be awesome to have one approach to this kind of problems yeah i mean the short answer would be we need more marketing people the long answer would be that um actually now for about a year or two we have uh nixops and and nix shell and so on and now i think finally this is this tech is ready to be used and one of the the biggest two companies is logic blocks and zalora and they have about 100 servers provisioned with this um and and the community is really growing the haskell community basically uh there were a few blog posts in the last few months explaining how people develop haskell with nixis and it went very viral and i would love love that the same happens to to the python community if we actually want to solve these problems and i think now it's the time that's you know this is we really see a lot of a lot of new users you can see that mailing list irc and everywhere and i hope that we get there uh i hope that we get to the same point it looks really interesting um the you said that it's supported on posix systems yeah that include mac yeah yeah yeah we have we have quite a bunch of uh unhappy homebrew x users that now use nixis for for the packages yeah all right thank you but we don't have that much free bsd users yet so uh it's more of a just one thing maybe was not clear what it does like what are the benefits of it's not a question it's just as uh what i missed from the the talk what are the benefits uh of actually using uh and starting developing uh nix uh they're using nix and with python it's uh you get in our company we got uh development environment which was we kind of switched to continuous development environments like not the only for deployments we use this thing but also for continuous uh development right so each time a developer comes like in a snap of a finger you have a development environment ready and no virtualization right it's just your system so it's kind of a lot different way and quickly getting started with new projects that was a major boost for us cool thanks hi uh thanks for a great great talk i had a few questions one is that you mentioned briefly binaries right do you provide like does nix itself provide binaries or do you expect the one wanting to use binaries for deploying stuff with nix to like roll your own binary storage or something yeah so the the hidra project basically you can host it yourself and then it will build the binaries of your customized packages or your uh projects in your company and you can point that to the official hidra and to your hidra and it will just ask about for binaries and fetch them another question that came to mind is more like a security related okay let's say you find you know things that would never happen you find a bug in open ssl or something and you get like like how i'm thinking it's like if i run normal like say depending on the box i get the new open ssl which has the same a bi compatibility and the only thing i need to do is just restart my services for them to end up in the same uh like using the newest version so how do you because i saw that you link specifically to those like a certain versions of software how do you solve this in nix do you this just mean rebuilding everything or yeah so um yeah this is this is one of the problems that we so basically yeah if you if you change the open ssl then hidra has to recompile all the binaries this norm like the last time that was a hole in the open ssl i think it took like one day or something uh that's of course unacceptable um uh but we have now an option called uh i think it's called security updates or something in xs and basically there is a hack around xs that the hash will not change so you don't have to recompile everything so you say the original the original library was open ssl and the new one is open ssl blah blah blah and then that will replace all all the everything that uses this open ssl library and it will not go and rebuild everything that's needed um and that way you can really really quickly you know update your server and because if you're using nix so as it will also know that which which processes we're using open ssl and go and restart those um the um the uh hash of of your package depends on its inputs which is can you be a little bit louder sorry so the hash for your package depends on the inputs which say for a pure python package would be python itself um so you can rebuild that and get the same version but is there is there anything that also ties it to the version of the nix toolchain that you used to build it because if there was a new feature introduced in the nix toolchain or something is that is there a way of basically rebuilding a package of like two years ago or something exactly as it was at that time so if uh basically a nix uh upgrades or something yeah um but that's like the whole nix uh is upgraded separately um so it doesn't affect the nix toolchain but everything else down to the gcc and glib and so on is basically a dependency then of of your uh python uh package i don't i'm not sure if that answers your question and so so um i believe the nix toolchain sort of if you build a binary will change change it our path or something in in binaries etc but if the so if they if that behavior slightly changes or something like that can you rebuild it is especially the version of the nix toolchain that you used in part of that hash just as uh as its dependencies are is that part of the dependency i'm not i'm not sure i understand the question so um i'll maybe let's yeah come to me and we can yeah okay any more questions just in your slides i missed the version definition when you listed the dependencies um where are they defined like so basically this is kind of like ubuntu style where the name is the version is tied to the name um and the name basically the version is is is not important um in nix at all because that's just basically amateur data um and when when the the nix packages repository changes you will just get the the new package uh and inside there of course there is the the version name but uh we don't do any like detection or something to uh about the versions but if you you showed this part next sample so i want some specific like gstreamer version and um it was part of the name um yeah so so when we have like when you have like gstreamer which you have version one and the the version before one then we have basically two packages uh and then you can pick which one you want to use um and you can always override the source and get another version uh if you want to change the upstream default for your project or server or whatever what is the difference between uh nix and docker that's so basically docker tries to isolate the environment from your system right uh and provides a very nice api on top of that whilst nix basically tries to solve the packaging problem and configuration problem so these are not like i think they'll those two things go together you can use nix as inside the docker if you want uh of course we're also using docker to solve uh the packaging problem and providing a huge binary blob but that's another discussion and in nix you don't have to do this uh you don't have this problem but it's still nice to have you know those lightweight containers and to experiment around that and so on that's very short answer cool well then one one question just one do you use nix in the web development because you showed a lot of stuff about the uh os dependencies and os package dependencies and even if python dependencies let's say uh right now it's for the back end but in our company we have a lot of struggle of packaging uh and deploying services with a lot of let's say uh for example javascript with bower and so on so how does nix apply to that i i know that you can uh it's declare your own sources and it can be javascript sources but uh do you have any for example javascript uh repository and how does it apply to the package so that for example python code finds uh those javascript libraries and so on because this is our crucial problem the debian for example can handle their own dependencies and it's fine you can do your own py peep repository and it's fine but gluing peep peep and bower for example that's a struggle and how does nix apply to that yeah yeah that's that's that's exactly where nix shines really good so we have a tool bower to nix i think it's called to generate the bower packages uh to generate nix packages from bower upstream in the nix packages repository and then you would go in your project do the same for all the extra stuff that you want and basically the nix knows all about about both sets of packages and you have those available and then you have all the python dependencies available uh and then uh you use the make derivation and nix shell to to develop on that and it will make it um it will expose those packages for you to use uh it's really hard to explain this without an example um but there are there is a blog post if you if you google around you will see uh uh how it's used for note packages for bower i don't think it's but it's the same thing i mean it's just the front end um and this is exactly where nix really shines when you have to combine two two stacks together