 attention please so welcome everybody I'm here again if I didn't see if you didn't see me before my name is Luca I'm gonna present you're gonna show you like ignition which is something that we mentioned already before I'm not the main developer ignition I'm just another guy coming for us if you want to thank somebody for all of these design whatever like go to this fine guy here he's the main developer in Montana right now I'm gonna give like it's a lot of content it's pretty packed as a presentation don't worry if it is something it's gonna be online so I'm coming from CoroS I'm mostly like an engineer I'm a developer used to do like Rust and Go I was I am in the free software ecosystem like since forever pretty much I work at Rated nowadays as you know after the acquisition I'm Italian I actually work in Berlin we still have the CoroS office there if you pass by say hi and before I was working in security this talk is pretty much like giving an introduction to ignition and then zooming into the details starting from like why we did that so learning from history in order to write something that fits better for the future so the first thing the really first thing is like what is the context of all of this the context is we ship a distribution and we have some programs we have some tasks to tackle and there are some projects that exist in order to solve these programs tackle these tasks and sometimes they are good they are like well done and they exactly fit in what you are looking for and sometimes they are kind of like everybody's using them they fit well in the like larger use case but for your specific things they are not exactly what you would like and our context is we're coming from shipping a distribution actually we we are still planning to ship a distribution let's say so back in the past which is called CoroS container Linux this is this distribution is a bit like different than the usual Linux distribution it is meant for what we would call an immutable infrastructure which is a place where you can quickly and like easily recycle machine so your servers are kettles you can throw them away and spin up new one in that environment you want to kind of like to kind of like a quick provisioning the OS itself it's something that you don't care about we ship it for you and you only provision it you apply your configuration to that to that extent the OS itself is immutable it means that like you cannot control it we ship slash user for you so for user partition which is read-only you cannot touch it at all and this user partition is as minimal as possible there are no interpreters inside so don't expect to run your own like Python program on top of the OS itself and there is no package manager so you cannot install other stuff but the nice things if you if you put yourself into these context is that you can quickly do out updates like you have two partition you use an active one you update the passive one you reboot into the passive one something goes wrong you go you roll back to the previous active one that's our context next problem is we made a distribution you boot it somewhere and you want to customize it you want to provision it the result for doing that which is called cloud unit it takes care of rendering like the early initialization of cloud instances and it has like a few let's say details implementation or design details the first one it's it's a collection of Python scripts and modules it takes some specific input in this case it's like some YAML user data and it is like a loose collection of services that runs at several points in in the boot so in the initial lifetime of a node but also like during farther cycles there is like an asterisk there is like as everything in engineering it depends it may run on the first boot only or it may run also later on there are few variables to play here with these two object in M like there are problems the first one is like the first set of problems like it's only relevant with these two elements together like first you need Python interpreters and we don't have you need Python libraries for cloud unit we don't plan to ship them cloud unit itself wants to control the package manager to install stuff from the US but there is no package manager and many other features that don't fit so well into this use case and basically this boils down to this distribution it's minimal it's read only and it's completely based on on system D then there are few more problems that are kind of like general to the design of cloud unit from our point of view of course the first one is it takes YAML as input which is pretty good as a human interface but as a machine interface so when you want to process it in a pipeline is not what you would expect it's configuration is a mix of declarative and imperative statements so you can declare something and then you can also say but I want to run these other commands arbitrary stuff the setup the provision itself rates with everything else so it's just like another service that is booting together with other stuff and it's running too late like if you want to configure the network but the network is being set up at the same time then you have like races and probably you're gonna run after the network is already set up and then again like it's hard to control like it can run a mood it can run like in multiple places multiple times or it may not it's quite hard to make a mental model of this and the kind of like the the drawback of this is that it can fail at any point in this process and you may or may not realize it like your node maybe left half configured but the service that you care about maybe they started but some other they didn't and you don't realize so it's like all you know it's pretty well as a configuration management for a mutable OS this is exactly what we are not doing and that's why we had to do something in order to first ship and the things that is we plainly ported it from Python to go so we've wrote something which is called chorus Claudine which is basically a subset of Claudine it takes the same cloud config but it's not written in Python in just like a go native library a go native application so it's self-contained it has like a few a subset a minimal subset of the cloud config so for example you cannot install packages of course and then we also did a bunch of stuff for like what we actually want to do which is like some sugar for it's in the flannel Docker whatever it was good like that that's what we ship for for a long time it was what initially everybody was using to provision container rooms problem is there were a lot of caveats like you had to exactly know the same things that you that you had to know for Claudine it which was not great and at the end like people were abusing it as like as a generous genetic split partner they were dropping their own logic as like imperative logic inside and that was like the context where we were starting when we started shipping something on top of that we actually had a few wish lists so it's like those are real problem but on top of that there are some things that if we had to start again we would do in a different way in order to get some nice results here and there so the first one of course is like the clarity of configuration we want to have like some documents that say you know the clarity way how the machine should be provisioned this document should be machine friendly that is should be easy to plug it in a pipeline when it is produced somewhere it is consumed somewhere else and in the middle there could be like other automated steps it should be an atomic provision so it should be either provisioning in a successful way or the node should not proceed into booting some services and the last step is this provisioning step is something that happens before the node is actually running and the services are actually running it's a precondition for that so it should not interfere with the rest of the boot and at the same time it should not be dependent of how the rest of the services are booted and are ordered and this is kind of like the whole setup of this discussion so what we did is like we started shipping something we took note of all the problems and details and then we went back to the whiteboard and started like drawing stuff and thinking about things the first one of course is like well we don't want to write C we need something that is compiled to native and at that time we're sticking to go pretty much everywhere which is a good choice simple to learn compiles to native binary the next one is well let's look at the boot of the boot process per se there are a few steps when you boot a machine and right now we were focusing like on the last part only like there are system these services and we can set up stuff in there and it's running together with everything else but actually before that there are a few steps there is one which is like very low level which is whatever your host fear where your machine is actually booting so like BIOS, UFI, some kind of net booting some other like more Arkane hypervisor things then there is a further step which is kind of like still related to the one before which is you need a bootloader and the bootloader is somehow dependent on the architecture and the way that you are booting it so in most cases is grab but there are a few cases where it's like pixie booting, ISO booting, some other things and then there is like one step in the middle which people are usually forgetting which is like there is an e-tron of FES so there is an early use space which looks exactly like a Linux user space but it's not the final Linux user space and the idea was like let's try to put something in here so that we can actually do the provisioning before booting into the real system the e-tron of FES is nice because compared to I don't know grab the bootloader or UFI stuff it's still a Linux system it's a minimal Linux environment it can still run a system D so you still have like multiple services and proper like supervision of them but at the same time like it's not the full system so it doesn't depend of like having all the details in the storage or the network or all the other application services being run and you don't care about them and at the same time you can initialize them here so you can provision you can configure them which means that more or less putting these ideas all together we are now drawing on our whiteboard something that looks like this let's focus on this intermediate step where we have an early use space let's make it like a similar as the final user space as possible so let's use system D and service units everywhere and let's stick a component here instead of clouding it in here so that provisioning happens here and if it successful then we continue booting and if not this is a real barrier so the node doesn't move the last idea is while we need a machine interface possibly something where we have like a schema to define it and that is JSON schema there are the discussion about like data interchange format is like unless you can pick whatever you want I was not there at the time when we pick JSON but my point of view is it's easy to consume and to produce from pretty much like any language that you can imagine it is actually a machine friendly there are a few gotchas that you learn after a bit but it's like in general it's way more straightforward than trying to process like YAML in a programmatic way but it's still human readable so it's like it's not binary it's not something really opaque and on top of that like it's nice because you have schema but it's not in-band you can like provide to producer and consumer an out-of-band schema and you can auto generate stuff based on that and those are pretty much like all the components that we put together when we were thinking how to do it and that's the idea that's the whole idea of ignition it's not complex there is nothing magic in it it's pretty much like put these together and then implement it and that's and now it's like we are just zooming into the implementation like the most interesting part was so far like what we covered already from here on it's pretty much is like implementation details a general overview is what we what we just saw so it's it's a first boot only also configuration tool it doesn't run on subsequent boot the user data is fully declarative so you don't take it's not like a runner for some script per se it just like take json file j sorry json document with some specific content and it renders it into the into the file system it runs in in it from FS which means that if you want to repartition your root disk or if you want to reformat your root file system you can because you're running in it from FS so it's still like live from memory and then you can boot into the real the real root file system it takes type json as input which is like important because if you want to process it automatically you want a definition somewhere and then there are a few like nice things that is given that is still like a full provisioning system it can actually like reach out to some other services like if you are on a cloud on a cloud provider you can reach to I don't know your bucket system with authentication and stuff and that's the URL if you want to check it yourself that's a very high level overview a smaller like a lower level overview is that ignition is actually comprised of several stages the the first one is not ignition itself is external there is something which is assistant degenerator plus some other logic which is actually able to detect whether it is the first boot or not and by default if it doesn't detect it is the first boot then it doesn't run then there is kind of like a common part which is not a stage per se which is like fetching this configuration I will show you later how to fetch this configuration and where it is stored but after this configuration gets some out into your node into the nitrame FS it goes through three stages in order right now a disk stage which is let's partition and format all the all the file system that you need if you want to reuse some file system some disk you can just like declare them and make sure that they actually match whatever you had pre provision and what the configuration expect then there is a file stage which is that's right some comment some some content sorry into these five system and then there is a quench stage which is this like magic atomic provisioning which basically means if at this point this service didn't fail so then the provisioning was successful if the service failed then we don't continue booting we stop here and we go into an emergency mode in nitrame FS which means that if you if you get to here you are very good you're not just booted and bootstrapping a cluster or doing whatever you're asking for that's like you know probably in a clearer way if you remove these mouse courses in the middle this description like stages and stuff and this is what I just like described you by words and this is actually like those are system these service unit and that in those are the real common for that one step back we mentioned before you need to get this input this user data from somewhere and that's the part this is something that we are mentioning a talk before like the images the instances themselves they know they can introspect on which cloud provider on which platform they are running so they can ignition can provide like specific fetch a configuration fetches platform provider for those right now ignition sports the usual suspect like all the Azure AWS we will cloud digital ocean blah blah blah it also supports like some bare metal or bare metal like environments like usual pixie booting or trivial FTP there are in some cases there are like back channel to the hypervisor like you know for example expose some back channel where from the from the gas machine you can ask the OS machine to provide you some data and this data is the user data and then at the end of the day like you when you are provisioning your node you can specify whether to use the whatever is the platform specific fetch or something different like I can be on AWS but I want to fetch some remote like my own HTTPS point and you can pretty much like mix and match we are almost getting to the end last thing is we are taking Jason here we are taking Jason here because again it's a machine interface you don't have to write it by hand but you can auto generate stuff that's actually what ignition does internally internally we've write the JSON schema and then we auto generate the code for that for ignition this schema is sorry the schema that we define so far is pretty much like whatever we as operator would expect from a provisioning system which means that you can provide a single configuration but then this single configuration can reference other stuff like you can reference another document to replace it or you can reference like farther snipset in order to chain them all together if you want to have some kind of like templating system which is useful for like dynamic generation so a node can ask like dynamically what will be my user data whenever I boot you don't know in advance but there is like a dynamic service providing it and it's useful for like stopping like there's a stub here go fetch it from some other I street back and it is version which means that like we don't break it when we break it we change the major some version stuff and we try to kind of like fast forward all configuration to new configuration without breaking compatibility that's the thing like we support out updates so you can start from like another version and we try to bring it forward of course like when there is a major compatibility break then like something is not supported we may not be able to do that again go check it there are a few examples and I show some in the presentation before final slide I think yes friends like these again I'm stressing these don't write these Jason configuration by and this is meant to be a machine interface and that means that we as the solution developer we have to provide you some tools but you as like a community you can write your own tools this is actually what happens like we have a tool which is like very basic again we are like we're startups we don't have like people located to project with cobalt things together whenever we need them the first example is the CT it's a configuration transpiler from Yamal to Jason it just adds a bit more logic into the final Jason but here's a user you've right then other people wrote provider for us like there is a Terraform provider if you are into Terraform in general you just define your TF configuration and then Terraform compiles down to Jason there are more advanced stuff like you can dynamically generate these and plug it together into a pixie environment so that's what matchbox does so that the pixie environment is a bit is a is able to provide dynamic configuration to DOS whenever they ask they are asking for it and then the last example is something that is currently being worked on which is you can install a full cluster this way which is kind of like the final conclusion of all of these you can bootstrap a cluster and you don't have to care about single machine configuration provisioning and installing single packages in these machines because the provisioning is done in a ton in an atomic way in any time fast and when the machine boots it only takes care of bringing up all the higher level cluster orchestrator and not like configuring the machine stuff you don't have to SSH into that which means that you can basically like start from scratch and bootstrap a full cluster in a declarative way and that's it the only things that I wanted to mention is like thanks to an exo-league of our of our Dalton which made some of the graphics for that and it's kind of like an inspiration for both the software and this presentation that's it for my side questions so the question is like I'll update the system safely so the answer to that is completely unrelated to this talk which is you as the infrastructure owner you have two approaches one is like your infrastructure is immutable which means that it never changes and whenever you want to change your configuration you do this like quick recycling of a new node and throwing away the old node so that you know that like one old one node is booting with a configuration and for the whole lifetime it will always have the same configuration that's one strategy yet but it's like that's possible in some cases in some other cases not there are other strategies which is like you provision the node in this way so you have all the nice things in here and then you have another component which is typically part of the cluster orchestration itself which takes care of like further further mutation of this configuration of the node that's okay like you have something that owns changes for the rest of the lifetime of it and it's auto-scope for how you first provision this node but then you have you may have a problem at that point which is like if you're not careful in what's going on you may have a skew between new nodes coming coming up with a ignition configuration and all nodes being updated in some other way that's that's what like the open-shift before the strength is trying to do which is like using the same ignition configuration for both the initial provisioning and having something else an operator in Kubernetes which takes kind of like updating the configuration whenever something change but in general like it depends where you are coming from where this is coming from or more on the side of like my infrastructure is immutable and we can recycle things and where typical like RAL or Fedora distribution are everything is mutable and you can mutate it at any point but then it's like it's harder to guarantee that everything is aligned I agree I see that I see that problem and they are kind of like different different talks and the problem of clouding is that it was merging the two together that was like configuration management and first boot provisioning so context not anymore I this the first three slides were kind of like historical context and those were like container Linux stuff so container Linux does it container Linux is still available but it's not something that we're pushing for them so it's not in anything Fedora know this is kind of like a generic upstream project you can plug it whatever you want but in Fedora world no so it is internally sorry the question is like which kind of storage technology do you support and are you provision that it is internally shelling out to a few utilities like SGDs kind of KFS or whatever and the thing is it can learn new stuff like you can do it can already do RAID in the future will probably may be able to do LVM things the problem is those utilities those things must be in the initramfs and the initramfs must be able to on subsequent boot to do the same kind of logic so like unlocking and putting everything together which is kind of like simpler in Fedora world it was way harder in container Linux world where the initramfs was immutable so we had to decide before before and what was going to be inside the initramfs this is something that we are actively like brainstorming how to do more things we already support something if you go into the schema documentation it says what but for example like it doesn't support encrypting the root of s because encrypting the root of s it's easy it's easy to unlock it when you are creating it but then you must have a full chain of things that are able to unlock it later and we are speaking about fully automated operation so not like asking the user for password for this that's clearly like simpler but it's not it's not in this code so the question is like how does this kind of provisioning reacts to changes in the external environment that are either hardware or software in your case and the and the answer is again like these two strategies you can say these changes don't happen like if I know that some problem for some whatever reason we just reprovision it from scratch with the new configuration which is kind of like again it depends how much control you have on these and the other answer is like well there are things that are always changing on the outside and you have to introspect anyway it's the case for example of like cloud metadata you cannot know them in advance and then in that case it's something else part of the US must be able to handle this kind of like mutability at runtime that's example for like chorus metadata it's something very simple and stupid but it's exactly that like there are things that are changing later and you cannot predict them when you're provisioning an old first and they may change later that's by design in that case like something else must take care of these mutable mutable stuff and there is no way that you can do it like from immutable provision at this from my point of view that's a no no answer but I know but in general but on bare on bare metal in bare metal is basically like a hard or bad because you are not in this like my servers are cattles like your servers are real hardware and you have a limited amount of them yes that's that's especially a bad case I know there was one last question so the question is like what about more advanced network configuration that you have to support there are two parts in this question one is network configuration in final OS and network configuration in the intramfs so it's like network configuration in final OS it's easier let's say because we have this in intramfs we configure whatever that also means like producing a configuration for your network manager which may not be network manager but your network manager and then when the system boots is basically like the full boot since the beginning already knows about the network configuration and that's the easy part let's say the hardest part is how do we configure the network in the intramfs so if you have something like special like I have three interfaces and only one of them as a gateway like an arcoded one and I have to use that in order to get the ignition configuration and that is the tricky case and that tricky case right now is handled by gluing together a few things including like kernel parameters and other stuff but that is again like it's a special yard environmental but it's trivially silly in cloud environment because cloud environment they don't have this problem like you know that you have the ACP you know that you have the first interface you know that you can reach the metalism point over that that's covers like 90% of the case and then there are the additional one which is like bare metal custom static configuration and those are like painful yes there are ways to do it and we my answer usually is your infrastructure should be an API as well so you shouldn't have a static configuration for that you should be able to spin a note and the note should be asking something else how to get this configuration and then in that case it works a bit better because we can now to discover that configuration apply it in a fast and proceed I guess is the last one that's all they take it out thank you very much