 Hi, good afternoon everyone. I see that this is probably my best attended session ever Let's first prove that I'm not going to be bullshitting here and let's Well, I could but so for those of you who know this This is the output that you actually get when you're booting from our chemistry so This laptop is actually running our chemistry trees at this very moment and it has been since January of this year I've told this to a bunch of people and they were like yeah, you should tell people about how you did this because it's not trivial and I can tell you even after this it will still be tricky and I can give some pointers, but It's still a lot of custom stuff, but we'll get to that So first note, I've heard that marketing doesn't like it when you call it atomic because it's not for the red atomic host So officially, it's I can only use the name our pmo s3 because that's what I based on And I'm not using the official atomic place. That's just to keep the marketing people happy Yeah, like it the internal name if it's displayed here Not so it is Is paid right, please That's the name I gave it I'm not very imaginative with names in case you wonder So a Little bit of the background limitations of why you would want to use it and why not I would set it up which is going to be the most tricky part And then some of my experience up last month So why would you use it well? The entire root file system is read-only and the entire thing is signed Which means that even if a single thing has changed Underneath the read-only parts You will know you will see that because if you notice the output it actually says Good signature from my signing key to the entire tree is signed and I'm currently running raw hide on this and I updated this morning Which I dare to because if it failed which it did you can easily refer to the previous one like they finally fixed the case where Dr. Nessie limits don't communicate and now they broke wireless Yeah wireless, so oh nice. Yeah So also if you want to know more about row height, there's a talk about that just across this wall And the last reason for me is it's fun, I guess it's a challenge Why not I like challenging things like maintaining open stack stuff like that Okay Limitations So there's no workstation trees available as of yet. I know that Dave is working on that But that means that you will most likely need to build your own trees Because you will want custom packages and you cannot install that currently I Heard they're working on layering, but as far as I know, that's not working fully yet Another thing is like Which I really noticed is that atomic or arping West free. It's out there but it has a lot of bugs and the fixes come slowly if If at all like there's a few bugs that I've opened which they're saying like that's Because another dependency doesn't want to use a library they won't use it like for example combs groups are not supported in tree files so you will need to Specify every single package you want in your tree by hand Because it doesn't support combs trees because lip comps won't use g lipsy Ask Colin Walters for more Well, I will come to that later because I've got a workaround Right, but they don't want to do that because they're not in the business of combs groups There's a whole lot of people that say like yeah, it's not my it's theirs So since you can't add packages You would like to use either docker or virtual machines or perhaps flat pack when I can finally get that working to run actual applications Because I do want to get development done Yeah, I do because I'm a software engineer actually that's what I'm paid to do So I've actually decided to go mostly the docker route and That means you will need to lead to learn a bunch of that To get started and make it work That's just things you should take into account before Going this route because it in the end it might be good. It will cost you a lot of pain to set up So setting it up as I said you will want to start with creating a custom tree Then you will deploy the tree provision and tests and everything and then Finally, hopefully do some work if you ever get there So for creating a tree right from the start you get a bunch of decisions to make like which packages do you want Some people will want to know I'd have I3 for example some people will want Emacs I use Vim and No, I'm not getting into any wires here. There is no war. Okay You will also need to choose which OS version you want to use because not everyone want to use raw hide How you're going to deliver it because You would want to compose it somewhere You could theoretically do that on your machine, but I prefer not to So I have a separate Compose machine which just runs a crumb job to every I think it's five minutes currently Try to compose a new tree and then if it managed to compose it push it to s3 Where I can then download it from because it's Cheap easy storage that has an embedded web server So you would want to decide on these things before moving forward because they kind of decided the rest of the files So then you will create you should create an initial tree file. There is some documentation on it But I will also give you the link to my own tree files because all of my setup is public and You can just base it off on that You will probably want some scripts because you don't want to rerun RPM tree compose and push it every time manually and Generate a signing key which you will need to keep secure for which I would suggest to go back in time and visit Nathaniel's help As I said RPM OS 3 files do not support comps groups But what I've got a python script that just ingests a comp a tree file resolves the comps groups and Spits a complete a expanded tree file That's also my repository. So you can just grab it from there After that you run our plume s3 compose to actually compose the tree file into a tree and it Hope that it works because I Will promise you the first few times you will hit upon a few this package could not be found in the repo and stuff like that There's also a few packages, which you cannot install like Dracood Rescue because it will break the entire compose There's a bug over for that, but They're not willing to fix it last I've heard They publish it install it which we'll come to next Test it and Let her rinse repeat you will likely go through this cycle a lot of times I've lost count. I think that my very first definition came to about a hundred trees before I started using it for daily use It's getting better, but Well I Was using for 23 at the time to start with so That wasn't the main issue the main issue is like You will find out that there are packages which should be pulled in but aren't like for example IP tables is no longer pulled in if you install Docker you will need a hard manual require IP tables because That bug is still open in bugzilla somewhere Yes, like now did the number of hundred or something that is like when it was ready for me for daily use so When it included Docker and virtualization and everything that I wanted for daily use To get a first tree booting is a lot less hard. It's just a lot of cycles to find which packages you want Because Like Sorry say again Yes, you could do that But then you get a whole lot of packages which default installed drowson which you might not want But yes, you could totally start with that and then Remove from that I Am I do when I point them and it's not Docker Because I'm not sure how well the Docker people people like it when you touch it their package Yeah, yeah, it was a requires colon IP tables that they were missing But yeah, I've hit a lot of other bugs that where they are missing dependencies I've I think I found most Yeah, I saw that you finally close the bug No, you reassigned it and now it got fixed I think So for deploying a tree My the two methods I prefer to use are a net install image where you just Put the Kickstarter on the file itself or what I do at home is a pixie boot So the fully automated regular image re-image is something that I actually do because RQM Westry has some Garbage collection issues where it doesn't actually clean up everything from previous trees So what I do on my laptop I think every month is just hook it up to the network booted from pixie and the entire OS will get re-imaged But all of my data will still be there because that's on a separate Sorry So That is this month that is used in kick starts to actually kick off a OS tree deployment There my full kick starts also in the repose so you can also look at that But kicks are slightly personal because of Sorry Partitions, sorry So what I've got is I have a Volume group which has a of 250 gigs 30 gig So it's part of slash boot slash evi 30 gig home boot 50 or 80 gig home and rest is left free for docker So the docker story setup just sets that up as the use storage Right except that your slice should be big enough for at least two trees The trees are one tree in the differences their diffs, okay, mostly but as I said, they have some issues with the Garbage collection so it will grow slowly over time. I Hope they will fix that and do time No diff is within the trees The Actually, no, I was saying it's correct first time it will store two full trees Because the entire tree is signed the entire thing starts because Yeah, yeah, I like it I Think that things that are the same will not be duplicated because it's a git like ish method they use Like they're using the same object system that get is also using The tree objects and the Pointer that tells it which objects are for that tree So where's that? Yes, I Think you do like the metadata should support it I'm not sure the tooling that's yet, but the tooling does support signing separately. So I guess Because that's that stored in the metadata and that's the very first file it downloads And that contains the reference to the objects so it's Top-down it signs them. I think it signs the metadata and then that points all of the objects what I Believe they were doing But don't pin me on that detail because that's an implementation detail and not a hundred percent sure Yes, so part one you couldn't Because read only and part two. Yes, that would break Because you would need to modify the underlying objects So for myself because I reprevisioned automatically I built a tool secure provisioning which just Automatically decrypts passwords Upon reinstalling So that I don't actually have to be there when it's provisioning You could even do that for larger scales setups and If you want to you can ask me more about the details and it also does all public everything of this is public Sorry No, well the the file containing the passwords is kind of if you know the server stack of my machine you can get the file The passwords themselves are not They're encrypted with the TPM in here and a you be key No because you be key for certificates doesn't use the press to Like it's the x5-9 sir smart card that it's using So My experience is that it's I Like it personally As I said, it's not for faint of heart. It will take Quite a while for you to get used to if you decide to go to route It will take quite a while to get set up until we get a tree from Somewhere else like David and It is quite nice that you can roll back like as I said I upgraded this morning It didn't work. You just roll back and you're back to a known good system Which is also ideal for example in a managed environment for companies you might want to use this because employees can't meddle with the local system all too much and If an update fails You can just roll back to previous one So that should lessen support cost probably Yes a question. Have you tried using ns poem to put at the same time Another point of the tree or the same one where you're sorry say again, you know ns poem the System this stuff to run like another container. Oh, right, right? Do you think it will be possible or have you tried using that to put another instance? I have not tried that that would be interesting thing to do Actually Yeah, that That sounds like a very interesting idea. Thank you Yeah Please come back in tomorrow or something Sorry I'm trying to figure out 200 desktops, right? Right all identically installed all kick-started Every machine will pull updates and install them from a local repository. That's not right interconnected, right? so doesn't pull for food or it pulls from what I stage into the machines and separate comps file to To specify, you know groups of local packages Right trying to think of how that works into this means where I Compose trees and those are layers on the tree. No, you don't do they layer? They're working on that. Okay, so eventually. Yeah, the idea is base tree and layers on top of those for different I think Layers has been able to different layer of packages you saw locally. Oh, so which is sort of a more developer thing or more like Deploying rotation you have multiple trees that didn't score post or that share packages. So it's the You probably compose a tree for each other. Okay. Yeah Okay, so I mean, how many of your things are you getting via the custom tree and how many of your applications are running this like desktop containers? Sorry, are you running anything in containers or you just running the tree? I run my development stuff inside Docker The only thing I run on the host system itself is SSH and Gits because my keys are not shared to the containers, which is explicit I Might change that at some point, but at this point everything else basically happens inside a container Yes That is installed in a Docker which runs as HD with exporting But this time really Important to the concept of this, right? I mean, I have no interest in running Docker containers on my user. Yeah That's just right now at least and so that's just for me because I still need to So yeah, the plan is as far as nation is that Right As soon as flatback work works out switch that I have not yet, I think somewhere not completely Right, that's the layered things Okay, you like to you modify Okay Right, it sounds interesting. I'll sure thank you for that Yeah, are you gonna talk more about specific? No, because most of them were either specific to packages like SSH Stuff or the trees that just people on top then I do have a specific questions If you are if you are successfully running most of your desktop apps and containers What what about the existing adora atomic? RPMOS tree Not sufficient for you earlier. We're talking for example of putting them or emacs in the tree Well, you're already running GNOME and the problem is GNOME or any kind of desktop environment like atomic hosts only contains bash and SSH and That like the main reason I'm not using that tree is just for Some graph going to face. Okay, so you need you do have the baby base Desktop interfaces in the tree. Yeah, and then you're running apps that point back. Yes, because You still need to run X server and stuff Sorry I tend to and unless I know that there's a specific bug that's blocking me Then I do it as soon as that bug gets fixed No, because it uses two trees and it always keeps the one that's currently booted So if you would update it always keeps the one that's currently booted So if you update once it will replace the non-active one if you update again, it will replace that one again It won't touch the active tree like I've done that a few times, but They are quite smart with that Well, I can show you It just settles like I keep trying it every now and then and then it just doesn't work Yeah, and this is a tree from 38 718 So Then you're missing a dependency in your package, here you go with the bug We're fighting back on during the talk Yes Right This was the latest tree that got composed and reason for that is that the current Arping Western is currently broken And I file it or I send a message to call and to ask like how do I debug this because the The failure is very annoying composing or committing at our no-search file or directory What father are you looking for? I have no idea because during Directly after this is doing so much in the code that you have no clue where is fail But that's why that's the latest tree for the moment. Let's see Well, this will only stage it like it will download it Okay, I guess S3 as an issue down. Oh, they kept the portal Right. Oh well I hate kept the portals, but So Etsy is outside Etsy is not covered under this Yeah Yeah, so what it does is it the Etsy is actually composed in the trees So it gets deployed and when you get a new tree it does a three-way merge between the previous tree the current tree and your changes How much what Quite a bit and you will see that during all of the install or during the Composers there's a lot of Postscripts that Provide warnings, and I've had I think just two packages that actually broke the rest is just all spitting warnings So yeah, so I've I've got two of them fixed and When I finally find out what the other messages are I'll probably get those fixed to but It just that's hidden Yeah, because it well it's in the anaconda logs somewhere, but that's by default hidden So one of the main things that you always see is things that talk to Etsy Linux because During the compose there's no as Linux enabled or there's a special kind of context, I think where it doesn't actually Where it can't talk to a lot of things so a bunch of resources these are URLs with my Tree files, that's the topmost one and that also contains my kickstart and build scripts Not my private key. I Think it does contain my public not my private key second one is the provisioning tool The next one is my set of Docker containers like the Docker files That I build everything from and then my website and how you can get in touch with me Pretty much, yeah Yep or find my pager email address good luck with that Yeah, it's not too hard to find but I hope not too many people Are there any further questions Yes, it is possible you can just run Arping OS 3 remote ad pool and then Deploy that tree and that will show up in grub so that should also work as a Quick way to get working true But for my Since that's explicitly not what I want. That's not what I did but yeah Right fair enough as A part of QE. I mean how you compose those trees to find out box very likely. Yes Yeah It's going to what the criteria are that would cause the problem you're saying but the what what? What would they definitely do if we are voting? Trees is nightly in the red as we can test those trees and that allows us to actually Test the same code that's going out to users. Yeah, you know, you can do a lot of tests right now You're testing something that's different from reading with you know with a DNF update Right those three then you actually test the exact tree that So that's for good doing three I Sorry using OS 3 on your It depends on what kind of For people who very often use yum installed with some graphical tool I'm not sure if you're mostly using text-based tools or like terminal or other Not changing your package chat all that often I think it's a pretty good way to make sure that you always have a stable system Right, but this is exporting over local You notice it somewhat but not too much Right, yeah, that's just how you decide to do that you could Just sure That depends on the project basically for some projects. I just have a generic Python and a generic I've got no jazz. I've got a couple of generic ones for back before stuff I don't touch all that often and then I just do yum install inside there every time again But for example for Ypsilon, I've got a static tree, which is exactly what we or I've got two sets of trees one One is just like what is on our website like this is what you need for development and the other tree is like This is what we install in production exactly So this is what we ship and this should work if this doesn't we are having an issue I'm using the second one mostly for testing releases and pushes first one for actual development Yes, and I actually even maintain up to the foot or ever base image, so yeah Is that wrong I think that's in 22 So yeah, it's definitely 23 and 24 It might be Yeah, yeah, I assume it installs faster too, right Yeah, well What it does is just it downloads all of the files and then run our chemistry deploy It downloads all well it gets the tree files from wherever you put the tree files I'm thinking This is exactly Over-recomplicated Maybe if you had a hundred thousand desktops, but I'll like 200 so you know As you can see for those You can look right now But after the initial one updates it just downloaded the diffs and that's so That should be they only have to completely reinstall a couple times for four doors Yeah I'm pulling it down from s3 in West Virginia, I think this region I It's it throw me in For deploying it's a lot quicker than normal anaconda, I think it takes about five minutes for a full deploy With a reasonably sized tree Of course, just wondering if it helps me Yeah, I mean obviously I know Right, but also I mean if I go from and oh, I mean you can do cross OS version updates with this, right? So you can yeah, so if I compose the food or 24 tree and push it and Yeah, yeah, I went from Years ago he actually was prospering between the door 20 and Ralph Yeah, I upgraded from Word 23 to raw height Without reinstalling because I'm trying to figure out does it make my deployment easier does it make the annoying updates easier? We're currently Hope the machine comes up and when it comes up it will install itself onto a new OS So I don't have to do it Easier if it was just a well, okay update yourself. Okay. Yeah, that's what you basically get here And then if I in that that's really screwed then it's like, okay, we move back. Yeah, yeah but also because I With a normal cluster of machines, you might get one that Might not get updated all the packages before someone hits reboot or stuff like that Whereas here, you know that every system has the exact same. Yes disc So you could just have one machine on your desk Test the new tree on there and if it works Deployed everywhere because that's exactly what I really want Accept if there's hardware problems. Well, of course, of course, but other than that if it works in one It should work on the others, too Well Yeah, any further questions