 So, hello everyone, welcome to the second talk of the afternoon session. I would like you to present Dan Langel, which will talk about why I prefer thick gels over thin gels. So, let's go ahead. Thank you. So my name is Dan Langel. I've been involved with FreeBSC since about 1998 when I got ADSL at home and someone said you're going to need a firewall. There's this 486 and these FreeBSC 2.2.8 CDs and you'll be off and running. Look where it got me. So I had no idea that thick gels over thin gels was even an issue until this year. I never thought I'd be talking about it. I first started using gels in about 2004 and I had to look that up. It's been 15 years. I had to look it up on an old diary entry and I encourage you to keep a blog or a diary of any type just to the everyday stuff you're doing because someday you'll be standing up here giving a talk and having to answer a question just like that. So gels are wonderful tools without a doubt. I love them. Absolutely. I use them everywhere. I use them on just about every host except this old I386 host, which isn't really well suited. I like gels because it'll let me do stuff which would otherwise require additional hardware and by that I mean I like to run NGINX and Apache on the same host and while yes you could do that on the same host for different things it's the additional overhead to get there doesn't overweigh the ease and beauty of gels. So I always like to start every talk with a disclaimer because just because I said that doesn't mean you should do it. Don't do it especially because I'm doing it. Do it doing it because it makes sense to you. It's right for me now but I may talk to you after the talk and you'll give me an idea and I'll be off on that tangent and what I'm doing now is of no sense to me at all. Instead use this talk as a catalyst for thinking about what you want to do with your own systems because what I'm doing now may be totally different in six months. It always helps to start with two basic points of terminology when you're talking about gels. The host is just basically a free BSD install and that can be anything. It can be Raspberry Pi, it can be a 48 core system in a data center or somewhere, it can be your laptop. They can all run gels. A gel is basically a VM on a host. Don't get technical with me. I know it's not a VM but we're going to talk about that on the next slide. So what are gels? According to Wikipedia, which is the true source of all knowledge, I know, gels are not true virtualization but they're really good compromise and the beauty of it is they're very lightweight and they impose very little overhead in the system and the other thing I like about gels is that when you're in a jail you don't know you're in a jail. You can treat it just like a host. Yes, if you have specialized knowledge you can look and see that, yes, I'm running in a jail where you can just run PS and see the J in that one column you know you're in a jail but apart from that it's just like being on a full blown piece of hardware. I like it because they're secure. You can't get out of a jail very easily. I don't know of any, anyone know of any current exploits that let you get out of a jail? Yeah, they're all applied. Okay, good. Jails are very simple and they're based on CH route and basically all that CH route is does is instead of you being at slash your user local jails slash damn and you're in there and everything you need is in there. Gails came around in 4.0 which is almost 20 years ago. PHK, we owe him a lot, not just for jails. The simple explanation like I mentioned before is that jails are just a CH route and I like to use them to keep things apart from each other. I like to have my web server in one jail, I like to have my database server in another jail and I like to have my commit processing in another jail just so that I can separate them all out, upgrade them separately without worrying oh I installed that and now that's broken that. I keep them all separate. Jails can't see into other jails so if you're in this jail you can't see what's in the jail above you or below you or beside you or anything like that. It's just only your own information. You can look at your own jails. You can look at your child jails. Yes, if you're running jails you can see the jails that you're running but if you're in a jail you can't see into the other jails running at the same level as you. Yeah, so no you can't just see in it. We talk about running jails and jails later. So I also like to use a jail for trying out new stuff rather than put it into my existing, an existing host. I put it into a new jail and then just toss the jail away when I'm done with it. Since we're talking about why I'm going from thin to thick jails I'm going to talk about what is a thick jail. That's your traditional jail. If you go back into the manual and you look at the old way of installing where you do a CD slash user source and then make install minus D some path that'll get you a traditional jail. It's basically the complete OS installed in some CH route. I manage that jail exactly like I would in a other main host. I don't treat it any differently. It's important to note that a thick jail is not a clone of something else. We'll talk about clones later and why I don't think you should be using clones for long-lived jails either. A ZFS copy of an existing jail is not a thick jail. Sorry, a ZFS copy of a jail is still a thick jail. Even if it started off thin, it winds up being thick after you copy it. But a clone, no. Don't use clones. What's a thin jail? Very easily. It's not a jail that's thick. Easy jail. Who uses easy jail? I use easy jail. I like it. It's very, very much, it's very much fun. So, easy jail is a concept of the base jail. And when I first saw this, I forget what year it is. It'll come up. I like the fact that it saved all this base OS. I didn't have to copy that into every single jail. And it turns out that one desire turns out to be the thing that got me to change to other jails. So, you can have a thin jail created via a clone. And the reason it's considered a thin jail is it's not a thick jail. It has a copy of something from somewhere else. If you have a jail that's designed to run exactly one application, it's not a thick jail. In other words, if you've trimmed out all the files that you don't need, and you only have one process running in the jail, that's a thin jail. There's a lot of jail managers, but I've only used two. IO cage and easy jail. They're both good. I've used them both. Both at the same time on different machines and both at the same time on one machine. And the reason I used them at the same time on the one machine was convert from easy jail to IO cage. And they do coexist rather nicely. So, easy jail. First released in 2005, but I started using it in 2008. It uses a thin jail, as we talked earlier about it before. And it does this thin jail by using a base jail, which is then shared by all the other jails. And I'll show you what the base jail looks like before. The beauty of this is that if I had to update a host, I would go in and I would free BSE, update the host, and then do easy jail, update minus U, capital U, the base jail, and reboot, and then I'm done. Everything's done, but not quite, as I'll point out later. Then I found IO cage. And it was first in the free BSE port between 2014, but I didn't start using it until 2015. It uses thick jails, but it can also use clones. But I only use thick jails when I'm using IO cage. It was originally written as a shell script, which I really liked. It was simple. You could read it. I could get through it all at once. Now it's written in Python, which isn't a bad thing, but I'm still using it. Now we get to the main point. Why did I convert? And I got to read this slowly, because there's a lot of different reasons. My main reason is that my jails were outdated. So I would go in and I would update the base jail and then ignore the jail itself. And usually things went OK, but every once in a while there'd be an RC.d script, which wouldn't work. Do you remember when I think it was user loc, etsy new syslog, then brought in user local etsy new syslog, and you could leave the base file alone and put all your other syslog files over here. The same applied for both syslog and new syslog. Well, I was missing those two extra include lines because I'd never done the merge master. So that's one example. There are others, and I can't give them to you now because I really don't remember them. I just remember encountering more than once a problem with not having run merge master. So it was just basically upgrade the base jail, and then I had all the other jails upgraded. But I couldn't easily mix jail versions. When I upgraded my base jail, I really had to upgrade due to the merge master on all of them right away. Otherwise, when I rebooted, applications which were looking for certain system libraries, those libraries were no longer there, and they just died. So I had to go and do a package upgrade of everything in that jail. That became a pain in the ass. Now, the other slight issue is disk space. It's not that I wanted to save disk space. It's just that there is no reason to save disk space anymore. Back when I first started using EasyJail, it was 2008. We were using FreeBSD7. And my hard drive at the time was a 500 gig hard drive. I know this because I look back in my diary. That's why you should keep a diary. But now I'm using FreeBSD12, and the same system has 90 terabytes. And an empty jail is only 2 gig. I have to run an awful lot of jails to get significant savings on a 2 gig per jail instance. So it's just not worth it to me to use base jails when there are the other issues associated with base jails. So why should you convert? First of all, read Michael Lucas' book FreeBSD Mastery Jails. That'll convince you to change to use jails.conf. It did me. I read that book on the train from Bergen to here and use it to update my slides. And you'll see at the end what my future plans are. So you should convert because basically you don't want to use clones for short-lived jails. If all you're doing is you're firing up a jail and putting something in it and having it sit there for a while, and then it goes away, and you never upgrade it, clones are OK. But don't use clones if you're going to do any sort of upgrade on that. Certainly not an 11 point x to 12 point x upgrade, because basically you're going to double the size of that jail because you're going to have the original files from the clone. You're going to have the new files from the 12 point x version. And if you keep doing that repeatedly, you're going to have this long line chain of files that is not good. I like running FreeBSD Update from inside the jail because I can treat it just like a host. Don't do that if you're running a jail manager because jail managers often keep metadata outside the host itself that you can't see from within in the jail. So you want to run whatever tool comes with your jail manager to update your jails if you're using a jail manager. In disk base, you want to say that two gig that I previously said isn't very significant over a lot of jails. But I just didn't like using base jails anymore, despite the fact that it's a very small savings. So the script. I used it to convert from easy jail to IOCage probably two and a half, three months ago. It's on GitHub. It's a very short script. It's easily followed because it's just basically it's a fine followed by an R-sync. What it'll do is it'll go in and examine the base jail and said, oh, OK, this is what's in the base jail. This is not what I'm going to copy over from your sample jail into your new jail, because all of that is already going to be there. And the way it does that is one of the parameters you supplied to the jail is a location of the sample base jail. And it says, OK, that's not going to be copied over. It is specific to easy base jails, but you can change it to use whatever other type of thin jails you are using. This is a typical base jail. The thing to have a look at is the fact that all of these main directories that you see in just about every install are just sim links to the base jail, and this base jail itself is a null FS mount of that into your jail. This part may be hard to follow, but if you have an easy jail system, it becomes very obvious. Now, the steps to running this jail conversion script are ridiculously simple, because I was sitting there watching TV and running these scripts and waiting for five or 10 minutes and then doing another bit. And I binge watched a whole bunch of TV series while I was doing this, because I had about 70 jails on about four different hosts that I wanted to convert over. Basically, the first thing you do is you create your new jail, and then you start setting the configuration for this jail based on what the existing jail is. Then I like to take a snapshot of the new jail before I converted it over, because I had a good working point now. And if I messed up my copy over, I didn't have to do the first two steps over again. Then what I did is I stopped the old jail, because I'm going to be copying over, and I don't want any of the data changing while I'm copying it over. And these three parameters are this is my sample base jail that the script uses to figure out what not to copy over. This is my old jail, which is where easy jail used to put stuff, and this is the new jail, which is created by that step there. This would run in about 10 minutes for, say, a mail jail that just relayed mail. But for, say, a database server that had several terabytes of data, it might take half an hour, an hour to run. But basically, I just ran it. The most confusing, the most difficult part was getting all of these things set over. But once you've done half a dozen, the next 20 become easy. And then at the end, I would just start up the new jail, and 80% of the time, it just ran. There is nothing to fix unless I mess something up in the configuration. Once everything was green in my monitoring system, I went in, and I turned off the old easy jail. I turned on the new easy jail, sorry, the new IOCage jail, and that was it. I was done. I moved on to the next jail. At some point, I paused the TV so I could concentrate on what I was doing. But for the most part, I just watched TV. Oh, wrong button. So we already did the post-conversion bit. Sorry. Yes, this is a different program now. This is the second slide deck. So this is part of the talk where I'm going to convince you to use thick jails. You get to pick and choose which jails are updated. You can update your host, put it on FreeBSD12, have all your jails still running on FreeBSD11, then decide when to update them. You can update them over a week, over a month. You don't have to do it all on one day. That's the main reason why I went to thick jails. And you want to run jails that are different versions. I want to have a FreeBSD9.x system on there because I have this old software that can't be upgraded. It's not advised. And friends don't let friends run clones. It'll go for a long time and work, but then when something gets complex and you decide you want to delete it, you may wind up in a trouble. This is all just anecdotal evidence, but don't do it. Now I'm going to convince you that you really want to run thin. You don't want to run thick. Thin saves a lot of space. If you're running 2,000 jails, that's 2,000 times 2 gig, which is 2 terabyte? 4 terabyte? I'm not good on math. So if all your jails are the same and you only deploy them once and you never upgrade them, then thin is perfect for what you want to do because then you just upgrade the base jail, all the other jails are updated. And then you're done. You can always update your IC.D later because no one really needs that right away. You can do your merge master later. But if you never upgrade your jails beyond patch levels, you're OK. If all you're doing is FreeBSC Update Fetch install to get the new patch level, thin jails are just fine. You don't have to worry about this side of it. But if you're upgrading between versions, that's when your user land scripts get out of date. I am a great candidate for template jails. What's a template jail? It's where you get a file system all set up, a jail, exactly like you want with the package you want, all the things you need. And then have it sit there, read only. And then what you do is you do a clone or a copy of that to the new jail. Because all my jails have a common set of packages, Sudo, Anvil, Bash, Joe. Who uses Joe? Who knows what? Joe's own editor. It's my favorite editor. It's like Wordstar, sort of a Wordstar interface. Who uses Anvil? OK, I'm not surprised because that's my own software. I'm not surprised nobody else uses it. Etsyresolve.conf, it's the same on every host, the same package.conf because everything installs from the same host. Yet I don't use template jails because I like using Ansible to update my jails. Basically, I do the jail creation, and then I create some super users in that jail. And Ansible then configures the jail how I need it. I don't really use Ansible for upgrading jails. I use it for configuring a new jail. Here are some jail monitoring tips that I didn't know about until I stumbled across them. There's this script that is installed by that port. And what this port does is it actually audits your base OS. And this is work done by Mark Felder related to VU XML. And he back ported all the vans for base. And any new vans in base are actually put in there. So you actually audit your base OS or your jails to see whether or not they've been updated. This will alert you if they've not been updated and a security vulnerability has been announced. Similarly, there's this script, which is just for packages, which is already on your host if you're already using packages. I've got some sample code in there on how to create a Nagios alert using both of these. And I use them all the time. I get notified about the recent XPAT2 voln and there is a curl voln. But this is how I get updated. Nagios tells me. This is how you implement those scripts. Well, this is how I implement those scripts on my system. One thing to take note is that I just say, do this on all jails. The downside to this is it will then run these scripts on any Pudrier jails that haven't been running at the time, which invariably fail the check because they don't have all the right scripts in them to do this. But I just ignore it. You could just list all your jails manually, but then I'd have to update this when I added new jails so I don't. This is where I convince you not to use jail managers. Sometimes the jail manager breaks because it's just software, isn't it? And people update it and things change and you've done something wrong and it messes up and fresh ports winds up offline for eight hours. And when it does mess up, your jails are offline, you can't get them online because your jail manager start command doesn't start your jail. So I first used jails without a jail manager and I'm pretty sure I can do it again because if I remember I started using jails in about 2004 and it wasn't until 2008 that I started using easy jails. So that's four years of using gel.conf. Well, it is there now. So the features in gel.conf now are very powerful and they also include default values. So your definition of a new jail can literally be jail name, curly brace, close curly brace and that's your new jail. That's all the configuration in it because all your predefined values have set it up to say that, okay, that's enough to define the jail. C3BSD mastery jails. This is where I convince you that you should always use a jail manager. And the reason why? The tasks managing a jail are tedious and time consuming. They're repetitive. ZFS create, mount this, install that, do this. So jail manager would do the same thing including all that for you. And if you don't use a jail manager, you're gonna wind up writing some scripts for managing your jails and you might as well let someone else do that. Otherwise you're writing your own jail manager and you don't wanna do that. I talked about my jails that got out of date. I needed to get them up to date before doing this update process. So it was my configuration files that were out of date so I had to run merge master by mounting NullFS mounting user source and user option and then doing the merge master. But instead of using merge master, I've recently found out about Etsy merge and Etsy update which I don't think are useful for this current situation but I wanted to mention them in case someone else can figure out how to use them instead. So you use them instead of merge master but it's from a port Cichl's Etsy merge and Etsy update which has been in base since FreeBSC 10 which is like ages ago and I only found out about it in the past week or two. They both do three-way automatic merges which means there's none of this merge master, Q, M, one, two, one, two, whatever and then you decide what you're gonna get in. It does almost all of that automatically for you. Again, I've not used these tools but they're very interesting to me and I wanna see if I can use them. At present, how do I update my jails? IOKage update because all my jails are now managed by OKage but that's soon to be replaced by FreeBSC update and for that I blame Peter. Why? Because weeks ago, not weeks, I'm sure it's years ago Peter tweeted that meme when we were in this discussion about jails and jail management. He was telling me how they update the FreeBSC cluster. He said just use plain jails and I ignored him for a while until I started reading Michael Lucas's book and that leads me to the one last thought. How are we doing on time? Where are we? How much time we got left? That's good. So this one last thought is very scary because it has to do with the dark side which is gel.com. Reading Lucas's book on the train from Bergen and if no one's taken the train from Bergen, please do. It's absolutely gorgeous scenery. I was very tempted by gel.com because it has this wonderful use of default values. I said earlier that you can no exaggeration declare a jail in three lines, two of which are curly braces and the first of which is just a jail name. To me that's just the epitome of brevity and so I'm gonna start using this. Some tasks are tedious though so I might write a script. Or two. And of course me being a packager, I'll package them creating a new jail manager which I don't really wanna do. But this is what I recommend reading. All of Lucas's book uses spoof of a very famous artwork. I don't know the original name of this one but all of his books, this is actually in the original art, this is a prisoner and this is the prisoner's child and they're coming to visit them in the jail. So do we have any questions based on all of this because we do have lots of time for questions? Come on up. Thanks for talking. One thing I enjoyed about the thing jails is that the base jail can be made read only. And so like this bakery base audit does a very good job of searching for break-ins. But if it's read only, there are no break-ins by definition. Did you like, well it may be, that's what I think. So did you like, have you had any concerns about this or have you tried to find any solution for that or was it even a problem for you when you were migrating to the things jails? I remember a few times when I was using IO cage where my jail would become read only suddenly. The whole jail would be read only and I couldn't figure out why but IO cage was turning it read only. That's not really related to what you're talking about. But there is a ZFS property with the ability to make a file system read only or even no exec. And I've found problems with no exec on file systems like slash temp. IO cage likes to run a script from slash temp. So I don't know a way around that. And what you could do conceivably is create a different file. Remember all those sim links in the base jail? You could actually create a different file system for each one of those and make them all mount them all Nala fast but then you're back to a base jail again. The beauty is that if they break into the jail they're not going anywhere but the jail. They're not gonna touch the host. They might be able to get into the database server if they can find the creds for the website in there but their disaster is gonna be linked restricted just to the website and the database server and they're not gonna be able to touch the host which means I can get into the host, delete the two jails, reinstall them and get gone again. So in that regard, I'm not worried so much about setting setting those directories read only but yeah, I appreciate the security value in doing that. Since there's no one behind me I can have another question then I guess. Have you tried Jalene ZFS file systems? I was looking into it for a while but I couldn't really get it properly I think. Did you have any experience with that? Was it good or bad? The most I've tried with putting ZFS into a file system is just creating a mount point in there. For example, I had a jail that was only for my mail but that would be a separate file system mounted a user home jail, sorry, user home den mailed her and that would just be a separate file system so I could snapshot it and copy it off somewhere. But no, I've never played around apart from trying to create a snapshot jail which you just receive snapshots from other hosts. That's one thing that I want to do but I think I got it working but never went beyond that. Thank you very much. Thank you. It's too major version. What would be your personal preference? Would you upgrade it from the host by passing the B as a path to the jail and from within the jail itself? When I was using a jail manager I would always use a jail manager tool. No, no. Yep, yep. But when the jail manager wasn't working as in just recently, I would SSH into the jail and do a free BSE update from within the jail. Once I go to jail.conf, I'm not sure. You can do the free BSE update from outside the jail and say this is your destination or you can be inside the jail and do it. And I don't have a preference in the moment because I haven't done it enough. All of my updates have been from the host using the jail manager. So I've done that a lot anyway. But recently I've done only a few free BSE updates from within the jail mostly because the jail manager was broken and that worked fine too except it screwed up the metadata, right? The jail content is now lying according to what the metadata says. So if you're using a jail manager, use a jail manager tool. But you're not using jail.conf. No, I don't have a preference. From within the jail then you need to allow flags to be changed inside the jail because otherwise. CH flags, yeah, you have to set security level, right? Secure level, you set it. It's often the tune, you have to set it to zero. I remember that from doing that myself, yeah. And the question, lately when I switched to tick jails, the shutdowns of a great number of jails always times up and maybe three or four jails are shut down and then the others are running and I need to bang it a few times to have them all shut down. Do you know how can I make it so they are really all shut down with single command? There are orders, the order in which jails should be shut down, you can set that. I forget where that is in regular jails and you can also set a timeout on how long it should wait and you can set that timeout to be a value so that it always just waits until it finishes because that may... Where is it set? It's in Lucas' book, I can't remember what it is. I'm sure it's in my notes because it would have been one of the things that I highlighted. Kindles are great for that. Oh, good, good, you're welcome. There was someone else in the line and then they went away, I think. We got time, it doesn't have to just be about jails. I guess I was just curious because you said you blame it to Peter Wem. For a lot of things. Sure, but I guess the question is what was his argument of why you shouldn't use a jail manager and to just use jail comf or whatever? He was saying that it's not that difficult to manage jail and to set up the jail is just this and to set up is just that and really the reason that I went to a jail manager and say, oh, this is so much easier but there's always a price for convenience and so I just decided I wanna go back to jail.com. I would have to look back in the Twitter history to find out what the actual conversation was about but I think it was about upgrading user land and upgrading etsy.com, rc.d. I think that's where it came from is that I had outdated jails and he said, well, stop doing it that way. And don't misunderstand me when I say blame Peter Wem. Peter's given me some very good ideas for how I do stuff at home, including the best thing I think I ever did which was the DNS01 authentication for Let's Encrypt and a hidden offline DNS master. And it was just so much fun to do and it just has been working forever. Yep, in the blog, look up Let's Encrypt. Look for DNS hidden hyphen master. It's basically a DNS server that is offline and is the master for all the DNS servers that are online and I can issue a Let's Encrypt cert locally and it pushes out the DNS and it only takes about 10 or 15 seconds. More questions? Thank you very much. Thank you. You're welcome.