 We're running a bit late, couldn't yell at Sam for being late, so we may drag lunch a little bit, they'll probably be moved there by the time we get out. This is Ike, he's from New York, he's not particularly dangerous, he fights us, and he's going to talk about gales. It's a very important talk I have heard in Canada, so forward to a pretty good show. Thank you guys, thank you for all these fantastic developers, so that you guys know who I am, if you just get into the good interview I am, if you get into the good intro, I am from New York, I am in this room, I am a user, in other rooms I'm a developer. This room is seriously a user. Jails accept to leave without the case of a server with software development purposes. I've been in and out of jails for years, and I'm not going to get into any religious OS virtualization bashing, but I do have some really strong points to make based on a specific threat model that has been the majority of my work, which are running production servers on the Internet, which is a wonderfully violent and fun place to be. Let me see. A lot of information for an hour, and the Ghost of Sam was made that less. So, questions after, please. Really fast, a lot of visual information, a lot of other information. I'll be around later, of course. Also, I'm trying to stay very close to classic Unix processes, ideas and stock methodology. I'm a real fan of history. I'm a real fan of not doing things out of the sake of just pure tradition, but traditions and patterns. Things come back for a reason. Things persist for a reason. And even in a lot of cases in my experience, those reasons aren't immediately evident. They often make themselves so hard to find or scale. So, I'm going to try to keep this all up very stock. I'm not really presenting anything that's not any stuff. No thank you for the craziness. I also assume you know the basics of Unix and perhaps free BZJL, but regardless, I have spoken about JL in a lot of different places, in different contexts before. And the last one, some of you were around it. Nobody in the room at your elsewhere, if you were in Tokyo. But I spoke about JL and servers and compared them directly to buildings in my city. I talked about comparing BHK to Otis and Otis Elevator. I talked a lot about the growth of cities and how that's very much like the growth of the internet and how these patterns are similar. And I tend to think of the internet and internet service providers as very much becoming like cities. I live in a great city. I live in a great city. It's always changing and it really has influenced my work with JL. But that's not the case at all. This is a big overview. So, we're going to start with an exercise that's about scale and patterns and complexity. Although if any of you have ever seen the Eames movie Power of Ten, a great film, 1977, Charles Ray Eames, and in this film, they start with this guy sleeping. The film is just a movie about a guy sleeping on a lake in Chicago, Illinois. And they zoom all the way out in these intervals of Power of Ten, way out into the universe further than I even show here. And then they sort of zoom back in. They get into the past DNA and all the molecules of quarks. It's pretty fun stuff. If you haven't seen it, it's terrific. And there's patterns that emerge at all these different levels of scale in this movie. And there's other patterns that emerge. This is something similar. This is like my Ike little version of the universe here. And we start way out of the big end of it. And here's the internet, you know. Satellites, somewhere in the GPS, you know. You know, clock timers and some things. You know, we start scaling into the net. And we're going to start scaling in towards my home in New York. And we slide past the nice book meeting. We slide into a data center. We're going to bowels. You know, a lot of us in the room know what data centers look like. And we get into the data center. We get into the cabinets. We get into the machines and switches. And there's a guy sleeping. And I don't know where we're at. We get into servers, the stuff we all touch, you know, very well. And our wearers know well. And what happens on the net reports and the software that runs behind these reports, these eco-physical interfaces. And this is where we get to be as the unit. And inside, we have all these daemons running these applications, various applications. And behind that, we may have all these other applications where we do all kinds of wild things. And this little guy, just if you see these around, they might have been on my clothing. We kind of say these everywhere in New York. And around these daemons. All of this stuff, just to post, just to make a web page happen. Okay, it's down. You look at it at a net scale. And there's more formal versions of looking at all these layers. You know, the OSI model. We all know this stuff. But my wears are better. And I'm not going to talk about hardware much, because we all know it's getting faster, cheaper, smaller. The two things in the match have, those are embedded web servers. Not only you that take them. So, units. This is what we're here for today. This is my little diagram of units. And it looks often complicated at first glance. But you can kind of see the devices over here, the kernel, and the rest of the user land. This is a simpler kind of view of units as we're going to need to look at it today. You know, you can take this view of units and think of it as something that's really complex. People like, I don't know if you guys saw the Brooks's talk yesterday. No, David? Yes. Some people need to think of the computers as very simple units. And we're going to look for just a second, just to kind of get into the room here, about where we're going with this. This is for the methods of the room's manner work fractal. And zooming in and out of the whole thing here. And seeing all of these patterns emerge. Patterns. Patterns. Patterns. That's the whole fun part about practice, right? And you totally forgot the list. Right when you get out of bounds. There. And so, you know, you start thinking about where we're going with this today. And well, it's a little bit cheesy style. But getting into virtual units is simple and things can be done in patterns and scale. And let's see here. So, these ideas and stuff that I'm going to talk about, the virtualizing machines. These ideas are not at all holds, all right? So if you look at the history of the units, the entire beginnings of the units was time sharing the system. Trying to share system resources. Okay? So, to me, jailing is a really old idea. First, computers could be used by just one person running one program. The local company's efficiency, so many people could use the machine. Time sharing was developed. The obvious next step was to allow each person to run several programs. Units is one of the multi-programming operating systems that now lets each person execute several programs simultaneously. But the usefulness of multi-programming was restricted by the existing terminals. And the first multi-programming program was built. There we go. Just quite explain the style of operating systems. Yeah, yeah. Okay. I think the idea. So what kind of real world context, more in virtualizing the entire operating system? External security threats and development messes. And the same mutually untrusted users. This is a bunch of hackers at DEF CON's capture the flag competition. Scary guys. Mutually untrusted users, though, aren't always scary guys. These are the mutually untrusted users. These are the mutually untrusted users that are important to me and from interpreting the word correctly. These are the mutually untrusted users that are the ones that get scared. They're made generally in the first place. You know, why are these people mutually untrusted users? They're all sitting waiting for common shared resources, the subway. And, you know, this guy at the far side here, he's not to be trusted in this woman's kitchen, period. He's not to be trusted with his motorcycle, conversant. That's just how it is. But everybody, you know, is waiting right here for shared infrastructure. But then again, there are these rooms full of wily hackers for the conference, isn't it? All right. So we'll get to some of the mutually untrusted users. I know these ladies' website, and I'm telling you, they hired a kid just to recompile Telman after. People do dumb stuff. You know, any system administrator in the room knows this. Look at this guy standing on a metal ladder in a swimming pool, barefoot, with an electric drip. All right. You see these scenarios at work, can't we all, right? And people just have habits that are out of control and you don't understand. People have bad names. Mutually untrusted users also have cultural differences that are astounding. In parts of India, rats, I'm told, are kind of sacred animals and treated very, very well. In New York City, they are not. So, you know what I mean? Like, you want to run a PHP minus QL admin? What? Okay, cool. It works for you. And then programs. Programs have all kinds of problems on their own. And they're mutually untrusted users as well. That's Grace Emlin and the first school computer boat. And then professionals. Professional system mates. Okay? Muscle memory kills. Who has the most accidents in the room? All of us, we do the worst things. We have the craziest stories to tell about systems we've taken down. Don't wait with it, you know? And all of this when you have shared infrastructure, do you? I don't know what you think. So, how do we solve all of these problems? How do we make sure that everybody sets the line because it doesn't create problems for everyone else? Well, America develops very bad. And let me just tell you really quickly that living in this kind of environment is about as much fun as living in really lockdown systems, you know? Like boxes with high secure levels and all kinds of turquoise rules. We have to ask before we run a particular line or we're going to be fired. It's not fun. You know, once upon a time, units was fun. And I still think units was fun, you know? And so all of these, these are all mutually untrusted users in my world that I care about. I care about letting them have fun and live their lives. I also care about protecting them for each other. And themselves. So more practice. Okay, go away from this abstract stuff and they're putting up with that. Maintaining old, junky systems. This is, you know, a client system, say with the, yeah, those are all one gigabyte scuzzy hard drive. This thing is sucking a lot of power and it's a big waste. It's running three web servers, a local use DNS cache, a file server that's used by two people in this office and two dev servers. And this, you know, it's like this all one little high density, one new box. You can get another one. Voila, they get redundant. This is pretty cool stuff. Now they're practical use. I don't know how things are in Europe but in states if you're around the data center seeing, wow, electricity. Wow, this is scary, man. It's bad. It's getting worse. Power problems. Consolidating hardware is a really great idea. All right? So, we'll get to jailing. Quickie Lil, just had to throw this in. It's kind of out of place but I thought we're going to write programs and do one thing and do it well. Okay? Jail doesn't. It's very simple. What is jailed to? Jail 8, manual section B. It's a user space utility. Just like if config produces virtual system image it's process tree-based. The jail system call is simply the system call that is, you know, both group and attaches it to an IP. It's a lot of the Siege Group code bound to an IP. It's pretty simple. What jail is not, it is not a classical machine-elit emulator. A little bit later, that's the macho land. You know, like machine-elit emulation. It is not a Siege Group. All right? So a lot of folks in the room, a lot of old DSP folks, call things jail all the time. It drives me nuts. It's hard to find stuff with certain engines, man. So, jail is jail. Siege Group. Siege Group. I'll grab you a beer and chill it out. Great uses for jail. Again, our resource sharing. You can securely separate untrusted users and processes. The great for learning environment. The great for development environment is testing. Hacking new stuff. User land stuff. The great, you have insane high availability to possibilities. Okay? So imagine if your system just hits the file system and you can start replicating it all over the place. Think of how you would do this and start going crazy up on it. You can jail the great for learning pots and highly vulnerable network services. Poor uses for jail. Anything where you need direct kernel access. You don't get kernel firewalls. Limited network interface access, depending on the context of your tuning system, limited device driver access. Poor use for jail. Wouldn't Siege Group simply do the job? You know, if you're running a name server and you've got all of the mechanisms there to Siege Group at RSC, it's pretty much a waste of time to set up a jail just to do that. Some applications require particularly low level system faults. Notably, Postgres still doesn't run securely in jails because it still uses these ridiculous system-climbed IPC faults and memory. If anybody wants to talk about that, I'm going to skip it. How to jail. I'm going to talk about just, I'm going to run through as fast as possible how to jail part. Realistically, a lot of us in the room have done this and you know, I'm going to have the man page here. The man page is the definitive manual for jailing which is short. There's no... Anybody writes what we really think about jailing? Probably not. No one. No one. Let me see. So what do you do? You compile your user land from source. You've got to get the source code. You create an IP alias on a network interface. Rocket science so far. Run the jail call with the IP user land and you kind of move the jail. It's not really moving, but it runs. It starts at RC. It starts at all. And a practical comparison again with the diagrams. What's different between these two systems? One's physical hardware. One's virtual. One's got a kernel. One does it. There's a device missing on the jailed system. Other than that, pretty much everything. So making a jail. From the host machine, this was done with a previous D6-1 machine and we didn't redo it for this time because it's the best version of these slides so far. Pretty polite. So you've got to get your source to see if it's up. Whatever. Make somewhere for the jails to live. Partitions, disc mounts, whatever. Putting your jailed systems at the very least on a separate partition is a very good idea for resource-based disc attacks. So when the jail uses all of the disc, it doesn't take the whole host system up. Simple little things. So you've just got to make somewhere for your jail to start managing the script. So I'm going to get to that in a bit. It's not RC. But starting all of your jails, the note here, starting all of your jails at the host system boot time is terrible. You have problems with masking jailed systems. Even if you start out just a few small jails, you'll eventually run a lot of them. And if you end up having a problem with one of the systems as it starts or you're in for trouble getting the whole host back up. So, pre-flame. And by the way, am I going too fast? Am I talking too? The man page again. Here's this important section about building the jail. Just a few lines. Define the source. Go through this. Define the source, where it's going to live. You make this directory. You go to your source code, you make world with your destination directory being where you're going to put your jail and systems. So we've got, you know, I'm going to put this in the usual open jails. And in the usual open jails, we just made this directory for our jail. And the next thing we're going to do is... What are we going to do? We're going to make... Go into the source code. You can see this up. Whatever. Make world with our destination directory. And you just let it cook. And you make distribution destination directory and you let it cook. So what do we just do? We took that jail, use-aland directory, and we just populated it with the use-aland. Cool. MFS in there. Then, quick note, I'm just consolidating a lot of parts. MOUDEVFS. You can throw it. MOUDEVFS is great. The devices are mounted instead of filled. When you mount it, you can throw all the flags you want in the world to restrict access to specific hardware devices. And really, you can give the jail access to only what it needs, which is the really locking box. This is a great idea. It takes a lot of time. It just takes a little time. And for historical purposes, there's no kernel. So, usually, you can pour that X error in. I would assume, because nobody knew what would happen if there was nothing called kernel in your root. Is that why there was a song? I think you won't need to see Autistic Issue. Autistic Issue. For historical purposes, there's that sort of error. I guess I didn't mean to do it. Right. Yeah, at least I can't figure out if there's anything else. Sorry. Which is the effective security amongst a number of those people. That's true. At that link, what do we do? We mount all of our devices. Why isn't there some automated system available for the state which runs the process? Shouldn't this just be like make jail here? There's too many variables in what it just explains. What if you're building jails all under the slices of your system? What if you're not building them on a particular slice at all? What if you're going to tweak out your main role? A couple of other flags to do something specific. What if... This isn't very far. That's a tough one. And also, again, we've got to configure... We're going to go back to the host machine component. Not the jail. Configure. Because in this case, we want to better IP address for a host and we want a really, really locked down host. We don't want to be running anything on the host. The host is sacred. The host is this place where your host system should never be rebooted. Your host system is very precious. And let me see. So what else are we going to do? We're going to throw some flags, nine at D. Obviously, by and they will know. We don't want it doing anything. We don't want our master system doing anything. So even SSHD, make sure you lock it down to your host's IP address. If the address is there, only going to be used on your host system. Only live. You don't want SSH listening on, you know, what about your SSH listens to the host and for your jailed systems. When your system doesn't listen to SSH. So those are about, that's about as crazy to get locking down your host system. Beyond that, you want to use the most extreme, the most extreme secure practices that you know to run your host system. You don't want to use it for anything. You know, like, I'm a big fan of seeing login screens of my hosts that don't fill up in my terminal. You know, after a month, because, you know, that place is not a place to play. Your jails are. You've got all kinds of problems here. Okay, so next, from the man page, we're going to run the jail command manually. All right? And all we do is just tell it where it lives, give it a host name, tell it the IP address, and we're on a shell. And this is kind of an analysis, a machine in a super user mode. Sort of. So, you run through this little list of things that are in the man page. And, you know, that's just a success. You have your host name, you could do it so fast, but it's just like setting a new system, add users, set a time zone, dot, dot, dot, dot, dot. And, we're going to look at rgc.com, SSHD is an annual, we're also going to toss in send meals mid-enabled, a personal thing of the like. And, but really, really good jail code. As you find it, you'll know that we're here, our base is none, we're just, you know, getting errors. Aren't doing your logs. And if you enable no way, that's all in the man page. So, what do we just do? We've got our user land devices, we've got our local kernel, and we just add users, you know, modify rc scripts, whatever is good. We're finishing. We're finished configuring the jail system. We sign it the IP address, we bring up the IP address, and, let me see, we're going to mount the ProcFest. This is just an old manual. I'll try to bring up the ProcFest in jail. Which, again, will only show the processes running in the jail to the jail users and to the jail group users. And, we've added our network interfaces. And, let me just start again. Let me see. A, what is this? Starts with B. We start in the jail. Manual. This is a little movie I'm starting. Does everybody know what this looks like in the room? Everybody's seen this kind of action? We're just installing a base system. Anybody installed previous D in the room? Okay, cool. The system has now started. So, we just finished starting our jail system. Now what are we doing? Well, SSH. We can figure SSH so that it's running. And, we can SSH into the jails. We can also list the JLS utilities. We can list the fire onward to list the jails available. And, in using the jail, SSH into the jail. And, to that, you treat it just like it's a running live server. If you're mucking about a lot, I'm just going to say quick side, which I'm going to hit over and over again. It's a small moment to continue. If you muck about in the jail from your host server, you're bound to run into all kinds of terrible, terrible problems. Not just with mucking up your jail system, but messing around with your host. So, if you say, how many previous D users in the room know my slash call, it actually is. Anybody? Use your local. Two or three still. So, with that stated, if you're trouncing into a jailed system, muscle memory kills, right? So, if you trounce into a jailed system and go into home, it's going to be softly to home under a drink. You know, to leave a huge number of things, and your host systems and home director, right? If I'd say user exists there, what am I doing? Create all kinds of confusion. So, if you really want to SSH, you can do it. Live in and put it outside. Let me see here. So, in the jail, and it's just like any new server. Yeah, there's all of our running processes. No, there's a bunch of things missing. Like, where is it? Oh, I don't know. And, wow, our processes start off all high. Hmm. Well, that's what it looks like when you're running inside the jail right now. Let's see. You also have root in your jail. And, as the jailed root user, I mean, you really are a root. Really? Really? And, so how do you know you're inside the jail? This control has this full of two of the variables. It also has this variable that tells you whether or not you are jailed or not jailed on your free business system. Bottom of this slide is a URL which we'll get later. For a catch, if you for some reason or another want to, you know, have that always be or not. If you want to always be zero, so starting and stopping the jail. So we're in jail and what are we doing next? Okay, so let's look at the jailed process. These are like really, you know, simple man-page things. They're all the jail-man-page that you should say about. There's ways to list all of these, all of the number of processes by jail ID. Let me see. These are all jail processes. You would kill a jail. You want to stop a jail. How do you do it? You would kill all with a J flag and the jail ID. And that's all there is to it. If using this in previously before, which people wouldn't be like me before that X system was in question, they would kill these scripts. There's a pistol and utilities that are really great in courts. Another little gotcha. Watch out for stacking mount points. Mount points are not like using mount isn't like if you're big. You can run that big and add that alias many times you bloody want and it will always be that one IP address and one IP address. Whereas mount points you can start and put this in here because it's really common to see all the time. Let's see what else. Restarting, and it's also been common over the years of new glitches and new bugs if you're starting and stopping deals from the RC scripts that as well are always mounted on mount points in the jail. Our screw scripts problems So, we're going to restart the jail with a little simple start script and there it goes. Let me see, if you look at this the jail IP, which is common we've talked about before JLIT has now changed to 6 JLA 8, 4, 5 JLIT has a little glitchy bug in it where if you kill a jail sometimes the jail will still keep itself listed on the list as though it's running. It's a little bit of a new noise it's really neat to know what's sure it's going to be all the processes you need. You can just start moving through the project that's where you're going to start with some scripts Let's talk for a second about Jaxx Jaxx is really handy in a situation it's actually tremendously useful in a situation where most of you are usually interested in programs. If you're consolidating a number of servers from a trusted network if you're consolidating systems such as jail systems in a trusted environment pretty much the reason you're jailing them is that you're trying to separate processes for various types of requirements for each other not necessarily for the users Now, Jaxx what it does is it spawns a process that attaches to the jail inside of the jail it's like, you know, it's using the suit okay? And I say, you know, Jaxx it's a JLIT and then the command so, doing this from the host, okay why is Jaxx that idea? Well, anecdotal story, jails were used for capture the flag at the deaf con a little bit more three years ago with great success the first time they've used it in a virtualized environment is anybody familiar with the capture the flag competition? it's neat it's kind of a war games it's more a network war games environment that happens at the deaf con conference in Las Vegas and it's pretty hard for some of the craziest people on the planet know a lot about the system exploits and in this battle, Jails were used for great success so suddenly they could do all of these things to really tightly monitor score it in ways they never have before every team is given a server to protect and then they have to go attack all of the rest of the other team servers tons of different ways to score points so in this environment the jailed systems for the first time people running the capture the flag competition had a God's eye view of the entire battle they could look in the directories and see what's going on they could take process samples and see what's running, what's not running what's how it's running they could do all kinds of fun things except immediately in this one immediately they ran into a problem with all of their scripts that they had run with Jails they had to build all of these scoring tools that were really sophisticated with Jails they would dip into the jails, do something and then pop back out with results and they would process and throw into the database to score it immediately all of the contestants everybody in the competition figured out what was going on because they would see this really strange process they couldn't quite see or figure out how it was attaching what was going on where they'd see some process they were using hang first with that it would be glitching matrix so immediately all of the jailed participants in the competition made all kinds of scripts to start immediately killing everything that came in like the other Jails tongue style so here comes J-exec the supporting tools and boom as soon as the hand goes in it kind of ran off so that was interesting to see that just for a practical perspective how quickly everybody adapted to this so J-exec is pretty bad in this case we kind of do things to go back into this system really what I was saying if you're running really good systems that are really truly compromise J-exec you're attaching a process that is somehow directly connected to the host machine to the jail there's a lot of impact service this is the situation which guy which bomb guy do you want to be if you're going to go in and you're going to figure out disarming this bomb do you want to be with the wires or do you want to be the guy with the robot who drives it down the street and blows it up alright so let me see the gap procted status contains the last field etc etc more process stuff so what we've gotten from this is that practical comparison for running applications in jail are pretty much here's pictures of like the process trade again with the diagrams here's the whole host server in proctafest the jail only sees the processes that it's allowed okay and proctafest here we at the end of the processes proct ID and then status at the very end of the file here you can see the host name you can see the jail that owns that jail process which for the host server you don't see anything you get a dash and that shouldn't be highlighted but the little j at the end will be status or ps output so you know that the process is jail is everybody getting it is anybody having questions for a proceed longer so it's the best practices and pattern stuff and opportunity so like I said you want to use your highest secure practices for your host server you want to know a lot about ssh you want to know you want to have if you're running a real production service on the net you want to have a contingency plan for what happens when ssh has a bad day for all of us you want to always see the jail file system in user land you can always see the jail file system in user land from the host server but again be careful all kinds of other pitfalls await you design your jailing system carefully which just means remember where you put stuff remember where you put your jail systems and be creative with the court because in a stock in a stock it's de-install and you have such an amazing tool chest to work with jail systems you don't need any external tools you don't need any jail system any specific jailing things from the courts everything you need really run great in jail systems just right there again like I said in 4.x previously we have jps, jkill and jtop those are jail versions all of what they describe and the 5 and 6 onward built into ps and kill so plus jls, jex, jtatch design your jailing system carefully note about null if that's what you're talking about that's kind of what we're going on and additionally any ps trees need can be useful xkl is really really useful just like disc images be a handy thing and I know I just told you guys that's a very good jail system so common weak points in a unique system let me see bosh jail post name lockdown resource attack just being full of just disc images of briefings wasted any of that as a problem port bomb and memory hogs secure levels and logging.conf which I get to in a second I'm going to do it on time if anybody needs to or wants to leave for lunch start feeling hungry feeling like it let me see process controls direct driver access flags do not matter what there's really nothing about all of these weak points that's really that different than any unique system to deal with these problems on any system that you're running that faces any possible friendly network of these problems so really the only difference between a jail security jail system and your regular system is the host name if the jail views are changed to their host name and is allowed to wow you're going to have a hard time fumbling around with the host trying to find what the processor is actually what do you have more than as many jails as you can have so some comments on isolation memory and process attacks there's a guy named Emerson Manto who solved this great problem he ran a thing called the open root project years ago it was a jail system and on one page the root logging credentials were posted and every hour on the hour the jail had a tar ball dumped over the top of it and it was restarted every hour on the hour and what a neat project and it's open to the world you can do whatever you want it's just a jail system there's a great paper that was insistent in the magazine the URL's here it's around the net and about how to get the one problem as everybody was using these systems to kind of break it and he found that was four bombs so he found that the combination of upping the secure levels setting the mutable bit on logging.com you can actually really pretty sanely deal with four bombs he wrote this great paper about it describing how to deal with it it was all very, very simple and at the end of the paper he had Emerson Manto as a 14 year old such a junior high school wow this is great and he realized wow this guy just trained a bunch of junior high school American U.S. terrorists this is cool so anyways that was a really neat project it's gone for now it's gone for a while here's the there's no city make sure this is available for everybody it's a really simple piece of code in front of my Brian Redmond wrote he used it to break just about every system in the world since 1986 that he wrote it and it's called hog you pass hog and it's a simple utility you pass hog and it needs to cure and it consumes that much memory in megabytes real simple so down at the bottom here you've got the simplest work bomb that you could possibly make and this is a great tool for stress memory limits look memory memory and process limits and process controls there's always been really difficult problems in the computer it really doesn't matter good friends and working a lot with some of those they told me great things and also told me there's heavy trade-offs and penalties for being able to control memory and processes this is still a really hard promise to solve and let me see and just as an aside this little piece of code it's been working with systems for a very very very long time so this is really neat so memory and process attacks here's the recipe for the lockdown step by step first step you simply edit blogging.com you log down all of your actual processes second step you've set immutable flags on the jailed the jailed system is blogging.com and then in step three you set a higher secure level for the jailed system and there's two of them honeypots oh yeah this has got the patch again so that your jailed system is kind of neat this resource control okay so you want to at least put your jailed system on separate partitions set separate partitions if you're trying to set separate everybody in separate partitions just from practice this can get really really good with a conversation not only in this manner we're going to be actually moving from partition to partition to blogging things down but when you need to make a jailed system disk larger it's really really inflexible fileback disk images are terrific they're really really flexible they're excellent tools in terms of controlling disk consumption but we've got severe disk IO and disk IO right now in the jailed system and I think a lot of systems realistically is the biggest bottleneck that you face so again there's trade-offs to secure your system so fileback disk we'll be able to do fileback disk but I'm going to skip it because it's in the previous handbook and the automation okay terrible packaging is really the simplest way to go about everything automated moving systems from server to server vacuum systems up simple it's reliable you've got to be wearing your dad and the broth pounds wear siblings the tar's fantastic it goes to use previously quartz mechanism you've never been in an environment but it's always something that's been talked about like wow it would be great for us to use the quartz mechanism not to put a jailed system in the ports that's silly but to run our own jailed systems from ports so then suddenly we would say hey we want the HACU web server jailed system for block project push button it's running automated that way or we want the light hvd server for this project we want that push the button it's running it also uses cbs and spn and for a time the web hosting company that I was part of I think we actually used cbs for a number of changes that we did calling the jails to our customer upgrading jailed systems simply use build world from the host system and toss build world the destination directory flag with the jails user end path and it's pretty darn simple editing syscontrol.com ah recipes so here's all the defaults and here's some movies that I think really know where I've talked about the grid buttons and knobs for control and deals set host name allow what is the defaults one which allows jailed users to be set their hosting which can be used for some time but again for any massive jailed system that's your first line of massive jailed system security jailed for stat fests this is to use mouth point information so in a jailed system you can show all the mouth points on the host you can show no mouth points what so ever or you can show just the mouth points that are mounted in the jails usually you also have a library on sockets things like paint now we can do it before that we can deny you can yeah allow access to router sockets you know I'm not sure who wrote this in but because of posterity this was written into allow system 5 IPC calls for jailed systems so if that meets your requirement requirements otherwise no need to run this I don't want to scrap security jailed changed flags allowed you can use that right there to lock down your logging.com so that the jail group can't do anything with the or they can't change flags on file and motherboard so over there's the stock values again firewalls quick comment this is an awkward question I can't run a firewall in the jail what's the problem? yeah you don't have access to the kernel boy would you really need if I can have a pf perhaps or something in a virtualized sense be able to only manage or give it to IP addresses network software but that's not there so all firewalls means to dump them in the closed server but again really what are you doing with your jails in many situations I don't know well if you're running a web server you don't have to open from the jails SSH or web server you can be able to control these values and maybe do more advanced things to the packets and the traffic a lot of that is not going to fall in the hands of applications as much as it would be really nice to be able to manage firewalls from the jail system so maybe it would be great to run some handy from jail systems that would be awesome but right now it's just not possible maybe there's all these things to be sure to get there let me see start script disk image starting from rc.dok I promise I would say something about this but for years we haven't wanted to use rc.dsc mostly because a little bit of time to use them when the system is booting it's just a really difficult idea to master the jail system it's only a few experiences running through things that we've never expected and secondly you know they've had a number of weird bugs and there's a lot of questions people would have never really pursued any of these bugs it's never really useful until it's about recently security vulnerability security vulnerability security vulnerability basically was that at Mt you had some super tricky bad little siblings here and there that placed other devices that execute bad that's very bad so the system starts suddenly you're executing code from the jail and the host game over for everybody so that's kind of bad there's also a bunch of new useful stuff in the jail rc scripts which at the moment I'm looking at personally I'm not committed to it personally but it looks pretty good it just checks for all of these bad siblings I don't know if this is a good strategy because these rc scripts to me down this kind of path of thinking can get really really really complex and I found buggy over the years in the first bunch if anybody has any comments about that comment with me let me see so okay that's about enough about the rc scripts for a long moment jailing prontab miscellaneous to the side of the prontable yell about a couple of things let me see miscellaneous gotcha can't delete the jailing directory this is a bunch of immutable blanks so you can clear all of that and you can do that that's frequently asked hello, what's wrong? is everybody how's everybody doing for hungry and for time should I continue on or should I go a regular pace or should I just go on slant okay true virtual machines and people really know what a true virtual machine is it's software that emulates hardware both and if anybody hasn't been sleeping through this talk so far you'll realize that's not what jailing is about jailing is about keeping okay and granularly being able to adjust not a virtual machine but virtual machines again they are the module place in computing because it's software you are computers this is tough stuff and some of the work that's done is absolutely great I will add so I am definitely not able to find a mode for a virtual machine however in the context of having a web facing production servers not the best idea in the world for a lot of reasons we look at these models we look at the models is that virtual machine wrapper around our operating system whatever operating system we can run on this virtual hardware Windows, Linux, Windows, Linux and then we've got our application from the top a lot overhead as you can see there's a lot more parts to this puzzle to manage which is you scale this out and say oh I have 100 virtual machines one piece of hardware this becomes a very very complex system I see it being very negative unless you would like doing bits but it should be a bit different and that's already on top of some based operating system whereas in previous in previous deals with jails you are running processes inside of one of the systems this is a kind of fundamental functional difference here and some negative parts of it are a, you know a developer head count on these projects closed source not as many people are using it not as many people are seeing it testing it, breaking it and to be really straight the same things that really need to be broken to fundamentally destroy jail are a CH room and TCPID sockets those two things are fundamentally compromised or have some fundamental vulnerability a lot of people in the world who don't care about jailing are going to be carried that's really important really important some zelots are very very nice and in the words of some of the Sun developers when they're not being swatted down by marketing guys it really came directly out of these jailed characters to my knowledge and some people who were set back to inspire me some zelots they've got nuts, they've got all kinds of interesting extra bonus things and there are lots and lots of features like at this point memory and process limits that are interesting and I don't know a lot about it or have any experience with it anybody who does in the room, I hope to hear about it cool and so like I said, as we refer to freebies to jail is an equivalent idea usually and mutual users try to be similar and the Sun put these custom inner zone IPC into Solaris because a lot of their market wanted this financial guys to go we've got too many servers and all this stuff were just very simple things, it's consolidated and they have all these trading applications and developers go out to the same hardware if we had some inner jail IPC we could do some more efficient communication and not be able to work this much cool, that's all great for your constraining programs that's usually interesting to use but it's an attack factor as well okay and I believe it's true it won't be turned off I'm trying to pick at it and Bill, we're going to head council and secure close source it's not an open source and let's look at Zandt this is the big one I don't want to be in fights with anybody except, you know, fun but Zandt's got a number of issues again, it's got complexity issues it's just a far more complex system from the get-go running the Zandt microbiology running the operating system and you know, if we look at attack vectors here we've got, again, these demand ideas, we've got the partitioned memory process in the I.O we've got the hypervisor itself which is an attack vector compared to the C.H. Rood and the sockets I tend to trust the blue forms let me see so attack vectors again and we'll head count man, there's all these things with Zandt driving me a little bit nuts developers love it I mean student hardware vendors love Zandt and this kind of weirds me out because knowing a bit about the history of virtual machines there's been so many that are coming on and just died over in the history of computing why? Well, vendors don't like it because if you have virtual machines you buy less hardware so really weird, why do vendors like Zandt so much? why does IBM of all people love Zandt so much? maybe we should try again but wow, they really like like this stuff well, there's a number of things let me see because suddenly now they well, this slide was created they were making specific hardware tweets for the hypervisor that were IBM only one new way to sell more stuff and this top secret VTCBU, which was Intel's idea you know, put all this hypervisor stuff right into the CPU and is anybody know much about cracking web? like you know throwing things like this into hardware was just a bad idea because now we've got an attack vector we've got something to attack that's really hard to change it's hardware that's annoying let me see and another company, Triangle Grants made an invented hypervisor as well and well, there's one more issue oh yeah, 2GB physical memory on the host machine is all that they can use wow that's, it's 2007 so you know, until a lot of these issues are resolved and I've asked developers who are very close to that community if there's been anybody who's addressed this so far, all I've seen is a lot of like so it's just something real I don't really care too much about it and now you got even Microsoft trying this to see whatever Microsoft was jumping into the mix and embracing the hypervisor not deciding sign for me personally and here's you know, an immediate like immediate exploits for hardware, software, combos that haven't even been released tomorrow you know, this was just a few days ago so this is going to be somewhere funny out for V-Train and Daggercenter and they're pushing all this Zen stuff again too you know, there's all this stuff in here just made me scratch my head going what's the game? I don't get it we're already doing this stuff from a user perspective business perspective, very successful a little sort of cool and we're a heck of a lot more sophisticated than some of the stuff that's about to be sold and so yeah 64GB of memory on Intel Machines using a also about this all these different other architectures that are going to be supported all this Zen stuff and I don't know about you guys but I'm kind of on the mind of doing this since I was a little kid since I was a little kid I've seen lots of CPUs come and go but I knew a lot about their internals but I'm not a real believer that Intel is going to be around 10 years in the same way it is now and oh yeah okay so in words out of true VR I found a cool lecture that he gave on this exact topic that quote, people love virtualization just laugh out loud wow that exactly echoes my thoughts because yeah it's not virtual macho virtual machines darn it's great and some future directions disk I owe I owe bigger disks more disks network disks disk disk DFS disks anything having to do with disk also anybody who's been doing work with in carpentry field systems from nail over both top of the adult kind of fail hacking most of the user lands to try to play with this stuff and it found some interesting things and more suggested reading Jail's confining the onto none good on roots it's exactly the content of my lecture is better and you know done this once before we've done this for two of them in New York there are three nice folks in these jamming parties which is kind of like the open root thing where we get box and start out a Friday night and everybody gets rude on this box who's talking in advance and we all make jails and set aside a bunch of some old down and we go to town everybody just jails makes jails screwed around and by the end of the weekend hopefully we destroy everything and it's a lot of fun and lots of learn so anyways interested in putting another one of these together somewhere I would be totally totally into helping it's a lot of fun and you don't want to learn about deals and systems that you can destroy a bunch of people on IRC to talk to about what you're doing is really helpful so we're at the end and lunch is on its way here special thanks we're in reality I mean crew great friends PHK and Robert Watson of course for writing or Jail and boy there's a bunch of loads of fun stuff there's a bunch of different crews that are very very important to me of course there'll be a DCON the Danish Mutates Use Group the CDK thank you so much for having us all today it's such a terrific opportunity to share and mostly special thanks to all the mutually entrusted users so I'll make sure that the links and hog is on the USB thingamajig in a while later on and also the lecture will be posted at the nice spot down over the site in the library as well just in case and I'm Ike and I really appreciate that all of you have done all of that enjoy your lunch