 How many people actually work with Chef or Puppet? Okay, how many people actually work with Docker? Okay, there we go, all right, okay. The idea of this session was, you know, so when I spent a lot of time over the last year, I always think I have four jobs. What I want is my family, which is a great job. The second job is my day job, right, which is, you know, today, and I'll tell you, and in fact, good time to tell you who I am, John Willis, Director of Ecosystem Development, which means that I'm in BD. I try to help partners integrate with Docker in the ecosystem, you know, so anybody who's trying to figure out how to use Docker, take advantage of Docker, that kind of stuff, I try to help them. I've got a team of three guys that we help them integrate. So that's kind of my second job. My third job is really, I've been an evangelist for many years, some of you know me, and I like to pick themes throughout a certain period. And one of the themes this year has been immutable infrastructure. And I'm going to talk a little about my idea, my thoughts about that. And then, you know, near the end of last year, hey, well, and my fourth job is this thing called DevOps, which is I spend a lot of time thinking about lean, thinking about like the meta layer of why DevOps is awesome, not the tools. And if all things being equal, if I could do that full time, that's what I would do. I would just join a think tank and really think about how to create organizational capital. So with all that, I started thinking about what kind of presentation would I do at scale that could be valuable, giving, except for my family. Actually, my family, because my son spoke yesterday, so actually all four things, how could I give a presentation that might take all that into account? And then kind of the driver was the unicolonial thing. So when we acquired a unicolonial company a while back and it's been killing me not to say anything about it, I mean, good friend, I have a friend I'm going to mention later, Gareth Rosgrove, he's like, I was in London, and he's like, you've got to come up and meet the guys from Cambridge. I'm like, yeah, that'd be cool. You know, the Cambridge guys are the guys from Roger. Like, I couldn't say anything to them, it was so hard. So I thought, okay, how do I put this all in perspective? Like, how do I put immutable infrastructure? How do I take my experiences of what I've done with Chef and put that in a presentation? They kind of, you know, called it DevOps in the operating system, really, which is how we dealt with this thing within the last at least 10 years certain, but maybe 15 years, from $2,000. So for those of you who don't know who I am, I pretty much live through Twitter in terms of my communication. I try to use that as my primary vehicle for communication. You know, if you want to get a hold of me afterwards, if you want to yell at me, tell me I'm an idiot, whatever. If you want more feedback on some of the things I mentioned, that's really always the best place to get me. That is Dr. Deming, that's not me. I do a podcast with one of my best friends, Damon Edwards. I've been doing this for an awfully long time, 35 years in IT operations. I actually was early in it canonical when they were just first starting their private cloud, like 2009. And then I was the ninth person in it, Chef Ops Code helped kind of define the customer-facing business there. The startup gods, that is to say, I spent 30 years of failed startups, but the last three to four years, let's say four years now, it's been very good to me because I've had two exits. And Strati has sold to Dell. And a year, basically next month, it'll be a year I sold a company called Saka Plain to Docker. We did like SDN stuff. I'm a core organizer for DevOps. So again, for the people who don't know me, I'm not bragging. I want you to get a perspective of why maybe this presentation will make sense. I was at the first original DevOps days in Ghent. I was the only American there. I've been part of that whole system, helped create the first DevOps days in the US. And then I also work a lot with Gene Kim. How many people have heard of Gene Kim? If you have it, and I don't make any money off of it, especially I will indirectly make money. I'll just see why in the next slide. He wrote a book called The Phoenix Project. And if you're interested in kind of the meta layer of DevOps, you should read that book. It's a novel. You will read it and you will say, in fact, remember when I tell you this, that you will say this because when you say it, you'll say, hey, that dopey guy said I would say this. You'll say, how did Gene get into my company? How did he sneak in and figure out what was going on in my company? And to that, we just finished a book. This is kind of my last shameless plug. We finished a book called The DevOps Handbook, Jezz Humboldt Gene and Patrick. We've been working on it for four years. It's been written about three times. It's finally done. I mean, it is like done, done this week. And it really, it's not because I've written it. I mean, if you've ever read anything by Jezz Humboldt or Gene, it will be the most comprehensive book on DevOps. And I've read pretty much everyone. And again, I'm not just saying it's mine, but partially mine. But the last thing I'll say, because I'm going to try to get into the rest of the presentation, is going to be kind of this operating system, more tully discussion. But like I said, my kind of fourth job is that meta layer about DevOps. And there's been a lot of definitions about DevOps over the years. And Adam Jacob, the founder of Ops Code, for years to me held the prominent definition, which was DevOps is a professional and cultural movement. Full stop. We're done. Nothing more to say. As I've been really working hard on the last three months of kind of cleaning this book, I've been going through chapter and chapter and rewriting stuff. And I realized that, you know, I kind of have my own definition now and I think is the one I like best, which is DevOps is a movement motivated to turn human capital into high performance organization capital. There's a lot there. And things that we're doing, with Gene with the DevOps survey, we've categorized statistically high performing organizations. How do you train people? It's the beauty. To me, it's a beautiful sentence because it explains, it's not just humans and it's not just tools. Tools are the glue between organizational capital and human capital. Okay. How many people have ever seen this screen? Yeah. So let's go back with the Wayback machine, right? You know, back in the late 90s, maybe early 2000s, as I get old, my memory gets a little high easy, but you know, most enterprises, I did a lot of Tivoli consulting. To kind of cloud and open source and chef and things like that. Puppet. I did a bunch of work with large enterprises doing Tivoli work and most large enterprise, most banks and insurance companies were predominantly windows based for most of their core service applications. There, you know, there were a few examples, like in design and things like that, sure you had Sun and other type of, but a lot of the business, the business of America was basically, a lot was windows and very little Linux. In fact, I remember, it was about 2002, where the Kevin Bank of America told me that Linux will never be in production. And it told me flat out, Linux will never be in production at Bank of America. And so anyway, so the idea was for the people who haven't used this, basically it was called Ghost, and you basically took pictures of an operating system, you saved it, and people would do their rollouts of desktops and servers on this model of a ghost imaging. And basically having images, a lot of people would do some cadence, maybe like quarterly, have quarterly builds. And then they would, early on, they would just script to Deltas. So you'd have a quarterly ghost build image, and then before the next quarter, anything that got added or had to get slammed in between that quote, hey, we got a new version of this and this has to go in or this new tool has to be in there. You'd script to Deltas and then you'd start over. And the problem with this was, for anybody who lived it, we had no management systems for cataloging images. We didn't even know we had to do that. And the chaos that happened was, wrong images got, like where's the right image, you could have a whole weekend rollout. Plus the technology to build the systems sometimes took a whole weekend. Like if you had 3,000 servers, in fact some banks, they almost called it the painting of bridge syndrome. You literally rolled out and then like you finish painting a bridge, you actually start over to go back again, right? And what happened then, we really, somewhere along the way, give or take, we had this kind of what I call first generation configuration management. And if you look at like, there was Tivoli, there was Blade Lodge, there was Opsware, there was what we used to call the big three. And if you look at it right, that's actually a Tivoli for installing an AIX package. And for those of you who have used Chef or Puppet, it kind of looks the same a little bit, right? And in fact, as you go back and you look at like the Tivoli configuration, I refreshed my memory, I went back to look at the manual. It was kind of interesting how many of the primitives were almost the same, right? But here's the key thing is this was not a convergent or it did not adhere to some of the desired state concepts that we use with what I'll talk about in a second is second generation configuration management. And that is, everything was kind of individually defined. Now some were smart enough to say, I'm going to do this, this, this, this, this is order. But you never thought of a system as a desired state or systems like this is the web server for all four. This is the proxy server. You basically built like, you had this blob and you built it up like this. So it was never a holistic view. And again, it didn't have convergence. It was very transactional. You did things like install, install commit, remove, root kit, you had pre-install, post-install scripts. So it was a lot of cool things. I mean, I made a lot of money. You know, I got my first boat from Tivoli, right? Like so, you know, but it wasn't right. I mean, it just, it was, you know, it was a mess. I mean, literally there were, there were companies that literally were doing the painting, the bridge insulation of infrastructure. And then, I think it was 2007, I'd go to Oskon. So I did this, I was doing a bunch of Tivoli stuff. I sold my business to my business partner. And I figured, all right, I'll just do open source stuff because I don't have a non-compete on that stuff. And I show up at Oskon and I'm really interested in Nagios versus Tivoli monitoring. Like this is what, like I really want to, and I go to this session called Puppet because somebody told me it was a monitoring tool. So I sit in there in a back row and I see this young kid that I'm going to exaggerate here, but he had pimples. He probably didn't. Luke Kniece and who founded a puppet lab. And he's talking about figures of management and I'm like in the back row thing and what is this young kid going to tell? I just left Bank of America and I, 10 minutes in, I'm in the front row and I realize my life is changing. Because what he's talking about is like so much better than everything I've done for the last 10 years. And so I get to meet him about a month later in Nashville and I do an interview with him. He tells me about this system, these guys in Seattle. This is early, this is 2007, right? So this is one of the early applications on Facebook called I Like. Went in one week. So they say, hey, let's do this thing on Facebook. It's a pretty good application. Went for half a million years and six million years in one week. Turns out there was some consultant in Seattle that basically enabled him to do that because there was no cloud. They did it with bare metal. They went from a half a million customers and six million customers in one week in bare metal. So I wrote this blog article. I'm like, why do people spend three million on Tivoli and they can hire a couple of consultants and use things like Cobbler and Kickstarter and Puppet and they will use Puppet. It turned out to be Adam Jacob, who's the founder of Chef. So he pings me, we become friends long story short, I wound up going to work for him. But this, one more thing before that the guy who came in as the CEO of Chef around the same time Jesse Robbins wrote this really cool, it's still out there. It's called Operations to Competitive Advantage, the secret source. He actually subtitled it to Tell to Startups. At a time, most people would think in a traditional operations model with basically in that last time graph they'd say, okay, I'm going to start up I'm probably going to have in the first eight weeks maybe 20 servers or 12 weeks, I'm going to have 20 servers and so how am I going to dedicate how am I going to build it? So a traditional would be minus technical debt, we would go ahead and just kind of build everything we had and then learn as we grow and build stuff. And what he tried to show was that if you really paid the technical debt particularly in dealing with the building of the opera systems, and when I say opera systems I'm using it loosely, right? I think you realize that now. I'm building what it takes to build infrastructure. And what he was saying is that if you spent a lot of time in that first couple of weeks building out the infrastructure in a certain repeatable way and he was at Amazon at the time and they would use a modified version of the CF engine, so I've been told. And then he showed graphically what could happen in terms of your ongoing costs. And in fact early on in the days when we started talking about these type of models we would say things like what do you think your server to admin ratio was? And you could go into the kind of Tivoli shops which I did, the BMC and Blade Logic shop and they would tell you one to 30, one to 50. And in the early days of companies that what I would call second generation configuration management. Really starting with Luke. Now CF engine was 20 years old last year. And I'll talk about it in a second but with Luke, Luke really put this kind of infrastructure as code desires to take configuration management on the map. And I'll explain that in a minute. But what we saw then is those ratios started going up to like one to 500, one to 700. Like in some of the web scale coming in today it's actually not uncommon to be one to 30,000. And that's pre-container and unicolonial, right? But what happened here just for those who don't know the history, 21 years ago Mark Burgess is a professor at University of Oslo inherits the data center to manage like who else should manage it? Yeah, the guy teaches Queer Science of course. And he realizes that the I'll try and only curse once in this presentation and then I'm gonna lie so I'll probably do it to a three. But he realizes that all the current tools are shy that actually use it. And so he creates a student project to figure out how to do this right and they start designing things like desired state configuration management, convergence, this idea that you actually treat the machine as a state. It's the web server and I want it to be tamed like a web server. So I have an agent that constantly looks and what's interesting is Luke Kines was actually kind of a power consultant of CF Engine. And he got frustrated because Mark Mark is a good friend of mine now Mark was in academia and he didn't give a rip about the interface. It was all XML, it was horrible and any time a user would say can you like, why can't you make this interface easier and he'd be like, not my problem. I teach Queer Science. And the big guys like Amazon and Facebook and all those guys, they didn't care because they put a human capital, if you will into the problem to make it better for them. If you went to a large company like Facebook and said, do you run CF Engine? They'd say, yeah, kind of. Like we've put like so many hours into CF Engine in our development that it really isn't CF Engine anymore. So Luke got frustrated and he wrote the whole thing over again in Puppet or in his language in Ruby in a language that he provided to the local puppet. And then in the story kind of continues where Adam Jacob was, like you saw it he was doing those eye-like and large-scale infrastructure with Puppet and he felt there were some things missing and the cloud was just coming along and Puppet was very nested deep in enterprise, red hat enterprise and all of a sudden, you know, cloud, Ubuntu and all this became and just little things like Chef, for those who have used Chef like Chef Knife was a great example. Chef Knife was almost a year ahead of Puppet in terms of being able to just type in a command and provision seamlessly an Amazon instance with no human interaction. Right? It would basically catch the exit, come back get the information back that the instance would start to get the instance ID turn around and start provisioning configuration. So you have this kind of what I call second-generation configuration and that's just the Chef recipe there. And I thought I'd put this in as a transition because again, I'm trying to walk through this you know, what have we done over the last 15 years? What have we done right? What have we done wrong? You know, I did at the DevOps Enterprise Summit I did a Docker for managers and I tried to go back and I saw lots of people do the history of oscillation lots of people do the history of cloud and then one of the things we'll talk about a little bit with Docker in a little bit is their kind of image copy-on-write model and I realized nobody really put one of all of them together. So at least as accurate as Wikipedia is and as accurate as my 56-year-old memory is this is the history of pretty much all the things we've seen. It's kind of interesting things like whenever I go back and look at this like I think it's zones I always think there were slur zones like in the 60's you know and then I look at that and say it was actually really his latest 2004 but the story really changed right in 2006 right Amazon cloud instances all that stuff and again I think you saw a lot of Ops code was able to take advantage of the ability to do this what Adam was doing seamlessly with I like with things like Kickstart and Cobbler and then kicking off Puppet you were able to basically drive the good stuff's coming guys there anyway and then it's funny and we said with the ghost stuff right we had like image sprawl that everything kind of calmed down and then we got into cloud and it happened all over again like there's a message here that we make the same mistakes over and over and over again so all of a sudden around 2000 line a friend of mine Randy Byers writes this blog about it and he's absolutely right is that we had the same image sprawl problem all over again that we had with ghost imaging and in fact this was PBS actually so there was a tool it's still used quite prominently called RightScale and it was a tool that basically abstracts and allows you to manage multiple clouds and finally Amazon and I'm sure they fixed this now but at the time there was a checkbox to say whether you wanted to be a public image or a private image and the default was public and unfortunately PBS actually put a private image public and had all the keys in it and it took them three days to clean up that mess right and again we had like if you read the article there's just images I mean even today really it's still not really easy to find the right image for the right I mean it is because you got the elastic and stuff like that but the point is it's still a problem and so what really drove us and Puppet at that point was yeah that is a problem that's why you want infrastructure as code infrastructure as code you kind of get your best case just enough operating system and then we build everything through a library or a DSL defined infrastructure we have cookbooks for everything we put you know I'm more familiar with chef terminology I've worked with Puppet fair enough but we create a run list and we kind of catalog that as the web server and then it runs these cookbooks which runs these recipes that from the kickoff of however it says you either it's to cloud in it with Amazon and and then you start firing off these cookbooks i.e. recipes and you start building infrastructure incremental and life's great right because everything's in GitHub and you got you know you load it you know where it looks like you hit a button you should look the same and Adam wrote a chapter and if you haven't read our web operations excellent book John Osbar you know chapter 5 where he jokingly talks about you know watching a movie tornado hits your data center you know you put the movie on pause I mean it's you know it's a little flippant you go ahead and you reload your infrastructure as code in you get all your cloud service turn it off you unpause you put down your popcorn now you pick the popcorn up you unpause the movie right but I mean that's how we've lived for quite a while now and I remember when this is I was an obstacle Netflix wrote this article called building with Legos and I'm like oh no here we go again which for all the things that like we like don't we humans ever learn right and what was interesting is it wasn't it was a head fake yes they were building AMI images as golden images like our two early examples ghosting and all but what they did cleverly is they were actually building like they were building war files so the whole concept of gyro war they just took that to the whole infrastructure so what they did is they'd actually and this whole discussion came up called bake versus fry do you bake an image or do you fry an image and a bake is a holistic it's built right there's advantages and when you start it up it starts up for the most part if you have to fry it like infrastructure code you got to work your way up could be 8, 10, 15 minutes I've heard it tell me 30 minutes they're saying I don't know we bake them but what we do is we bake them just like we bake application code we put it in nexus or an artifact or wherever they put it and when you go to provision you pull the latest release and then of course there's some services they built to build the scaffolding for convergence between multiple systems clusters and all that but in the end it was like oh wow this is nice and actually this was really the birth of even though they didn't call it at the time immutable infrastructure this was really the birth of a concept of immutable infrastructure where you had this concept of immutability and let me say immutability as in the terms of a metaphor not exactly immutable no computer system is immutable things change the minute you turn it on but immutability in terms of the sense that somebody could take a system and throw it into some provision and be assured that for the most part the bits are the same for the important parts and that if you need to change it you just re-provisioned it or you rolled back to the earliest release and this is work for Netflix reasonably successful although they've had to put like everybody else who does this really well a lot of human capital into making this work because if you go out and look at their open source projects there's like a trend out of code that they've written even on Amazon where you're not that was the whole point you had to go to Amazon you didn't have to have a whole group to build all this infrastructure and so I'm sure everybody in this room knows this because I'm going to transition now into kind of immutability in containers and of course I'm going to talk about Docker but just to make sure we're all on the same page there's really what free unicolonial we think about the kind of three types of virtualization that we deal with on a day to day we've got type 1 or vmware, ESX, Hypervisor, APB indirectly that's Amazon Rackspace although I know some of those guys have converted to KVM at some point but in general what we saw over the last at least most of the time over the last 10 years have been Zen based and then type 2 which is our KVM virtual box vmware workstation and indirectly Vagrant right for everybody who's using Vagrant either on a virtual box and then we saw with Docker the rise of what was called OS level virtualization which is this is hypervisualist virtualization the compute is really Linux processes OpenVC was a good example and Docker was really an abstraction on top of LXC with some other cool stuff and this is a really good slide from an IBM paper that came out about a year and a half ago kind of depicting the difference between a type 1 type 2 and Linux containers people ask what's the big deal about a container well it may or may not be a big deal but you're type 1 you have the hardware, you have the hypervisor and then you have the VMs and the VMs contain everything the operating system it's a full stack right these are big fat images and then your type 2 is your hardware operating system and then the hypervisor is on top of the operating system and then from there it's really the same concept and then where OS level virtualization the predominant model for that is implementation of that is Linux containers you have the hardware operating system and then you share the operating system kernel and then the images are reasonably sparked small compared to big VMDKs or large AMIs and so what you get with the OS level virtualization is you get millisecond provisioning so you can provision a container in 400 to 500 milliseconds depending on how much is in it and all that kind of good stuff there's some as low as 150 to 200 milliseconds 240 milliseconds is probably the smallest shortest I've seen it is bare metal performance so you don't have the hypervisor getting in the way you don't have the queuing all the things in the hypervisor have some value at scale it is for the most part VM like for most of the things that you see people using today and what you can use you know most applications name a few MySQL Nginx we can spin them out they all run in containers pretty well every once in a while there's some idiosyncrasies that we have to do certain things a little bit differently but for the most part it satisfies most of our requirements for running it's lightweight and of course there's a rising tide list all boats in that there's been a lot of people contributing the viral growth numbers on Docker or I think that in October they had their billions of downloads for Docker Hub so that's two and a half years right this isn't an apples to apples comparison but I can't remember how long it took Facebook to get to a billion users and it isn't an apples to apples but I'm just saying a billion downloads is pretty crazy and here's the here's the crazy thing in January I remember that we were bragging actually it was April our CEO did a blog post where he was bragging about our growth and it was a half it was basically 500 million so we had in our first two years we went from zero to five million and then in the next six months we went to a billion that's a hockey stick you know so if you're not filled with containers I mean you're basically each root file system it's called a container you share the kernel you get isolation of the process man we use name spaces so you build name space isolation actually 191 and now 110 which is in release candidate one right now is actually added a user name space so now you actually be root in the name space but not on the host that sounds a lot of it the 110 is a really significant security release I mean really significant SE cop support everything there and then since most of people new Docker here I wouldn't spend too much time here but it's a thing what I want to transition into the thing I think has become really interesting is immutable infrastructure and how some people are starting to deliver infrastructure in a way like the Netflix building with Legos concept but even even crazier if you will and it's because of this isolation and one of the things that containers so one of the things that happen with containers and you know again my preferred container dialogue is going to be Docker of course there are others and you know pick your favorite but what developers the thing is developers love containers and I'll just say developers love Docker there's a lot of reasons one is imagine your developer, your last line of code something makes a mistake, you're testing and you gotta rebuild your infrastructure and it's a converged service like so it has your part of five or six different services you probably can't run that on your laptop maybe but even if you could the spin up of that service on a good day could be 3, 5 minutes could be 12 minutes and I remember I told you I did a DevOps cafe podcast and I remember we interviewed a lot of people and what I heard over and over was from developer managers would say that our developers actually get mad because once they transition to Docker they get mad if their infrastructure is converging in like two seconds you talk about spoiled right like probably it was like 5, 8 minutes right so there's this idea that you can spin everything up really quick to build your environment the fact that you can actually run your whole environment on your laptop so that there's four or five other services that are owned by four or five other groups you can load them all into your virtual box infrastructure test yours against it and here's the part that gets really exciting is everything there is basically immutable so that means that everything that I do when I'm done with my infrastructure for my service and I've basically pulled the great latest almost to think about the Netflix model where everything's loaded into an artifact factory or some place where it is immutable at that point everybody else is at least at this release is immutable I test my service I ship it off and we'll talk about integration here in a second because that gets even more exciting but for the most part for me as a developer I'm pretty safe in knowing that as it goes through the pipeline integration CD and in-production the bits I tested here are the same bits I tested there and we say in DevOps developers should wear pagers imagine you wear pagers and you pretty much can know really quick whether it's your code or something else at least you now know a lot of the like the edge cases are going to be so insane more likely it's going to be an infrastructure problem or it's going to be the error code problem that's going to be easy to figure out so this immutability thing has really driven it's not for everybody it's not for all applications and I thought we'd don't go home and tell your manager we need to throw out all this legacy we've had the last 40 years and turn it into immutable infrastructure but Greenfield new microservice term architectures this is a predominant model that's getting a lot of success we talked about the lightweight again Docker just made it simple like Liz could tell you it's been around forever Docker just made it really brain dead simple it's three years ago three and a half ago a good friend of mine wrote a book called test driven development chef and he talked about using containers in a CI how to do scale out CI testing with containers imagine you create a postgres table and get to that same table space a thousand times in integration test without having to fire up a thousand VMs test it, that's VM test it, fire up the next one two, three minutes, four minutes per VM imagine you could do that in like 400 milliseconds right, like an hour so you could do these scale out the thing he didn't ever really explain how to do it and I tried to figure out how to do it and I gave up and then the first time I got a chance to get to an early copy of Docker when it was still Docker cloud I read the read me read me and literally I had a container up in like you know, seven minutes in fact my first run I thought it didn't work I was like I needed to say anybody just run the Docker run I said Docker run Ubuntu or Docker run I figured it was one of the base images and it just ran did nothing I'm like ah, it didn't work again it did work, it just didn't help the way so and then we talked about community and all that so and then Microsoft, that same paper I showed you earlier they did this thing about when they tested they did some benchmarking the same paper that had that but they did a serial bring of 150 Apache servers just simple web servers very simple nothing to it but they brought them off 150 in 36 seconds and the teardown time was 9 seconds right, so talking about the speed and immutability, starting thinking about different types of workloads you're probably going to have a shorter TTL right, time to live and anyway, so this is just a little more of the architecture of Docker I took from the training in case you hadn't seen Docker but basically Docker engine runs as a daemon that interfaces with these processes that are encapsulated by namespaces and all that stuff and allows it to interact with the operating system kernel physical layer and then I figured I'd throw this in the client architecture does everybody understand how the Docker host and Docker containers work in general anybody done okay but another important point of this model, so you have containers and then you have containers right and one of the clever things that the Docker people did, it's only been a Docker year so I take no credit for any of this the you know it's the most systematic container right, yeah, but the other thing I did you go back to one of the things I put in that timeline earlier, which was you know some of the things that came up like ButterFS I don't have any UFS there but the idea of these these copy and write file systems and what was cool is this opened up I think it has a lot to do with a lot to do with capital L a lot to do with Docker success in that it used these copy and write file systems what do you call Union file systems out of the box and the idea of Union file system is everything's layered so you're always instituting a layer on the thing and you inherit the previous layers and the top layer is the right of the layer so for those of you who work with a Docker you run a you run a container against an image that image is kind of a binary of everything that's been stored that's immutable and then when you fire it up if you go in and start installing packages or making directories that's all going to be in the copy and write layer and then if you don't commit that image or commit that sorry commit that canane or you lose all that right and we'll talk about Docker files in a minute and how you can do an alternative way to do that but the bottom line is that's basically kind of a memory copy of what you have now why is that awesome you can create immutable infrastructure that contains everything the application the middleware all the things that you need are encapsulated by the way I didn't mention this earlier if you didn't know that when you instantiate Docker it can run on bare metal it can run on private virtualizations and KVM it can run on Amazon it can run on GC it basically can run anywhere right and immutable because it doesn't care I mean the only thing it really cares about is the sharing of the operating system so you might want to be consistent there but in general if it's running Ubuntu on bare metal 14.04 and it's running Ubuntu 14.04 on GCE or Amazon for the most part Docker doesn't give a rip right but here's the thing like so you have this image thing with the copy on right so you got the immutability which is awesome but think about like what is the other killer thing of why people like Docker is because you can do these insane things in your CI loop testing and one of the reasons you can do that is because of that copy on right letter I think it's the next slide I've got a minute but there's a great Gardner article about I mean imagine if the testing state of a table you got a Postgres table you want to test that state then you want to crush it and start back at that state again a thousand different variations if all you had to do is rebase the memory like the image is already layered it's already on wherever your infrastructure is probably a laptop or maybe it's in some Jenkins slave that's running a Docker inside of VM and all you have to do is basically set the image level back to the original so now you're talking probably less than a hundred milliseconds you know we're not even loading up the whole thing now we're just basically instantiating back a layer of image lower and starting over you get into some really cool scenarios where you see people telling stories of doing this like crazy web scale horizontal testing it's the eye that really couldn't happen and so to dive back to the operating system discussion what you basically do how many people have built images from a Docker file a little less but it's the same so so this is the Ubuntu this is actually the current 14.04 Vacker file for Ubuntu so one of the things we have is we have this thing called scratch scratch is basically it's an image defined that is nothing really it's kind of a base plate building layering like you can't do a pull scratch and you can't do a Docker run scratch but it's how you can start it sets up the foundation for building for kind of creating a container with nothing and so this example we actually pull if you actually go to Ubuntu that's the same image you would do to create an AMI if he was talking about scratch you would create for trustee in 14.04 you would pull the call cloud image so and if he doesn't know when you do an ad against a tarGZ it just basically untarsen untars it and that becomes the directory structure so you just lay down of course of nothing the instantiation of a cloud image and then they just customize the init and stuff like that because containers don't really have an it inherently and then ultimately batch so in a sense all your Ubuntu when you talk about like Ubuntu you're really kind of some modified version of Ubuntu based on the cloud image and you can go to docker hub you can select any docker file so you can look at sentos or fedora now some of them you have to see you got to go back to the source code of when they built the image like the sentos one it's just a kind of a docker tar file for sentos that gets pushed over and built so we have these things called webhooks we have webhooks implementations so that you can put in your github repository a file to say every time we change this or we do a commit go ahead and rebuild and point it at the docker hub create your image and these actually have to be what they call official images and then there's how many people have heard of busybox now I'm testing the numbers keep going down so yeah so busybox was developed early on as this kind of I mean we wouldn't call it so I don't think there's a phrase just enough container but just enough operating system it really was a really minimalistic operating system early days of docker it was used heavily for training it was used for just kind of you'd certainly use it for verification processes in fact I think the docker getting started you run a whole little world and it basically runs busybox but here's the cool thing looking into the world of you know where we are and where we're going unicernals is that a lot of people will take busybox as a base to start thinking about very small app concentric container images there's been some really good ones where java so the other words is notice I'm not loading the cloud image from your bantu right or you don't see anything here but if you look in there I'm not loading anything from I do this model I'm going to take this base image where they've just put the basics in and then I'm going to go ahead and I'm really only going to put it's really an operating system less operating system right and I'm really what I'm going to do is I'm going to start adding only the things I need and I'll show you an example we did this with so my start up with socket plane we required you to run openv switch so we had what we built is an SDN for docker and so what we wanted to do is ship with our install a container that had openv switch in it so openv switch is the most popular open source virtual switching fabric and we went ahead and used the busybox implementation so we kind of made our own little socket plane one because we had a couple tweaks there but in general we added just the things that were required I didn't do this, Dave Tucker did this we reverse engineered openv switch and found out only the things that you needed for openv switch so we had and we weren't doing it we'll talk about unicorns in a minute we weren't doing it for all the kind of things that are making unicorns top for you right now we did it because we wanted to make it as small as freaking possible because we knew it was going to be something that somebody had to add as part of the installation of our product it had to be there and we wanted to make it as tight as possible to make the image as small as possible so that's what we did but again you saw a lot between 2012 and 2015 a lot of people who would do these type of you know alright let's just roll up our sleeves and figure out what this application needs it's going to be this layer of Java instrumentation this kind of libraries and when it's done and in a sense it wasn't a unicorn because it still ran under the OS level of virtualization under a true host and I'll talk about unicorns in a minute so it wasn't a unicorn but it was very special purpose so there was this kind of special purpose you know and I don't really want to call an operating system right and I just wanted to mention that for those who don't know that it's been around for a while in the Azure but Windows Server 2016 Microsoft has actually written their own container implementation that is 100% Docker API compatible so basically out of the box you will basically have Docker run Docker pull, Docker push you know the image stuff is still not as fluent as because you don't like they're not as going to have as many images and there's still some work to be done on on how to run Docker files and build of images given the licensing constraints but other things too but in general all the things that you know about Docker run all the primitives of the Docker command will basically be sorted and you will be able to create images and in fact Microsoft has even gone as far as creating the nano servers right so the joke is they took the Windows out of a Windows operating system right and then I talked about immutable infrastructure you know I talked about the Legos Martin Fowler wrote immutable infrastructure let's see what am I at I'm at 45 minutes I'll probably go a little fast to get to this I'm going to skip this I have links to all my immutable discussion this is an important paper and it proves basically from really a true computer science Steve Turgat of why you want to do immutable infrastructure and I've done many presentations on this and you know you read it and you get a sense of what's the difference between a congruent and a congruent basically is going to be an immutable infrastructure and I've written some articles so I want to get into Unicronals because we've got a few minutes left right so that's all the buzz right you know so for those who don't know Unicronals and I'm not a Unicronals actually I like you it's thrown on my lap like everybody else but some of you might have been working on it for years but I think most of us are like oh god what's this now now I got to explain this one to my boss yeah so you know Unicronals are specialized works we'll see so we saw this happening with the containers and a special you know purpose like this was building up right and I would tell people in their infrastructure like you know if you want to do this right really go ahead and have an infrastructure you know not even talk about Unicronals if you're going to use Docker get an infrastructure group that builds base images for your organization this whole thing about Steve and do your own vulnerability so I know we're doing some stuff with Nautilus and you can file that project for vulnerability scanning but you know own your infrastructure so like actually you're going to do this at scale start having people to think about don't let people pull images from the wild we don't let people pull good hygiene chef shops don't let people pull chef cookbooks from the wild like they vet them they infrastructure them they're supposed to but the like I say that with containers like you know roll up your sleeves figure out what the base images are for your organization build those make those as you're this is the base Java image that you get for containers and you know if you need something more let's put in your quest and let's create it that way so Unicronals are like taking it on steroids where you're basically going to get a language to take the parts all the parts and you basically build an image that really is the kernel there's no concept of user space kernel space it's just all kernel space again I'm not an expert here but if you think oh my god that's the worst thing that could ever happen you know you would say well think of Erlang it has that kind of model of Erlang where it's designed for failure you know so mileage varies definitely on the type of Unicolonal but the idea is that what you're taking that concept I was describing with containers to it's furthest point we are actually now going to basically develop images that just strip out everything and you can see well I have some slides on the benefits here and obviously I'm sure everybody knows this now we acquired Unicolonal systems were the ones that do Mirage OS out they were doing some of the most significant work with Unicronals out of Cambridge they were here all week most of the class they're either Unicronals perspective right like so we went over this we got like our type 1 hypervisor we'd run a VM with the full stack on top it if we're running containers we're running a host that we could run bare metal host with the container and application in the container find the image or what most people do today unless you're really really scale and you've invested human capital people run containers in virtualization environments with the isolation VM and then you either throw a whole bunch of containers in it or in some cases the guilt story they run like they actually run an Amazon image per container so they get real they get like a T1 small I forget what they get they get a real small image and they just their microservices base and now what we're going to see now is this weird thing where like everything's gone right like now we just have the hypervisor and this Unicolonal out and that has been compiled and I'm getting some weird spaces faces who probably have never heard of Univirals like they want to come up and choke me but we'll talk about why you might need to do that in a couple of minutes but imagine this too there's always upside down side like anytime new technology comes out like it sounds awesome it's not binary right there's risk reward but some of the upsides there's plenty of downside I'm not going to smoke them anyway but some of the upside is this idea now people can do actually time compile of an application you know before we had this static image whether we were using big VMDKs or even if we were using Docker with the host image and then we thought we were doing just in time with the application because we could take what's in git we could file it running bang and we could do a thousand deploys day or we can do Amazon blows to one point they did a thousand deploys in one hour right we do talk about just you know people develop an infrastructure really fast in that like all that underbelly stuff is also just in time right it gets really crazy right and this is this is a good paper it's any blog article you start reading about Unicorns it kind of points back hey you probably should read this paper before you read my blog and it goes into like the history more than you want to know anyway just get a sense I'm not going to go through it in gory detail but you're basically going through this compile process and then you're creating this image that's what's there you know lots of vulnerabilities there are certainly are other Unicorns other than Mirage this is some they all have some form of language that they use to build this matter this is a scenario everything is compiled you're not pulling somebody else's base of something and then only building on top of it you're building the whole thing that is a VM now but not a VM in the traditional sense of like a VM where or as in it is a VM in the sense that the hypervisor is going to launch this thing and there is no operating system in theory of course most of the testing development is going to happen under a VM right now but in theory you have a bare metal machine you have a type 1 hypervisor and you run Unicornal VMs and then the Rump kernel is another one that should be looked at very interesting stuff there with these guys done so and so you know why Unicornals, why is everybody going not so Unicornals right now clearly the performance is interesting right we go from 400 or 500 milliseconds under 100 milliseconds okay you know will that be significant maybe I mean I think the place where you might want to think about significance of that is if anybody has done anything with Amazon Lambda and if you've seen some success in those in certain workflow or certain patterns of work that's where this will make a difference the ability to instantiate something is if it is so ephemeral that it doesn't the fact that it started to stop is almost equal to the time of the application the instantiation time is marital but a lot of people are getting really interested now I think Unicornals are going to you know it's like anything else we you know as humans we bought pet rocks and everybody is going to be running around about Unicornals about my god stop everything and I'm not one of those guys I love new technology but I love it for technology so but there's some truth and there's some non-truth in this the truth is that we are fighting a horrible battle with people who figure out who could get into our systems and attack us and understand our pattern when your system is completely compiled there's no known foot print of directory structure there's no utilities so as an attack you basically decrease the attack vector off the chart right because you can't like I can go into any variant of a Debian system and I can figure out things as an attacker I'm not an attacker but and I wouldn't know the first thing about what the attackers do other than that they screw me up sometimes but the you know Verizon this is this always blows me away 2015 Verizon report basically 97% of every compromise so they have a pretty bad summary that they did survey 97% of every compromise was due to 10 known CVs if you don't know what CV is it's in this database they tell you here's the one that really kills you not just a 97 with that one down to 10 8 of them were over 10 years old this is why people are going to get really excited this is why your boss is probably going to come from some Gartner seminar and scream at you why we aren't running Unicorns just they're going to hear this like there's no starting points for a balanced character is amazing where's the puppet he's been involved with Unicorns very early on he's been with Docker very early on I wish he were the Docker you know but he gives you the good and the bad I love his quote he's saying somewhere along the way why are people going crazy part of this is a numbers game you run a reasonable system you might need 50 different services and 200 packages and the attacker has to compromise but he does the counter balance he's got another blog article about security and what he thinks and again it's not all it's not binary it's not black and white but here's some examples like now we're talking about again it's right a DNS server that's 446 it's a web server I mean this will make a difference things that start up in less than 100 milliseconds are at this size but it can run only days of what Netflix was doing with Amazon in terms of auto scaling and infrastructure none of those three are even close to that level of what some of these really good implementations of auto scaling infrastructure and true orchestration modeling of infrastructure and on it infrastructure is going to be even worse because we thought we had it it's going to be twice as three five times as hard for unicarnals if you bargain the next slide will make you laugh but maybe I can guess what the next slide is I will send you something really cool but it gets on to but on the upside I said opportunities like things that we can make better even though they're bad now if you like infrastructure this is your baby you're not changing anything there's no tools there's no utilities all the bad things that you can debug those are all the good things if you like commutability you're a win this was the slide I was hoping somebody would guess Brian Cantrell I think he's the funniest guy on the planet he's actually a very personally nice guy he does rant a little bit so I think when it was sometime this week he went off chart about unicarnals but here's the thing I read it and he's right but the things he's going to lay out and that's the main thing there's a whole bunch of things he's going to say that he's going to have to take back in a year for now when Jorian goes and hi Brian if you're watching because he did the same thing with Docker when Docker first came out he ranted about how Docker was so horrible and it was going to destroy he never was against containers but he thought Docker was not ready for project and in a sense he was right the year that Jorian implemented into their infrastructure in a year for now I guarantee you I guarantee you that Jorian will have a form of Docker's unicarnal implemented into Jorian but he's a brilliant man and read this article because I'm not going to blow smoke up here until you kind of know the best things ever Brian is like 10 times smarter than I'll ever be and his one statement that reigns absolute truth is something we're all going to have to deal with if we want to use unicarnals is unicarnals in capital are entirely undebuggable there's a list of things you're not going to get so I thank you very much