 Hello, everyone. I am Eric. I'm just some guy, but I've been working with containers for quite a while. I have a Twitter handle if you want to bother me about things after this talk. This is going to be a brief romp on containers, a little bit of history, not a lot of technical details. No CLI flags were harmed in the making of this presentation. What I'm going to talk about is what changed when containers took off, the big picture, the semantics, why things changed, and what we can learn from that. I'm actually going to try to target things that are not container-specific. I don't want to get bogged down in the details of namespaces. I want to pay attention to the user experience of containers. Can I actually change slides? Technology. This is the first time I've used Google Slides. The one thing I'm going to take as a ground truth is that containers are on to something. They came out of nowhere. In a couple of years, they are so ubiquitous in our industry and our conversation that I don't even have to define the word, do I? About half the talks at this conference are about containers. This is kind of funny because none of the technology behind containers is actually new. A lot of people at this conference being very technical probably know this already, but C groups are not exactly new. Resource limits not exactly new. The namespace features of the Linux kernel they've been around for a while. Other operating systems have had similar things, too. But containers as a wave, if you'll let me use that term, really took off in like 2014 or so, maybe 2013. And so what I want to focus in on is what changed around that time. Was it just marketing? Probably not. It took off hard enough that something's got to be behind this. So the big question is, what actually? What changed in 2014 that made our whole industry go, whoa, containers? I don't know if anybody's ever seen the video of Burn After Reading, but there's this nice little scenario where the intelligence agency is talking about what did we learn from this whole experience? And it's, I'm not sure, don't do it again. What's it is a whole question. I think we need to stop and ask ourselves some really basic questions like that. What is containers? What is it? What changed? What problem are containers solving? There's probably a couple of different answers to this, but I want to know what the biggest ones are. Let's see. Can I have individual slide transitions? Yay, okay. Did containers catch on because of security? I'm going to say no. I heard a no in the back of the room. Yes, I agree. Security features have been around before containers. The buzzword came way after the core features. We've also had other sandboxes for much longer. Some folks in this room might want containers for security, but that's not what took off. Another way we can point that out pretty easily is one of the most popular container engines, when it was beginning to take off, would pretty much open a TCP socket on localhosts and let you do things. This was a good time and got corrected fairly quickly, but during this popularity wave, CSRF on localhosts could root your box. Whatever. No, security is not why containers took off. Is a resource limit? Maybe a little. C groups and stuff came into the Linux kernel because some big cloud providers wanted more resource management. This was definitely important, but again, it predated like C groups are a decade old or more already. They didn't happen in 2014. They were not the independent driver of this phenomenon. I would say this wave took off when developers became the early adopters. Cloud stuff for containers is beginning to kick in again. Now we've got Kubernetes and Habitat and all these other things, but those are like the last two years or so. They're kind of in the take off mode and the containers wave predates them. So it's none of those things. Containers are famous for shipping. That's a big word. So this talk is going to be the five Ys a little bit. What is shipping? What does that actually mean? We use containers effectively to ship a bunch of files to prod. If we look at what's underneath containers, it's tar balls and trutes. There's a lot of other things too, but it's tar balls and trutes. We've had tar for a long time. We have been able to push bundles of files around since longer than I've been able to use a keyboard. But this is what's fundamentally behind the container wave. So what changed? Is this really going to be this simple bundling files, pushing them around? We've got to be missing something, right? But no, this is pretty much what happened. When containers took off, we got tar balls, some layer formats, a bucket on the internet where people would push their tar balls and then pull them back from in an alarmingly centralized way at that. And we got trutes and suddenly containers. So this seems too simple. This doesn't seem to make any sense. But let's compare it to what shipping was like before containers. It depends on what exactly your user story is. The most basic, simplest user story is probably you had some kind of system level package manager and you would ask it to install things for you. This package manager would select the actual packages that happen. It would install all those things and it would help you maintain your system. And for a single user with a single host, this works pretty good. This got us through like some decades. Not bad. But as time passed, we've ended up layering more systems on top of this. So now, if you're somebody who manages a fleet of any kind, you probably have a configuration manager of some kinds, a chef, a puppet, an ansible, whatever the heck. And that thing has an entire programming language, which we pretty much use to write specifications for how our package manager should manage packages. Wait, what? Yeah, we've wrapped a lot of complexity around this. Why did that happen? Are we happy with that? It seems we needed it because we need a way to tell package managers how to behave reproducibly across a large number of hosts. But I also have this rather snarky comment here of Prey. I think this system isn't working real well. We had some other talks earlier in the day from somebody at Facebook which had an amazing screenshot. I wish I could edit back into my presentation about basically running YUM in a loop. And I don't want to pick on YUM, but this is something that comes up a lot. Config managers tend to have to run the package manager in a loop because the package manager has a lack of item potency. The side effects are global tier system. They are unbounded. They're unenumerated. This is something that's changing with containers. But in the past, this was an issue that we had to deal with. And things would be even more fun if you have to run your own SaaS shop, right? However you shipped things, it might be the system package manager, but for reasons of a whole other talk, it's probably something else. So a lot of complexity was in our old story for shipping. Containers took a lot of that away. They really forced us to streamline by not giving us good tools, which sometimes is a good thing. So we've got snapshots of an entire system now. And this is just always the way with containers, as we've come to recognize that word. And this is different mechanically, but what's really interesting about it is how it forces the rest of the ecosystem to look a little bit different. Now, when we run a dependency system, when we run any kind of package manager, effectively, we've moved out the step where we resolve and pick dependencies. This is absolutely colossal. So now we always, with containers, we have a build phase, and then everything stops moving. It is serializable, it does not change at all, and then we fan out deployments to a whole fleet. This is the real money of where containers changed things. Now that we can ship everything pinned, we've removed a huge source of potential surprising changes and inconsistencies when deploying things. It means rollbacks are guaranteed. There's even some nice little bonus features of the way we ship around tar balls now tends to mean you can just stack some more stuff on top for your local company things, and it sort of works. But the big thing, that dependency selection step moved. There's a couple of other things that changed too. The big picture of this is we can do fleet deploys with confidence. Snapshotting forced it to be something that we can be confident about. There is no possibility left with containers of having an install script at the end. Your dependency manager happened, it's saved, and now that's it. This also cures a problem where a package manager could leave wiggle room. This is something that happens in a lot of package managers. One that bugs me a bunch is NPM modules can install another module and you can have pinned hashes for your first module, but then that thing can list more transit dependencies and it can just say, go get it from HTTP, no S, no hashes, no pins. We fix that problem with containers, you just can't do it. We've also from trutes somehow gained the ability to install multiple versions of programs. This bullet point makes me feel a little weird because why was that hard? I think there's a whole other talk in here, and I would love to hear the community discuss this. But one thing that containers also definitely did for us is make it easier to detect those things now, so I hope that's going to continue to change a conversation about multi-versioning. So containers changed a lot. Trutes and tar balls, seemingly simple, really shifted a big picture. Did we really need containers to get these things? Because we've kind of got a Katamari domicis style, like a big ball of stuff, right? Now we're pushing these massive tar balls around. We're sort of hoping they diff and layer well, sort of and hoping both very, those words are doing a lot of work there. People like the number of blogs on the internet about how to make the right Docker base image. It's rough. So maybe we have some lessons to learn. And I would say the big one is to look at what happened to our package managers and take that to heart. If we make the dependency resolve step separate, like very separate, the system stops, is serializable, immutable, and you can push that state out, you can copy it around, ideally store it forever, then we're going to have much better systems. This is currently really rare, and I would say if your package manager doesn't do this, it is failing you. Those are harsh words, but it's what we're getting out of containers. It's apparently what we've always wanted. It would be really nice if our tools now sat down on how to think about that and could do it better. We don't need this big ball of stuff if we can get the data about package selections to be an immutable artifact itself. We could store the blueprints instead of the buildings. It might be a little lighter weight. It might be nice to work with. Crazy thought. Another thing I think we should probably learn from containers is end users, the people receiving the shipping, really like tar balls. Simple is good. Zero step installs are good. You can't screw them up. They're inherently deterministic. If you're not doing anything, then your install is deterministic, and that is really helpful if you're a config manager or some other system trying to anneal a system into place because you don't have to look for side effects. There aren't any. Containers are also not the first time that users have told us this is good. Installing stuff on a Mac is drag and drop. Drag and drop is a good heuristic for success in your packaging. If there's a script that comes after that, it's complexity. Did you need it? I don't know. If that drag and drop doesn't work to any location but slash, consider filing a bug for your internal team. If you really needed that, please ask why. The biggest lesson from all this is simply, yes, actually, these are problems. Containers sweeping the industry happened for a reason. It's because there was a huge gap in the value of what we have in our existing tools and what containers can give us. There are some folks in the industry who have made really big numerical bets on that. There might be some funny money when you hear billions of dollars valuation going around, but this is a bet that people have made about the gap between where we are and how these tools should behave. It's a huge number. That is the magnitude by which we have work to do. So the big takeaway of this talk, we may need to let go of some idols. We may need to rethink some very basic things about how we treat package management. We have not got it all figured out yet. It might be time for a drawing board session. That's my attempt at what did we learn. I'd love for everyone else to have tons of questions too. I don't think I asked all of the good questions yet, but that is the body of the talk. Thank you. I'll put up a couple of other quick links here if people are bored. There's a podcast called the Manifest FM that some people started about package managers and asking good questions. I think it's really cool and you should look at it. Also reproducible builds are the bomb. Everyone should go back in time one hour and watch Holger's talk about that. So you said that the dependency management problem moved out, but what if your application, for example, depends on a database and now you need to have a database running somewhere that still needs to be something that keeps track of it. So didn't we just move the problem? Well, yes, we always just move problems. I think if I process your question correctly, you're asking about how things should work when we have a whole data center worth of services. Again, I would say microservice versioning is another whole layer of issue. It probably gets a lot easier to handle if we can get systems where we snapshot the whole system dependency instruction graph. I don't know, there's no name for this format because nobody's tools do it, but if we had that per system, then it would probably be a lot easier to build a bigger graph of that snapshot for a whole data center, but I don't know if that answers your question really well. But is it really a different thing or isn't it just the same problem but now with a different name? It probably is the same problem, but baby steps.