 All right. Hi, everybody. Sorry for the delay. Pardon our technical inefficiencies. New computers and old video systems. We tried many dongles. It ended up, I mean, needed a different computer altogether. So I'm Forrest Carpenter. I'm an engineering manager with the Cloud Platform and Services Group at Cisco. And I'm Robert Starmer. I run a technical consultancy. We help people build clouds, determine what kind of clouds to build, et cetera. And so we are breaking a cardinal rule of never present from someone else's slides. This is a variation. We're presenting someone else's talk. Yeah, so we thought we would talk a little bit about containers. And the title is Will Containers Save Us? I think we wanted to talk a little bit more about how containers really fit in the community that we're talking about, specifically the OpenStack community. Yeah, they're definitely, it's a loaded question. It fits that paradigm of if it has a question mark at the end, the answer is no. But the answer is no with nuance, because there's different use cases, different needs, different requirements, and containers fit a number of those. It's a matter of assessing out which one you want to apply it to. So Robert? Yeah, I think we can start first just by talking about what containers are. Because I've seen a lot of people talk about containers. Containers will save us, because it's the future of all technology. This is often coming from an application development team. And they're looking at it from the perspective of having had to manage, potentially, from an operating system level up their infrastructure. And one of the things I often see people think about is they think of a container as a mini virtual machine. And you can build containers like that. And in fact, the LXC container environment was a lot like that. You sort of built a full operating system environment just to run your application. But I see containers as a fundamental shift, because they're really more, at least the way I've seen them being used effectively, they're really more about an effective packaging and management technology, and a more efficient way of managing upgrades, and potentially even downgrades of the applications that we build. So that's, I think, a fundamental thing. If we start thinking about these as a packaging technology for an application, rather than as a replacement for a virtual machine, I think we immediately get into a better thinking place about what containers really mean. And I think that people think of containers and how they can interoperate. And you can kind of mix and match a little bit. Whereas if you're coming at it from a pure virtual machine perspective, either you need orchestration of a number of virtual machines, or you need to bake everything into your virtual machine. And then there's a lot of overhead in wanting to upgrade one particular service, one particular piece of technology within that virtual machine, and it has a cascade effect. Whereas containers have the promise of a little bit more easy interoperability and a little bit more plug and play nature to it. Right, and I think that that's actually a key point. The packaging is really important. And the segregation that a container provides looks somewhat like what a virtual machine provides from a single application point of view. For those of us that have built open stack environments, or built some of these massively complicated environments that all live on one machine, often on one physical resource, we've seen the interaction of trying to upgrade one component. So for example, you try to upgrade Keystone because Keystone has some new authentication features that you want to use. And having the rest of your environment, which you don't want to touch necessarily, blow up. Why? Because Keystone needed a new library. Keystone needed something different than all the other tools that you were currently running. Or trying to upgrade and not being able to because of a collision between different libraries. The nice thing about containerization is that we bundle the specific libraries that that one application requires in that one image format. And as you say, you bundle your libraries together, but you don't necessarily have to be concerned with the underlying operating system of a virtual machine. You don't need the expertise of saying, oh, I understand rel enough to build these virtual machines. But now I've got this other package or service that is really engineered toward Debian. Now I need to learn all about an entirely different distribution and build that virtual machine. Theoretically, all you need is just the bare minimum of the libraries, like you say. And then you can package it up. And we're actually seeing a shift. If you look at even what Docker, the Docker community has done, the initial container models for a lot of the services that you were getting from Docker started with a container that was already a model of an operating system. So you started with a CentOS base or a rel base or a Debian base image, which was already on the order of 300 or 400 megabytes. And it had a bunch of utilities. All of the command line utilities that the sysadmin in us likes, because I can sort of get into the container and still type ls. I can run some bash scripts to fix something that's within my application. But what's happened is that as the community has learned how to use the containerization of these resources more efficiently, we're getting to lighter weight operating systems effectively as our base. So people are talking about using Alpine Linux. Well, you might not want to use Alpine Linux as your base operating system for your entire environment, but it's a nice compact Linux environment that in the case of a container doesn't even have the kernel, and it has very few of those additional utilities. So you suddenly don't necessarily have a shell to work with, because you don't actually need it if your application is just trying to run within the container. So there's a big shift that's going on in this sort of condensation of what was a virtual machine down into getting to just the pieces that your application needs to run. And it is changing how we manage and operate our environments, too. Debug certainly changes once you put things into containers. And it's something to be aware of. It's not the exact same world. Like I said, once you containerize that resource, you probably don't have the tools you might have had in the past to debug what was going on if that resource breaks. So it's another thing to sort of be aware of. And you mentioned the sys admins and all of us, and DevOps was a big sea change a few years ago. Now we're all sitting here talking about Kubernetes and Docker and containers this year, and actually years past as well. But the sys admins, too, from that perspective, using the containerized services allows them to have consistent environments, bare metal environments, operating systems that they are familiar with, that they prefer, that use the debugging and the different utilities that they are familiar with. But then they can containerize the offerings that they're putting forth to their consumers, their internal customers or external customers, and still have the flexibility of offering the things people want while being able to maintain their data centers in the fashion that they're familiar with. Yeah, well, I mean, one of the things that comes up when people start talking about containers is a question of security. And that simplification of the underlying operating system, because you're still going to be running an environment. Hopefully it's massively stripped down, because really all that you want is the environment plus the container operating engine, and potentially a container operating environment as well. So you want something like the Docker Demon services to actually run your containers. And then you might layer on top of that, the most likely candidate today is Kubernetes, to actually manage containers across multiple machines. We can talk about also how this sort of interacts when we start talking about using OpenStack as a containerized application, sort of also fits into this. But the point I wanted to make was that at the security level, even though there are still concerns, although I think they're pretty much being eradicated in terms of the security of an individual container and the container's service, if I can remove a number of the resources that used to be in the underlying operating system as well, because I don't need them, I can potentially make the operating system more secure as well. So you get multiple layers of security that become more efficient, easier to manage, easier to keep upgraded, and that's a huge benefit. Yeah, I think at that point you almost distill security down to really a question of networking, because that seems to be the one thing you really can't get away from and that you really do have to put a lot of thought and effort into is what sort of service, what are your services networked to, what can they access, what can't they access, whereas the processes themselves are pretty well squared off. But networking becomes sort of the thing you really want to put the most scrutiny on. Yeah, so that's sort of some of the basics about containers, right? So what are these pieces, what are some of the value propositions of containerizing services? Now if we apply containers to something like OpenStack, right? OpenStack is a distributed application. We happen to often deploy it as a distributed application all running on one machine. How many of us have built an all-in-one OpenStack instance? Everybody, right? Built DevStack, right? Everything is running on one environment. So that means that all of my 75 or 80 plus processes, each with their own requirements of potentially slightly different libraries, are all running on one operating system, right? If I containerize that, I can either do a sort of VM Lite version, which is, again, the thing that I don't think you should do, but I could say I'll take all of my Nova services and act as if I were to put them into one virtual machine. And that was actually a classic process even a year and a half ago where you would say, well, I'll still provide some segregation of these resources so I can do something like upgrade Keystone or upgrade Nova without necessarily impacting the rest of my environment. And my first level of segregation is virtualization, right? But the next step I could take is to actually map all of that into a container. Again, this is the thinking of containers or just Lite VMs. So there's an approach. It does work. It's not necessarily the most efficient, though. And so if we really want to use containers to their fullest, we start looking at individual processes in individual containers. And that's where we get to 70, 80, 90 potential containers still might be running on one machine providing that initial resource. And our interaction with those resources then is via network connectivity. So we're either talking restful interfaces, so the APIs that we know and love within Nova and Neutron, et cetera, or the back end processes, things like Rabbit providing its communication message bus services. But they're all coming out of individual containers that we can then upgrade potentially more at will. And so the many containers on the one host model as you're describing it, like you said, it's primarily, it's the virtual machine Lite. It's essentially using the same, I hate the word, paradigm just with a different technology, right? But once we have that working and proven, then we can really take advantage of containers and where we go to something like a Kubernetes or a Dockerform or some sort of orchestration system where now we're distributing it across many machines, we've got the high availability built in, and all of a sudden this is where containers really shine and bring us some value. Right, and if we look at the sort of the migration of process there, when we built systems that were monolithic, so all of our processes on one machine, we've probably used a tool like Puppet or Chef or Ansible to distribute those processes and distribute the application components onto all the physical resources that were actually running our environment. So classic model of three controllers, I have to have something replicate my control plane resources three times over. The same thing is true for containers. If I want to have three controllers, I'm going to still have to deploy the containers three times over, and I can do this manually using effectively the same model that I used to install the initial packages, right? Because often I would say, hey, go ahead and do a yum install Nova API. Well, I could do the same sort of thing and say Docker pull Nova API and then Docker run Nova API to get that process started. And so I can automate and enable all of these different containers on the specific hosts that they belong to and build out my distributed processing environment. One of the issues that comes up though, of course, is I have no way of managing those containers after I've launched them, other than again, writing additional Ansible to try to keep making sure that my containers are running and my services are running and the systems are running. To say nothing, I'm upgrading the services. Yeah, and if I want to upgrade them, I have to deal with this exact same sort of process. And this is, I think, where the container operating environment of specifically Kubernetes comes into play. So if you've gone to a bunch of the other sessions, which I'm sure many of you have, one of the sort of the key selling points that people are talking about today is OpenStack on Kubernetes. So the containerization provides value just because we've simplified the process of upgrading a particular component within our system. We've bundled all the resources that are what requires all the libraries together. The next step then is to say we can deploy these and we can use tools like Ansible to deploy them. And the call Ansible project is one model of doing exactly that, getting these containers out onto the specific hosts that they need with the configuration that they need and getting them running. We add the next layer, which is Kubernetes component on top, and now we have a management model for all of these distributed components. And all that Kubernetes is really doing is making sure that the particular version of a container that we want on a particular host is running where we would like it to run. And that's incredibly powerful because now if I lose a machine, I can replace it and Kubernetes can actually do the job for me of making sure that the processes get restarted on that new machine. So it becomes very powerful from that perspective. Absolutely. So at what point do we stop layering it? We've got the Kubernetes OpenStack, Kubernetes on top of that, we could just keep going forever. Well, I talked to somebody just the other day that said that they have Kubernetes, managing Kubernetes to manage Kubernetes, which can then run OpenStack, right? So we're starting to see multiple layers already. And it's kind of interesting because you can start building domains that have these different resources. A number of the folks that I've talked to that are looking at deploying OpenStack really wanna use it as a way of managing their entire data center, from bare metal to virtual machines to containers. And usually in that kind of a conversation, we start talking about things like the Magnum project as a way of launching a Kubernetes environment on top of OpenStack. So this is the other model of using containers. Not so much containers for managing the infrastructure or a particular application, OpenStack in this case is just one of the applications that you can install this way, but managing a custom built application on top of a flexible environment. So you do sort of get into a chicken and egg question, right? Which one do I want first? Don't wanna have Kubernetes and then OpenStack or do I wanna have OpenStack and then Kubernetes? Do I wanna have Kubernetes launching OpenStack, launching Kubernetes? I think there are lots of different approaches to this, right? And to me, the biggest value here is that having Kubernetes on top is the thing that the application developers are looking for. They wanna start building applications using Kubernetes as the model for their application deployments. If you're an operator and you find that something like Kubernetes makes sense for you, for other applications as well, it might be viable to deploy just one initial Kubernetes layer and then do something like layer OpenStack on top of that. And I've certainly seen companies that are banking on that model as the right approach to deploying these services, right? I'm familiar with one or two of those. Yeah, there might be a couple. But so even if you said, well, I wanna get Kubernetes down first, we still have the classic chicken and egg problem. How do you get the systems launched? So we have to have some way of building out our bare metal environment. Then we have to get Kubernetes installed and we have to have something monitoring the Kubernetes environment. So you have to think about that whole process. But once you have it, the models of containerization and Kubernetes driving the containerized processes, even though it's only two and a half years since the initial release of Kubernetes, it's actually become a very powerful and stable platform because they're very simple primitives driving that. But I don't think most applications, most enterprise environments are quite ready to say, oh, let's get rid of all of our virtual machines because we just managed to get some of our applications into virtual machines. So we still have a ways to go before we can just say, let's scrap all that old infrastructure. So I think there's still a lot of value to having an open stack layer, whether it's the very bottom layer in your infrastructure or sits on top of a Kubernetes infrastructure. Again, it all comes back down to what is the use case that you're trying to deliver. And it's not going to be a universal answer for any one situation. And luckily these tools interoperate so well that depending upon whether you need the portability at the base layer for upgrades and deployments or whether you need it at the application layer for your application developers who can't possibly have the bandwidth to understand their operating systems, there's a way to configure it that will line up for what you need to deliver. And I'm seeing people, especially when they're building very small systems that are not expected to grow, the largest I've seen that I think fits in this model is usually like about 16 compute nodes, maybe an equivalent size of storage because often these are big data targeted application environments. You might want an open stack environment running that just because you want flexibility in rebuilding the system as you understand which applications you want to use to run the environment. It's maybe a little bit more efficient to do this with virtualization. And in that case, I've seen people very effectively just use the call of Kubernetes model or rather the call of Ansible model, which is just let me get my containers built and then deploy my containers sort of as a one shot deal to get an open stack environment running. And then I'll use open stack to manage the rest of that infrastructure. So for small systems, that's pretty effective. Once you start looking at larger systems though, I think the benefit of having the Kubernetes underlay to help operate the open stack system that is then operating potentially thousands of nodes is incredibly powerful. So there is sort of a break even point where I think below it maybe you don't get a whole lot of advantage of having a Kubernetes layer and above it, you certainly start to see the benefits, especially in terms of keeping the system upgraded, keeping it at the current state that the environment needs to be. Expanding a little bit on what you just said there, I think there's a certain value in having the Kubernetes layer at the bottom where we have this enormous tent. It's not even a big tent in open stack of all of these different projects and services that are available to us. One project. And having that Kubernetes base layer allows you to say, well, I need my Nova and my Keystone and my Cinder, but I'd like to experiment with heat or I'd like to try out Trover, I'd like to do this and you can just sort of pop the container up, see how it interacts with things, decide whether or not it's good for you and then it's a little bit more user friendly. Well, and the containerization segregation in the past, if you did that, if you deployed heat into your environment, it was not necessarily easy to clean up what it used to actually get itself installed. Once you've gone through the process of containerization though, it's very easy to say, I can just remove that container and it won't remove the database resources that it may be created, but those usually are bundled fairly tightly into a specific database resource that you can drop, but it will get rid of all of the application code, all of the additional libraries or the specific libraries that that model of heat required, right? The other thing with containers that I think falls into that same sort of line is that if I decide to try to upgrade heat, right, I kind of start going through the same process. I have to get the software installed. Well, in the container model, all I have to do is copy the latest image onto the system. I can do that with something as simple as Docker pull or I can use the Kubernetes engine, if I've gone that far, to actually automate the process of trying to roll forward to a newer version. If something breaks, the nice thing is I can roll back and the restart time, this is something that often people talk about, you know, we always talked about how virtualization was nice because it would take 15 minutes to boot a physical server if you were lucky and it would take a minute to boot a virtual machine. Well, I can take that down to microseconds to launch the process in a container because I'm not booting an operating system, I'm just kicking that one process off within the operating system and that's as fast as any other command we might run within a particular machine, right? And because of that speed, if I upgrade to a new version other than things, again, like database manipulations that may have to go on, the process of turning on a new version is very fast. The process of turning it off and turning on the old version is just as fast. So it's another very powerful value and one of the reasons that so many people are saying, hey, containers are something we have to figure out how to leverage in our overall operations paradigm. So will containers save us? Yes. I actually, no, I'm actually quite serious about this. I have, you know, I used to build OpenStack systems and it would take days, right? We got to better and better automation for that and we got it down to the order of hours. But it would still take four or five hours pulling packages or pulling source code from a remote source to get an OpenStack environment built. With containers, if I have the containers pulled down from an online resource, I can build an entire OpenStack system in about five minutes. If it doesn't work, I can tear it down into the same amount of time, if not even faster. And that gives the ability to, from an operational perspective, to start cycling through trying things so much faster that it makes it possible to understand how things break. When things break, you don't necessarily have to fix them. You can often just replace them. It changes how we think about infrastructure as a manageable service. So I do think containers can actually save OpenStack from the blight of OpenStack is hard into something that is OpenStack is more manageable now. All right, so it's actually a very powerful tool for us. Yeah, I think that iteration improvement is really a value that's sometimes overlooked and is definitely, at least to my mind and to the applications that I'm responsible for delivering within my company and things that I'm aware of in that space. It's really the value for me. It's not the end user flashy, oh, I can just deploy a container thing that I was at DockerCon a couple of weeks ago. That was really great to see the things that they could do, but I was more interested in it from that ROI perspective on having my teams being able to deploy and test and iterate as quickly as possible and not have the overhead like you described. For me, it's the cost savings and I think that's where I can use the word save us very literally. How are we doing on time? Do we know? I don't know, but do we wanna have questions? Does anybody have a question? Was this useful? Are you guys awake? It's Thursday, it's after lunch. Kevin has a question. Oh, I'm not sure we're gonna let you have a go. They're gonna be a crappy one. So talking about that immutability aspect of it and just replacing a broken container, what do you see as the impact on the upstream code base if your operators are no longer debugging these things and they're just saying, screw it, throw it out and give me another one? That's a good question. So I think the upstream community is there's still going to be debug. I don't mean to say that the only answer is if it doesn't work, just get rid of it and replace it because at some point we still have to figure out why did it break so that we can figure out what to replace it with. I was thinking about it more from the perspective of especially if I'm trying to upgrade or if I'm trying to change from one version to another. I have the ability to say that didn't work. Let me just go back to the previous version where I know things did work. Now at the same time, many of the processes are not even quite that simple to upgrade. Often if I'm going from one release to another of OpenStack, I'm gonna have to change how the database is structured somewhat. And that's going to add a bit of complexity. I think what we are going to see though is because of the speed of that process, because it doesn't take me an hour to build out the latest version of OpenStack and upgrade all the services, when I go through that process in a test environment, which is potentially a much lighter weight environment now too, I can understand what the database changes look like. I can understand how to validate that I can actually back out from that before I actually go and try this in my production environment. So that's gonna be important. But we can also start testing multiple versions. The last operators mid-cycle in Milan, one of the things that came up where people were saying, we finally got to a point where we could do an upgrade and it would work. And then the next version came out and we tried to do that upgrade, which would work if I had started blank at the current version, the now current version, but that breaks. But I think it's going to be easier for us to start testing those multiple release cycles and making sure that they work before we put them into a production environment and have them break there. Yeah, building on what Robert said, I think the iteration actually should provide us with more information, more data, more understanding of how different things do and do not work well together. There's, of course, the onus on human beings to report those things and fix those things. But I think back to, I was at VMware years ago, we had a giant matrix of all the possible guest operating systems and VMware offerings and we had to test them all. It was a commitment of time and resources to do that, that if we can iterate more quickly and we can see these processes really exponentially speeding up, then we should be able to collect more data and actually improve things faster if we're exposing where things aren't working well together. The other thing I would say about the container upgrades or the debug process, debugging containers is not the same as debugging processes on a normal operating system because often the container has segregated those processes. That's part of what containers do and understanding or getting into the container to see what's going on if you don't have the right tools installed is potentially an issue. So it's something that you have to understand the new model, new paradigm of doing that debug because it's still possible, right? And you can still get that information for the developers that they can help you fix it but it just requires a new thinking about how we're actually doing that. So just something to be aware of. Question. Yeah. If you have a container registry and you basically just pull your containers and install them then of course containers are great operationally but in real life I guess often you will have to also build your containers you have to maintain them and I guess in some way basically the problem then just is shifted from having to manage your installs and your upgrade scripts to having to maintain your build environment and all your Docker files or whatever. Is that something that we should be concerned about or do you expect that basically the upstream communities would provide those containers and whoever's using them would always just pull pre-built containers from some public registry or from whatever vendor? Yeah, I think you're gonna see both models. I think you're going to see some people, again the simple use case, the Kala community for example currently does have a current version maintained set of packages. I actually haven't talked to them to understand what they do about like interim patches so like patch releases if they're updating the containers for that. So that's maybe an open item there, right? But I do think there's an expectation that there will be somewhere a well-known source much like the cloud images that exist from a lot of different sources that that will continue to sort of exist. At the same time the containers may not be exactly appropriate for your environment. A very common one, picking probably the hardest project of them all, getting Neutron to work appropriately for your particular environment potentially integrating your networking services often requires building your own container from a specific model. The thing is that that's a model that can actually much more easily be integrated into a continuous integration process without some of the headache of the classic packaging problems that I think people have run into in the past. Making sure that you have an RPM packaging environment or a Debian packaging environment often was yet another whole orchestration and operations project that had to go on whereas a CI containerization process I think is a fairly lightweight service model. So building your own containers, having your own container registry isn't as big a problem in my view as the classic packaging process of the past. They're probably of similar scale though to a certain extent. The biggest advantage though is that when you then go to try to apply those packages to your production environment or through your staging into production environment the containerization of those resources means that you have sort of a known working set that isn't going to impact any of your other known working sets directly. And again that's the container advantage that's why I think so many application developers love containers. I know that I have the right libraries, the right binaries bundled together to run so long as my underlying container service runs. That's kind of what I see. Run Viko. Great talk. You brought a nice point about pushing intermediate patches to registry. So today in call of the PTO pushes just one day at the end of the release just once per cycle and there's a plan to automate pushing master every time container changes so in the form of intermediate builds and pushing the patches. So how do you think we can automate that or even use it in a production environment where like do we need to even try to build and push intermediate patches and how do we do it with multiple developers and even upstream is figuring this out. You have any thoughts about that? That's definitely an interesting problem because one thing is just because you've pushed a new container into let's say it's the Docker hub registry that doesn't necessarily mean that I even know about it or that I have a hook watching for that change to occur. On top of that I might not wanna just roll out that updated container unless I have the process in place to say okay let me roll it out and try it. Again if it's just a patch I'm not likely to be changing database structures so it should be something that I could very easily roll in and out if it doesn't work. But having that process is something that I don't think that's been fully automated in the call a project for example or any other project that I've seen to sort of maintain that relationship. And another question is like even if you automate it how would we notify the users who are to consume the patch so right now there's no like email mechanism and registries like let the users that new patches have been pushed so the upstream is also thinking about like pushing our email notification. I think the term that we all like to use now is microservice right? I want a microservice that. Microservice and email models. Cool thank you. Yeah all right. I think that the right probably answer there is if we can get to it having an API service that you can call a hook against and say hey I would like a notification message anytime something changes. In the long run that's probably the most efficient way but is that going to work for everybody? Probably not. There's some people that have restrictions about what they can see on the public internet and all those sorts of things. So we may have to poll. Exactly so you might not be able to poll either right? So okay. Well I think we're about at time. Any last questions? Okay. Thank you for sticking with us and thank you for coming.