 Alrighty, can I say anything? Sure, sure, I just. Yeah, it's fine. I tuck it off, no worries. Containerize OpenSack, first steps. So, there, hello, okay. So there are many things that we can say about Containerize and OpenSack, and OpenSack is that, I would assume you guys are familiar with OpenSack. Who's not familiar with OpenSack? Okay, so basically no one. Okay, cool. So, container, OpenSack is quite a big service, or piece of software or community, whatever, to deploy clouds and provision infrastructure and other services. So I just have half an hour, and plus I'm Italian, I like to talk about many things and probably ramble a lot. So I'm probably not gonna cover all the details about Containerize and OpenSack, and what I'm gonna try to do is keep it on the basics and mention a couple of things that are important for you to keep in mind where when you are containerizing a service just like OpenSack, or if you're playing too, so you can do it. I'm not gonna talk about any tool. I'm not gonna talk about either triple O and simple containers or coal or whatever. I'm just gonna talk about the basics of things you have to keep in mind if you want to do it even by hand. If my keyboard works, I'll be able to switch slides. It doesn't seem to be working. There you go. That's me, in case you didn't notice it. Thank you all for being here. It is Sunday. It is the last slot. So the fact that you're here is admirable. I work at Red Hat and that is my Twitter handler. If you like to talk, tell me, I love feedback. If you don't like to talk, that is not my Twitter handler and you don't know me. I don't wanna know it. Free to interrupt me at any time if you have questions, I really don't mind. I would actually rather answer questions when we are on the slide than having to go through every slide during the talk. Just a couple of disclaimers. I tend to speak really fast and I tend to speak really low because again, I'm Italian and well, Italians. And sometimes I drop F pumps. So I'm sorry in advance if I do it, but I get really excited when I start talking about technology. So the why is why we're deploying an open stack on containers, why are we doing this? So I'm gonna kind of like mention four or five, I don't remember, four or five motivations for doing it. I don't think this is an exhaustive list, but I do believe those are the motivations at a list. I find more appealing for doing all this. And deploying an open stack is quite an easy task already. I wouldn't say it's easy. It's relatively easy, but it's kind of like a solved issue. The problem are operations, date to operations. So these motivations are actually based on that assumption, which is managing open stack and keeping it running is the actual hard task. Moving from one version to another, that is a hard task. That is not a solved problem. We're still hitting issues there. And then they're not all related to open stack services not being able to actually migrate databases in the right way. But the fact that you have a huge piece of software that is still, it has to run and you don't want to have any downtime in your infrastructure. Hopefully you won't have any downtime. So to maintain an open stack running, that's a hard task. So service isolation. This is one of the things that I find more appealing about it. And one of the things we've been very careful with in the open stack community is not duplicating code and trying to reuse as much as we can from each other, each service in the community. This is great. It helps us out in speeding development. It helps us out in not duplicating and not repeating ourselves. But it is a problem when you create a gazillion of libraries that all of the services depend on and then you collocate those services in the same hardware and then you have to upgrade one of the services. It'll try to upgrade the version of the dependency that the libraries or software it depends on which are gonna cause a lot of conflicts with other services that still depend in older versions. So being able to isolate services, that is sexy as far as open stack is concerned when you're deploying it. Because you can upgrade a single service without breaking the other service that is already running or risking to break it because of some version incompatibility or something like that. It's of upgrades, still in the upgrades train of thoughts. If you isolate open stack and you have a container image, you are able to upgrade one service to a newer version, probably roll back in a more easy way, I guess in an easier way. You still have the problem of migrating the database and doing all the database migrations and schemas and data and if you fuck that up, okay, sorry, if you mess that up and then you'll have real problems. You're gonna damage your data, which you don't like. So that is not a problem that you'll solve with containers. What you will solve with container is the fact that you are upgrading to a newer API, probably. And then if you hit a security issue or something and you say like, I need to roll back, it's gonna be easier to do it. So, okay, it doesn't solve all the problems, but it does solve part of them. Deployment flexibility, it allows you to more dynamically kind of decide where you wanna run these services. One of the things that I like about containerize and open stack is giving up the concept that open stack is this massive, really hard to run service or project and start thinking about it more like just like an application that you can run. I know this is like oversimplifying what open stack is and how hard it can be to run the entire cluster. However, if we start thinking about open stack services as an application, we'll find it easier to reason about the entire architecture that we'll run and to decide where we want these containers and services to run on. I will probably talk a little bit more about this in a bit. And these of scalability, I was saying this is gonna make Nova API go faster. I'm just saying it's gonna make it easier to scale it horizontally and add more nodes if you really need it. Whereas in a non-containerized environment, if you have a very fixed set of controller nodes to actually be able to scale it out, you probably have to add more nodes to it. Whereas if you have a set of Kubernetes clusters, for example, it is easier to just tell Kubernetes, hey, you know what, I need more nodes of this. I need more open Nova APIs running and they'll do it for you and they'll decide where to put it and add more processes of it. And that will help you scale in the entire infrastructure. And again, like this is not gonna magically make a Nova API go faster or glance API being fast when you're downloading, uploading an image. That is not a problem that this is gonna solve. But it is the requirement of adding new nodes or more nodes in a dynamic fashion, I guess. So those are the motivations that I find more appealing again, like I'm pretty sure the list is longer or shorter if you don't like containers, but not my problem. But those are the topics that the points I find more appealing about the whole thing. Now what do you want to, when we talk about OpenSack, an OpenSack cluster and an OpenSack deployment, we're not only talking about OpenSack services, we're not only talking about Nova API or Glance API, we're talking about all the services that OpenSack depends on for functioning correctly. We're talking about the database, we're talking about RabbitMQ, which is known for being a slacker in the entire cluster. And we're talking about Redis, we're talking about Mencached and all these services that OpenSack depends on. If you just scale the API nodes, you're not really scaling the entire cluster, you just come part of it, but eventually you're gonna hit a different bottleneck which are all the services that OpenSack depends on. So when you're deploying, when you're thinking about containerizing OpenSack, one of the questions you'll hit is, so what do I actually want to containerize? Do I want to containerize just the OpenSack services or do I want to containerize the entire thing? Like, do we want to containerize my IDB, Mencached and all that? So you can containerize some of the OpenSack services, you can, I've seen this, I've talked to people that have done this, they keep running the database service and the host and they just containerize the API nodes or they run the database in the host and containerize all the OpenSack services that are not only Nova API, but also Nova scheduler and Nova Conductor and the same for Glance API or Cinder Volume, Cinder scheduler. The motivation behind this is having more control on the database node and where the data is, the same thing for the compute node. The people doing this sometimes leave Nova compute outside containers so that they can control entirely the compute node, they have full control of it and for them it's easier to reason about the storage or feel safer about the storage of all the data and while they're still able to scale the API nodes which are the ones that users normally hit more often for various queries whether you're running instances or not. So like I said, you can containerize some OpenSack services and there are other folks that are containerizing the entire OpenSack cluster living outside all the third-party services. So they would containerize from Nova API to Nova compute from Cinder API to Cinder Volume, the same thing for all the Swift services and Neutron. This is great, one problem with this is that the difference between these and just containerizing some of the OpenSack services is that if you decide to containerize services like Nova compute, you also need to take care of containerizing Liberty and some of the other third-party services that Nova compute depends on. You can't just leave Liberty outside or OBS. The recommended thing is just containerize both things and make sure that they're running in the same node. Don't try to run them in different nodes because that's gonna cause you some serious issues if you have some network splits or something like that. So if you're running all the OpenSack services in container and living outside the database, you need to take care of this. It adds a little bit more complexity when it comes to creating the container and the things that you have to take care of when you're containerizing the service but it'll give you a more consistent architecture because you know all the OpenSack services are running in container and just the third-party services are running on the host. However, my preferred option is just containerize everything and put my ADB and my SQL, whatever you're using, PostgreSQL, put RabbitMQ, Mancustee, and Redis, put everything in containers and it's not really, it's going to change the way you deploy your infrastructure. It's going to change the way you reason about your architecture and it'll take you some time to switch from the mindset of having a bare-metal OpenSack deployment to having a containerized deployment. However, when you do that and you start reasoning about it in a more dynamic way, in a more app-oriented kind of like mindset, it'll actually make it simpler to locate services in different nodes or co-locate some of the containers on the same node. It'll make your architecture entirely consistent, which is something extremely valuable when you're running such a big infrastructure. You don't want to kind of have to deal with, I don't remember if this service is running on bare-metal or if it is running on container. You just know everything is running on container and when you upgrade things, even if you're upgrading your container orchestration engine, you know you're upgrading the thing that every other service depends on. So it's easier to just reason about it and have the consistency in your architecture. So I would definitely recommend this in our experiments. This has proven to be easier for not only reasoning about the architecture, but also building tools that will eventually deploy and manage your OpenStack deployment. So to how, again, I don't want to deep dive into different tools. I'm not here to pitch any tools. There are plenty of tools that you could use to create either Kubernetes templates or creating OpenStack images. There are many different OpenStack images already for containers and different tools that are doing this. What I want to talk about a little bit is probably tips and tricks when you're doing these kind of things. And again, this is not an exhaustive list. We're kind of like short in time, so bear with me. One of the questions is that do you want to use a container orchestration engine or not? Who knows what a COE is? Okay, cool. Awesome. For those of you that don't know, Kubernetes is a COE. It is a software that will schedule containers for you in different nodes, in different ways. Docker swarm is a COE. So one of the questions you'll face is do I want to rely on a COE or do I want to deploy the containers on the nodes that I want to do in a more manual way? There are many ways to do it. You can use Aensible Containers. You can use your own Aensible Roles, but I don't know why you would do that. You can do anything, bash scripts, whatever you want, to deploy these containers even pop it. You deploy these containers in the nodes you want, if you're not using a COE. However, if you use something like Kubernetes, you'll get all these scheduling capabilities and node placement capabilities and a whole bunch of other things that come with a COE that will make your architecture more, I guess, reliable and controllable or scriptable, if you will. Whereas if you run things just in containers on a, I don't know, like a Docker runtime, what is really changing is where you're running the service, what you're really gaining from this is isolation and ease of upgrades. Because the service is still running, you're still deciding where to run the service. You're not reasoning about OpenSYC as an application, you're still reasoning about it as a stateful distributed cluster of cloud provisioning, basically. So, and again, like I would really like that, one thing I'd really like about this is stop thinking about OpenSYC as this massive, safe whole application. I want to think about OpenSYC as these just apps, right? It relies on databases and messaging systems that are gonna take care of the rest of the distributed problems that come into play. So, if you just use Docker or different runtimes, container runtimes, what you're really doing is changing where you're running the OpenSYC services. You're just containerizing them and still deciding what nodes you wanna run them. There are still processes, basically, that are running in your nodes. However, if you use a COE, you're given the decision of where to run these, or most of the decision of where to run these containers to your COE. You can still use fancy node placement, logical strategies if you want to. I would recommend doing that. We'll talk a little bit more about this in a couple of minutes. But you're giving most of the control to the COE, which I believe it is a good thing. From the last talk, give more power to robots. To label your nodes, you want to label your nodes. What labeling nodes means is you still want to know what your compute nodes are and what your controller nodes are. The difference is that you're not really, you're not entirely attached to this concept of controllers and computes and storage nodes. You do want to label your nodes because you want to tell, that you don't want Nova compute to run, well, at least I believe this is how you should do it. You don't want Nova compute to run in another node that is not what you consider your compute node. There are people that are running Nova compute in a different node and they just have Libvert running in the compute node and just talking through TCP. You can do this. I don't think it is a good idea because if there's a network split, then your Nova compute service won't be able to talk to Libvert and you'll still have a Nova compute getting requests from other services when you're trying to boot instances that is not gonna be able to boot anything. So I believe as far as open stack is concerned, Nova compute is the compute node and not exactly the hardware. So you wanna collocate those in the same node and this is where labeling your nodes comes handy because you can tell either to your scripts or to your COE, hey, I want to run these two containers in the compute node, don't run them anywhere else and you want to label your controller nodes or worker nodes, let's call them worker nodes. You want to label your worker nodes because it is easier to tell your scripts or your COE where to run things than telling it where not to run things. So you just want to run a service in your set of worker nodes. It's easier to tell that to your scripts than telling it run it anywhere but in this set of nodes. So it's easier if you just label them and you know what your nodes are about. But don't overdo the node labeling. Don't try to label everything. Try to keep genetic labels. Again, if you start labeling all your nodes with extreme granular labels, you'll end up having a very strict architecture that's not really gonna help you. It's not gonna make, you're not gonna take advantage of all the power that a COE will give you or having things in containers and a more dynamic kind of like deployment or architecture will give you. So trying to not overdo this and be more flexible about your architecture and start thinking about OpenStack as an application that you can run pretty much everywhere or anywhere actually. Well, everywhere if you want. One container per service. This has nothing to do with the philosophical discussions on whether a container should have more processes or not. This is just because it is easier to reason about your OpenStack architecture if you just have a service for, sorry, a container per service. And again, like a good example would be Nova Compute where Nova Compute depends on lip-bird. If you're not running a lip-bird, well, assuming you're using KVM lip-bird as a driver for Nova, if you're not running a lip-bird, you're not gonna be able to run any virtual machine. So one thing that we did, this is probably, we took this from the Color Project and it is that we have a container for lip-bird and another container for Nova Compute. We could have run those in the same container, basically, if we wanted. And just have Nova Compute talking to the lip-bird socket instead of talking to the lip-bird through TCP. And I guess we can still do that by mounting the socket in the Nova Compute container if we really want to. However, not having them running in the same container makes it, again, gives us more flexibility and the ability to upgrade just lip-bird if we need to upgrade just lip-bird or upgrade just Nova Compute if we need to just upgrade Nova Compute. Sure, we have more images and the process of building images is probably longer and we need to maintain more images and all that. And if there's some upgrade that needs to happen throughout the entire architecture, we'll have to upgrade more images. This is probably one of the downsides of doing something like this. But again, it'll give you more control on what is running, where it is running, and how to upgrade each of the services. Try to avoid container placement. Again, it is fine to place your Nova Compute and your compute services or your volume services because you know what servers you want to use for compute, you know what servers you want to run your virtual machines on. But for the rest of their architecture, my preference, at least, what's proven to be good for us or in my experiments, at least, is to not try to place all the services of OpenSack, just like if you're using a COE or your scripts that'll decide dynamically where to run the container, just let it do it for you. And again, think about OpenSack as an application that you can run anywhere in your architecture except for a few services that really need to run in specific servers. Trying to rush through it because I'm running out of time. So separate bootstrap tasks. You can either build a container that will initialize the created database, do the thing to be, and then run the service or you can separate the bootstrap tasks for each service into different, either containers or steps in your deployment tool and in your deployment process. I would recommend separating these tasks. This is what's worked for us very well. The reason is that if you run the service, I mean, you don't want your process, I believe at least, you don't want your process, your Nova computer, your Nova API process to just die because the CNDB has not ended. If you separate the bootstrap tasks from the containers or the processes that are running the containers, you'll get to parallelize the entire deployment process easily and you'll be able to run all the API services and all the open stack services, even if the database has not been synced already or even if the end points have not been created in Kiston yet. And as soon, you run everything, you basically sell your scripts or Kubernetes, hey, this is a set of containers, pods, whatever that I need to run. And these are the jobs that I need to run as well to actually make sure that all the databases have been created and all the TP-syncs have been executed and that my sister is in a state that I want to. Then you can run all the checks to make sure that your service is, thank you, that your deployment has succeeded by checking either the database or the different set of services that are running already. Use embedded DNS, try not to depend on the IP addresses, but rather depend on the DNS that are provided by your COE or your entire infrastructure if you are running on your own runtime containers. And this will make easier for deploying your architecture because you don't have to go and query the IP address for each container, you just know that that's the DNS name that your services are going to get, like either Glance API or Nova Compute or Nova API, and you just pass that to your container. And don't abuse mounts. It is very easy to start mounting a lot of things into your containers, even in Nova Compute that you could mount, deliver, socket into Nova Compute, but again, like you don't want to do that. It makes, it tags both containers together, which is probably not so reliable. And I think that's it. If you have questions or you want to apply to everyone that has just come in, feel free.