 I'm going to talk on one of the use cases where I've used Docker, but I'm using Docker here, but I can mean like Rocket or any other thing which may come up in future, but I mean containers in general, Galera, okay, let me start, I'm from Perkona, we do MySQL development and cluster development, backup tools and a lot of other stuff associated with MySQL. Ah, so this is what Galera means in the language in which it was developed, in Finnish I think, yeah, so that's what it means. It's a collection of people all growing in Unison and it stands for Synchronous Replication, that is all the nodes which are identical in nature with respect to the data. So what is Galera? It follows, it's like this, it looks like there are multiple MySQL nodes and there is something called a Galera plugin for the Right Set Replication API of MySQL and it follows something called Eventual Virtual Synchrony, which is a type of synchronous replication with other two being one copy equivalence and the PAXA style, whatever that is. Yeah, so there are a few concerns here that I've listed and the replication here is data centric rather than your, as usual, async replication that you have in MySQL and yeah, I've written that the causality and synchronous, the difference being that when we say causality and when we say synchronous, when we say synchronous it means that data is present on all the nodes but it does not, we do not say when it will be present, it means suppose you are doing a query A followed by query B and you want the query B to require query A on all the nodes, then that's when you do something called a causal read, the causality needs to be maintained, that's what it is. So I just wanted to separate the difference between the two here so that's why I mentioned it here and the last factor is the latency which is very, very important in case of asynchronous replication because people want data on all the nodes but they don't want the necessary delay there and this whole replication is based on the optimistic concurrency control rather than a pessimistic one. What happens is suppose you do a write and there's a transaction, it goes to remote nodes, this node does not wait for that transaction for all other nodes to acknowledge it because the end-to-end communication in a van or a LAN environment can be quite expensive so this does not wait for that. What happens on the other end is suppose there's a conflict between what it is doing locally and what it's getting from remote end and this is a process called certification and if there is a conflict, the local one is aborted and the remote one is applied so that's how the optimistic concurrency control model applies here and that's how a high throughput is ensured. Anyway, that's about the short description to Galera, the synchronous replication behind MySQL. Now I'm going to talk about containers and I'll come back to that later. So basically what do containers provide as all the talks before me have detailed all the namespaces, the PID, namespace, network, namespace, mount and user one. Out of those things I think I prefer, I really like the concept of network namespaces because each Galera node requires three ports and if you want to set it up on the same box, for instance, you need to mark the range for each of them and it can become tiresome. Having network namespace eliminates all of that and then yes, C groups. C groups are used by Docker mainly for metrics from what I understand and yeah, as I said, good fences make good neighbors for a clear separation of CPU, disk and other resources and for the manipulation of the containers, usually, I mean, there are places where your usual container injection does not do, then that's where you can use like NSenter or NSenter is the command that you can use and that's what I've used for injecting something like Q disk, traffic control Q disk into a container and that's what that was required for one of the QAs that we do and these are the other sys calls that can be used like Unshared for a new namespace for a child and other things and crew as the LXD talk in the morning mentioned, crew can be used for migration of the containers while ensuring high availability and that is something which is very important for us with Galera containers because high availability is one of the goals for a container, for a cluster, sorry. Okay, this is just I'm mocking the downtime veil that Twitter is to show, so this is like the uptime veil, you know? Anyway, so why did we choose Docker and about Docker? So, yes, simplified networking was one of the main things that was required for us. I was earlier using QMU KVM and with DNS mask and it was not fun to play with all of the nuances of the network because I'm not good with that, but again, it's very complex and, you know, I didn't want to get into that. But again, there were other things like volumes that I can share between multiple containers, multiple nodes and the storage options like Butterfs and what is available newly in 3.18 kernel that is overlayfs which also allows for page cache sharing and stuff like that, something that you can use only with use as KSM with QMU KVM, KSM being kernel or same page merging which can be quite CPU intensive as well. So as I said, batteries are included and, you know, you can use different batteries if you want. Okay, so here I've written it as symmetry and replication and deployment. That's what I have written here. So synchronous, I needed, so synchronous replication is with Galera is nothing but there's a symmetry in replication, there's all nodes are equal, there's no leader, there's no slave, nor there is any leader election or nothing like that. All nodes are the same, they can write, you can write to any node and read from any node you want. So that's what we required symmetry in development as well. So when I say symmetry in development, I needed to generate containers just from a single config docker file or any other configuration file and, you know, generate any number of containers from it with a definite configuration, possibly with a cow semantic, copy on write semantics. Yeah, so since I talked about both, so now I'm talking about how I'm using all of them. So one of the things that we require is elastic requirements. Suppose when I'm doing benchmarking or testing or deployment, there are times when there is a need for changing the amount of containers deployed on a node or on a hardware device and that's what I mean by elastic requirements. And the second one is fast scalability. That is also something that's very important to us, especially with testing where I can, I want to start like 20 containers, 20 nodes in a short period of time and that's where the performance aspect comes into picture. And packing for saturation with this what I mean is whatever the mishaws or docker swarm have, wherein you can ensure that there's a where you can define the pack packing of the containers on a node to ensure the maximal saturation of the hardware. And snapshot transfer is something that is used when you want to transfer the state of a node to a new node. And that is simplified a lot with this when there are copy on write semantics available with the docker and others. And runtime instantiation, by that I mean that even though you have created docker images through a docker file, it's possible to change the parameters at runtime either through environment variable or through the command line itself. And that is quite helpful, especially when you want to do something like LD preload and stuff like that. Okay, so why again, why is we are using this? So I wanted to keep it minimal and inheritance by what I mean by inheritance is suppose that we have images for release experimental and testing. And I don't want the docker file for experimental to reuse whatever is there in the release one. And I want to instead use it and build upon it. So that's what I mean by inheritance and extensibility. And by immutability, I mean that whatever the docker model ensures with its layers, that is when you change something, you get a new container. And as you keep on changing, you don't change the existing one instead you get a new one like a functional approach to containers. And the registry is also very important in that it's I've written the go get the in the go you can just mention a URL. And as Brandon talked about it with this image discovery, that was that's also very important so that we can we have a single point from where we can fetch images and and do stuff with it. So there are some proof of concept material available in the GitHub URL there. And the docker images are over there. So in the networking part, linking is what docker provides with it. But nowadays, they're thinking of doing it doing something with open v switch. I think socket plane is the one who is working on that for because with linking what happens is there is a node and when you link the node to it, you know, it's it's one direct it's unidirectional. And that is something that can that is something which is good for client server paradigm but not for a cluster where you need it to be bidirectional. And that is something I change with using a DNS mask container my own. So and passing IP addresses to the DNS mask container through a volume and helping it regularly. So that's that that's what I had to do. But nowadays, there are new newer solutions coming into place. One of them is open v switch. There's Veev and others. So and that is something I am considering because networking is some is one of the main things with the cluster because with the bad networking, you can get poor latency and the cluster right or read actually right is as fast as the slowest node that it has got. So that's important here. Some of the issues that I ran into it while doing it is, you know, these are some of the current issues. For instance, if you have IP address for a container and you you Docker restart it, it comes up with a different IP address. And that can cause the issues when there are membership membership changes in the cluster. And currently, even though you can plug in a lot of other stuff, you cannot plug in DHCP CD, at least not that easily into the cluster. Because I wanted to use DNS mask as the DHCP CD so that I can have deterministic IP addresses that I earlier used to do with the QMU KVM. Okay. And this is another thing, the socket interface, you know, Docker exposes when it exposes to the host, it exposes it through a port and a couple of IP address and port. Whereas I needed a socket interface like a unique socket interface, that is something that is required. Currently, what I'm doing is I'm using so cat and you know, doing that bridge because I require that for with something like this bench through which you can you can spray your load through in a round robin fashion through all the sockets that the socket interface there can be helpful. And as I said, linking is not that you know, there are there are different patterns available like ambassador pattern and stuff like that, but those were not ideal for Galera cluster. And this is the last one that one of my colleagues mentioned when I was talking to him when I was introducing the concept of containers to him, he said what happens you build on what happens if you build on a kernel which is new, like 3.18, which has a new syscall something like get random map. And you transfer that container to a older kernel, but with the same user stack that it is carrying what what happens there, I mean, what will be the behavior there and or in case of some syscalls with degraded behavior in older kernels, but not with in the newer kernels. So the current the concept of kernel ABI even though it's considered pretty much stable across you know, over a longer period is not something that's addressed in Docker or any of the container specifications that I've seen so far. And that is something that needs to be done. Otherwise, since the concept of since Docker, etc. news, we may not be seeing it but over the years, we may start seeing that issue. Anyway, these are the use cases where I'm using where I'm doing it. I'm doing something like chaos testing and stuff like that with the multiple Docker containers like 2040 or stuff like that. And and that's one of the use case. The benchmarking is another one. Though for benchmarking, the hardware is much better than what is used for testing. Of course, this production deployment rolling upgrades for a cluster, you need a high availability. So with the rolling upgrade, what we can do is we can start another container on the same device and bring this down and bring that up with the same set of with the same volume, you can pass to both. So that is something which is very which is a very interesting use case and very helpful as well. And for rapid deployment, this is something that I was for new for new join is in our team and others. This is what helps us because because sometimes they want they have a tree, a code tree, and they want to test it, but they don't want to know about any requirements that they have to install. So what this allows for us is to start a bunch of containers, one for like this bench, one for DNS mask and other for Galera cluster. And without any on a bare host to be able to do all kinds of testing, just give with a code tree and build and even doing the building inside the cluster inside the container itself. So doing rapid development is rapid development and testing is one of the use cases where we use it the most. Okay. And packaging testing is another that we are using it for though in case of Docker, you cannot do like system D inside it, at least not trivially. And for that, we were we are looking at something like a system DN spawn or some something like that, because that allows for it. And you can actually convert between the Docker images and you can run it, you can export it into a system DN spawn instance. Okay, this is the orchestration part. Currently for in our project, we use fig because it's quite simple and it's well integrated into Docker. I know there are other solutions, but fig is very much integrated. And there's a Docker comp compose coming up and which is essentially a rewrite of the fig and currently it is in Python, but there, I think in future they'll move it to go. But fig itself has some limitations. For instance, I use fig scale, which allows you to define a minimal YAML file and you can bring up any number of containers with that with the same configuration, but it does not allow for something like a substitution variable and stuff like that. But so that is the limitation that I'm facing there. And missiles and Docker swarm, that is another thing that we are I'm looking at for and flannel and Kubernetes flannel, I think got recently into coro s in its own container, I believe. And that's, that should be helpful for deployments into AWS, the deployment. And these are the ones we have been looking at for deployment. One is the EC2, which was recently announced in the Amazon conference AWS conference. Google container engine already has it. And one of the first to have it. And digital ocean provides for coro s containers. A joint has something called smart cloud. Who contains the containers? This is a picture of those Russian dolls. You can put one inside the other. So this, I'm talking about the environment here. So there are many available. One is the project atomic speaker of whom is sitting here with this RPM OSD and stuff. And coro s, Docker swarm, and swarm are not necessarily the complete environments, but they provide you to manage it. And this is a specific case that I wanted to talk about is with coro s. One of the nice thing about coro s that I found was that you didn't have to learn anything new if you had already built your build services for your cluster with system D, system D services. And it also provides at CD, which is a distributed coordination mechanism and fleet for deploying services on other nodes based on various criteria. So this is something that I required for bootstrapping a cluster because bootstrapping is something that is not symmetric. That is you need to bring up a node and only then you can add other nodes. After that, it's symmetric, but bootstrapping is the initial part. And for that, you can use with HCD, you can use something like queues for storing the, it provides the concept of queues, which you can use to store the IP addresses of every new node in the cluster. But then the test and set feature is something that is very helpful here because that allows a node to see if there is already a bootstrapped node, otherwise bootstrap itself. And that is very important. And the concept of side ticks, wherein you have docker containers, which you are starting with the fleet CTL service, but you want to do some external things like, as I am saying, like bootstrapping. So side ticks can be used there. And others are like LX, ALXD, which there was a talk in the morning about that. System DN spawn that we are considering for packaging testing because the concept of, because system DN spawn allows you to run a system inside it actually. And that is good for our testing of our RPM packaging. And Rocket as the talk was about in the morning again. And this is the concept of security. It's from plan nine, Glenda, I guess. Yeah. And what you see is what you get to attack. That's what I mean by that. And the security that we need is provided through like S-word, SACOMP, which I don't think is integrated into any yet. I don't think docker has it yet, but having SACOMP will be very good. And the capabilities, which are already there now since a few docker versions, because running a privileged container can be a bad thing. And the verifiability up to bare metal, and that is also something which is very, very, which is a requirement for us, which is not there actually till now, but which should be, but it is in process from what I read. Anyway, so that's about the talk. Thank you.