 Let's do this. All right. So my name is David Vossel. I work at Red Hat and today I want to talk about Pacemaker and specifically how we're using Pacemaker to power OpenStack high availability. So first off my title, it might be a little bit vague. I'm going to discuss a little bit later on why I think Pacemaker is OpenStack's PID one and what that even means. They want to be clear about the purpose here. The purpose is I want to discuss OpenStack high availability and how I think cornerstone Pacemaker is the cornerstone that makes our OpenStack high availability architecture possible. But before discussing the specifics on OpenStack high availability, I want to do something a little bit different. So it's easy for me to rattle off how we're doing HA and it's really not that complex. It's actually simple, which just makes it powerful, but without some background understanding why we made certain decisions. I don't think it's going to have the same impact. So I'm finding a lot of times the questions we had to ask to arrive at a solution are often times more important than the solution itself. So to illustrate where we've come from and where we're going, I want to start with a story. So the title of our story is the future of HA and how Pacemaker saved the day. And it rhymes. So I guess that's cool. So the story goes like this. Once there was a database, it looked like this. This was a special database. It wasn't like the other databases. This was a distributed self-replicating database. So everyone saw this and they said distributed self-replicating database. This is exactly what we were looking for. The fault tolerance is built directly into the application itself. If one no goes down, the rest of the nodes can pick up the slack. So there's no expensive shared storage and use. Everything just kind of works. It's magic. And everyone was happy. But then they started to think, okay, this is really neat. But how do we low balance this thing? So they wondered maybe HA Proxy could help. I mean, HA Proxy, it's a low balancer that's kind of what it does, right? And sure enough, HA Proxy said, no problem. I got this. Low balancing is what I do. And everyone was happy. But then everyone started thinking again, okay. So what happens if a database node dies? Does that just mean some client request aren't getting processed anymore? Or what's really going on here? And so this HA Proxy said, no problem. I got this. So if a database instance dies, HA Proxy is actually able to monitor for unresponsive database instances and route the traffic to the remaining database instances. So that works. But everyone wasn't quite convinced. Thinking more about high availability, everyone asked HA Proxy another question. So what happens to the load balancer dies? And to this, HA Proxy said, nothing. And because HA Proxy was unresponsive, everyone got to see exactly what happens when the load balancer dies. Client requests are no longer processed. Everything grinds to a halt. And it's not exactly what everyone had in mind for high availability. So the problem's obvious. How do we have redundant load balancing? We need to have resilience built into our load balancers. So the application or the database is fault-tolerant of itself, but load balancing isn't. So to fix this, initially, we use keep alive D. So with keep alive D, we have the ability to have a virtual IP address assigned to our HA Proxy instances, which we have redundant HA Proxy instances. So if a HA Proxy instance dies, the virtual IP moves to another active HA Proxy instance. So we can have HA Proxy instances failing, database instances failing, and clients have no clues what's going on. Everything just keeps working. So this is kind of awesome. This is what we were looking for, right? Kind of. There's a problem. So everyone asked one last very important question. What actually happened to the unresponsive nodes? The ones that are grayed out here. The ones that keep alive D is telling us are dead and they're routing traffic to the virtual IP address to the other instances. So to answer this question, keep alive D says, what other nodes? Except the answer came in for an echo, because the reality is the other instances, they weren't unresponsive at all. There was a network split and two partitions informed. So at this point, they're having data consistency issues, clients are writing things and they're able to retrieve them, and just other weird, undeterministic behaviors occurring. So everyone hung their head in shame. They said we've made a huge mistake. But they couldn't figure out where they've gone wrong. I mean, they had a distributed self-replicating database that had default tolerant built directly into it. They had redundant load balancing. They're still having data consistency issues. So how do we fix this? For us, the solution was Pacemaker. So Pacemaker handles the prevention of split partitions very well. It's native into how Pacemaker works. We have some very sophisticated mechanisms to handle this that I'll touch on in a little bit. Besides just being able to handle this very well, there's quite a bit more that Pacemaker has to offer here. So Pacemaker gives us this concept of system-level high availability. This is a holistic approach. It completely eradicates the need for keep alive D. With Pacemaker, Pacemaker is managing the virtual IP address, the HA proxy instances, and the database instances. So if an HA proxy instance goes down, Pacemaker is able to relocate the virtual IP address for us. It's also able to not just monitor the HA proxy instance and say, oh, it failed. That's too bad. It's actually able to invoke action. It's able to say, the instance failed, let's recover it. Same thing with the database instances. It's able to invoke action. So it doesn't just sit back and watch. Okay, so Pacemaker lets users define a policy to define what services to run, where to run those services, those relationships between services and potentially the locations that they run. And Pacemaker is able to enforce that policy to produce a system-wide deterministic behavior. So there's no unknown states here with Pacemaker. Pacemaker has a complete view of the entire system. If we want to simulate what happens if a resource fails, we don't actually have to fail that resource and just watch what happens. We actually know it's determined. It's predetermined. We can actually feed in that failure to our simulation program and actually produce a graph that shows how everything's going to be reallocated throughout the cluster. So everything is predetermined. It's a finite state machine, and there's no unknown states. So going back to our story. With Pacemaker, we don't question the scenario for two reasons. One, we trust our core provider. It's very good at handling these tiebreaker type scenarios. And two, we don't question what happens because we have stoneth. So stoneth is Pacemaker's fencing daemon. And fencing is a large part of why Pacemaker's able to produce deterministic behavior. So stoneth is an acronym for the other node in the head. If Pacemaker's ever in doubt about the state of a node, a node's fenced. So fencing, if you're unfamiliar with it, it's just a process of taking a node and separating it from an active cluster. So you're taking a misbehaving node and separating it from the healthy nodes. Typically, this is going to be like a power fencing where stoneth actually talks to a physical device at the power switch and flips it off. But it's not just limited to that. There's lots of different types of fencing devices. You can have, like, virtual fencing where Stoneth is going to talk to a script at the virtual machine host and kill the actual virtual machine PID. So stoneth is really smart. With stoneth, we don't question the state of the misbehaving nodes because we know the state. They're dead. So a quick recap. Pacemaker, without Pacemaker and stoneth, we potentially have split partition issues and we have no centralized way of recovering resources. I should point this out, though. It's possible to handle these split partition issues with keepLiveD. It's not native necessarily to how keepLiveD works, and it's very difficult to get this behavior. It's really hard to do what Pacemaker natively has very sophisticated and simple ways of doing. So with Pacemaker and stoneth, we don't have to worry about split partitions and we get this centralized management of all our resources. So takeaway from this part is Pacemaker and load balancing. They're not mutually exclusive. Pacemaker handles actual resource management. It's actually handling starting and stopping of the load balancer. Load balancer is just handling distribution of traffic. So in a lot of ways, they're meant for each other. It's kind of cute, right? Okay. So that was my intro. So now we're back to the title of my presentation and I'm going to get to talk about some more interesting stuff. So before I talk about OpenStack specifics, I want to explain this part. Why do I think Pacemaker is a distributed PID-1? And what do I even mean by that? So to understand what I mean by that, let's take a look at what a PID-1 or a NIT name does. In this case, we're going to look at system D. So system D, it launches services in parallel. This is what we get our super fast boot up times. You can boot your laptop in like two seconds rather than two minutes a few years ago. Yeah, it's able to observe strict ordering between services as they boot up. So if you're looking at this diagram here, RabbitMQ and Glare, they don't depend on each other. Nova depends on both of those things. So RabbitMQ and Glare, they can start at the same time in parallel. We want Nova to wait until those services are up. The last thing system D does with respect to research management is able to monitor and recover failed resources. So that's all really cool, but there's a problem for us. OpenStack services, they're not centralized to one node. They're distributed. So in this case, Glare doesn't exist on one node. Glare exists on a distributed set of nodes. So we need to be able to say, start Glare cluster, form that cluster, gain quorum, make that Glare cluster available, then start everything that depends on that cluster, like Nova. So system D is limited to one node, so it can't coordinate this. The pacemaker can is what pacemaker was designed to do. So with pacemaker, we can actually say, start Glare cluster across three nodes, form that cluster, once it's available, then start Nova. In fact, if we go back and look at what a PID one does, we see that pacemaker actually does all those things. Pacemaker can launch services in parallel, yet observe strict ordering between those services. They can monitor and recover failed resources as well. But unlike system D, pacemaker can do this crossing number of nodes with any number of resources. So what allows us to do this? An important aspect of pacemaker's configuration is resource constraints. So with resource constraints, well, first off, we define the resources that pacemaker is going to manage. With resource constraints, we can draw relationships between those resources, so things like ordering constraints. Start X, resource X, then start resource Y. This is what we're dealing with Glare. We're saying start Glare, then start Nova. Colocation constraints. We want to say, put resource X with resource Y. This is what we do with HA proxy in a virtual IP address. We want the virtual IP address to be placed with an active HA proxy instance. So the big thing to get about this, though, is these two types of constraints, they work independently of each other. So start resource X and start resource Y. That doesn't mean they have to be on the same node, even. But they could be. You could also combine them. You could say start resource X and start resource Y and make sure they're collated together. So it's kind of an either-or sort of thing. There's a lot of power in that. The flexibility of being able to do, have those work independently of each other is, given pacemaker, a lot of flexibility that we are for OpenStack High Availability. So speaking of OpenStack High Availability, let's dive in. So how does this work? If I could break down our OpenStack High Availability architecture into one sentence or statement or collection of words. I don't know. There's some characters in there that aren't really letters. It's going to be pacemaker managing virtual IPs, load balancers, controller services to maximize the availability of OpenStack APIs. Basically it's this pattern over and over and over again. It's probably understandable why I told that story earlier. So the pattern starts like this. We have a virtual IP address. Every virtual IP address is assigned to a back-end service. So there's a virtual IP address for every API. From there, we have an HAProxy instance which takes this front-end virtual IP address and distributes it to the back-end services. Wherever an active HAProxy instance is, a virtual IP address can run. Lastly, we have our services which most of them run active-active. Some of them are still an active passive but we're working with community on that. These are the services you're familiar with. This is Nova. This is Glance. This is Keystone. There's also things like Galera or Redis. It's an underlying infrastructure. So one thing I want to point out here and it's important for scaling is pacemaker's use of the clone resource type. So all these services, they use clones. And what a clone is, with pacemaker, we find a resource once, just one time in the config, and we say how many replicas of that resource we want. So with HAProxy, we want redundant HAProxy instances. So we want to spread out all our HAProxy instances across all the nodes. So it's clear at once and say replicated across all the nodes. Same with our services. So if we want more capacity, we just add more nodes and pacemaker can dynamically grow our configuration. And one thing to point out about this is when pacemaker dynamically grows the configuration, the ordering and co-location, all those constraints, they still apply. So it's really, it's growing it, but we're still having the same behavior. This is a more detailed look, I guess, at how this works. Again, we have a virtual IP address. Things that want to access the back-end services. They all talk to the virtual IP address. That's how we're providing high availability. So Glance and Nova need access to the back-end database. They're always going to talk to the Galera virtual IP address. And we're always going to make that available. So let's talk about some of our deployment strategies for this pattern. First one I want to talk about is the collapsed architecture. So with the collapsed architecture, all controller nodes are identical to one other. This requires kind of more powerful hardware because every node actually has to run every open-sack service. It's really easy to deploy. It's really easy to deploy because everything's just a clone. So if we look into this a little bit more detail, we can see our pattern all throughout the collapsed architecture. So there's Nova APIs. I'm trivializing some of this. I know there's a lot more to Nova than just a little box. But the virtual IP address is signed to Nova. We have the same thing with Glance. And it's back-end Glance services there. And if we want to scale our collapsed architecture, these are all clones. Remember what I said about clones earlier? Scaling is just a matter of adding more nodes. Everything's dynamic. Well, as far as pacemaker is concerned. So let's look at startup order in a little bit more detail. So remember what pacemaker is giving us here. We're getting reliable load balancing through managing the HA proxy instances in the virtual IP address. But we're also getting cluster-wide startup ordering. And some of the startup ordering, some of the things pacemaker is doing here, are really complex. So trying to bootstrap a Galera cluster, that requires a lot of hand-holding. They've improved things upstream, but it's still difficult. So I'll tell you the process, and I think it's neat that pacemaker is actually able to automate this. So to bootstrap a Galera cluster, you have to look at every one of these instances. You have to pick one of them out. And when you have to look at every one of them, find which one has the highest write sequence number. Once you've done that, that's the one you want to use to bootstrap. So you start that one first. Then you start all the other instances, and they have to connect to that and sync. So there's a lot of steps involved. And then these steps are distributed across lots of nodes. Pacemaker can actually automate that in a very deterministic and safe way, where previously it was pretty difficult. So we can say bootstrap Galera, this aren't anything that depends on Galera, like Nova. Likewise, we can unwind the process. So if we want to stop Galera, say we want to do a package upgrade. We can say stop Galera. It's going to do a graceful unwind of the shutdown order. So it's going to shut down anything that depends on Galera first, like Nova. Then it's going to shut down Galera. You can do package upgrade, whatever you want to do. Another example of complex start ordering is Redis. So we're using Redis for Solometer. And Redis is this master-slave replication thing for us. So we have to start every instance of Redis in slave mode. Pacemaker's able to pick out one of those instances, start and master. All he wants to master instances up will start anything that depends on Redis. Okay. So that was our collapsed architecture. Second deployment method is our segregated architecture. And for this, every service has its own dedicated hardware or virtual machine. So this is going to scale much further potentially. Of course, lots and lots of nodes, but it's kind of flexible in that because you can add capacity where it makes sense to add capacity. You don't have to just add capacity and get all your service replicated. You can add capacity in just certain areas. So taking a closer look here, you can see like you're breaking out the glare cluster. You could have your own dedicated hardware just for load balancing. And the list just goes on. But I don't know how useful it is to have a pure segregated architecture. I think it's going to be a lot more common for us to see things like this mixed architecture where you have... Most of your architecture is in this collapsed view where Pacemaker's managing everything in its clones on maybe, say, three nodes. But then you might break out certain parts of the cluster like maybe you have dedicated hardware just for load balancing or just for the database or whatever else you need to add capacity to in a separate way than the collapsed architecture. So let's review some of the advantages of Pacemaker managing controller services. So these are advantages that are on top of Pacemaker's solid, quorum, and fencing features. These are things that we just get in addition. So we have to have quorum. We have to have fencing. We have to trust those mechanisms. If you don't have that, it's not good. But we get more than that. We get the ability to automate some really complex bootstrap processes like Lair and Ritas. We get to have ordered stop and starts so we can gracefully shut down a service. And we know that everything dependent on that service is going to shut down as well in an ordered fashion. So you can do package updates or whatever. Same thing with a node. You can gracefully turn off a node and know that everything's going to gracefully migrate somewhere else. So you can do upgrades on your node. We also dynamically grow capacity by adding more Pacemaker nodes because we're using the cloned resource type. And we have this centralized view of the distributed service state. And this isn't something I've talked about yet. It's really cool. So with Pacemaker, you can look on any node and look into the cluster and you can see exactly where all the resources are running, where resources have failed, what nodes are online, what nodes are offline. And in a lot of ways, I mean, that's kind of like what something like Nagios does. So I mean, you're getting a view into the cluster similar to something like Nagios provides, except Pacemaker's actually able to be a Nagios that actually does the recovery as well, which is really cool. Okay. So switching gears a little bit. So far, I've been talking about the OpenStack control plan and high availability. But now I want to talk about something different. So when I start talking about compute node high availability, I'm often encountered with some ideological differences that make the conversation difficult. So I'll explain what I mean by that. I'm sure you guys are familiar with the pets versus cattle argument. So you have your pets, they're unique, you name them. You might want to have a couple of instances of them. You spend a lot of time and energy maintaining them. If you want to scale, you have to throw out your old hardware and get faster hardware and scales only vertically, as opposed to cattle, which cattle have lots and lots of them to make a herd. They don't have names, they have numbers. You kind of feed them water and you don't really give them individual attention. They scale horizontally, so you just throw more and more instances of cattle to gain more capacity. So here's the problem. I feel like sometimes we've told ourselves to HA it's only for the pet scenario. And somehow the herd saves us from needing high availability. And that's not true. Certainly we want to deploy different methods for cattle than we do pets. I don't think pets have their own high availability techniques that don't apply to cattle. But that doesn't mean we don't want to detect that there's dead cattle and automate the recovery of those instances. We want to maximize the availability of the cattle, just like we did the pets. It just looks a little bit different. We also want to be able to do things like isolate dead cattle instances or misbehaving instances from the rest of the herd. So I'm not necessarily talking about bringing kind of these old, rigid, high availability techniques of the past. I think we need to bring something new in, something that's tailored to this use case. So I think PaceMaker, it bridges the gap between this old, high availability world and the new. And we can see this in how we scale. So the core of PaceMaker infrastructure is traditionally being limited to 16 nodes. And this is a limitation on the core sync membership layer. Core sync has this mesh network that grows exponentially and complexity. So the more nodes you get, the more complex it gets. And once you get to around 16 nodes, sometimes people can push beyond that, but it starts getting a little bit crazy. So for pets, 16 nodes, that's fine. Maybe small herds. But 16 nodes for cattle, that's a joke. So we did something about it. We recognized the PaceMaker's policy engine, the finite state machine that's making all the decisions about resource state and how to react to failures and things like that. That's our most valuable asset. And it turns out it's not limited to any sort of 16 node limitation. It can scale much further than that. So how do we break that piece out of this 16 node limitation? And our solution was PaceMaker Remote. So PaceMaker Remote is not bound by any sort of messaging limitations. And with PaceMaker Remote, we can scale PaceMaker to hundreds, potentially thousands of nodes. So what is PaceMaker Remote? It's pretty simple. It's just a daemon. This daemon, if we're going to get down to it, it's just a lightweight way of integrating a node into a PaceMaker cluster that doesn't require core sync. PaceMaker Remote cluster services, they can run on remote nodes in the exact same way they can run on normal PaceMaker nodes. To PaceMaker, a remote node and a PaceMaker node rank core sync, they look the exact same. PaceMaker treats it as if it's a single partition. So services can be distributed across all the nodes, remote nodes, or PaceMaker plus core sync nodes in the exact same way. While it can run, PaceMaker Remote can run the exact same nodes as PaceMaker does or did, I see the use case that PaceMaker Remote's going to thrive in is when we start using cloned resources. And this is for a few reasons. The primary one is that once we start using cloned resources and we have lots and lots of nodes, PaceMaker's policy engine, that finite state machine, it's much more efficient at calculating transitions. And we need that to be snappy. We don't want our transitions to take seconds to calculate. We want it to be milliseconds. And cloned resources help us with that. So the use case I'm envisioning here is, I'm not sure if I made this clear, PaceMaker Remote requires a small PaceMaker cluster. I'm envisioning three nodes for your PaceMaker cluster. That cluster is going to be making all the decisions about everything else. There's one that's producing all the transition graphs and things like that. Those three nodes are going to power a much larger set of PaceMaker Remote nodes. PaceMaker Remote, it's monitoring services. It's doing whatever the PaceMaker cluster tells it to do. It's just reporting back. So if we look back at our, or if we look at our OpenStack ComputeNone architecture, we can see this is going to make a lot of sense for us. So looking at our collapsed architecture for the controller nodes, we already have a small PaceMaker cluster. It's already powering all of our OpenStack services. So we can utilize that PaceMaker cluster to power a much larger set of Compute instances. So our PaceMaker Remote instances here are running our Compute nodes. And you can see that we have four services in there. And this outlines another cool aspect, feature that PaceMaker has. So we have clones where you define a resource once and you can replicate over and over and over again. We also call it a group, so a set of resources. In this case, we've defined four resources to make up our Compute node, do all our Compute node stuff. And we've said this is a group, launch this group as a clone across every PaceMaker Remote instance. So what does this solution give us? We can maximize the availability of Compute instances. So PaceMaker is not only, it can recover Compute instances when they're having trouble or whatever, but there's a trick we have. And this is where the value really is. PaceMaker is monitoring both Compute nodes. So it's able to see when bad things are happening on Compute nodes and it's monitoring the controller nodes. So it's monitoring NOVA and Keystone and Glance. And it's able to coordinate between the two. So we're actually able to do some kind of fun here. If we detect something bad is going on a Compute node, we can talk to NOVA and say, this Compute instance is just bad. It's gone. Don't even worry about it anymore. And we're able to trigger the evacuation of the Compute instances where we're on that Compute node much more efficiently and say, NOVA, okay, this node's gone. Start them somewhere else. So we're actually able to coordinate the evacuation of virtual machine instances or containers or whatever. So there's one last thing that PaceMaker is able to do with Compute instances. Sometimes Compute instances go really bad. Maybe something catastrophic has happened. They're unresponsive. The node's failing in a way that can't be recovered. So for this, we need a way to isolate that Compute instance from the rest of the herd. We don't want that Compute node running virtual machines anymore. We don't want it to be available to do that because there's something wrong with it. We don't necessarily know what's wrong with it, but we don't want it there anymore. And to do this, we have a secret weapon. Stone us. So PaceMaker and PaceMaker Remote can both do fencing. We can fence PaceMaker Remote nodes in the exact same way that we fence PaceMaker nodes. There's nothing special about them. So a Compute node is just going crazy. We can cut it off. And then we can trigger the evacuation of those VM instances to a healthy Compute node. Okay. So I'm kind of winding down now. So I want to leave us with this. So what's the future for PaceMaker? The problems I'm most interested in mostly involve scaling. So I want to know. I want to know how many services we can run. I want to know how many nodes we can run. A few years ago, PaceMaker, it was primarily used just in two node clusters. We understood the power of PaceMaker. We understood the power of the finite state machine that exists in PaceMaker. We knew that could be expanded beyond that. So it's been really rewarding to untap that power and make it available to cloud infrastructure like OpenStack. I think PaceMaker is a solid foundation for OpenStack high availability. And I'm really excited to see where it goes from here. Questions? Can you guys use the mic? Is PaceMaker remote planned for OSP7? I don't know if I should talk about that. Okay. Thanks. What if your process is no longer a process but it's something running in a thin container like a Docker? Do you think we're going to have the tools to monitor, restart processes in thin containers like that? Yeah, you can. So there's a Docker resource agent, actually. So we can launch services in a Docker container and monitor them across the distributed set nodes. That's actually the thing I'm really interested in lately. And we have some new features that are coming out with this. So we're going to make it a lot more transparent for people. But they haven't really been announced yet, even upstream. The works there, I just haven't documented it. So I can't really point you guys to anything. But the Docker resource agent absolutely exists. So you can do a lot of that today. I have kind of a specific question about RabbitMQ. I know traditionally they had talked about using PaceMaker, but nowadays has its own internal mirrored queues. Does it make sense to use PaceMaker with RabbitMQ these days or just stick with the mirrored queues that they provide? I'm not sure that's mutually exclusive. So PaceMaker is managing starting RabbitMQ. And it's actually coordinating how it works right now is PaceMaker is actually making sure that RabbitMQ successfully forms a cluster. So we start RabbitMQ maybe across three nodes or whatever. And we're actually PaceMaker's coordinating to make sure that RabbitMQ cluster forms quorum and everything is available. So, yes? Okay. Hello. You were earlier on in the talk talking about the similarities between SystemD and PaceMaker. And one of the neat things about SystemD is the ability to declaratively describe a service and its dependencies and how it starts, how it stops and all those things like that. PaceMaker, what I've seen in the past, at least, is giant shell scripts with large collections of PCS commands that you run, you end up with some dependencies and things in PaceMaker. And if you decide you don't like what you've installed, you have to kind of reverse that. Is there any way to provide PaceMaker with a declarative, basically a map of, here's a bunch of services, here are the constraints for these services. I want to give them to you so you can install them and I want you to be able to take them out of your PaceMaker configuration very easily. I'm not sure I completely understand how we don't do that already. So PCS is our configuration mechanism and you were talking about that earlier. How can you not take the constraints back out? I can write a shell script that will do some of that and then I can write another shell script that will undo that, but that's a different thing that those two things aren't magically synchronized and if I make a change in one, I have to make a change in the other and it's not a declarative syntax either. It is a step-by-step execution of shell script commands. So I am asking, I guess, if there's something like a unit file for PaceMaker. Oh, I understand what you're saying. Yeah. No, we don't have unit files for that. So what you're doing when you're using, I mean, I understand what's going on in that shell script, you're invoking PCS commands, which actually write to the PaceMaker configuration. There's a distributed database within PaceMaker itself that holds this configuration. So you're committing changes to that database and you want to be able to reverse them. Yes. PCS helps with that quite a bit. It abstracts a lot of that away. If you're doing something like you create a resource and you build all these dependencies between it and you want to take that resource out of the configuration, if you do something like a PCS delete, it's going to take out that resource and all the constraints associated with it. But yeah, you do have to go in and you are manually drawing a lot of these relationships yourself. So there's no such thing as a unit file for PaceMaker. Thank you. So first question is related to the PaceMaker versus PaceMaker remote. Just to make sure that I got it right. The non-PaceMaker remote machines, they're actually the nodes, they're actually responsible for like forming the column. Nope. Wait, wait. Say that one more time. To form the column. The non-PaceMaker remote nodes? Yeah. Okay, so I'll explain how that works. The one thing that PaceMaker remote nodes don't do is place and quorum. That's why you need the small PaceMaker cluster. So your three PaceMaker nodes, they form quorum. PaceMaker remote has nothing to do with the quorum of the PaceMaker partition. Okay, that's actually what I was saying. The second question is actually, I'm not suggesting the architecture, but a SpaceMaker able to like go and look into two different levels. Suppose like you're deploying Keystone with the MySQL database and you want to have HA on the MySQL database and on top of that, to have like HA on the open stack level, you actually depend on that service. Can you manage like across the two levels or would I need like two different PaceMaker clusters there? You only have one PaceMaker cluster. Okay. If you, it wouldn't make sense to have two PaceMaker clusters on the same sets of nodes. You have one PaceMaker cluster and it's all the services. Just like you have one system D. Okay. We can talk more later. Okay, cool. So how do you stop Stoneth from dueling in a bifurcation? Say that one more time. How do you stop Stoneth from shooting each other? So we don't. It's actually CoreSync that does that. So CoreSync has very sophisticated tie breaking mechanisms where if there's split partition, it's very good at telling one side that it won and the other side that it didn't. So that's how it's prevented. The one that won shoots the other one. Got it. So my question was specifically for PaceMaker monitoring services. Okay. So is there any feature where PaceMaker has some advanced way of monitoring services as opposed to just doing service status? Like if, let's say for instance, my service has some endpoint applications which I can do monitoring or health checks to make sure that the service is up and running. Because we run into issues where the service is up, but it's not doing anything. It's throwing errors. So we want some more advanced monitoring for PaceMaker to do on the service. So is there a way for PaceMaker to do that? I'm not sure I can fully follow that question. There is. Yes. Okay. All right. That's what that was good at. Yeah. So PaceMaker has the ability to use a lot of different resource types so you have, we can control system D. We control like LSB, the old init style scripts, but we also have these OCF scripts, which we write that allows us to do some really detailed monitoring. Yeah. Okay. Cool. Thank you. So I have a question you have talked about the compute nodes being, you know, sort of having the infrastructures they are being managed by PaceMaker, but PaceMaker's policy engine can scale really well and has all these advanced features like utilization-based resource placement and so on. Have you ever looked at using the PaceMaker policy engine also for placing the VMs or containers themselves so like getting a little bit into the NOVA territory? We haven't looked at that. Why not? Maybe that's a good idea. I'm not a part of the NOVA project. That could be an interesting use case. So as we relate to containers, is there any advantage to running PaceMaker instead of a fleet or Kubernetes? Yeah. Good question. Yeah. You get really deterministic and proven behavior with PaceMaker. I mean, Kubernetes and all these new like fleet and Kubernetes, they're neat and everything, but you're not getting the same deterministic behavior. It's difficult to use Kubernetes to define. You can define ordering and things like that, but PaceMaker is a finite state machine and it's locked. All these things are locked and we have really fine control over the exact state of the cluster. I think they're recreating PaceMaker, if I'm being honest. I had a question about like fencing and stoneth. If you've got your three node deployment that you're talking about and you lose say glance on one of the nodes dies, is it granular enough to just shoot that glance process or is it going to shoot the whole node just because glance died? And then are you like propagating failure to places that shouldn't be? Does that make sense? Yeah. So deterministic behavior, we do what you tell us to do. In the case of glance, we're going to try to restart that node. That instance, excuse me. So it's not necessarily that we're going, what happens if glance fails? Do you want to stoneth? Do you want to fence that node? You probably don't want to. What you want to happen is just have requests no longer routed to that glance instance. So HAProxy is going to attack this unresponsive and it's no longer going to get requests to it. PaceMaker is going to try to restart that instance of the glance. If it can't restart it, then it's just going to market failed and it's not going to get fence just for glance failing. I mean, if you want it to, you can go in and say if glance fails, some fence is no. But by default, no. We don't do that. Hello. Can PaceMaker make use of multiple heartbeat network? For example, I think there can be multiple rings in the crossing configuration. Can PaceMaker make use of that? Or how can it deal with that? That's the layer below PaceMaker. You'd want to talk to CoreSync about that. So PaceMaker is going to utilize whatever CoreSync gives it. So you're talking about redundant rings and I'm not the person to talk to about that. This is actually because, for example, you have a compute node and you have storage network and you have management network. If your management network is down but your storage and tenant network is okay, you won't have to restart or shoot the compute node because this will trigger unnecessary service interruption for the application and the workloads. Yeah, that's a good use case for it. Yes, so if PaceMaker can use multiple heartbeat rings and if we can deal with that in the resource agent, it will be very nice. And this is actually a problem for us. We are using PaceMaker a lot. It's CoreSync. Oh, all right. Thank you. I think how much are we done? I'm sorry, we're done.