 Hey guys, hey guys, hopefully you can hear me at the back. So I'm going to talk a bit about highly-available foreman. There's a lot of decisions you have to make, a lot of choices. Hopefully you'll be able to go away and build them to cluster yourself in the future. So this is originally what we're kind of working off. So right in the top there, I've got post SQL, post SQL database. You can obviously use my SQL or something else. I'm not going to cover any kind of clustering SQL. The PostgreSQL guys had a really good talk about the application yesterday. If you didn't see it, look at the recording. It's really good. So then you're going to have your load balancers, because you've got a cluster of databases, hopefully there. You've got a few foreman applications there. That will scale up quite well. My diagram has three. Yours might have four, five, whatever. You have load balancers there, just underneath. That's everything that wants to talk to the foreman application via the load balancers. So see my laptop there. And the smart proxies also would go via the load balancers into the foreman and load balancers to the applications. And then down the bottom, I've got two separate smart proxies clusters. My laptop, I've only actually got one set up and ready to go because it's a laptop. But you could set up more. And then underneath the load balancers, underneath the smart proxies, you have load balancers. And the blue things on there, they are meant to be clients, and they would connect into the smart proxies. So the foreman application. So each node, obviously, you're going to have the same SSL certs. If you've got different SSL certs on each node, you're going to have issues if you go via load balancer. And one day you get one, the next day you get another. Let's keep them all configured in the same way, same certs. Use DNS help names. So you might have your node one, your node two, your node three. And then your name might be foreman.example.com, or whatever it is. You're going to make sure all names are signed so you can test firstly connected, but you're all going to forcefully connect to one single node, or you connect to any. There's a little file there as well, encryptionkey.rb. Not many people might know about this, but this is used to encrypt passwords in the database. Each node needs to have the same key in there, or whichever one writes into the database just won't be able to read the other nodes, so keep them all the same. There's also another directory, a file there, local secret token that's used for signing cookies. So if you've got one session open, and then a server goes down, and then you suddenly get redirected to the next server, because the cookies are valid for the users. There's also some settings. Foreman URL is normally used in the provisioning templates. That's for the clients to connect back to Foreman to say, OK, we're built, or there's a couple other users as well. That needs to obviously go by the load balancer. That defaults to the FQDN of the first node that sees the database. So you're going to have to change that manually afterwards. On attended URL, similar sort of thing. Actually, I think on attended URL is used in templates for URL, I can't remember. Change it. Foreman Mencache plugin. So we've got a plugin that will connect to Mencache to store user sessions very similar to the local secret token for the cookies. When one server goes down, you obviously want to be able to, your users have to be transparent to your users. So when one server goes down, you connect to the next one and users have just no idea. Again, it can scale horizontally. So go back. We can have three, four, five, 10. Don't go crazy. No need to go crazy, hopefully. Also, you could use the same principles as well, not just for scaling for HA, scaling just for scale, because you want to have multiple nodes. You don't make much of a care about availability, but you just have a big, large estate. So the smart proxies. Each node has to have the same S certificates again for the same reasons. So there's two sets of certificates here. So Foreman Proxy is a service that the form applications will connect to the proxy to do stuff like create DHCP releases. There's a couple of other things as well. Get data out of puppets. Foreman needs to trust those certs. So if on the Foreman servers, they're obviously going to be using the same certs or come from maybe a CA, you're going to generate those certs. Obviously, using DNS names. So you can see there, the first one, I've got CentOS 7 Foreman Smart Proxy. That's my name for my Smart Proxy cluster. I've got DNS names. It's got number one and number two on them as well. So I can always just make sure I'm hitting one if I want to test puppets. So again, each node uses the same S certificates. Generally, you would generate these, let puppet generate these as part of the installer. Obviously, your puppet modules, they need to be the same on both nodes, or three or four, whatever those they are. If you've got different nodes or different versions of puppet modules on different nodes, many clients are going to get different things back, potentially, whichever node they hit. This would be bad. Let's make sure that that's all good. So R10K is a really good tool for that. I've set up R10K on my laptop. I've just, as a demo, just installed one module. Again, you've got an option as well for SRV records. So the puppet agent in the puppet config file, you can set them up to set which is the puppet master. I've set that to my Smart Proxy, the low balancing address. You can set that to use SRV records. So you can change that dynamically quite easily without going around to your machines. You might have hundreds of machines you don't want to change that manually, because normally, obviously, you do that by puppet, but if puppet isn't working, because your master's are down, you can have a bit of an issue. SRV records are a really good way to go as well. Puppet CA. So my question is, do you really need high availability with Puppet CA? So when the Puppet CA is down, the only thing you can't do is deploying new nodes, so to sign the certs. So that's probably not an issue, hopefully. Might be an issue. But let's keep things simple if you can. So maybe just have one node as your CA, and in your puppet.conf you can take this one node as my CA server, but use the low balancing address, which can go to either of those two for the master. A couple of thoughts I've had about if you want to have highly available Puppet CA. So just mirroring ETC puppet labs, puppet SSL, just mirroring that directory between the two servers, and the Puppet CA server is enabled. It should work as well. You can use CRON, you can use rthink, iNotify, and I think GFS2 would probably work quite well here. The certs are small, but they won't change very often, depending on how often you deploy nodes. There's a couple of options. Obviously, you can use the shared storage. I really don't like shared storage, because it just moves the problem away from the application to somewhere else. Yeah, you can have highly available storage, but just avoid it, keeping simple if you can. Again, you can use SRV records. So if you've got a puppet CA server set to smartproxy1.example.com, you lose that server, and you want to point it to two, so you have your certs, and you put them on two, and you want to point it to two. You can just change SRV records, and again, you just don't have to go out to multiple machines all the time. That's just going to be a massive pain in the arse, like I said. So a couple of lessons to learn about my experience doing this. So a lot of these actually apply to any kind of highly available system. Use configuration management for that system. If you've got multiple formal application nodes having to change something manually on one, two, and then three, you're not going to do it. Forget it's error-prone. Use configuration management, whatever you want to use, I don't care. I use Ansible. Don't terminate SSL on the load balances. I think this is probably possible. I think it's really complicated. Foreman has some really good documentation, but there isn't that much about terminate SSL outside of Foreman. You probably could. I just kept it simple, but we'll try it anyway. So I don't know how much you guys know about Rails, but part of the store form when it runs rake, db seed, and db migrates, these migrate the databases or just put data on the database. You don't need to run this on every single for the three nodes or however many you have. You only need to run it on one. So don't need to run it on one. Just one, so set that flag to false on the store. You can easily, so if you have lots of plug-ins, for example, it can take a long time. If you're using the packages, I think that the RPM packages and then the Debian packages, they run db rake and db seed and db migrates. So I think we should probably change that, but for now, that's going to be an issue. Hopefully, depending on your size of how long that takes, it might not be too much of an issue. Clear tasks are frequently often, so again, it kind of relates to scale, not necessarily HA. All the tasks, if you're using form and task, anyway, it's stored in the database. They don't get deleted, I don't think by default. So I added a cron job in there just to leave about 30 days. If you have a massive table full of millions of tasks that you don't ever look at, it's necessary, it could cause problems with the database, maybe. Just get rid of it. Tuning, so you've got a couple of tuning things for Postgres and Apache, just to make your cluster run more smoothly. That's not going to go into this too deep. So this is some of the Ansible that I used to it, so hopefully you guys can see that. At the top, I've got my two hosts. They're my two formal application nodes. My diagram had three at the start, but I only have two. So if we just skip the first few, because that's a scenario if you're using the installer, I hope you guys are always using the installer and not doing fix-mangly, because the installer's really good, easy to use. So there's an encryption key variable, and that will ensure that all my nodes have the same encryption key, and then you've got a new hash form and installer scenario answers. So this roughly translates to if you're using the installer and you've got a variable name, so this translates. So if you look at the first one, that form and then form and URL would translate to hyphen, hyphen, form, hyphen, form, hyphen URL on the installer. A store in this way, Ansible makes the installer, it knows when it needs to run and when it doesn't need to run. Obviously, it's really hard to work out. There's lots of variables in there. Form and URL is just set to my mobile Ansible address because I don't want to default into the FQDN. Server name is used in Apache in the vhost config. I have a password, dbhost, that's my database cluster, various database passwords, just names. Oof key, Oof secret, they use to be the same on all the nodes. Form and proxy, or the smart proxy, they use Oof to authenticate to form it. So again, if each node has them differently, the prompt proxies are only going to be able to authenticate to one, or maybe two. So make sure that that variable, that is the same on all your nodes. Puppet, DNS alt names, I use Puppet to generate certificates, so I like to set that. If you've got certificates coming from an external CA, you could form a memcache plugin, so that configures a memcache. I've pointed it to my memcache hosts, a certain namespace that expires. I've installed memcache on my application node as well. You don't have to. I did. Just made it easier. Some of you may notice the thing that's missing on that one. I mentioned before about there's a file, local secret token, .rb. That file isn't in that ansible role, but I'm going to do that in the future. So I migrate that manually, but don't do that. So this is my smart proxy cluster, similar sort of thing. So I've got an installer scenario. That scenario doesn't actually exist. I wrote it myself. It's just like one file, really simple to do. Hopefully future versions will have a smart proxy scenario in. Again, answers again. So formon-proxy, formon-base-url, if you're using the installer manually. I'm going to set that to my formon cluster, which is going to be the load balancer. Trusted toasts. So I've probably got a bit over here there, but I'm set to trust all of the smart proxies in its cluster and the formon nodes that I have. You shouldn't need to trust the other smart proxies. So for example, smart proxy 2 wouldn't be needed on smart proxy 1, but this made it simple. Same config. That's easy. Oof consumer key and Oof consumer secret. That's going to be the same that I had on the previous slide to match such how it uses to fence cake. SSL key, SSLCA, they're the certificates that you run. If you're using your own CA, obviously they're not going to get them here, but I'm using the self-signed certificates. So when you run that command, the puppet search generates on one of the formon nodes. You'll get given certificates, and they're the certificates you've got to pass in there. Templect URL, again, defaults to FQDN. You want it to use the load balancer at the bottom. So that's for the clients. Formon URL, obviously that's going to be the load balancing address. So a certain name. You don't have to set that, it's just a name, but I'd like to make it the name, the load balancing address. And I've got the two nodes behind that. And the DNS code names again. So I'll talk to you a bit about the future in a minute, but does anyone want to see this, Emma? I can shut down my nodes and see what happens. So this is my, so truth be told, I haven't got a load balancer on this machine. I'm using a very primitive source of load balancing, which is host files to direct which node I go to. Obviously you're not going to do that. So here I'm using, so it's this one here, so it's 29, which is node one. You can see here that I can kind of go here, I can do stuff I'm logged in, I can go around the UI. As a user, if my Formon1 node dies, in fact, I don't actually really need to do that because I can justify this. So if that node dies, it gets taken out of the pool by the load balancer. I've now pointed to the new node, and you can see here, it doesn't work a bit later. But you can see that my session is still valid. I don't have to log in again. My cookies are valid. To use it, hopefully, if you're using a load balancer, that would be seamless. I'm just going to show you from the clients as well. See in this on the client, the public configured server and also the CA server is that one at that address, so it hasn't got one or two in. It's the load balancing address. I'm going to post file 3, which is SmartProxy 2. So if we don't mess around that nicely, I think it might make some NTP changes, actually. Yeah, so it's started a NTP service. But again, nothing will happen. If we change that now to, so now I'm going to point that to SmartProxy 1, just change the IP, Puppage 1's T. Nothing's going to change again, hopefully. But to prove it, I have got the same catalog. If I stop, we restart NTP, because SmartProxy should then talk to forward and get data from the ENC. Have you hit the same? Again, in this scenario, if the CA server goes down, you can't sign new certificates so you can't deploy servers. But it helps with HA, obviously. If you don't care about that, don't worry about it. So in the future, we want provisioning. I would like to see provisioning done. So DHCP, the default DHCP daemon that we use, does support a secondary and primary server. I would like to see us be able to provision in a highly available manner. So Forman will configure DHCP on both nodes. So both of the SmartProxy's could respond to the DHCP and receive many DHCP requests, though, given. Again, TFTP could be quite simple. Forman would just need to know that these SmartProxy's here in a cluster. That's not currently possible within Forman. So Forman needs to know that. So when you deploy a new host, it can copy those any files that you're going to serve over DHCP to both nodes and not just the one that it knows or thinks it's going to use. Discovery, I think they need small changes in there. If you guys use discovery, there may be kind of dragons with other plugins, but I think on the whole, it should kind of mostly work. I think Catello, I don't know if you guys use Catello. We need to kind of move that to use split deployments we're calling. So Catello has made up a multiple components. So you've got pulp, you've got the formal application, you've got candlepin, you've got cupids. There's like five or six different components. Being able to split those out into different nodes and then make those nodes independently, highly available would be really nice. Also super cool if you want to start using containers and containerizing Forman or Catello. I think kind of HA and containers are kind of a must, really. Do you want to have any questions? Yes, you can do that as well. That's one of the nice features. Probably that would be better than directly kind of getting Ansible to cool Puppet, but it's a bit kind of weird. That's a super nice thing. One thing I forgot to mention is about upgrades. So not related to really to your question. But when you upgrade, you can upgrade, hopefully, upgrade one node. If you can upgrade one node and then run the DB1 node, you can upgrade one node and then run the DB1 node. If you've got three nodes, you upgrade one, so the UI is now one to version two or whatever the version is. You upgrade a secondary node. That will upgrade the UI and it will run the DB, Rake, Seed, Migrate. You've now then got two nodes in your cluster there, the new version and the database, the new version. And then you just have to upgrade the third one. If you can do that nicely and time taking the nodes in and out of the load balancer, well, you can get almost close to zero downtime when you do upgrades. But definitely a former's gut, yeah. The only thing is, obviously, you have to have another Puppet Master, which some people might not want. So the question was about Puppet CA and making that highly available. So I think, initially, it's kind of low on the requirements, if you like. People mostly care about being able to configure systems than they do about deploying new systems, HA. Definitely, if you're going to start doing provisioning, being able to make that HA is probably going to be needed. But it's not saying it needs to be solved within the Puppet community, I think. There's a couple of ways of doing it, like I did mention, but it depends on scale. It depends on what you're doing. There's a lot more choices there, I think. I don't know if there's necessarily a particularly right way to do it or a wrong way. Probably not, no, not initially. I would like to see it done. I just haven't got the expertise to do it. It's just quite simple. So the installers all configure SSL for you. So sorry, guys, the question was about SSL, why not just terminate it at the load balancers? The main reason for that is for the stability. I would have to go in and work out to get my certificates and put them at load balancers. Obviously, that's quite simple. But the installer supports passing your SSL certificates. So that was just as easy for me. I think you probably could do it. It would complicate things, especially with Puppets. Because on the form of proxy, you have two different SSL certificates. You've got Puppets certificates, and you've got the form of proxy certificates. It just adds another thing. I don't know, I mean, I don't think it's necessarily wrong. Any other questions? Does everyone understand the demo? Because I don't know, it was quite, I don't know.