 So, tonight we're going to talk about tiles of distribution and of course, I can get an intro to this slide. It's going to work really well. I'm Scott Colton. I'm based in Sydney, Australia. I work for Puppet and I work on core engineering. Actually, I'm now the tech lead for core engineering to do with cloud and containers. So, I actually don't work on Puppet server itself. I work on other products besides Puppet and the new roadmap and new technologies that we're going to bring out as other products besides Puppet server itself. You can find my GitHub at Scotty C or at Scott Colton. And that's my little boy who he's three now and he also wants to code. So, he's got all the stickers and stuff on his laptop. He's just learning to type at the moment and read. So, that's pretty cool. In this talk, what we'll cover is basically, I think it's already decided that containers will be the new unit of application. I think that we're pretty much the ecosystem is moving towards or has decided that that will be the way that container runtimes, that runtimes of applications will be packaged. But, how will we distribute those? And there's basically, I've spoken about this before. I'm going to go through two choices. There are other choices, but I don't see them as much of a choice as the other two. So, what options do we have to solve the distribution of the actual container image or the image or the container? So, first of all, we'll start talking about the orchestration layer and why that choice is so important. The network layer. So, we have to have a look at securing communications between our microservices and then persistent data because not every application can be stateless. Some sort of state has to reside somewhere in applications and if we need persistent storage with the orchestration layer, what do we have? So, in the orchestration layer, like this was actually, Brendan Burns said this from Microsoft. If the container is a new unit of application, orchestration should abstract away the complexity of underlying infrastructure. So, basically what that means is you should be able to run any infrastructure that you want and you should just have an API that you can deploy an application to. And I'll talk about it a bit later right at the end where Puppet kind of fits into this. But I think this statement is true. From an application development point of view, they shouldn't care if it's running in CentOS on Neve or Ubuntu or even Windows. They should just care about the orchestrator. The orchestrator should then take away the complexity of scaling and should look after routing and should look after a whole heap of sort of infrastructure tasks. So, I'm going to talk about two choices tonight and I see these as the main two. I know there's Messos as well, but kind of that's like a niche market. I want to talk about Kubernetes. I'm going to talk about Docker Swarm Mode and the differences between the two. Although most people will talk about them in the same sentence, they're actually completely separate entities and the way they solve problems is completely different. The thought process around the way they're developed is also completely different and the way they're actually marketed now will be also completely different and they're becoming more fractured as time goes along. So, why is the choice so important? At the moment as it stands, they have totally different methodologies of deployment. So, if you have a look at pods compared to services on Docker, they're completely different. They're no longer interchangeable. They're architecturally different from the applications point of view. So, again, if you define an application as a deployment on Kubernetes, that won't work on Swarm Mode and the Docker-composed version three file that works on Swarm Mode won't work on Kubernetes. So, although the reading, they both look YAML, they actually behave completely differently. The network stacks are no longer interchangeable. So, I think since Docker 1.9 when overlay networks come out, Google decided that that wasn't the networking methodology that they wanted and the overlay networks was no longer used by Kubernetes. They created the CNI interface and there's a lot of network stacks that work around that. So, that's not interchangeable anymore. And the stricter key value store is configure your own or is included. And that's a plus and minus depending on the skill set of your team and also what you're actually trying to run. So, we'll go through that as well. So, we look at choice one for the orchestrator as Kubernetes. I should just sign down a picture. This is some pretty dry reading. This is what Kubernetes looks like from an architecture point of view. So, basically it has the ideas of a pod, the kube API, then it's got the kube proxy, kube DNS, that pretty much makes it up. You can bolt in things like Prometheus, CD Advisor to get some sort of metrics and things out. But basically, you've got a controller node and you've got a worker node and they all bolt together. It's pretty simple architecture. But Kubernetes pods don't look like anything that's in the Docker world. So, straight away when you look at it, you can see here this is just a pod that deploys elk. This is just elk, that's all it is. This will get elk up and running for you. But unlike Docker where you kind of in the service world call things a little bit differently, in Kubernetes it gives you a lot more stuff around things like metadata. You'll notice the API version is also mentioned there. The namespace that you actually want to logically put the application in. Then you've got things like, if it's privileged or not, it's probably the same. But the way you do container pod is completely different and the image is the same because they both try to use OCI compliant images. So this methodology of deploying elk would not work on Docker or Swarm. And then that's just for a single application. So that's just a pod and if a pod has a foyer, it won't restart itself. It'll just die and that's it. So then Kubernetes come up with services. Services define how many of the application you want running. In here, we're just running Vault and we're just exposing the ports. If this Vault container died, it would then restart because the service would be defined that we want it one running all the time. So there's some sort of like self-healing here. And this worked really well. And this can work across multiple nodes or a single node. Kubernetes then was like, what happens if we want something a bit more complex in services? So they come up with deployments. What deployments actually give you is multiple services. So underneath deployments, it's actually running multiple services and those services are then spread across the nodes. So it's kind of a hierarchical layer of the way you deploy your application. And underneath services, sorry, I forgot to mention, are just pods, multiple versions of pods running. So what Kubernetes come up with the idea of a hierarchical data structure that you can then deploy applications with. There's nothing like this within the Docker world. Docker stack is a single layer. There's no hierarchy of it. There is like self-healing and desired state, but there's nothing as complex as what Kubernetes has deployed, which has got its pluses and its minuses, especially when you first start to use Kubernetes. Or if you come from a Docker background trying to go to Kubernetes, it's kind of like a little bit complex. You can see as well that this is still on extensions beta one. I'm not sure if it's off yet. I still think that the deployments are on beta one. But yeah, this actually passes data down through to the service and through to the pod. So as I said, it's a hierarchical structure of how to deploy your application. So that methodology is completely different to the Docker world. This is a major one that when I talk to customers or I talk to people that are having issues with Kubernetes, it always comes back to this. It comes back to that they have issues with the RTD cluster. And that could be whether they don't know what RTD is actually doing. They haven't thought about a backup and retention strategy for the cluster state. Or it's just they've got themselves into a state where the election has gone wrong and a node that knows about nothing has now become the leader. And that is by far the worst thing that can happen to you in the Kubernetes world. So one of the first things that like when I started learning VMware, someone said to me, if something goes wrong in VMware, it's always the storage that made it slow. I found if something goes wrong in Kubernetes, it's almost always RTD in the back end. So when we first started testing this back before it was one zero, we actually found that there was with an AWS when Flannel was the only interface for RTD for networking, sorry. The network latency between RTD and Flannel actually cost the cluster to keep failing over and reelect the leader. When we up the AWS instance those, and I mentioned again, this is before Kubernetes one zero, when we up the instance slides, then the issue went away because of that dedicated networking. So we traced it back to being between RTD and Flannel and it was seen nodes as down. So that's quite tricky, was quite tricky to troubleshoot. And that's why we start looking at swarm because at the scale that we were using in that in my past job, we didn't need to go for the complexity of Kubernetes for that particular use case. And we were in health, so we had a highly regulated compliance that we had to also adhere to. And some of the stuff in Kubernetes, we couldn't explain like removing the Docker bridge zero and then having to use Flannel and install stuff. When the auditors look at the install process, they're like, why are you installing something removing half of it and that. And that was just the way Kubernetes was done back in the day. But I have still seen issues with with STD, even running Kubernetes on top of Rancher or something silly like that. I've seen people get themselves in trouble not actually knowing what STD does and understanding what STD does for Kubernetes. They just like STD is the back end for it. But without actually understanding what the back end does, they don't understand the complexity and what it actually the data it stores and the importance on the cluster that STD has. So if you're gonna go down the Kubernetes road, I definitely would start with fully understanding what STD does. I'm looking at a backup and retention and strategy and also a resource strategy because if you've got a massive cluster of couple hundred nodes, maybe a thousand nodes and STD goes a bit wonky, you wanna know that you can able to restore the state of a cluster on another cluster if you need to and then migrate your workload onto that. And as you can see here, it's based on rafts. You have to use STD for Kubernetes. There's no other like choice you can have, like you can't use console or anything like that. You have to control and STD yourself and it is API driven. You can actually tap into STD yourself and get metrics out of it and check the health of it. And there's a whole lot of stuff built into Kubernetes to make sure that this is running properly. But a lot of people just like sort of glazed over STD and think like it's just a part of Kubernetes where it's probably one of the most important parts. So I really stress that if you're gonna go down Kubernetes and use Kubernetes, understand what STD does, understand the technology and play with it. Outside of the Kubernetes cluster, have a look at it as a distributed key value store, whether it's API calls to get applications or just to play around with it and test it out and see with it. Because it is its own entity and it has got its own complexities and you should really understand what it does. So now we'll look at trace two for orchestration and we'll look at Docker swarm. Docker swarm has a kind of a similar architecture as such where it has the master and then a worker node. So it has that idea. What Docker has gone for is a simplistic method of deploying the clustering. So they actually take the rough consensus store and they look after that for you. They encrypt it for you and they look after TLS. It comes between all the parts of the cluster and that's all turned on by default. So one of the things that swarm gives you is out of the box, a simple cluster that's easy to set up. There's been massive conversations about does swarm scale. I know Docker internally has done like a proof of concept that it runs on 2000 nodes. 2000 worker nodes. Again, it has the same limitations of the raft consensus that Kubernetes does. You shouldn't have too many managers. Probably seven would be enough because the more cluster operations you have the slower that the consensus will be. You can get yourself into trouble that way. But again, that's no different to Kubernetes because it's again using raft. But for seven manager nodes, you can do 2000 worker nodes. Kubernetes, obviously Google have done that at a much higher scale than that. But the beauty of this out of the box is that the internal distributed store is looked after for you. The raft consensus is looked after for you and built into the Docker engine is a snapshot functionality and also a restore functionality. So they kind of take out the complexity that NCD brings in to Kubernetes. And one of the things that Docker has really worked on at the moment is security by default, which on Kubernetes you can get the same security, but that's something that you have to take on yourself. Docker ships with better security out of the box. So they ship with key rotation straight out of the box. And Daigo from Docker, the head of security, has done a great talk about this on Container Camp in great detail. If you want to find out more about how this works, he done like a half an hour talk, pretty much just on this and how you can like mitigate men in the middle attacks. The CA can be hijacked and you can still, your cluster can still live. You can rotate the keys on the fly. How the key rotation's done, how it creates two CAs, puts a trust between both and can get a key off each and then removes the other one. There's a whole mechanism that's gone into this that is super, super cool and you could do a whole talk on itself. I really, you should check this out because it is something that Docker has worked on that I think he's really understated. And I don't know any distributed application at the moment that straight out of the box has this functionality that works. So not only is like the raft consensus being encrypted, that keys then and CAs being shared, worker nodes then get the key. The other thing that worker nodes can't do that I know in Kubernetes, obviously you can't do either because you need to control to actually access the cluster. But on Docker swarm mode, you can type Docker commands on a worker node. By default, it's got least privileged access. So you can't even question from a worker node how many nodes are in the cluster. Won't let you. It's got no privileges at all to any admin functionality. That even includes just questioning like simple facts about it. You can only do that from the master node. So that is like, they've really, really thought about probably more the enterprise grade clients. They've really thought about the UX side of things about how to make things really simple. Like you can literally set up a swarm mode cluster in about 30 seconds. Kubernetes obviously takes a bit longer. But there's more flexibility with Kubernetes. What, by making this more secure by default and also taking away some of the flexibility of it, it's very opinionated. And that's the only way that they could do this. So it doesn't give you the flexibility that you might want with Kubernetes where you can choose the networking stack. You can choose what you wanna do with ETD. You might actually wanna shard that off and use that for application as well to get key values from it or something like that. All that flexibility that's with Kubernetes is not there with swarm mode. It's opinionated and they've done that for a very simple reason that they wanted the UX to be really good and they wanted the security by default to be really good. They couldn't do that if it wasn't as opinionated. So how the application looks. It looks fairly similar to Kubernetes way because it's YAML but it's actually not hierarchical. It's a flat file that has all your services that call each other by default. So this is the only way you can deploy services across the cluster and it's really basic. And again, I think they've gone for a really simple UX and experience that someone can pick it up really easily and go, Redis needs DB, this is what we want, this is the constraints we want and that's all we're gonna do. There is a lot more options that you can put in there but yeah, it doesn't have the hierarchical sort of attributes that Kubernetes have where you have deployment services and pods. But again, it will get nodes into a desired, sorry containers into a desired state. So Redis you could put, you want replicator of two that always be two running if one died it would respawn another one. So very similar to the Kubernetes way of looking at HA. So the networking layer, this is where it gets quite interesting. We'll go to Kubernetes first. Kubernetes has a massive ecosystem of networking providers, absolutely massive. You can pretty much do anything on the networking layer on Kubernetes. So if you look at Contrails from Junipa and understand native Junipa, VLAN, Tagging and BGP open Contrails, the open source version of it. Flannel again has got a heap of functionality around it. I won't go through all of these. That was kind of the first sort of networking stack for Kubernetes that was external of Kubernetes also made by Chloras. You got Contiv, which is open source. You got Caliguram that does BGP. There's a whole heap. So if you have any networking needs that are really super complex, Kubernetes can handle that for you. There's no issue at all. But again, with that Kubernetes complexity comes responsibility and configuration. So it's a double-edged sword. I like, I think this is a really good way to go. But again, understand the stack that you need understand your use case and understand your business model and what you're trying to achieve. And then understand the choice you made and learn that technology. Because between the different networking providers, it's hard to swap and change with Kubernetes. It's not super easy to rip one out and do another one. It'd probably be easy just to build a new cluster, migrate it across and do some sort of a blue-green and things. But one of the things that Kubernetes really is about and relies on is the fact that it's an ecosystem. Dockus Warm is kind of like part of a Dockus platform. It's their ecosystem that's internal to them. So there's not many other providers writing network plugins for overlay. But Kubernetes has created the CNI, which is an open initiative for container networking. And a lot of people have made networking providers for it. So yeah, as I said, I personally, I think that the trace of networking providers is really awesome. So depending if you're like a massive telco that needs BGP to route stuff from internally to externally to understand the routes through the container network for services for something like CIP, Kubernetes can do that. If you want to just have feedline tagging to go all the way through from your physical networking, Kubernetes can easily do that. So yeah, that's really powerful. And the fact that you can do more complex networking on a more granular level in Kubernetes is a really interesting idea, especially if you're doing multi-tenancy clusters or you're looking at deploying multiple applications in a multi-tenant environment. You can do it with Swarmode. It doesn't give you the granularity that Kubernetes will with any of these products. It's just like kind of a more of a flat networks scheme. And some of these providers as well, they support VXLAN and encrypted between nodes. Some of them don't. And then Kubernetes Ingress is pretty easy. It kind of works like if you know AWS, the Ingress path, you don't have to use this, but you can. It will do layer seven routing for you. And that pretty much goes internet Ingress services. There's other applications that you can use. I just went for the default one. Kubernetes, as again, is an ecosystem. There's other front-ends you can use for it as well. But I just went with the actual Kubernetes Ingress. Kubernetes DNS service discovery is kind of one of the things that if I got to change something in Kubernetes, this would be it. I would actually, by default, out of the box, have this built-in to Kubernetes installing, not as part of the add-on manager, but as part of like Kube API, Kube Proxy. Because this kind of needs to be there. It's like, yes, you have a cluster up, yes, you have an API to deploy do, but if you deploy an application and can't find each other, it's kind of useless. So I would have this as part of actually the core part of Kubernetes, but that's kind of the flow. It's an externalized application that sits inside the cluster. It works as a deployment, so it runs on every node. It'll spin itself back up. It's actually got three containers in it. So it's got a sidekick, DNS mask, and the DNS server itself. The actual traffic goes to the Kube API and then to the DNS report and back out. And yeah, namespaces and services. It works fairly well, but as I said, I would like to see it as part of the core of Kubernetes offering, not something that add-on manager looks after. You can get into some issues with Kube DNS not working after deployments. There's some open issues on Kops about that and a few other things. But yeah, I think this would be best moved into the actual core parts of Kubernetes, but that's just my opinion. Other people might have different opinion. So choice two with Docker Swarmode is the Docker networking stack. So basically, it's overlay. By default, again, opinionated. Again, it runs on the raft distributed store that is not looked after by you. And again, it is encrypted by default. In saying that, of course, it's very opinionated. It's not as flexible. So there is an application by Cisco that's been just released or was just released to DockerCon that can actually plug into overlay. There is an actual network API and they're the first people to jump on it. So that will be able to understand security native vlan tagging and any of the routing on the Cisco side that you wanna pass through. But out of the box, Docker only has overlay and overlay vxlan. So this is basically what it does. Creates Docker zero, creates a v and talks to each container and it exposes a v as it's zero inside the container. And then if you wanna use vxlan, this is what vxlan does. It just does an eight to 11 queue, one queue trunk and through to each of the Docker containers. This actually works really, really well. And if you're going through like a compliance audit, this is super easy to set up and it's super easy to get a compliance across it. If you look at something like one of the providers to CNI, it's harder to explain what that's happening and there's a third party application talking to it. And then you'll start to get questions around the interface, how it talks, how that traffic's encrypted, whatnot. So again, Docker spent a lot of time making this secure by default. But again, to make it secure by default, it has to be opinionated. So there is definitely a use case for this. Some people don't need the complexity of all the stuff that the Kubernetes network can do. It might not need any of the BGP, might not need any of that. They might just want to have like six or seven different like lands inside the Docker swarm network and they'll be on their own IP addresses, services don't need to say anything besides what's on that land segment. And this will do the job perfectly. And it's one command to set it up. So yeah, there is a real use case to use this. And not take on the complexity of like the Kubernetes networking. Because it all comes down to really business decisions around how big your team is, what skill sets you got in-house and what deadlines you have around what you're trying to actually deploy on top of the cluster. You don't want to spend all your time admitting the cluster or not actually making improvements on the cluster itself. Docker, the DNS server discovery is, from my point of view, a lot simpler and easier to set up and get it working. It just works out of the box. So on 127.0.0.11, there's a DNS server in every container. That address then is mapped to the service and the container name. And that's stored in the raft consensus store. That's all looked after for you. So as soon as you have swarm mode enabled, DNS service discovery set up. You don't need to do anything. And that's just out of the box. Again, you can't actually really do much to the DNS server like you can with Kubernetes. So you can actually go in there and you can actually add records. You can actually curl it. You can get a whole heap of responses out of it. Docker, you kind of can through a GRPC interface, but it's not supported. It's really difficult. They wanted to make it kind of closed and simple to use and opinionated. But again, it hasn't got the flexibility, but this will work out of the box. This works flawlessly. This is probably one of the best things built into swarm mode that I like. It just, yeah, it works really, really well. The other thing that Docker has that the Ingress does is mesh routing. So basically you can hit a service on any one of the nodes in your swarm cluster and it'll route the traffic to where the container is. Doesn't matter if it's on that node or not. Kind of what Ingress does on Kubernetes. Except you don't need another service or set up another pod to do it. It's just turned on by default. Again, this is really cool. It's out of the box, works. The only confusing thing, this originally when it was released was called HTTP mesh routing. It's not actually HTTP or HTTPS mesh routing. It can only do single port. So if you've got two services running on port 80 on each node, you need to do something to handle the layer seven traffic above that. This will not do that for you. I don't really understand. You can only have one port. So either 80, 443. It doesn't understand. It will actually just like, it could go to any of the nodes that respond. So it doesn't understand that. There is a project from Evan Hazler, the works at Docker that is called interlock. It'll actually sit there as a service that'll listen to all the swarm events and it has an engine X cluster on top of it. When something migrates or moves, it then updates engine X. So when the request comes in, it handles the HTTP headers and points to the right port. So Docker did change name of this lately to mesh routing because when they released it at DockerCon 2016, they said HTTP mesh routing and people were like, awesome, put heat support 80 servers on it and they're like, oh, this is not working. And they're like, oh, that's not actually HTTP if you use more than one HTTP port. So they've made it mesh routing, which it actually is. Again, it's opinionated, but again, this is like one command runs swarm in it and this is all working. So everything I've spoken about is this one command. Persistent storage. This is actually a really, really interesting conversation to have. I'll speak about Kubernetes first. Kubernetes actually believes that all clusters should be stateless. And Google are talking about it and you talk to anyone, they say they should be stateless. Yet they actually have the best options for persistent storage out of any orchestrator. So although that they say that it should be and that's probably what they internally do, they know not everyone can be like that. So they've got the host pass, of course, they've got cloud vendors. So there's a cloud plugin for Kubernetes that understands GCE and AWS. I hear Azure is coming now that Brendan Burns is at AWS. Sorry, Azure. It understands internal clouds like OpenStack. It understands Swift and understands VMware storage. And there's third party ecosystem tools built around it like Portworks that can be an intermediary API that can talk to these if you don't want to use the native cloud providers. So although they say that all containers should be stateless, they also have a real good ecosystem around what it can use. Docker Swarm Mode. Unfortunately, at the moment, Docker Swarm Mode only supports NFS and host pass. Docker has actually acquired a storage network, a storage company. They have stuff coming that's in the works but it's not been released yet. So they are a little bit behind the eight ball. Talking to them at DockerCon this year, one of the first things that they're looking at is an image cache that spans the Swarm cluster. Because one thing Kubernetes does and one thing that Docker Swarm does is there's not an image cache that's cross node, which is super annoying because you have to download the same container on every node or the same image on every node. So I think that that'll be a real value add. And then from there, I think they're gonna build it up to have some more native sort of persistent storage. But at the moment, I see the people at Docker actually working on an image cache that's across the cluster. They see that as the biggest issue, not actually persistent storage. There was actually a really, really good application called Flocker that was out by cluster HQ. And unfortunately, they solved this problem before everyone else thought it was a problem and went bankrupt. So they actually had a really, really good application that was Kubernetes or Swarm Mode agnostic that spoke to all the cloud providers and would actually create like an EBS volume you could attach your container to it. If the container, then that host died and the container spun up on the other, and none of the node, it would attach to the same EBS volume, which was awesome. I actually use it in my past job. And I thought this is one of the best things ever that I had not many people use it because like, although it was kind of like an enterprise-y product, there's not that many people using containers in production yet that have seen that this is an issue and they kind of solved the problem way before anyone would pay money to say, actually, I have this issue, which was actually really sad. It's still open source, it's still on GitHub. You can use it, but it's just not under support. How about Rexray, the bloggable architecture that you can see came up with. You can pretty much plug every possible storage array into any swarm node, worker node. Yeah. And then there's like an extension separate that allows you to configure it as still a postcard. Yep. So there is Rexray. I've heard really bad things about it to tell you the truth. I'll do. Yeah. Yeah, so I've not heard good things about it. The guys from EMC Australia that are on open source have been trying to get me to test the pages, but yeah, we haven't used it yet. Yeah. You could add Rexray. I have heard of it, but yeah. What do you see in practice? I mean, from the perspective of real-world largest on-prem, not public-bound importance, what do you see people using NFS? So I see people using creating LVM volumes on NFS and then using device mapper to map across that. That's what I see on internal. So just two different non-points and then? Yeah. So yeah, real-level, primitive sort of networking stuff. But yeah. So I'm actually working with the hardware vendor at the moment to install UCP on top of their hardware. And that's what they're going to be using. They're going to be using different mount point, mounted as an LVM volume, and then let the engine do something pulling across the top of it for the Docker, for Swarm. For it to use the converters. Yeah, yeah. So that's what I'm seeing customers use. Yeah, but yeah, there is Rexray. I should probably add that. Thanks for that. No, that's all right. So what's the best tool for the job? I'll ask you. I don't think there's a right or wrong answer. Really look at your business use case. Look at the team and skill set you've got. And then also look at what you're trying to solve. Ask yourself that. Try. I know at the moment there's a massive religious debate about Docker and Kubernetes. And they're kind of like being the Twitter thing and all that. But try to step back from that. Actually have a look at your business value. What you're trying to achieve. The complexity that you want to take on. All these should go into the designed decision of what you're using. And they could probably possibly use case in your business while you use Swarm from one application and Kubernetes for another. It could be possible. Not saying that that would be something that I would do. I would probably standardize on a container orchestration. But there's no right or wrong answer to that. And at Puppet, we have both covered. So we have modules written that will control Kubernetes. That is at the moment. I'll show you actually the first demo of that ever. And that will be released to PuppetConf this year. We also have Docker Swarm modes already out and on Puppet forge. And we have Docker UCP or Docker data center already out on the forge as well. So no matter what trace you make, we support it. My team actually looks after all the modules to do all this. So if you have any feature requests or anything like that, just let me know. And we'll look at it. And internally and probably we actually have just decided we're actually going with Kubernetes. So we went through all these, tested all of it, and decided which one would go with. And Kubernetes was the trace that we made. So that was the right answer for us. For various different reasons of the applications we're deploying, the complexity of what we need it, and like back end, and happening to be a classic of both cloud and internal, Kubernetes seems like a really good play for us. So I should show you a quick demo. This is nothing really that exciting. Actually, I might not demo because the internet actually is a little bit on the slow side. I probably should have done this. But this is basically, we're controlling Kubernetes. The Kubernetes module that we're going to play will handle all the TLS out of the box for you, whether you want to use external CA or hard code the cluster. So hard code the certificates as a file type. We'll also look after the management of all the actual Kubernetes parts. We'll have options to install the network and provide it that you want with one single parameter. We'll also do you want the kube-dns, sorry, the kube dashboard, all the sort of bells and whistles that go with it will be a parameter yes or no flags. And this will basically be a supported module from Puppet. So if you have issues with the cluster, it will also obviously configure HCD. If you have any issues with the cluster, you can actually get support from Puppet if you're a PE customer, which is a really good value add. At the moment, this is just going to install a whole heap of packages. I'll probably should keep this off a little bit earlier. Yeah, but I think there's a real strong method for. I think there's a real good play for us because setting up a Kubernetes cluster from an operations point of view, like an SRE point of view, is not the easiest task. And there is, like I said, spoke about the complexities. So we're trying to abstract the complexity of building the cluster and parameterizing that and making it more user-friendly than Kubernetes to install these at the moment off the bat. And as I said, the story that I went through tonight was the testing that we did about which one would go with. And as I said, we put it all together. We actually went with Kubernetes internally. And that was for a few different reasons. But I won't go through the different reasons why we went through it. But we did test one mode as well. Actually, probably one of the biggest facts is probably we wanted to make ourselves agnostic of, so as you can see here, it's just going to install Docker 112. We want to make ourselves agnostic kind of a vendor. The fact that Kubernetes is not a vendor product. It's actually owned by CCNF. And it will very shortly probably run on solid container in RunC, which, again, open source technology is not owned by a vendor. As we also shipped software, we didn't want to be tied down to another vendor's release cycle for us. And yeah, Kubernetes is completely not vendor-related. Although there's different vendors supporting it and kind of developing against it, there's no one vendor that owns Kubernetes. There's no one vendor that owns container D or RunC. So this is just installing the package from app. It's using KubeLite to do a system D service that will control the cluster. It's just going to wait now until the cluster APIs up. Once the APIs up, it's going to install KubeDNS and then KubeProxy as well. And then that's it. The cluster will be there. So with that one command, I'll just figure it out. It automatically deployed the server for me and deployed Kubernetes. And that's all the work we did. At the moment, this is alpha, as I said. So some of the parameters are still hard coded because I'm still developing this. But by release time, there will be just a few parameters. Probably about, actually, there'll probably be about 30 or 40, 30 plus parameters that you can choose to configure. Or I'll put some same devolts in there of stuff that most people won't want to like configure. And I wanted to give it the flexibility that Kubernetes has if you want to be a really advanced user. But then if you only knew to Kubernetes, it would just work out of the box. So that's some of the UX stuff that I've been working on the module. As I said, this is going to depend on how quick the internet is here. So it's downloading all the containers it needs. So am I taking a second? While I'm doing that, I'll just build Swam on the other side on the Ubuntu. You can see as well, straight out of the box with public code. I've just, there you go, Kubernetes is built. So if we go, whoop. I did mean get not get. There you go. KubeMaster's ready. So simply typing, like spinning up a box and automatically applying the class Kubernetes to that box. Pop it just in sort of Kubernetes. That's one of the easiest in sort of Kubernetes I've seen. And that's one of the things I've really been working on. Because I think there's real value in that to abstract some of the complexity. For most people, I think that they're not going to need all the complexity and configuration that's there. But I want to allow them to do that as they grow, or if they need to grow. Again, it's installing Swam on the other side. Swam mode and doing it to play a service. Again, I've just applied the Docker class with some Swam configurations. So I've just applied the listening address and the advertised address. And this will build the cluster for me. So again, with Puppet, we're really working on taking out the complexity of cluster orchestration and really, really working on being able to make it easy to use for anyone that uses Puppet. So even though you've used Puppet before and you might be new to Docker or you might be new to Kubernetes, through the same language that you've used to define all your other infrastructure, you'll be able to find more advanced tasks in cluster orchestration. And through reading the code that we've got, you'll be able to understand and bias most of this, learn more about that product itself than having to go to the GitHub repo or use something that Kubernetes has at the moment to deploy it. So I think that's really powerful. While that's running, I'll take any questions. What do you need to know about it? Um, are the codes going to be released? That's not good. Are we unable to connect to our public web repository? OK, that file, because it can't connect to our public web repository, which is not good. Like, when you choose there are a lot of, there's a mini queue, there's payoffs, there's a lot of platform modules. So are you seeing the module that Puppet lack is? Yeah, so we're trying to, first of all, we're trying to get ourselves as a, like, a partner with people to get into the container market. So we're actually talking to Google at the moment and some people in CCNF to get our, like, stuff on to the Kubernetes website. This is more for people that are using Puppet that want to go to there, but we're hoping that also we're drawing new customers with the ease of use of this particular module compared to something like Terraform. Minikube is good for development. So basically what I've kind of ripped off in this module is I've taken the bits I liked on K-Ops, the bits I liked on Kube Admin, and Kubernetes itself, and I've taken all those bits I liked of those three projects and put it in public code. So yeah, the shortcomings that either all of those had. Again, I probably could have contributed to those products, but then it would tie our module release to that particular, so I wanted to couple that, and that's the only reason I didn't contribute back to those products because I wanted to couple and not have a reliance on someone else's development cycle to ours. Yeah. Could you break me if I'm wrong? Puppet, if I'm not mistaken, does install package upgrades. It's more of like a mutable upgrade, something like many years ago there was a package manager. So Puppet is a beautiful server client architecture that allows you to kind of make sure that your app is up-to-date to the developer's requirements if there are pre-rex changes for them. While, say, Terraform or someone else just basically does what a builder would do. It just completely rebuilds the image towards the destination platform, and it can plug it into the Canary scenario, and you can have like a gloss break button to roll back. How does the rollback work with Puppet when it comes to containers and updates of the prerequisites? So how do you ensure? Do you rebuild from scratch as well with all the plugins for, say, Kubernetes or Docker misses? Or the prerequisites updates would go through the client within the container, which has to be part of the image? Sorry, you're talking about the application running on top of that? So we're not playing the, we do have a module that will deploy pods on top of Kubernetes, but I don't see that there's real value for us at the moment to play in that. That would be something that, like a CD processor would play in. What we're worried about is the state of the cluster, the version of the cluster, updating the cluster, the security controls around the cluster, all the stuff that between the cluster and the OS, that's what we care about. Upgrading the cluster, rolling updates of the actual Kubernetes binaries and stuff, that's what we care about. And if there's sort of pod deployment that's not core Kubernetes-like functionality like Kube DNS, Kube Proxy, Kube API, Puppet at the moment doesn't, we shouldn't care about that. We're looking more from the operations point of view or the SRE DevOps point of view from behind it and the actual applications itself, although we have a module that can control it. There's a lot of functionality building the Kubernetes to do this that's already there. We don't want to play with that because it's good. That was my quick thing, thanks, Frans. Yeah, yeah, we don't want to actually... There's a peer line. Yeah, yeah, so we play below the Kubernetes API. Another question you talked about upgrading the cluster. Have you looked at the bootcube from CoreOS, like the self-hosted Kubernetes deployment that is rolling updates of Kubernetes itself? No, I'll catch on it. I think Tectonic is using it, but like... So the idea is to bootstrap, like to create a bootstrap Kubernetes cluster that then bootstraps the cluster itself. And then if you do need to upgrade the version, you can kind of upgrade. No, I haven't. That might be interesting also. But it's from CoreOS, so they do mostly like imutable. So it's basically managing communities. Yeah, it's interesting. Yeah. There's some other cool things I've seen from Kube Admin using LinuxKit as well, for fully immutable. So yeah, there's some interesting stuff around LinuxKit as well from Docker. LinuxKit is one of the best projects I've seen for a long time. I really enjoy it. There's some cool stuff that can be heard. So around that whole really immutable infrastructure, I don't have any answers to where we'll play it in that space. But this is more towards people that are already running, like, CentOS and stuff. According to the customers that I speak to, there's not a lot of people that are running CoreOS that we see. So. So what I understand is that basically your module allows you to have a couple of roles within Hovered. So you would say one machine, if I got seven machines for the Hovered, sorry, Kubernetes monsters, and then the rest of the way is for them to be workers. Yeah. And what I'm trying to achieve is basically to integrate in their role now, Kubernetes cluster into the usual Hovered run. Yeah. That's your aim, right? Yeah. So my question would be then, if it related to the corresponding system. So if we run, supply should make large cluster, like, couple thousand nodes. And then, obviously, we want to have all the usual things within Hovered. Like, for example, we would want to have kind of a deployment of new version of the Kubernetes management and then having the whole step of S1 through it before we actually roll it out. Are you aiming to, like, with your module, are you aiming to simplify that? Everything except what you said about testing. So we won't ever look at testing the cluster. Yes. So basically, hopefully that we would be able to use something like Hira to extrapolate and say you got seven, the first two nodes are going to be this version of Kubernetes, the other, and upgrade. And then you would have to externally run your tests. And then on the next run, if the test was successful, you could have, like, a webhook that would then upgrade the other nodes. Okay. I think this is not learning. What I mean is, say, we, like, write about our application plan. And then we want to cluster, like, do that. On the, sorry, on the cluster or the applications? On the cluster. Yeah. We obviously would want to make it, so that we don't have to shut down the applications. Yeah. So we need to kind of roll in update. And I wonder, how your energy is with the talk. I guess, like, let me help here. I guess the cannery, off and upgrade, it's your whole vision. Yeah, sure. Kubernetes cluster. But there are dependencies within the cluster. They are not easily kind of cannery-ball. They are single points. Yeah. So you really have to have F, UAT, separate. Yeah, that's not what I want to have. I think it would be easier, right? Yeah, well, sure. The architecture of Kubernetes itself is prohibiting the cannery testing for certain functionalities that are cluster-wide critical dependencies. So you basically cannot, within the orchestration layer that Puppet provides or any other vendor provides, you cannot just upgrade a piece of Kubernetes cluster, test it, and then roll back. Thanks. I'm either, but I'm pretty sure that there is a point. What I mean is, suppose I define in here, somewhere like, and I want to have in that side, I want to have a different version, and then I would probably be able to pass the variables through your module into an actual other program. Yeah. Right? Puppet takes a picture. Yeah. And then I want, of course, from the host, I want them to have an ability to inform the application of what I'm running on right now. But this is a cannery deployment experience. The usual way how you would do that in the Docker universe, you pass the variables from the host into container. And so my question would be, and I'm just now thinking about that, to be honest, my question would be, would you support something like that within your module, or like, do you ever think about something in that area? Not really, because I don't quite understand what you're doing, so take away Puppet itself. So if you did an application of like Kubernetes, an upgrade of it, and say you use Kube Admin, for example, and then you wanted to tell your application running on top of it, how would Kube Admin know to tell your application? Yeah, but the environment there, what in the pod? But the pod deployment's not what we're looking at. So it's not looking at pod deployments. We're not looking at pod deployments. Yeah, that's what he's saying, yeah. So it wouldn't matter if it was Kube Admin, or it wouldn't matter if you were using KOps. The underlying infrastructure that you're talking about is not the concern of this module, nor would it be of KOps or Kube Admin. That would be passed as an environment variable through the pod to have a look at maybe the Kubernetes version. So that would be something that your application logic would have to know, not something that this module would support. Because I don't think KOps or Kube Admin would support anything like that either, from my understanding. Yeah, because yeah, I don't see a massive use case. I think what you highlighted also, it's the focus on the management of HCD, where KOps, I mean, by default, there's not a lot of configuration on the HCD side, and that would be a focus on the private modules as well. Yeah, yeah. So that's one of the things that I'm really very concerned on is translating something into complexity and adding some of the functionality to it. Now, on the first release of this module, it won't be there, but one of the plans I have is to be able to implement a backup strategy of the cluster state by passing a flag of a snapshot time and some sort of functionality to do a local snapshot to a node or nodes, or an S3 bucket, perhaps. I'm not quite sure of the architecture of how that's going to work, but something similar to what the raw snapshots are in Docker. Did you play with HCD operator? No, I have seen it, yeah. I mean, when you set up an HCD operator, you also specify a snapshot interval, and it basically automates all of it, and it does disaster recovery. Like, if your HCD cluster loses consensus, it will see it from the snapshot and look for recovery. So it might be something like I could actually use that as the mechanism that you use to pop it to control it, so it's completely in control. So pop it, we'll have a look. Use that, check the backups there, check everything's all there. So kind of continually do that for you, but take away the operational overhead that you have in the check and stuff. And then if the pop it run fails, because the backup's not there, you can get a stack notification or some of that from pop it. We've got a whole heap of infrastructure around that sort of messaging service side. But yeah, it's kind of like making it easy for people to bolt all that stuff together, because at the moment, unless you're like fully, like in the Kubernetes project yourself and needy-apen it, from someone standing outside, that stuff doesn't look so easy. And from the customers that I've gone to, because I myself took advantage, took a throw advantage that everyone knew about Kubernetes and containers as much as the experience that I had, and then going to actually out to the customers and speaking to people that are trying this, there's some people are really struggling and it's really hard to get your head around. And when you're inside that bubble, it's like, you're like, oh yeah, I'll just use this and I'll do that and that. But when that's all foreign to you and it's like kind of speaking another language, it's hard to get your toes wet and know what you're doing. And that's what I'm trying to look at for, like our clients, for example, is writing something but that they can take and look at the code that's in there. And by osmosis knowing pop it, can easily read what it's doing and then learn about Kubernetes that way. And then they might move on to something because they might not use this module to control Kubernetes anymore. They might grow past it. They might not be, they might find out something completely different than they've written themselves, but they still use it to learn. And I think that's kind of cool as well. So the license of Puppet itself or the license of this module? Yeah, so the license of this module will be MIT, it'll be open source and it'll be free. But there's Puppet Enterprise and then Puppet open source, so that's a completely different conversation. But yeah, the actual module itself will be open source. It's going to be available until October. Yeah, so it's going to be released at Puppet Prom through my Puppet Prom speak. You guys are the first? Yeah, well, it's only in Alpha at the moment. So I'll actually call out to some people and try to get some beta testers before October. So if you're interested in that, you can see Terrence and yeah, we can try to get some people to estimate it. As I said, you guys are the first to even hear about this and see it. I actually only started to play in this module a week ago developing it because I knew I was coming here and I wanted to do something cool because I traveled all the way from Sydney. So yeah, I found out I've got the Puppet Comp to talk and we had the business decision to write the module and this is a week's worth of work. So yeah, I just wanted to demo something to kind of show the UX experience and how simple it was. So that was kind of, there's a lot of hard coded stuff in the moment that I need to parameterize and I also need to write the part for the worker node. It's not written, only the master. But as I said, that five days work and I had no ticket at the time for that. That was like just doing it in my lunch breaks and stuff. Any other questions? That's all. All right, thanks and thanks for having me.