 So, we have our next session by Tyler Urbeck and Kanza Kavili, and they will be talking about it wasn't DNS in CICD with Kubei Redd and Tecton. I'm just waiting for them to join in. Hello. Hello. How are you doing? Thank you. I'm fine. How are you? Good. Just waiting on Tyler. I think he's trying to connect, but let me see if it's here. There we go. All right. I figured it out. Perfect. You guys have the floor. All right. Let me share my screen to start. Present. Yes. Screen two. Is it all good? Cool. All right. Let's kick it off. Welcome, everyone, for our talk called It Wasn't DNS. We checked. It's CICD for infrastructure with Kubei Redd and Tecton. Hello, everybody. I'm Tyler Arbeck, I'm an architect at Red Hat Open Innovation Labs, focusing mostly around site reliability engineering. Kind of have a passion for this space of automation and pipelines and things of that nature. And as you'll see soon, making things do things that they were probably not planned to be, how they were planned to be used in the first place. So, if you want to feel free to reach out, if you have any questions after this, here's all the places you can find me. I'm here with Chansu today. Thank you. My name is Chansu. I'm a site reliability engineer at Red Hat. I work with companies to help them adopt DevOps cultural practices, help them to grow a SRE mindset within their organization. And well, it is at the beginning of September. And as Tyler said, you can reach out to release our links. Feel free to send a DM or reach out to us if you have any questions. So, let's get off. Okay. What are we going to talk today? So, we're going to talk about problems that we have today and how can we move us closer with the stuff we have already. So, let me draw you a picture. So, I'm sure that we are all enjoying our CI CD pipelines, you know, the benefit of having an automated pipeline is places already known and experienced many of us, I'm sure. So, it checks my, well, my automated pipeline checks my code quality. It runs tests for me. Then it checks some more security stuff. Then it deployed. But then again, it run bunch of additional tests, some integration tests, performance tests, et cetera. And it's all good. It promotes my code to the next stage. So, it helps us basically live in the good life because it just saves me from myself. But yet, I know that some of you can think some examples, even though when your pipeline is green, everything was fine, but still something was wrong. And why is that? Because having a reliable application pipeline is only half of the battle. You need some reliable infrastructure too. And I know that we all do our best for our infrastructure and we all practice an infrastructure as code as much as possible, which gives us the option to share the understanding over our infrastructure. They happen to us test our infrastructure in a repeatable way. But do we really have the chance to test everything? What do you think, Tyler? Yeah, absolutely not. I mean, there's a phrase saying that it's always DNS for a reason, right? There are just holes in the testing strategy that we see in a lot of places today where an application may be fully tested from front to end. But any time you start touching on infrastructure, oh, that's too complicated. That'll cost us too much money. We don't have the resources. We don't have the hardware to be able to test this out. So from what we've seen out in the field, there's just this gaping hole around infrastructure style testing that we decided, hey, it's time. We need to have a better way to do this so that we don't need to scream at the sky and say it was DNS again. That's why we wanted to introduce you with some, I can't say enough analogy, but it's a nice way to put this in, which is ghost ships. Tyler, would you like to say a few words about ghost ships? Absolutely. Let me weave you all a tale here, right? So for those that aren't familiar with the term ghost ships, ghost ships are just these, for one reason or another, whether fiction or fable, are these ships that just roam the sea with no crew, whether something tragic has happened to this crew, or if it's something like the Flying Dutchman where it's a haunted ship, right? But ultimately, there are no live passengers that can be hurt. I kind of came across, you know, I was probably watching a movie with my daughter and there was a ghost ship on it, but kind of came up with this similar idea of, hey, why can't we do the same thing with our technology environments, with our hardware, with our infrastructure environments? We have these ghost environments that ghost ships look like a ship, float like a ship, kind of go along the water like a ship, must be a ship. Same thing with these ghost environments. Looks like prod, feels like prod. Just no real people operating in that environment, right? So we really like, we like talking about these environment, these ghost environments. Hey, you have your DNS systems, you have your things like mainframes, any kind of like hardware based system, you know, infrastructure that you have, or not software based, you could do load balancers, things like that. But, you know, spin them up, make them look just like prod, but no people there to be impacted with it. So that's pressure one. Point one is like, hey, there's no people that can be impacted by this. So even if something does go wrong as part of our tests, we are not hurting anybody, which is the key part there. The second part is, you know, on top of that, this type of testing allows us to get to the point of, we don't need to actually have this hardware that's solely dedicated to this. That gets super expensive when you need to dedicate hardware, you know, dedicate, you know, bare metal just to do these things. Why can't we just spin this up in, you know, in a virtual environment or on a cluster with, you know, various amount of tools that we have, so that we can, you know, one, take advantage of common platforms, but also we don't need to, you know, have all these expenses to take care of, you know, these dedicated resources. So we decided, hey, what tools do we have that will allow us to do this? So that brings us to bring our first members on board this, on board these Go ships. And Chanse, I'll let you take it from there. Yeah, it's Go ship, no people's gonna hurt, but we introduced, we onboarded to our favorite tools into our ships. One is Cubebird, which is absolutely our favorite, is basically here to help us bring our virtual machines into container workflows by running a virtual machine within a container. It's super exciting, it sounds super exciting. Whenever I try, I work with Cubebird, it just makes me super, you know, hype and because we can develop, manage and deploy virtual machines side by side with our containers and we can take advantage of all the simplicity, all the speed of containers and Kubernetes and this really, really empowering. This really empowers us with all the flexibility, all the agility we gain with containers and, you know, it's easy to deploy on top of Kubernetes in no point shift. It brings some custom, custom resource definitions. It's easy to spin up a virtual machines and it's really helped you to standardize things and it's helped you to use all these best practices from Kubernetes like scaling, overcommitment, high availability, all these things. And another thing that we onboarded is, I'm just a bit teared down, sorry for that, but we love it, it's our peaceful cat tecton and we use tecton for CI CD part because we again wanted to go with another Kubernetes native framework. So tecton is here to help us build our pipelines for testing and tecton pipeline is quite, to be honest it's quite different what we used in terms of CI CD tools because there is no engine to manage to start with. There's a core component of tecton, there's pipelines, runs as controller in Kubernetes. There is no engine or plugins that you need to manage. There are bunch of custom resources that you need to deal with which is, it seems intimidating at the beginning but we're gonna go through it in a minute. So once you get used to it, it seems super handy and alongside with Kubernetes, alongside with Kubernetes, we can make this ship work as our needs for our needs. And one thing, we get this anytime we kind of start talking about this. Oh, does it need to be tecton? I've got all these other CI CD tools. What about Jenkins? What about system XYZ? And really it could be any tool that you would like. This doesn't just work. This isn't a tecton specific thing. This is a whatever CI process that you want to use. We really like tecton because I'm sorry, I'll use some buzzwords here. We like the cloud native approach. We like being able to declare, to literally define all the stuff. We love our YAML, so give us all the YAML you got here. But tecton is just one approach here. You could substitute in your CI system of choice but what we'll show here today is going to be tecton. All right. It's a good thing that this is live but let's see if we're gonna sync or... So I'm taking my ship away and we're gonna do a little live demo of how this is in. So which one would you like to start with Talon? Would you like me to go through the code first or try to see... Yeah, let's look at the visual first to give folks an idea of what we're looking at and then we can make them break out the spyglass for looking at some YAML here. All right, so we have our pipeline here which is, well, we tested right before we come so it's all green. So basically this pipeline, what does this pipeline do is it gets the... Let me start from over. So we're gonna test our Ansible Playbook which we use for installing the DNS and configuring the DNS on top of it which is a very, very core component and important thing that you need to test whenever you do some changes or you need to do some tests before you roll the changes on the production you don't have to necessarily have a test environment or just for a single line changes in your playbook you don't really necessarily have a test environment to test this. So in this pipeline, we get our inventories, get our playbook and check our playbook if it is healthy or not. In the meantime, we are creating the virtual machines inside the apprenticeship and well, it takes a bit of time so we wait for this virtual machine to be out and then since this is a new virtual machine it has a new IP we get this IP to create our inventory and then we test our playbook towards this and then verify our configuration with an additional tasks to see if that DNS is configured and work as we expect. Taylor would you like to add something more because you love this. No, that was all good. It's definitely this whole process is why are we doing this? DNS is a fun thing to poke at because everyone has scars from DNS. And the way that people operate DNS isn't always the... What's the nice way to say it? It's not the most practical or best maintained system. It's always let me go in and update this record and let me go in and just update this config and human hands make human mistakes. We forget the bracket. We typo the name that we want and we'll end up with errors. We start impacting our users which at the end of the day is the thing that drives this whole idea of why we want to do this. Creating these ghost environments we're able to spin this up just like our production DNS server and then at the end we check to make sure that we get the responses we would like based off the changes that we're making and if it's all green then you can build out your pipeline from there. You can automatically progress it through your other environments until it hits production. So that's the whole thinking behind this and why we are using this example because we see this a lot so we want folks to kind of latch onto this idea. But I promised a live demo so it's better if I should kick this because it takes time and then talk about it a little bit more detail so let's see if I have my little code here. While you're kicking that off I'll explain why we keep saying it takes some time. So we are running on a fairly small cluster this is just one of our dev clusters and also we're running in an emulated environment so the performance gets drastically better when you're not emulating when you don't have emulation turned on with Kuber so if you're working on bare metal or systems that you don't need to turn on emulation for you're going to be in good shape performance much better but since we're just virtualizing and emulating everything a bit slower which is why here in a second when you see how long it took for some of these things to run you'll see like a 40 minute runtime what are you dealing with? Give me a sec I just wanted to check if everything is fine with my webbook it should have triggered this one I don't know why it didn't but because it's live demo it just worked it's not 45 minutes ago as it's in here but not working that now because it's demo God's it's just hating us okay now we are going to back in good shape sorry for that we've done this like three times some of the problems three of them were different and that's why we have CI to catch those problems it's because I didn't push it I committed but I'm too excited for it sorry I didn't push it anyway so what components we have of course we have pipeline I'm going to come in a minute but Tecton gives us the a component called event listener which basically gives you a webbook for you to set up in your g3 repository and you can set up for different events listeners for different actions like for one push and each of them have these definitions for trigger which one you're going to trigger and what sort of values that you're going to bind from your payload so and we have we have that's why we have all these trigger binding definitions and some trigger trigger template definitions yes it's hard to remember this and these are basically helps you to utilize from utilize getting the values from your payload and put them into the into the pipelines that you're going to trigger which is this pipeline and this pipeline consists of all these tasks that you just saw and it basically takes the values and pass them to the tasks and you can do this some sort of ordering or some parallelization or retries et cetera all these kind of things and you have these test definitions that's why I said that it's a little bit hard to get on board with Tecton at the beginning but once you get used to them it's it's fun because one it really helps put around base scripts I don't have to deal with Grimby and all the other stuff I love Bash I write base scripts to do some stuff in here but don't be intimidated with all these long stuff because we didn't write some of them you basically and thanks for this for the Cupid community we got some ready tasks from Cupid itself like creating the virtual machines waiting for if virtual machine is ready or not cleaning up for the virtual machines all they need to do is just provide some values or our virtual machine definitions or the name of the virtual machines things like this and thanks community again for this it takes care of it all and we just add additional tasks around like cloning the guitar, ensemble playbook creating the inventory based on the new virtual machines and taking off this ensemble for that and it should be still running should be still running yes but also I can tell about this Tyler yep no I just echoing what you said we prefer to be sticking with the ship theme here we prefer to be pirates we much prefer to plunder things that we can find in the community versus having to write our own I think throughout this whole pipeline there was maybe one task that we wrote ourselves everything else was just things that we were able to pull from the community provide our parameters to and off to the races so you know this may look like a lot of yaml and it is a lot of yaml but we didn't need to write all of it right we were able to just pull it in apply it to the cluster and it was available for us to use and while it is still running the ensemble part is also coming from our own infrastructure and simple playbooks because we use ensemble heavily to manage our own infrastructure and look at the number of these ensemble roles and playbooks it's it's a lot the idea was coming from basically this these are we have all these environments okay we have open stack environments we have our virtual machine environments but I just want to check something based on my little changes like trigger something and to see if it works fine to see if it is fine etc and you can't really have different testing environments for it or you have a bunch of people working on different stuff you can't have well you can't have some sort of consolidated test environment but still it might not be enough for you so with all these sensible playbooks we basically come from that idea saying that just want to test this little stuff I don't want to break anything else but it's on our other side and we all know that reliable infrastructure is really important so yeah so let's actually take a look at our pipeline while this is still likely going to be running and to take a look at what it's actually doing right and I think this is a really good example of how to bridge the gap of some of your infrastructures code your config management tools like Ansible can be bridged with a Kubernetes native way of testing some of these things because we're not just testing the deployments the things that pop into our cluster but we're also in this pipeline we're testing the things that help us build out our entire environment so we're not just testing that our manifests look good we're not just testing that we can spin up a VM inside of Kubernetes if we dig over to the pipeline one thing we're actually doing is we're linting the Ansible because if something goes wrong and there's something wrong with our Ansible we don't want to waste all the time to see a failed VM build right so we're testing everything the whole way through our process from the things that we're going to use to build our VM to the building of the VM to what pops out after the Ansible has run so you can see we've got Ansible linting happening there and a lot of times there are things we like to ignore so you can see that you can even throw in a skip list of Ansible linting roles that you don't like and this is just a warning but it's still a good warning it says hey you're referencing a file that doesn't even exist in there so it probably should be more than a warning but it still works nonetheless we obviously are not referencing that part of the play book so it's cool to show even running inside of Kubernetes running inside of OpenShift you're able to test out this full platform and again this takes a while because we're testing it from front to back but then one thing you could do even in the future is okay I don't need to test out the building of my DNS server but I just want to test out the configurations on top of it well Kuberit has a bunch of Ansible linting tools that allow you to just bring a custom image so you could have that fully baked DNS image your kind of gold DNS image and then just spin that up so now rather than waiting the 20 minutes for your DNS server to build now it's okay this thing just spins up in 3-4 minutes and then you can apply it you can test out just the configuration end of it so they're really cool tools you can speed only test the pieces that you actually want to test that's a good point basically you can take the image of your virtual machine and load it in here and then you can test on top of that so yeah good shout out and here I think we're seeing the Ansible run of where things are actually happening so we're going through our install we're installing the Python packages we needed all the supports all the Ansible goodness that we're looking at here until we get to the end that's where we just configure we add our zone in there that we'd expect we just have our dummy zone in there but that's just one part it's cool to see that you've installed a DNS server but we actually want to check things we want to make sure that things actually happen so I don't think there's a lot of good I don't know but I can't show let's take a look at what's actually happening in that execute in VM again yet another great task that we were able to pull from the Kuber community and it just it literally connects to your VM from your pipeline based off the secret and stuff that was generated as part of the pipeline we generate random SSH keys because you should not submit your keys to your repo speaking from experience but if you if we look at the last task as part of the pipeline Shansu we'll take a look at what we're actually testing for and I'm if we just go into the pipeline itself not the tasks sorry yep no you're good yeah so it's not a very complicated test I'm just digging to see that I get a response from the zone that I configured in our case it's just our labs kind of zone that we configured and I said hey if it's empty it's not there and it's broken obviously we see it's green so it's there but the cool thing there is they're they're also tasks you can you can start you know probing at records you know it's making sure you get the right C name or the A records or any type of records you want to kind of poke around that and really you know that that's that this is a very plain example because we wanted to kind of give like an approachable example to this really the sky is the limit as much as you want to test and poke around at inside your VM the tasks are there you just need to inject what inject your logic right since this is running and we have only a couple of minutes I can just jump into the next slide and quickly saying that the echoing like the other side the possibilities are endless well sort of endless as long as you have any sources to run firstly in your virtual machines or if you want to wait like 20 minutes like us during a talk so that's fine too but now that you know that you can run virtual machines in Kubernetes you can pretty much emulate you know we can pretty much try to run everything in here your imagination is really just constrained by CPU and memory at this point okay I'm just going to fast forward and we also have this slide for talking about what sort of things that we do we were doing while we were preparing this demo and well basically minus resource if you are running in virtual environments and using the emulation so I would just repeat the second thing is about the tecton and having to retry many process and volumes if you would like to use the same volumes for the parallel test or if you would like to run a couple of same pipelines that's using the same virtual same persistent volume to clone the playbook or something like this not everyone has retried many volumes so this might be a constraint for you what else Tyler we can say quickly yeah the two things on the Qvert side is these are virtual machines but they're virtual machines built on top of your container platform so take advantage of that a lot of the times your pipeline will try to connect to a VM not ready yet well kubernetes open shift we have things like readiness probes so we can sit there and wait until this thing shows it's ready we can just run a kubectl wait for condition equals ready it'll sit there and wait for your VM to be up at the same time you might have seen I think we've got the link in here so you can take a look at the repo we define a virtual machine custom resource it's part of our pipeline that spins up a virtual machine but the actual running VM itself is a virtual machine instance so if you don't say spec or running inside of your spec to say running the true it'll create your VM but it doesn't spin it up because you'll have to have another task that sets that to true or kicks your you know it's like vert CTL start or something like that so the thing that you want to test won't get created if it's not set to true so keep an eye on that we only have two minutes so we just had to run this really quick so if you have any questions please feel free to reach out to us the repo is out there also just for the example tasks or not for the example playbooks but it was fun to prepare the demo yeah I guess kind of our call to action right get out there write these tests write these pipelines for your infrastructure it is worth it it will save you all the sharp edges in the bleeding that you will experience testing and prod so hope to see everyone kind of taking this approach thank you again to all the community folks for letting us take all the tasks and having you know kind of being able to assemble it from that and also take for the live events opportunity it's the closest things that we can get for face to face so that was fun thank you and thank you so much both of you for such a fun and knowledgeable presentation it was definitely fun watching all the visuals so thank you for that and I don't think we have any questions yet so I think that means that it was a flawless presentation love you guys thank you thank you so much for coming in and presenting have a good day bye bye