 Hello Q-Con, welcome to this session on Cuddle. Pipe full of Funkit number 7? Pipe full of Funkit number 7. That's right, Pipe full of Funkit number 7. This session is taking a look at declarative testing of your Kubernetes cluster using Cuddle. A project that I am one of the committers on. And I was fortunate enough to, very last minute, get a hold of a co-presenter. And let me introduce you to him. It is the loud, mild schnoot. Look ahead! Alright, alright, that wasn't fair. You're right. Let me introduce you to the Rooster. Foghorn Leghorn. He's going to be co-presenting much of the material with me. And you might know him from Who Framed Roger Rabbit or Space Jam. He started out in 1949, but he is gravitated today into just doing some commercials. KFC or GEICO. And I'm the other rooster. We don't need no more roosters around here. We got one in our midst. My name is Ken Syke. I'm one of the distributed application engineers with Day 2IQ. That's how to get a hold of me at the bottom, Ken Syke on Twitter or Ken at Day 2IQ. As senior to instruct junior roosters in the ancient art of roostering. Let's get started. Well, Cuddle stands for the Kubernetes test tool, as you can see here. The origins of it is that we were creating Cudo, the Kubernetes universal declarative operator. With a focus on declarative operators, we thought it made sense to create our test in a similar way. So we went after declarative testing. When we look at testing, there is a number of different types of tests. Developers are probably used to unit tests. You, of course, have integration tests. Both of those really focused on developers. Both of those kind of fit in to the developer workspace. And then you have your end-to-end tests. I'll say, what in the name of Jesse James? Do you suppose that is? Well, that's a really great question, Foghorn. It turns out that end-to-end testing goes from end to end, from front end to back end. And when you look at it from that perspective, it's not necessarily from the developer perspective. And one of the advantages of looking at an end-to-end test from a declarative standpoint is you're looking at conditions or setups and assertions that are from the perspective of an end user. But also, usually when you have developer tests, you have one cluster or one version of a cluster or one style or environment that you're testing in and that you're repeatedly doing if you're doing the right thing in a CI environment. But wouldn't it be nice if you could take a test and testing on a lot of different clusters but then also test it maybe in the customer's environment? You know, probably the best way to look at this would be to look at what you, Foghorn, have done. I believe you were one of the people involved with creating the imperative tests that are in the Kubernetes proper. Let's go take a look at that. Well, here we are, Foghorn. First of all, it should probably be worth stating that they can use their R tests. That's fantastic. And someone probably did what they knew. But what we have here in Kubernetes test, CMD, CRD, and there is a series of them, of course, in this directory, is an SH file. So a shell script of some sort. Imagine what you need to know in order to accomplish your end goal of just testing end-to-end. Here we have JSON that's kind of interweaved inside of this SH file. Perhaps it would be easy to mistakenly put in a comma or an extra line and you could test it, you know, by running the SH file. But there's really no way you could give a validator to validate this shell script, but it would be really challenging to validate these files, the JSON or YAML files. You can also see some crypticness, some imperative way of managing certain aspects. You can see here, I'm going to get the object assert of custom resource definitions. A variety of things here. And it's worth taking in. Keep this in mind as we start looking through Cuddle and taking a look at another approach at managing this kind of concern. That dog's as subtle as a hand grenade in a barrel of oatmeal. Well, what does it mean to be declarative then? Well, when we say declarative, what we mean is the setup, the actual creation of state or existence of objects in the cluster can be declared just like any Kubernetes object would normally. So here we have the creation of the doghouse, which is an engine X pod that's going to run. So this is step one. It's just a YAML file. So we can use a YAML editor and all the features of beautify and prettier and all those things that you might be used to as a user. We can use it for testing. What does it mean for assert? Well, it means we use a YAML file. And in this case, we need to be explicit about the thing that we're after. In this case, we want to know that the status phase is to running. That's it. And what you can express here as we will see further in the presentation is you just express the thing that you're interested in asserting. Anything else is ignored. We have been flam flam. And we can go further with those assertions. The assertion here is we have we have some special types within cuddle detailed that through this presentation. So here we have the test assert and we can change the time out. We can add some collectors. We'll talk about what that means. You can, of course, have a series of YAML in the manifest file. This as you can see is 00 assert that YAML and it is setting the time out, setting up a collector and it is making sure that the status is running. So I ask you the question, if you could do things imperative or you would do it declarative, what would you do? You know, the funny thing about introducing Froghorn in this presentation is this is the one thing I remember about him is he said the words I do declare and I cannot find one example of that. All I see is the meme. So that's kind of comical. I'm not sure what happened. I don't know if those are lost or if my mind just made things up. But why don't we why don't we dive in further into the rest of cuddle? Now, whose response I say who's responsible for this unwarranted attack on my person? So who is the target audience for cuddle? Cuddle was originally written for operators, Cuddle in particular. We quickly needed to use it for helm charts and for other operators. It's basically for anybody putting a workload on top of Kubernetes. And as such, cuddle is being used in Kudo, of course, but also has been newly added into the operator SDK for the scorecard. So if you'd like, you could read those manuals. Now that I see that's no way. Well, I'll spend some time showing you how to do exactly what's in those manuals. So how do we get started? Well, first do a brew install or a crew install. Probably be the easiest way to get cuddle on your platform. Well, I say, what are you trying to trap them on? Well, let's get started with the basics then. I might run through this fairly quickly, but through the magic of video, you can go back and re-listen. I want to make sure we have plenty of time for demos. So let's get started. The cuddle CLI is a plug-in to KubeCuddle and is invoked through KubeCuddle, cuddle, and the subcommands are listed here. The main one is test and it's the test harness that you would usually run up against a live test cluster. There's other options which are pretty fairly well documented, but the demos I'll do in here are against live clusters. When you have a test commonly, you're either in a directory that has a test suite file, which we're going to see, or you will specify all the command line arguments and it's very common to pass like the test suite folder into this as the first argument. The other one that might be useful to understand is the assert. I recently added that and where you normally pass in the test suite to the test subcommand, with the assert you pass the direct YAML file of the thing you want to assert. As you're starting out and you're building up certain state within your cluster and you want to assert something to be true, you can use this assert to verify that along your path. You could also use it to verify that the state of things is exactly the way you intended it outside of cuddle altogether. Then we have the major subcommand which is test and as I mentioned, there's a bunch of options here. The one that you see at the top is KubeCuddle, cuddle, test and I'm passing in a test folder called test. Within a given test, there are test steps and those are your YAML files. So if there are any other file extensions within that directory, maybe like your readme files or some kind of setup details or your licensing files, all of those are ignored. The only thing we pick up are YAML files and then it has a specific nomenclature to it. So we have an index dash and a step name. Oh, it's important to point out that the test e to e is our test suite. The example is the test and then we have test steps underneath the test. So 00 pod would be during the creator update phase of a step and then the assert is the assertion or errors phase of a step which we're going to learn more about. It is also very common to see multiple YAMLs within a manifest file. During the creator update, this is critical for people new to cuddle. We do a match against the GBK group version kind given the name or a filter for the unique object. If we don't find one, we create it. If we do find one, we update it which can be very confusing or create interesting challenges when you're switching between namespaces and I'll talk about that a little bit later in this presentation. So if you create an object in let's say step one and then you assert it in step one and then you want to augment or change that object then you just need to provide the delta of the thing you want changed and that is that's an important concept. If the object already exists, it's just going to augment it. If you normally run with multiple namespaces and then run all in one namespace very common for people to have failures that are surprising that's because usually because they were used to having test isolation through namespaces and now they don't and they didn't have proper cleanup after their tests. So this is a really important concept. Any of this getting through that little blue bottom of yours? And then we have the assert or errors file. Again, an index and the assert all of the YAML within the assert within a certain period of time the default is 30 seconds. The state that is defined within the assert file will have to be true. Otherwise, we fail the test and the errors is just the opposite of that. It's the state of something that should not exist. So you want to check test for the absence of something maybe even the absence of the object even existing. Let's go through some of the terms now. I've already covered some of them, but let's put a nice bow around it. The terms of service. We have the test suite, which is a collection of tests. We have the test themselves, which is a collection of test steps. We have a test step which has multiple phases kind of the create update and then the assert phases and then we have the test assert. What's also important to note here is that the test suite, the test step and the test assert are all object types defined within Cuddle. This bar is all mixed up. Now there is a couple of areas that could use some improvement within Cuddle. This is one of them. There are two concepts that are conflated together. I'm going to just draw them out here. So a test suite. The test suite is, as you see on the left, a folder in this case is called test data. That is the collection of tests. So every folder that you see underneath there are tests. The test data is the test suite. But also we have this object type which is the configuration YAML of a kind of test suite you see on the right. We're looking at a test suite in that particular example. We can actually create a number of test durrs which would be a series or an array of test suite folder locations or what's conflated together is the startup of the test harness which should be separated out from the concept of a test suite. But today this is what we have. You've got your signals mixed but I know what you mean. Glad you got it. Okay. So then the test as you see here we have a test pods folder that's within our test data test suite the test folder. I'm sorry that the list pods is the name of the test. So when you see that reported within the stream of logs for a test it will be listed as the list pods test. It's also very common for people new to cuddle to actually point to a test folder when they are starting up the test harness. So they say keep cuddle, cuddle test and then they specify the location of a test. We look for the test within the test suite not finding any. It won't run anything. So that's a very confusing point for people starting out. You need to point to one folder up which is the test suite itself. It will discover all the tests with underneath the test suite and it will run those tests. And we have the test step and again in this case there's two concepts to it but they're more or less match. So in this particular case we have our list pods and then under that we have files. We have the 00 pod and the 00 assert. Those are the steps and 00 is the 00 step. You can have any number of files. Of course you want it to be somewhat manageable but within that then you could also be very explicit and declare a test step. So if you wanted to do what you see here you could define within the 00 pod a kind of test step which would actually apply in this case some commands assuming that you had some commands that you wanted to run maybe creating a service account creating a service itself doing something that's outside for whatever reason of your declarative files. Then finally we have our test assert and again this falls into the same idea as the test steps. We have an assert file in this case it's 00 assert. We can have multiple things in it. We also have a test assert kind. So within the assert file we can say that this particular test doesn't take the default 30 seconds it takes 20 seconds. In addition to that we want to assert the pod named test two has a certain quality of service and it will assert that to be true within 20 seconds or the test will fail. And then finally we can run the test. So keep cuddle, cuddle, test and then a location of the test directory. Before we get to a demo there's one more thing I want to show you and that is this other concept that's super important to understand that by default cuddle will create a namespace for every test. So we provide test isolation through namespaces. We understand that there are certain environments for which you may not have permissions to do that and we have ways of getting around that by focusing tests on one test namespace. That said, this is the default nature of it. You will get nice isolation of your tests through namespaces and you may need to make adjustments to your test suite if you change from this. You gotta, I say you gotta keep on your toes. Toes, I did. Well, before we get to a demo let's talk about what you need to do to do your very first cuddle. You have cuddle, CLI installed and nothing else. What do you do? First, let's create our test suite. So we create the tests E to E folder. Next, we need a test. So let's create the doghouse. Within the doghouse we'll create the 00 install.yaml file which for us is just a very simple engine X. The next is that we'll create an insert file 00 assert.yaml and in this case we're looking for a pod that has the name of doghouse and has a status of running. If it has the status of running then this test passes. Next you have an option. You could just use the test the way they are and point the test harness the first argument of the test to the folder to the test suite folder or you could create a cuddle test.yaml file a configuration file of test suites and then include directory that you're it's supposed to be pointed to. If you have this file then it's nothing more than saying cuddle test and we'll discover where those tests are and run them. In this particular case we have two tests. You can see them running right here. We have the doghouse and we have the henhouse. Both of them have been started. You can see that we created a namespace for the doghouse and we created the namespace for the henhouse. So we have test isolation. This is the default behavior. What if we just wanted to run one of those tests? Well, we could actually specify dash dash test and specify the test in which case only one of those tests would be run. Let's get to a demo. This would be more fun than a barrel I have with a monkey. Well, as you can see I've just created a a cluster with kind and there's nothing running in it. So let's start by putting something in it. Let's run an instance of engine X and then let's create an assert and I've already done that. So there's the assert file. We're going to assert for a pod named engine X that has reached the running state. Let's switch back to the command line. In this case we're not going to run a test. We're going to just validate the assert and we're going to specify a short time out and we're already asserting it to be valid. Let's take a look at what a failure might look like. Maybe we misspelled something. I hit save there. Obviously that's spelt in correctly and let's run the same assert. There's something going on around here that just don't add up. So in this case it takes three seconds. We had to wait for that. We get a full diff which seems like a lot. There are actually two errors that come back. This is a very verbose diff of everything that was not specified. That was present. But then you get a second targeted message which is this resource for status.phase had a value mismatch. It was looking for running with one end but we actually found running with two ends. Seems pretty simple, right? Why don't we take a look at our larger project with the e2e tests and see what we might be able to learn here. The first is we have two tests. We have the doghouse and it's step zero. We will have created a pod which is named doghouse and is nginx and the assert is very similar to what we've seen already. We're just checking to see that it was running. The henhouse actually has a couple of other new features that might be interesting to look at. The one is you may have you might have some YAML that you repeatedly use and either steps or tests and you may have it out here all by itself like this henhouse example right here in the pod.yaml. Here within the 00 install we say apply and then we provide a list of the YAMLs that we want to include so I can use a bunch of them across tests. I could put pathing there this could say dot dot slant dot dot assuming you know where one's at it could just as well set that it could be in sub folders it could be in a variety of places in this case this is the location that's close to this file which is the pod. Notice that we've defined a test step so there's a number of things we could do within the test step. One of them is commented out but I could have put in commands and that would be a list of commands. Then we have our assert which is a little bit unique as well worth showing off a few things one we could have changed the timeout so I could have switched to maybe 60 seconds. We have a collector and I'll talk about that in just one second. So this changes the test assert configuration and then this is the assert and could be a series of assert. This also shows multiple yamls in the same manifest file with collectors. It's common or interesting sometimes to collect additional information if a failure were to occur. So in the case of say this test failing we have two options with collectors currently one is pod and one is commands. So if it was command you could just run any command so you could gather whatever you wanted and that will just be dumped right in the log with the test. With the pod you're specifying that you want to collect the logs out of the pod. So the pod that is the henhouse pod will have its information dumped within the log file. Of course if there were multiple containers you'd have to specify which container there's all the configurations that you can imagine that you would need to differentiate or there. You don't stop talking so much as tongues on fire. And that's it. We could then just as well run that test. Let me clear the screen. Cuddle test and in this case we want to run the test and now we have it running eventually passing one more really cool feature that I'd like to show you before we leave this demo is I've taken the example that you we've seen here the henhouse and doghouse and the test week that it is. I have tarballed that and have put that up on GitHub. So we could run the test cube cuddle cuddle test and specify the URL of the test week. It will pull down the test week extract all of the files and then run the whole test week. So we're running the exact same test week we were before except for now we've pulled down the archive from the GitHub. That's it. Super cool. Right. Let's look at some additional features. The first is that we can obviously do a delete. You can do a delete based on a number of things. So as you see here in the example I can get rid of a pod. It's my pod or pod with labels of app engine X. We can delete things as part of a test step and you just include that into your step. Additionally, we could add commands and you it's an array of commands. The example you see on the first one is we can do a standard apply if you wanted to. We already do applies. It is not necessary to do that. But in the case of Kudo installing an operator without the instance might be something useful and there's some danger here. This is dangerous. You gotta know how to handle it. One wrong move and you're done for. Pay attention son. This is for your own good. As I mentioned before it is possible to set up cuddle so that it doesn't create a test a namespace per test. The danger here of course and the thing you need to be aware of is you may have to do some additional test design. You may need to delete things and then assert that those things are gone using the errors prior to moving on because you do not you no longer have the isolation that you would normally have. But otherwise it is a useful feature for many people. As we've shown before you've I'm so dumb I didn't realize that I had this in my slides. He's so dumb. He thinks a Mexican border pays rent. The example here of course is that we can apply. Some more tips. Everything is accessible to cuddle. So here we're asserting an event occurred. Of course remember that events are short-lived things but the test is only going to run for a short period of time. It's not something usually that's going to be on for longer than an hour. So in this case we're asserting that an event occurred and that it was created. The source of that event was a cubelet and that the involved object was the pod named my pod. Another tip is we're going to take a look and see how you might start up your tests with CRDs but perhaps you have CRDs that you're going to be creating or adding along the path or along the way. Prior to using the CRD is to assert that the CRD exists what you see on the left the 00 assert.yaml after you've asserted it then in the next step you could use it. So in this case we've asserted that the CRD exists prior to it being used. A common use in Kubernetes world is using Helm. So here we see examples of using the command and using Helm to do installations and potentially even setting up service accounts and role bindings and that kind of thing. And as I've indicated we can pull test suites straight from any given URL. We can even include them in our test configurations. Notice how great this is that you could have a suite of tests available to any of your customers and clients and they can test in their environment. And what I've not shared up to this point is that it is possible to add some flags so that it doesn't delete anything. So the creation of something establishes the state according to what you desired and then asserts that that state is there and then leaves the cluster without tearing down the tests. This can be a very useful feature of using Cuddle. Another great feature that Cuddle provides is a report. If you specify a report you can specify either XML or JSON. If you specify XML it'll be a JUnit XML report. You can see two examples of an XML report here. One with all succeeding tests and then you have an example here a failing test in the doghouse. Since Cuddle was originally created to test operators Cuddle in particular let's take a look at the features that support that. The first is we need to enable the ability to install CRDs. Cuddle has the ability to install manifest or CRDs or apply them within the Kubernetes cluster. The difference is that when we see CRDs within the CRD dir Cuddle will wait until the CRD has been realized within the Kubernetes cluster. Then when we talk about working with or testing an operator there are two different styles or two different ways that an operator might be being used within the cluster. The first is that it is a released operator and the other is that it's an in Dev operator. The example you see here is that we have a Cuddle init that is initializing the controller within the cluster and the wait process will wait until it's complete. This is how we establish Cuddle running within our cluster and then we'll have a series of tests that follow. In Dev we do it slightly different. In Dev we actually apply the CRDs and there's a wait period of time for that and then we invoke a command on the code that is part of the CI environment where we invoke the manager and we put that in the background. What's important about this is that the standard out by default will be part of the test logs. There is a way of opting out of that. Now you can see how Cuddle supports the released testing as well as the in Dev testing of operators. Let's open the leave some more time for Q&A. You're way off. I say you're way off this time son. Yeah right. Let's put some closing remarks around this. First of all there's a lot more I would love to share with you or show you. If I would recommend a place to get some feel for the different tests I would recommend this location under GitHub underneath Cuddle itself if you want to get some more cuddling in. The other is the first release was March of this year. We've had the 07 release. I just got out in October. We have a release cadence that happens about once a month and if you'd like to get involved with that you can find the project in this location. You can find us on Kate's slack underneath Cudo. You haven't had a need or demand to drive it out to its separate channel. But if enough demand happens we'll do so. We do follow a KEP process and we do once a month meet and talk about community issues. I'd love for you to get involved and with that that's all folks.