 Hey everyone, am I audible to everyone? Cool. Hello everyone, welcome to my talk. Today I'll be chatting a little bit about testing Kubernetes end to end on your bespoke cloud platforms. So do you want to test Kubernetes end to end on your cloud? Also curious to know how other projects in the Kubernetes ecosystem do this? If so, you'll love this QBtest session. Let me tell you a bit about myself. My name is Priyanka Sagu. I work at VMware as a member of technical staff 3. Currently, I'm one of the release lead shadows for the ongoing Kubernetes 1.26 cycle. I was enhancements lead for the Kubernetes 1.25 and have been on the release team since 23. I'm also one of the three technical lead shadows for the special interest group contributor experience in Kubernetes project. Go by P.Sagu on Kubernetes Slack. And I sporadically use Twitter, but that's my Twitter handle up there, underscore P.Sagu. I'd like to talk a little bit about what is QBtest right now. So QBtest 2 is a framework for deploying Kubernetes clusters and running end to end tests against them. So it's a testing framework maintained by the Kubernetes special interest group testing, or we call it SIG testing, for launching and running end to end tests on Kubernetes. It's intended to be the next significant iteration of QBtest, which we'll talk about in a minute. So effectively, QBtest 2 manages three things. It manages cluster configuration, that is test cluster deployment, end to end testing and log collection, that is running the end to end tests against a newly deployed test cluster and collecting logs as a test run through. And eventually test environment disposal, that is the cluster decommission or release of other test resources once the tests are finished. QBtest 2 uses a binary plugin mechanism for detecting plugin deployers and testers. The binary plugin mechanism is very widely used across the Kubernetes project. The canonical example is kubectl. So the way it works is there are three binaries in QBtest 2 and the binaries are named in a particular standardized format. So in this case, any binary with the name kubectl hyphen, your deployer name would be used for the deployer binary. Any binary with the name kubectl hyphen tester hyphen your tester name would be used for a kubectl tester binary. And just kubectl is the binary for the framework itself that received flags and arguments to discover and work deployers and testers in your system path. So what exactly is a deployer and what is a tester? So QBtest 2 deployer is responsible for the test cluster lifecycle. That is the deployment of the test cluster all through its decommission locally or on different cloud providers. And QBtest 2 tester is responsible for running end to end tests against those clusters that are deployed by deployers. A typical workflow is broken down into multiple phases. So QBtest 2, we start with build, go through up, test and down. Now build is an optional phase. Build by QBtest 2 build phase means here we build the Kubernetes artifacts from the source code. Up is again building up the test cluster. Test is where we actually run the test suites against our newly deployed test cluster. And down is decommissioning the cluster once the test flow is done. Now this is the command line invocation how you'll use QBtest 2. So we use it as QBtest 2 followed by deploy name, followed by flags for up, down and test. The flag for test follows the name of the tester and the test arguments. So one of the examples we can take here is for if you want to run upstream CNC of Kubernetes test against a GKE cluster, this is how you'll do it. So GKE is the name of the deployer. So the flags up, down says that we want to deploy a cluster, a GKE cluster. And then we want to use Ginkgo, which is a tester to run the conformance test. So the focus as X says go and look for any pattern that says conformance in the test suite. And then once we are done with running our test, then down flag will bring down that cluster. So like we just learned in a while back that QBtest 2 is the next significant iteration for QBtest, why that iteration was required. So QBtest 1 or I'm just calling it here as QBtest version 1 is a framework that Kubernetes project currently uses in for continuous integration testing or CI and runs about like thousands of job daily. It comprises of a single binary encompassing all different test environments and scenarios using the Kubernetes CI. And it fall organically from a bunch of legacy script to the point that it has now become unmaintainable. Which is why one of the primary goals of why QBtest 2 was created is to move away from legacy dependencies and to build something that is easier and extensible and pluggable. So the community supported the progression and QBtest 2 was deprecated in favor of QBtest 2 on July 14, 2020 and was placed in maintenance mode. It also allowed reinstanting community contributions to the tool because it has become so complex that people were not able to follow through the code and they were not able to contribute back to the tool itself. So it allowed reinstanting those community contributions as well. So overall the intent behind this new design of QBtest 2 is to minimize coupling between the suppliers and testers to encourage implementation of new suppliers and testers out of three. So we have I think three or four suppliers that come right away from the upstream QBtest 2 project, but there are a lot of out of three implementations as well. And we'll talk about how to write your own deploy later in the talk. And lastly to keep the dependencies and surface area of QBtest 2 small. So how the upstream Kubernetes project integrates QBtest 2 with Brow. Now for just a brief, Brow is the Kubernetes CI system, CI CD system, which is used for GitHub Automation and provides a chat of flavor of a way of triggering automation. So something like slash foo style commands. So let's try to see how the upstream project does it with Brow. So user starts with a GitHub PR on one of the Kubernetes project repos. They add a comment on their PR something like test all and if it's visible. And the comment will trigger the Prow or the Kubernetes CI CD base pipeline in the Kubernetes test infrastructure. So Prow is a Kubernetes base CI CD pipeline. That means Prow runs inside a Kubernetes cluster and QBtest 2 will also run inside somewhere in the Kubernetes cluster and where it does. We'll just see. So inside the Kubernetes cluster, a proud job is created. A proud job is a custom resource and anything in Kubernetes runs inside a port eventually. So there will be a port that will be created for proud job. It's like any other pipeline. Now inside that proud job, there'll be containers. And in this particular case, the image that we are going to use for this particular test will be Kubekin. So all the all the jobs that require testing given it is end to end will be created through an image called Kubekins. And we ship Kube Test 2 as part of that image and not just Kube Test 2 other test binaries as well. So once we have the Kube Test 2 binary in the port container available, we can use that Kube Test 2 binary to create test environments to run end to end tests and then to finally clean up test environments. So this is the broader overview of a flow of any job that test Kubernetes end to end in the upstream Kubernetes project. So what are the primary features that Kube Test 2 provides? Consistent cluster lifecycle. A typical testing flow of Kubernetes looks like building some binary from the changes that are coming from a PR. So you have raised a PR on say Kubernetes, Kubernetes repo. That is you are adding some new changes to the Kubernetes repo. We'll try to build a new Kubernetes binary out of the source code that is also having your changes. Then we'll using those binaries will deploy a new test cluster with that with those binaries will run some tests against that test cluster and then tear it down. So that's a typical test flow. In Kube Test 2 we have we have standardized and codified the lifecycle that is our up flag, test flag, down flag that does that for us. So up flag will bring up and the build flag also sorry. So build will if required build will build the source code, the binaries from the source code with all the changes that are coming from the PR. Up will bring up the cluster with those changes. The binaries created with those changes test will run tests against them and then down will bring down the cluster. So we have standardized and codified the entire lifecycle in Kube Test 2. The next thing Kube Test 2 provide is decoupled the deployer implementation. So Kubernetes comes in many flavors and offering so for different distros for different installers and providers and you would want to run at least test against some of those flavors, right? So Kube Test 2 provides a way to not only run your test against the different deployers that comes ready ready ship for the Kube Test 2 project, but also allows you to extend the framework to include your own implementation of Kubernetes. So for example, right now Kubernetes Kube Test 2 provides the deployer for GKE, GCE kind, but if you want to say test use Kube Test 2 to test test your Kubernetes on say any cloud, Linux or anything, you can just go and find out how to bring up a cluster in Linux and extend Kube Test 2 to do that for you. And we'll try that in the end. One of the key features of Kube Test 2 that improves the Kubernetes developer lifecycle is being able to run CI testing jobs locally and as much as reproducible as possible. So as part of the goal of minimizing legacy dependencies, Kube Test 2 provides a unified command line interface to be able to reproduce a system under test in both the CI and the local environments as closely as possible. And lastly, Kube Test 2 also integrates seamlessly with other tools that Kubernetes project uses such as Boscos. Boscos is a tool used for leasing testing environments and dependencies at one time. Most canonical example being your test requires you to have access to a GCP, a clean GCP project or a clean AWS account. So Boscos will release you that account for the duration of your test run. And once the test is finished, the down flag will release it to be later on cleaned by the Boscos janitor. So let's go ahead with the demo. So we'll try writing a custom deployer for Kube Test 2 now. Kube Test 2, as I mentioned, natively supports GCP, GKE and kind deployers, but it also enables writing custom deployers out of tree. For this particular demo, I'll try to walk through the steps to create a custom Kube Test 2 deployer for Azure cloud. I'll try to provide a code walkthrough that will outline the key steps that I had taken to extend the Kube Test 2 framework and implementing the deployer interface for Azure cloud. Also, the demo will be slightly rushed demo because of the time limitation, but the project source code, slides and demo recording will be available on GitHub, links in the later slide. So this is what we need to implement. This is our deployer interface, and as part of the demo, we'll need to implement this deployer interface. So if you want to add a new deployer, all you need to do as a minimal requirement is to implement this deployer interface. For the purpose of this demo, I'll try to implement the following methods. So up, down is up, dump cluster logs and build is optional, so I'll skip that for this demo purpose. And we'll be calling a new deployer as Kube Test 2 AKS. So what I'm doing here is I'm creating a new directory called demo. I'm cloning my fork of Kube Test 2 here in the root of my Kube Test 2 fork. So Kube Test 2 at its root have a make file that have multiple make targets to install the Kube Test 2 binaries individually, the deployer and tester binaries, or you could install all of them altogether. I'm doing that here by install all, and what it did is it installed all the Kube Test 2 framework binary as well as all the deployer and tester binary as well. So you see that it was, well, I missed it, but it will run again. So when you run make install all, what it will do is it will look at all the folders that are named in the standardized format Kube Test 2 hyphen deployer, or so anything with Kube Test 2 hyphen, say GKE, GCE or kind, it will treat them as deployers and any folder in the manner Kube Test 2 hyphen tester hyphen the name of the tester, it will treat that as tester. So when you run this make install all, the Kube Test 2 binary will also be built and it will be able to detect the deployers and testers also in the path. So we have these at the moment and we do not see anything, something like AKS at the moment and what we are trying to do as part of this demo is to add that. So what I'm trying to do is I have three remote branches available and I'm trying to check out into one of my branch, which is a step one. So at this particular step, what we are trying to do is add a wire framework for Kube Test 2 AKS deployer. So there is already a deployer present in Kube Test 2 project called noop deployer, which does nothing. So we are taking that code and starting with that. So what I've done is I have created a new folder here called Kube Test 2 AKS and everything that is sitting inside noop is there. So there is a main.go file and I'm importing the deployer that you see there at the top. We are importing that deployer and we are adding that deployer to the actual app package. So now Kube Test 2 framework will later be able to see this new deployer as well. Let's also see what's there in that deployer folder. So we just have one go file there. It's a deployer package itself and currently it's exactly what's there in the noop. So I've changed nothing for the sake, but we just wanted the AKS deployer to do something at the moment. So we just copy pasted everything that the noop deployer had. So yeah, we are at this part we are implementing the deployer interface. We have up, down, is a dumb cluster log and build there which are doing nothing at the moment. That's what noop deployer does and we'll try to implement that in the later steps. So once we have added this new folder with name Kube Test 2-AKS and if we go ahead and run that make install all again, it should be able to see a new deployer now. So that's what we are trying to do now. We have compiled the Kube Test 2 binaries again and since we had unfolded by the name Kube Test 2-AKS it was able to now see a new deployer. If we give that argument to our Kube Test 2 binary now, it will be doing something but it will be doing exactly the noop thing. That's what we see here. It's just showing everything for the Kube Test noop operator at the moment. So let's go ahead and enhance it further in the next steps. So I'll check out again in my another branch which is the step 2 and at this step what we'll do is we'll try to add some logic for later on implementing our up and down function. So even in the upstream Kubernetes project, the GKE deployer and other deployer, what they do is they have a script somewhere sitting, something to bring up a GKE cluster for example and then there is another script sitting somewhere to bring down a cluster. So we are trying to add something like that here. I've added a folder called scripts here and I've added two scripts in that folder called KubeUp.sh and KubeDown.sh and those scripts just have the logic for bringing up Azure Kubernetes cluster. So this one is a KubeUp.sh. So what we are doing is we are just trying to check whether our container has an az binary and qq binary. We require some environment variables to configure our cluster. So there is a VM size that we can give it while invocation or it will take the default values but there are some required variables as well, environment variables which is the resource group and the cluster name itself. So it will see if these environment variables are available in the path or if they are not then we'll be anyway doing that adding those environment variables as part of our deployment implementation. So once though it is able to see those environment variables it will go in line number 20 and create the resource group and line number 30th onwards it is creating the Azure Kubernetes cluster and once the cluster is up we'll try to fetch the kube config for that cluster and save it somewhere in our path so that we can interact with the cluster. Now similarly we have the kube down.sh as well. Again it will look for az and jq binaries and the same resource group and cluster name it requires that to interact with the cluster that we just built and if it is able to find those values then it will go ahead and use the az binary to go ahead and delete our cluster. This is a simple very simple logic to bring up Azure Kubernetes cluster and bring it down. We'll use this script in the further step to implement our up and down function. So this is our final step. So here we are just trying to enhance our placeholder deploy.go to actually deploy and remove AKS cluster. So now our main.go is ready. Our scripts are ready. We'll go and add some more stuff in our deployer folder at this step. So we have added a lot of new stuff in our deployer.go file which is we have now at this point implemented the up function, down function and cluster down is up. So we'll see that now. You see we have online number 37 and 38. We are giving the name to the resource group and cluster name that will be used by the script later on. We have also changed our deployer name at this point. Now it is called AKS. We have deployer struct which has some values that will be used later by our up, down and other functions. So here we have finally implemented our up method for the deployer interface. So what we have done is at line number 84, we are trying to compile the path to our kubeup.sh script, bascript and down there on line number 91 we are actually running that script. So the kubeup.sh, this file, when ran it will go ahead and bring up as your Kubernetes cluster in the as your account. And we will also try to check whether whatever that script did, whether it eventually ended up into bringing up a cluster or not. So that's the validation at the end of that. Now we are doing exactly similar for our down step as well. We are trying to compile the path to our kube down.sh script and down there we are running the same script again. That will go ahead and delete the cluster that we are just deployed above. Now isUP, so we are doing a very naive assumption here that if there are nodes available, we are our kube CTL binaries able to interact with our cluster and give us some information about nodes. That means there is a cluster running there. So what we are doing here in our isUP function is just running kube CTL, get nodes, giving it the kube config file to our newly deployed cluster. That is, if it gives us something, that means the cluster is up. We have two other files here as well, common.go and dumplogs.go. Those are utility files basically. So we needed these environment variables during our kubeup.sh and kubedown.sh bascripts. So we are setting those environment variables at this point so that our deployed.go file can use them during the script invocation. So we have the resource group cluster name and the kube config environment variable set as part of common.go. And the dumplogs.go what it does is it has this function called dump cluster logs which creates a log directory. So it will create a folder somewhere where we'll store our logs from the time when we are trying to bring up a cluster. If we are running some tests on it, it will also collect the logs during that test run and also tell the cluster decommission. So, yeah, and this kube CTL dump actually take that dump. So what it again, it runs a kube CTL command which is kube CTL cluster info dump and it takes the kube config file to our newly deployed cluster. So that's what dumplogs.go does. So with all this in place now we have our up, down build is up methods implemented for our new deployer interface. So that means our deployer can bring up a cluster now and bring it down. So we ran the make install all again and it will have all those functionality in the newly built kube test to AKS binary now. So now if we run kube test to AKS at this point now everything is changed. It is kube test to AKS deployer no more, no op and now we can give it the up, down flags as well. So when I give it up and down it will first bring up a cluster and then followed by that it will bring down a cluster. I'm not doing the testing part at this moment because you can use the up, down test individually as well. So you see it just did that one thing only it just went ahead and executed the kube up, down test script at the moment and very specifically that AZ cluster create command taking all the values from the environment variables that we had set. So it takes a bit of a time and it entirely depends on what kind of kubernetes flavor you are trying to build up. So in this particular case it takes a bit of time but for the purpose of demo I have fast forwarded. Let's see but it will still take a second here. The run is still there but since the cluster creation started I'm able to see a new cluster in my azure cloud account. So it's the same cluster information that we just saw there. And once the cluster creation is done it will go ahead and take the kube config of that cluster and save it somewhere in path, in my local path. So when kube test ran the AKS up and down cluster creation and decommission so it also created this underscore artifacts and underscore runner that occurs where it had all the entire information and under the underscore runner we had the cluster dump that we had taken as part during the up and down functions. So I'm just trying to check whether my cluster was deleted also. So just using the az binary to check that and there is no cluster. So yeah that was a quick demo. Now that's not the entire extent of what kube test 2 deployers does which comes shipped with the official kube test 2 project. They do a lot more secure way of bringing like a better way of bringing up the cluster and bringing down the cluster and do a lot of stuff during the time. But this is something if you just want to go ahead and try building a new custom deployer for kube test 2 for your own Kubernetes platform you can start with these. Like you can have some way to bring up a cluster and some way to bring down a cluster and you can use them to implement your own deployer. Additionally some good news. So as of 27 days ago there is also an official out of tree implementation of kube test 2 AKS deployer available. I'm not yet checked it but that is available here. Yeah thank you everyone for listening to my talk. It was my very first time presenting at a conference so sorry for the hiccups. I blog at psaagu.com so if you want to just check about other things that I do you can check that. Again that's my Kubernetes Slack handle and Twitter handle. So kube test 2 is part of the Kubernetes project under the testing oriented special interest groups or SIG testing. These are the links to the kube test 2 project itself and the SIG testing community. And if you want to get involved in SIG testing or maybe in the kube test 2 project itself the community is very welcoming. So please feel free to come and say hi on the hash SIG testing in Kubernetes Slack and you'll be very happy to get you started there. Yeah that's all. Thank you everyone. I had a slide for Q&A so if anybody have any questions I'm happy to answer them. And if not you can always find me on Kubernetes Slack here. Hi thank you for the presentation. I actually just in the process of understanding kube test 2 a little bit more and use it so it was very interesting. I have several questions and most likely we can talk later but what about Sonobui or Sonobui? Like I know that in later situations they also integrate kube test 2 for some aspects. Like if I'm starting with a new cluster and I want to do and test in what to use. Like go straight with this and implementing a custom deployer or maybe using Sonobui that has some helpers I think. I have seen a little bit like they're doing some plugins or something. I know that it's outside a little bit of this but given that it's VMware I just I don't know if I have more information about it. So I'm not very much aware of Sonobui. I just know that it's some, can you correct me, it's testing tool or security tool something? Yeah it essentially runs the conformance test by default from Kubernetes. So I think you would want to extend kube test 2 for the tester part. There are a lot of tester available for kube test 2. We have Kinko. There are very specific to the purpose node testers available to run some scalability test against Kubernetes. So we have tested for that. I think if you want to run, if your focus is on running tests against or using Sonobui then it would be extended kube test 2 for a tester, a new tester. Yeah. And just another quick question. Is it possible to reduce tests that were already created basically for Kubernetes? Yeah because the part of running the test itself, how you actually sort of get those tests and never to integrate it with your custom new binary basically that will run the deployers and so on. So I'll try to repeat the question and try to understand it again but are you trying to ask where the upstream Kubernetes tests are written? So if you want to reuse them. Right. Let's say that I have custom tests but I want to also run maybe a switch of tests that are already implemented in Kubernetes basically. Yeah. So this is where the upstream Kubernetes tests are implemented and they're invoked at where kube test 2 picks them up. That happens and so this is another repo we have in the upstream Kubernetes project. This is test infrastructure and at some point in the talk I mentioned about proud jobs that eventually that have that kube test 2 binary and picks up the test. So you could find those jobs defined here. So just to cover up these are all the projects under Kubernetes. So if you want to very specifically go and look at how those jobs are written you can. These are the name of the repose. So Kubernetes will be Kubernetes slash Kubernetes repo 6 will be the you would find all the repose under Kubernetes 6 under that Kubernetes 6 and so on. So I'm just trying to look for under Kubernetes. Yeah. If you like if you click on any of these you would be able to find out the jobs definition there. So for example say I should pick up. So we have this release. So we'll be sort of essentially vendor in those Tesla saying in your own repo that you have kube test to imagine. So kube test 2 comes as part of the container image that we used to bring up the proud job board container. So kube test 2 comes from the test infrastructure side and then when you write a cube. Let me I'll just pick up any and I can just quickly do I have some minutes. If not we can talk later. Okay. Thank you. Okay. Yeah. So this is one of the jobs that we'll use kube. So what it will do is for example you have your test written somewhere in a repo in this particular case you have some test defined in your container the container the repo. So if you give this line from line number one to 26 to your proud CI CD what it will do is it will create a port using this image. And so your proud job will have the kube test to binary and all other testing information. So all your tests are in other stuff and it will also have a folk of container the container the or any ex ex repo where you have defined the test and. You can also give from which branches you want to pick up the test from and yeah. And once you have all those things handy you can go ahead and just do a run a note s h run a note s h is part of the kube kens e to e image itself. And this is this argument the dot test slash build s h it will look in the repo that it just forked over. So if you want to use cube test to for your own repo test like your test that are sitting in your own repo you want to write something like. This if you are going to use proud and basically here your repo name will come and here your test invocation will come. Thank you. Thank you.