 Welcome to this talk. Today's session is on who broke the build. How many of here are developers here? Managers, fortunately. Good to know. Okay. Have you ever seen this from your manager? I hope everyone as a developer would have noticed breaking a build. So then the developer would say, how can I actually fix that? Okay. Is there any better way of fixing the broken builds? So let's get started. So quick intro about me. I'm Ram Mohan Rao. I'm a senior software engineer at JFrog India. I'm passionate about open source and I love table tennis. Okay. A quick introduction of what JFrog does for people who don't know what. So JFrog is founded in 2008 it's a public listed company. It has around 1,100 employees. Around nine location with 7,000 customer base and six products, which has get to K8 Kubernetes, all hybrid OSS and multi cloud. We have a universal DevOps stack. And most of these developers are community champions as well. Okay. As you know, software runs on everywhere. So you can consider on your mobile. I think most of you use iPhone or Android. You frequently do an upgrade. So Google, Apple uses our backend to push those software updates for you. Okay. So JFrog mission here is to power all the software updates in the world. So whenever you use any device, you would like to have updates to that so that the latest versions are actually being used. So aligned to that, I think most of you have come from these backgrounds where, where you have used a waterfall model into below before 2000. And then in 2002, nearly to 2010, people still use agile. Don't mistake me by putting a slide over there saying that it is still 2010. But post that most companies started to do releases daily. That's more important. So let's get to the agenda's today's agenda on the talk. So as most of you are here, our developers, the major concern that people tend to have is how can I actually push my changes to production much faster? Okay. So I would just give an overview of how a development environment would look like and then cover end to end testing as most developers feel that they run unit test locally, but not end to end test. So my talk is more focused on how can developers leverage a tool and then use and run end to end test locally. And we have a tool called Cuttle. We discovered it and we were fortunate to get direct from direct contact with the maintainer and we were able to understand and integrate into our CI. And a quick demo, that would be very interesting and a quick summary of what we would be learning. Okay. So an ideal development environment would look like, say, I might be a developer joining a new team, and I might be given a Windows laptop or a Linux desktop or a Mac. I don't really care when I want to set up my dev environment. I should be free enough to set up my environment in such a fashion that it runs on day one. So the onboarding process of development experience is something that we need to focus on. Like when I joined as a developer in most of the previous companies, what we used to do is we used to have a Confluence wiki page. There would be instructions to set up a dev environment. You would copy paste all the instructions and it would take a day or two to set up the environment. Then most of them are manual, I would say. And it would run into some issues. So let me quickly get into how a development environment should look like. It should be a single click automation. And it should be able to develop and test locally as a developer. And it should be as same as production environment. So as I said, I used to generally take a couple of days to set up a dev environment previously. With automation, you can quickly set it up in 15 minutes or 20 minutes. So what does automation save here? It saves time and the resources as well. When you have no manual steps, it would implies error-free. And as a developer, I would like to quickly reload my changes locally. When I mean that, I don't want to deploy it on an external server to test my changes. So I should be able to deploy my changes very locally. And the dev environment should be very much equivalent to a production environment. Okay, let us understand the problem that we have with a current feature branch development. I think most of you use Git as a source repository, right? So when you develop a feature as a developer, who would first analyze the requirements, then create a feature branch, then does his coding around that, and then writes his unit test, test its, and then pushes the code to a Git repository as a feature request. But when he does some test, he only does the local unit test, not the end-to-end test. And end-to-end tests are such a pain that you can't run them locally. And say, as a developer, when I raise a pull request or a feature request on a merge request, the CI would just trigger the end-to-end test on that branch on a remote CI CD server, generally. And it would take at least couple of hours to run the end-to-end test. Say, if something breaks there, then developer is actually losing couple of hours to understand what has actually broken, okay? So we thought, I mean, we were thinking, how can we actually solve this problem? Okay. So let me iterate the steps that I'm trying to help you here. Say, for example, if end-to-end tests fail on a remote CI CD server, what a developer would do is, he would test it and commit it and then push the changes to a Git repository. And then commit the changes back again and see if it works. But there is no environment to test locally. So the pain point is developer is unable to test everything locally. So this round trip continues and how can we avoid this? So let me quickly take you to the remote end-to-end test. See, when a developer writes its unit test, if something breaks, he would fix them and commit those changes again and raise a pull request. Mostly, the CI CD server runs those tests and the time to come back to the developer is huge. Only when the CI tests pass, you would be able to review or merge the code. So is there any solution that we could think of? Then we started evaluating just a heads up saying that most of our development happens on Kubernetes, cloud native. So we were evaluating a couple of tools and see if we can help us to reproduce the same thing. So what we thought was instead of running end-to-end test remotely, why can't we leverage a tool or something that can enable us to run the end-to-end test very similar to what we can run locally? So how would this help a developer? So instead of actually pushing it and there might be some environment issues on a remote CI CD server, that may not work sometimes. So to leverage that, have those end-to-end tests locally, then if tests fail locally, you would be able to run those tests, commit it, amend it, and then run it and fix it. So what I'm proposing here is instead of running end-to-end test remotely, run those tests locally. So how can we achieve this? So as I said, we've discovered a tool called Cuttle, which is a cloud native Foundations tool. It's under Kudo Builder and a tool Kudo and there is a specific Slack channel in Kubernetes called Kudo. You can refer to that. You can probably refer to the references at the end. So what is Cuttle? Cuttle is a Kubernetes test tool. It is basically used for writing test, mainly designed for testing operators or custom resource definitions, controllers, or simply put it as declaratively test any Kubernetes objects. And it's YAML-based, so you don't need to learn any new language over here. If you are comfortable with Ansible or anything, it should be very easy enough. And it accelerates the ability to set up an environment very easily. So to get started, how can you actually install Cuttle on your desktop? If you are using a Mac, you can use a brew tab, Kudo Builder tab, and then brew install Cuttle CLI. If you are using a Linux, you can use kubectl, crew install Cuttle. And Cuttle also provides API integration. Say you are writing, you are a Go developer and would like to integrate it directly into your Go code as an end-to-end test framework. So you can use Go GitHub, Kudo Builder Cuttle as well. So Cuttle is, I mean, if I am a being a developer, what is it used for? So if you are an application admin who wants to automate the creation of Kubernetes clusters, you can still use Ansible for that. But this is more for testing and testing some Kubernetes operators specific code. And if you want to test Kubernetes applications on different versions, okay? And if you are a developer who would like to easily test operators without writing any specific Go code. So Cuttle, let me explain how Cuttle framework would look like. It has three main objects. One is the test suite. You can see here, it's a CRD, custom resource definition, where the API version is Cuttle dev v1, beta one. And the kind is a test suite. So this is a test suite. And you can see kind, start kind as false. So by default, Cuttle uses kind as a local cluster to run all those tests. So if you have a Docker desktop, you can use kind to actually start integrating, start your test suite without having any Kubernetes cluster externally. And I would actually demo the entire test suite and actually how it works. So you can see it also has specific commands where you can actually add commands as a preset of thing. Then next is a test step. So you have a suit declaration, then you have a test step. Say I would like to install some specific app and to see whether that test has successfully thing. So Cuttle has a test step and an assertion as well. So a step would just install or specifically do an API testing around it and assert would do an assertion on the expected values that it needs. Say for example, most probably you would be knowing Artifactory. So as part of my test, I would be installing Artifactory and then asserting whether Artifactory comes up with a single replica. So this is a test suite structure that I have for my demo. So you can see I have an end to end test that you can add into your project structure. So Cuttle by default provides parallel steps as well. So you can run eight parallel tests install is one step, install is one test, scale is another test. So you can parallelly add eight tests that can concurrently run eight. And there is a Cuttle test configurator that I've already shown you. So let me quickly run through the demo and how Cuttle works that would be interesting. Can you see my screen? Okay, so let me show you the project structure here. You can see I have the same structure that I've shown you. And I would like to quickly run a test Cube CTL Cuttle test run. And what it would do is let me actually show you the code as well. So you have the structure that I've shown you. So I've set the kind as true and I have my Docker desktop running. And this would mean it would use a kind cluster to set up the entire cluster and run those tests. So you could see here it has actually created a kind cluster. So let me quickly copy this configuration. I would like to export this cube config file. And I use a tool called canines, which is easy to view the Kubernetes clusters. Okay, so it's an open source tool. So what I've done is I've actually run two tests as part of my end to end test suite as part of the demo. And each test would run parallelly and on a different namespace in a Kubernetes. So you can see there are two namespaces that are getting created. One is Cuttle test extract goldfish, Cuttle test sort cowbird. So both of them tries to install Artifactory and then do an assertion on top of it. So the first is a basic install. Okay. And we'll do an assertion on the replica count. Next is a scale thing where you would install Artifactory with multiple replicas. So the replica count is two over here. And then do an assertion saying that multiple replicas came up. So this is a simple example of how you can actually test a simple Kubernetes object or an application. But this can be integrated with with API test or anything to do with. So this is anything, I mean, it is language agnostic, I would say. So even if you are a Go developer or a Java or any Python developer, you can still integrate end to end test using Cuttle in your project. So let us quickly walk through the demo, how it is running. So what it tries to do is first, it runs the test suite. You can see it ran a Helm repo ad. And then it is trying to run two tests. It is trying to run two tests. One is install, another is the scale at parallely. So it is creating a namespace and then installing Artifactory. You can also see on this K9's tool how this installation is going on. The reason I asked you to stop the streaming of the Wi-Fi is the Artifactory image is around 700 MB, 700 MB and which would take probably five to six minutes to download and install. So bear me for a few minutes. So in the meantime, I would actually show you the documentation of Cuttle. So as I said, this is a cloud native sponsored even free project and it is built by Cuttle's team and it has a Slack channel called Cuttle. And you can see the release cycle and it is very active. This is not maintained by JFrog by the way. This is something that we have been using and we are currently working with the contributor Ken Saipan. He is actually from US. So if we have any issues around Cuttle, feel free to reach out to him on Kubernetes Slack. So and there is a documentation, it's very specific to Cuttle how test suit, test trip and test assertion would work. And there are very specific setting how can you actually integrate any Kubernetes cluster, not only running it on locally using kind, you can use that. So let's go back to the C. So the Artifactory started downloading and it has ran successfully as well. So once that test is completed, the end to end test would actually delete the namespace as well. So let me quickly go back to the configuration part of the test suit where you can see the timeout that we have mentioned is 900 seconds. So it nearly took us around, sorry, it's already done. So it took us around six minutes to complete the end to end test locally. So the main idea around showing a demo of Cuttle is this is simple install that we were able to parallely install two tests, parallely test two tests without any complication, without any complex logic. Simple, easy to use. This can be integrated into any language specific code, either go Python or Java. We started using this in our internal projects and we were able to see the leverage that it has. So so it completely, it completely deletes the namespaces as well. Let me quickly go back to the presentation. So the Cuttle references are, you can see Cuttle dev.docs and there is another GitHub reference that I've shown you and there is this very specific Slack channel, Kubernetes Slack channel called Kudo, where you can post any queries. And there is a tool that I've shown you called canines that you can install using brew install canines on a Mac, which would actually show the graphical view of Kubernetes clusters and access them and manage them as well. Summary. So Cuttle is an open source tool. You can contribute as well. It is used for local end to end testing. With that, we were able to minimize the builds that we broke as part of our code. When you test end to end test locally, most of the code that you have done the changes as part of the feature branch development would work. When we have few broken builds, we can probably release much faster than you could do before. And that would mean happy developers. Any questions? Happy to take them. Could you please repeat? Yes. No, these run on locally using Docker desktop as a kind Kubernetes cluster. Okay. His question is we have the test suite that we have configured where we have provided Helm ad. Is it running on a specific container or is it running on your local machine? It's running on your local machine. Any further questions? Right. So the main idea around is we should, yeah. Okay. So his question is how can developers able to run local setups effectively? How can we actually set up dev environments which are very equivalent to production? We have done in our previous company where we had a setup of long setup. We were using RPMs to set up a, our production was using Ansible to deploy RPMs on a production VMs. So we were able to replicate using Docker containers. So we were using CentOS Docker base image and we were able to just compare it so that, for example, I have some shell scripts that we have written on our own that would actually set up Docker containers and install RPMs on top of it. So it's replicating the same thing as, so cut till does that very effect effectively on Kubernetes side, on cloud native side. So if you want to set up a Kubernetes in a go, you need to write go code to set up, set a context and then create a namespace and then do that. I would, I would recommend, I don't think most companies open source most of the dev setups that they have locally, but few companies does that saying that we have done that and you can probably try it out as well. But most of them use containers to do that. Any further questions? Okay. Thanks everyone for joining this talk.