 Hello everyone, my name is Mauricio and today I'm going to talk about debugging Kubernetes E2E test with DELB. One of my first tasks when I onboarded my team was to run the Kubernetes Storage E2E tests using a CSI driver installed in a cluster with a Windows node pool. My mentor Jin had the time, gave me enough instructions to do this and after some time with her help I could create a cluster with Linux and Windows node pools. The next steps were to install a CSI driver and run the E2E tests. After I read the instructions, I learned that you can compile the E2E tests into a binary and run the test with Qtests. With this I could run all of the Kubernetes tests, however I was interested in running only the storage tests. Jin suggested to use the focus flag and with that I could run a single storage test. However, I saw that this test would sometimes pass and it would sometimes time out and fail and I didn't know why. This suspect that it was related with Windows and suggested to go deeper. In this diagram we can see what I was looking at in the cluster. I would run Qtests sometimes and the test would pass, sometimes it would fail, sometimes it would pass and fail or fail and I was not sure why. I had a few things in my mind to debug it. I could read the test source code and try to correlate it with the test logs and in the meantime I would try to see what was happening in the cluster. There were a lot of objects created and deleted in the cluster and this happened too fast for me to understand what was happening. Another option was to add a few statements like slips or printfs in places where I suspect that the test code would be failing, however for every little change I had to recompile the binary. I thought that there was a better option to do this and the option that I thought was to instrument the E2E test binary to run in debug mode with help. The advantages are that we can set breakpoints in any line, analyze the test objects that are defined in the code, analyze the cluster objects and if I make this, this would be a one-time setup, however there were no docs in coordinates about how I would do this and I was not sure if I would have enough time to do it. I also have a lot of questions about Qtests and I didn't know that even if I could compile the binary in debug mode, if I would be able to run it with help successfully. My next step was to find out what Qtests does under the hood and what it's doing, it's calling a few shell scripts which in the end call the E2E test binary through the Ginko binary and the idea that I had was what if we called it E2E test binary with Delve instead. I copied all of the flags that we had from the previous command. I compiled the binary in debug mode and I executed it with Delve instead of Ginko. And with that I could see the Delve command prompt and I could enter a few statements like break and continue. Break in this case is stopping at one line in the volume expansion code in the search test suite and what this test is doing, it's installing the CSI hotspot CSI driver and it's creating a pod and a PVC. After the test run, it stopped at the line that I wanted to debug. And at this point I thought it would be a good idea to analyze the cluster. I could see that the pod was created and I could also analyze it in the Delve prompt. In this case, I'm trying to print its name and check if it matches what I'm looking at in the cluster. This is just a simple example of what are the things that we could do with Delve. After this, I understood what was happening. The pod was getting scheduled sometimes in Linux and it would pass and it would sometimes get scheduled in Windows and it would fail. So my mentor suggested to add a note selector so that we could schedule the pod to run in Windows all the time and hit the issue and debug it. And I thought that this learning could be useful for everyone and I added a new M-bar that everyone could use to run their tests in debug mode. But we can continue adding more debuggers. We can instrument the Q-controller manager, which in this case I'm running on the right and I'm connecting to it through my editor on the left. We can also debug components like the Qubelet in this case through the same commands. I added a breakpoint in the volume manager reconsider. So let's add a debugger to all the things. Thank you.