 Hello everyone. Good noon almost. It's almost noon. Happy to be in Amsterdam and I'm here to talk about Minicube with you. First of all, who am I? My name is Medea Gazizade. I've been maintaining Minicube since 2019. I have released 86 versions of Minicube out of 140 total Minicube versions. I'm also a software engineer at Google slash manager. What is Minicube? By show of hand, who here has used Minicube? Does everybody almost? I think most of the people's hands are up. So I don't think we need much intro to Minicube. In this talk, we're just going to go through years of my experience as a Minicube maintainer, going through a kind of like a history of Minicube to the latest news of Minicube, which is Minicube GUI. Minicube is a tool that quickly sets up a local Kubernetes cluster on popular platforms such as macOS, Linux, and Windows with a special focus on developers, application developers, and new Kubernetes users. And our own survey data confirms that as well. 42% of our users are the ones who use Minicube to develop applications. Whenever I go to KubeCon, people recognize Minicube with the emojis. They're like, oh, that's the one with the emojis, right? So I was like, yeah, yeah, that's the one with the emojis. And who are the people behind these emojis? Other than me, there's Steven Powell. There is Anders Bjorklund, Peratrak Rogovic, and honorable mentions of previous maintainers. Dan Loren, you probably know his name from Chingard, and Thomas Stromberg, and more than 800 individual contributors. Thank you to all of them. Minicube started with a GitHub issue in 2016. So the idea was setting up Kubernetes is hard. Can we have something that just users just set it up? Because you don't need it. Most people don't have a Linux workstation. You just want to play with Kubernetes. That was the original GitHub issue. And Dan Loren responds like, hey, I want to take this issue. I want this. And he actually said, I have a physical Windows machine under my desk. I could use this test lab for that. And the reason he said that, because Minicube, the design of Minicube is you use, at that time, use a virtual machine to start a Kubernetes cluster. And virtual machines could not be tested in Pro or in a container or regular testing. You have to have actual physical machine that supports virtualization. At that time, in the cloud providers, there was no nested virtualization. So you could not even use a cloud provider at that time. But there's a tweet by Dan Loren that says, me, seven years ago, treat your built system like production systems. That's me today. Seven years ago, he was talking about his computers under his desk. These are actually the real pictures of those computers that Dan Loren initially was under his desk. Unfortunately, we lost this test lab to COVID, like maybe 15 seconds of silence. And the reason we lost this is because you could not go to the office to do things with them because they required manual interventions, like restarting them, changing the cables, stuff like that. So they did just all that. We still have them. The Jenkins machine that we're hooked up to them, this Jenkins machine is still running. Their first release was in May 2016. And fast forward, 2023, 1.8 million line-of-go-lang code. And that does not include the website or a documentation code or comments. 1.8 million line-of-code is the MINIQ project, just a repo. The vast support matrix for popular platforms and superior architectures and all the drivers that you could have. One of the advantages of MINIQ is it will meet you where you are. If you do not want to use Docker, you could use any other virtualization technology as your driver. And we also have some special drivers, such as Bear Metal or SSH Driver. And the green ones are the ones that you have integration tests with. And we officially support them. Of course, MINIQ also gives you the option of specifying your runtime, which could be container D, cryo, or Docker, despite Kubernetes taking out Docker from their official code base. And also the CNIs. I was talking in a meeting that said MINIQ is just not one project. It's the sun with an orbit of projects. And I try to visualize that sentence in this slide. And MINIQ is the sun. The top one is the MINIQ ISO. This is part of MINIQ code, but we hope to take it out of MINIQ code and make its own separate project. This is just enough Linux for Kubernetes. It's a handcrafted Linux distro that we added kernel module one by one through the past seven years. We just only added modules that we needed just to run Kubernetes. And we made it work for ARM64, for X86, different platforms. And there's a whole build process and build automation around that. We hope to take it out of the MINIQ repo, but that's one of the main MINIQ's work streams. The other one is the machine drivers. This one is the... If you guys remember, many of you might not remember. There was this thing called Docker Machine seven years ago or six years ago by Docker. It was a library that Docker created to create VMs. And MINIQ actually used that driver. The problem was Docker stopped contributing to that project deprecated in 2019. And MINIQ actually forked that project and kept it alive. So we still use the Docker Machine's lib to create all kinds of virtual machines in MINIQ. And that project is a separate MINIQ codebase, but, officially, MINIQ is the only user of it, or maintainer of it. Another work stream of the MINIQ's orbit is the official add-ons. You might know that MINIQ has more than 20 add-ons, and we officially maintain some of these add-ons, like AutoPause, Search Provisioner, or GCPOS, DuVizor. And those are another work stream of MINIQ. The test infrared tooling, we're going to talk about that in a different slide. We transformed the infrastructure, and, unfortunately, we cannot use Prowl, because Prowl only supports testing in a container, and MINIQ wants to be tested in a virtual machine or a hypervisor. So we developed our own test infrastructure tooling for that. That involves GitHub actions and self-hosted Jenkins agents in different clouds. And also, another set of MINIQ projects is MINIQ maintainer tooling. You're going to learn about some of those tooling, such as Gopro, Flacrate, CISAT, Triage Party, and other tooling. The other work stream is the benchmarking tools that we have developed, about three or four main benchmarking tools for Kubernetes. They actually use it in some other projects as well, but they're mainly used for MINIQ's development. And MINIQ website is its own project. It's in the MINIQ codebase, but it's really its own effort. That goes with MINIQ GUI, which you're going to see a demo of it today, and set up MINIQ, which is the MINIQ's official GitHub action. Other than the orbit that we talked about, which was internal part of MINIQ, there's external projects that MINIQ depends on. The main one, of course, is the hypervisors, all of the hypervisors. Hyper-V, virtual box, KVM, KMU, all the good ones. And Kubernetes upstream. Whenever Kubernetes changes something, we have to adopt. When they take out Docker from their codebase, we have to adopt. When they change something in Kubelet, we have to adopt. And then, CryDockerD, since Kubernetes-124, we have been depending on CryDockerD to continue support for Docker runtime in MINIQ. Docker runtime is very popular because it provides you the fastest way to build your image directly on Kubernetes and we continued supporting that for our users. Kube-ADM used a Kynes-based image for Docker driver, Cryo, Mavi, BuildRoot, Podman, ContainerD, Lima's socket VMnet, Kudos to Lima that helped us to provide KMU driver, CNIs, and that goes to the next slide. During the past few years that I've been maintaining MINIQ, if you ask me, what are the simple principles that you follow are these things. These are the ones that I follow myself. Empathy and inclusion, open source, backward compatibility. And for the coding part, I follow XP and Yagni, which is extreme programming and you're not going to need it. These are like principles that I follow for MINIQ and when I do the code review, I've been following that past few years. And this is the transformation of our test infra. I'm not going to go through the slide, but we massively invested in improving the test infrastructure for MINIQ. We have more than 46 self-hosted agents in five different clouds. That includes clouds like Equinox or Max Stadium, just to actually physically test MINIQ and those machines. We have a full list of the integration tests in our website that is automatically generated, too. So, state of MINIQ in 2019. This was actually the real GitHub issues and also this was the MINIQ Notes. We have a meeting called MINIQ Roundtable. This was the subtitle for meeting, burning the legs of the developers since 2016. This is real. One day, my manager was telling me, my laptop's fan is always on. And one of the other maintainers, Thomas Strumberg, he had to ask, are you running MINIQ? Seriously, that was so sad that we had to ask people, are you running MINIQ when their laptop's fan were going on. It was very disastrous. So, we invested in developing tools, generating flame graphs. Actually, those tools are open source. You're going to see the link for them in the end of this presentation. And we identified how to conquer the CPU usage of MINIQ. The left one is the CPU usage of MINIQ over time. And the right one is CPU usage of MINIQ compared to other tools. I know this is so small to see, but this is not the point of this talk. You could see the full benchmarking on MINIQ website. Simply go to MINIQ website, go to benchmarking section. You'll see all of them in a bigger, nicer way. Another thing that happened in 2020, we reduced the MINIQ start time significantly. It was the same time that we introduced the Docker driver and the unified preload images for all driver. So, all drivers of MINIQ became much faster, 86% faster. Today actually uses about 19 seconds, but this is in 2020. That was 21 seconds. As a part of this effort, we developed two tools, Slow Jam and Time to Cades. Sorry. Slow Jam helps you to graphic visualize the stack trace for your Go application. And Time to Cades helps you to identify and measure your Kubernetes creation performance in different categories. We have identified about six categories and I'm going to link to all of this project in the end of slides, both Time to Cades and Slow Jam. That helped with making MINIQ start faster. Another effort that happened in 2020 and 2021 was automated benchmarking. Every MINIQ PR, okay, let me tell you a story before I tell you why we did this. We made all of this work to make MINIQ faster and nicer. We merged some PR that made it much worse. And we have to stop that. We did all of that work to make it nicer. And so we developed a tool that when you make a PR, it comments on MINIQ PR and says, hey, this PR is going to make MINIQ two seconds slower or two seconds faster. So we know exactly if the PR are making MINIQ slower or faster. And those benchmarks are automated. And they go through daily, weekly, per release. And this is an example of it. In 2021, MINIQ go through being used by many other products such as scaffold, such as cloud code, such as many other tools, cloud shell, many other products that use MINIQ as an embedded tool. So you had to invest in embedded tooling, stuff that machines need, not humans. For example, JSON output. So every MINIQ command has a JSON output or something called schedule stop. That you could say, hey, MINIQ, do this task five hours later or an event mechanism that you can parse MINIQ's events in a step format. You could say, oh, there's a step one, step two, step three. So the machine could tell if MINIQ, what step of the progress MINIQ is. We also added auditing feature for MINIQ. So you could audit who did what if there are multiple users are using MINIQ at the same time. You could see who did what. The advanced weights logic, dash, dash, weight. Watch flag, license, command, supporting C groups, using in multiple different types of environments and machine readable exit codes that actually the MINIQ exit codes are published to our website automatically. So if you go to our website, search for exit codes, you could see, oh, the exit code 79 means this and these are the possible reasons. So this is like, we made everything usable for machine usage. In 2021, we also invested in making MINIQ work in the CI. Even though that's not the main usage of MINIQ, there's a lot of demand for that. You could see examples of MINIQ running in all of those CIs, like CircleCI, Travis, Azure, GitHub, GitLab. And you could go to MINIQ, CI, Oregon, GitHub. We also invested in setup MINIQ GitHub action, which is maintained by MINIQ maintainers. So you can test your PRs and GitHub actions against MINIQ by simply adding mediaGH slash setup MINIQ. And there's very good examples of actually building your Docker image against MINIQ on your PR. I encourage you, if you are interested in that, go to this repo, setup MINIQ GitHub action. There was a talk in 2019 by Thomas Stromberg, MINIQ bringing Kubernetes to the next billion user. I was very brand new in MINIQ at that time, but that was very inspiring to me, that talk. Thomas talked about the next billion users that are going to use Internet. They're not going to be English speakers. They're going to be in Africa, in Asia, and countries that are developing. So it's like, if you want to make Kubernetes available to them, you have to make it ready for them. You cannot assume they speak English. So we invested in a translation framework for MINIQ. So without having any knowledge of coding, you could add your language to MINIQ. This one is Japanese, but you can see other languages like French, Spanish. I wish I had Dutch, so we could add it for Amsterdam. But it can easily contribute to that. Just go to the MINIQ website, search for translation. It will tell you exactly how to add your language to MINIQ. Another thing we invested in was including people in the triage process. MINIQ has more than 8,000 GitHub issues. So it's very massive user interaction in GitHub. So we developed triage party and every Wednesday, there's a nerdy party called triage party. You can join and we will sit together, play a group video game of triaging MINIQ. Issues. In 2019 to 2021, we also overhauled or started the MINIQ website. Before that, everything was, you know, GitHub back down pages, not really usable for public, but now that's another milestone that happened in MINIQ lifetime. And of course, we added the Twitter, which is automated mostly in the Slack channel. Another big thing that happened during my time in MINIQ past few years is investment in developer velocity. So I talked about massively overhauling the test infrastructure. We have so many tests in so many environments and so many run times, different drivers. And we have about 300 integration test cases. The left one is the raw test logs of Golang. So let's say there's about 35,000 line of code. There's one test that is failed. You want to copy that test log and just see that. First, you have to identify that, which test was failed and then copy it. So we invested. I invested in making this thing called Gopog. Don't tell me what it means. The name, maybe I'll tell you. It's embarrassing. The name is just an embarrassing name. It's science for go pretty or go home. So the idea is making Golang tests pretty. So I just say go pretty and then I auto-completed myself or go home. So the right one is Gopog. So it just makes it foldable to categorize it by different results. You can see all the failed ones, all the past ones, all the skipped ones. And you can copy them and also can give them permanent links to them. And also sort them by duration. You could say which test took longer. This helped us to squint less when we reviewed a PR. This was one of the developer velocities that was a game changer to making Minikube serve millions of users. As you can see, I like charts and I love charts actually. So another thing I invested in was developer velocity for test flakes. One of the problems that we had in Minikube, I don't think there is a project that does not have flake tests. If there's developers here who maintain a project, do you guys show off your hand if you have flaky tests, like the tests that flake? Yeah, yeah. Let's see. Encouraging amount of people. So we have flaking tests that sometimes flake. You don't know if they're actually real failure related to this PR. And the biggest part of it was we had to manually look at the history of the tests. It's like, oh, yeah, yeah. Don't worry about that test. That's, I know, that fails all the time. So that's not related to this PR. But it was such an inaccurate thing to do and also very brain-consuming for the maintainers. So we developed this tool to identify the flaky tests in Minikube against the head. So this tool will comment on the PR as tells you this test fails 2% of the time on master or head. And so that means most probably is related to this PR. But if it says 98% of the time it fails on the head, that means this PR probably didn't do anything that broke that test. So we actually have per test, per environment of this flicker is automatically published to our website and also commented on the PR. That was another huge investment that I did for developer velocity for Minikube. Another investment was Minikube bots creating PRs for us. Do you guys remember the chart that I was showing you that was external dependencies of Minikube? I'm sorry. It's like scrolling so fast back. But this one. So any of these projects when they bump something, we also had to bump something. And some of this were just basically mechanical work that we had to do. And we had to be tested too. So whenever QADM constants changed, cops and Kubernetes, we had to bump things. So we just made 15 different automation that Minikube bot would make PRs for us. Actually, Minikube PR made more than 500 PRs. And we made it to push the code in midnight. So when we are in the office, test results will be already for us to look at. And now we'll not use our traffic test infrastructure. We also sync the Alibaba images for users in China because they don't have access to American cloud providers. And building the ISO and Docker images to the PRs. And actually, I'm a little bit concerned because as you can see, my name is the first one. But Minikube bot is catching up to me to the contributors guys. Minikube bot is top six contributor in Minikube, more than 500 PRs. By the way, you can see all the main maintainers of Minikube. One thing weird happened in 2022 that I thought I would never see that as a Minikube maintainer. Users asked us to have Minikube without Kubernetes and we delivered. Dash-sational Kubernetes. And the storyline of that Docker desktop license changes, a lot of people are using Minikube just merely as a Docker desktop replacement. And it's like, I don't care about Kubernetes. Can you start Minikube without Kubernetes? And we said, sure. So that happened in 2022. 2022 was also the year of ARM64 for VM driver. We delivered our ISO in ARM64 because of the Mac M1 machines that are mainly ARM64. So we could actually today use KMU driver on macOS with ARM64. Well, now we could actually talk about Kui. I'm glad that we have time to look at the demo of it. So Minikube.gui, this is how it looks like. It's just a tri-icon. Oh my God, I should have closed these things before. Don't look at my desktop, it's so messy. So this is Minikube.gui. It stays in your tri-icon like that. And you could use most of the Minikube features that can restart it, you can stop it, you can pause it. I didn't talk about Minikube pause, but this is another thing that happened in Minikube. We introduced this thing called pause. It actually pauses the Kubernetes, but not your application. So if you deploy an application to Minikube, if you pause Kubernetes, it will pause incoming requests. It will pause taking new applications on your Kubernetes. It will significantly reduce your CPU and you can delete it. And you can use the popular features such as Docker N. Right now if I do Docker PS, this is everything is inside Minikube. You could use a service command, you could mount a folder tunnel, you could SSH to Minikube, and then you could do the dashboard, hopefully that works, and then the add-on. So you could enable different add-ons in Minikube. This is a new project that has been under Minikube's milestone for a while, but we're finally happy to have the first release of it. And you could actually go to the repo, Minikube GUI, if you want to install it and give it a try. But be patient with us, this is like, and we usually welcome your contributions. This is written in C++ and QT. So if you are familiar with C++, come help us to make this Minikube GUI better for the rest of the users. So I would like to hear. Look at the, I like charts. I made a chart of, this specific chart is from Dan Lauren. He maintained Minikube, he started Minikube. And you could see 2018 is when he retired LocalCube. LocalCube was a project that was used for bootstrapping Kubernetes. So we are using Qubedium right now. Everybody uses Qubedium. And you could see in 2016 we had rockets containing runtime. But now we have cryo and continuity and Docker. But this is a timeline that generated from the time that I started maintaining Minikube. The star ones are the more major changes. You could see the 1.2 is when the translation framework happened, the registry add-on was added, and auto-setting extra configs. And 1.5 is actually a very interesting release. I like that release myself. That was auto-selected drivers for you. Before that you had to say Minikube start, dash, dash, driver this, dash, dash, dash. You had to know what driver you want to use. But 1.5 Minikube just auto-selects it for you. It looks at your system and says, oh, that one is the best driver for you. I'm going to choose that for you. This release is interesting, the 1.7, because that's when we introduced the Docker driver and a bunch of other things, such as driver-on, C-groups. This version 1.7 beta, that is also interesting. That's when we introduced PaaS. 1.8 is when we introduced the preload. By the way, you don't have to read all of that, but I'm including all of this as appendix, so you could download it from the coupon PDF. The preloads were a game changer in Minikube. What does a preload do? When you pull an image using Docker or any container on time, it first downloads the image from the repository, calculates the shock, matches the shock against that, and then pushes it into that and loads it into the memory. That was time-consuming. We already did that and loaded it to a tar-bile, and then when Minikube starts as a sidecar, injects that into the file system. So both for VM drivers and Docker driver, we unified this preload mechanism that saves Minikube more than one minute in creation time. Also, here we have more auto-selection, so we auto-select memory for you. Before that, if your computer had a lot of memory, so you had 16-gigabyte of memory, Minikube would still choose the minimum one for you, but now Minikube says, okay, you have a lot of memory. You could be a little bit nicer. You could give it a little bit of more extra room for space. Based on what you have, you auto-select the CPU and the memory here. At 1.9, we have this user experience improvement, such as delete on failure. So Minikube, one of the biggest problems in Minikube at that time, I remember, you start a cluster and it will fail halfway. Always the problem with the solution was delete it and start it again until it will fix it itself. So Minikube now would do it for you, just delete on failure and redo it for you when it was safe to do. Another important release was this one. This was a huge massive release for networking of Minikube that helped us deliver multi-node because we introduced Minikube's internal networking. There's some interesting add-ons, such as Metal LB, but the CNI work in this release was massive. Minikube released one major overhaul in this release. In this release, it was when we introduced GCP Auth add-on, which is automatically, before that you had to manually add the secrets to every pod if you wanted to use a private registry. You guys probably, if you are familiar with local development, you probably know what I'm talking about. If you have a private registry, not a public one like Docker Hub, you had to manually inject the secret to your pod. So this add-on would help you to not do that. It will do it for you. And this release, 1.12, is when Docker driver was generally available. And we supported more overlay FS. 1.15 is when we were working on the embedded use. Remember the slide about JSON and schedule stop. This release is when we, 1.17, is when we start working on ARM64. And there's a new driver called SSH here, which means Minikube will SSH to a remote host and start a Kubernetes cluster for you there. Oh, and this is one of the cute ones, Autopause. Autopause is an add-on for Minikube that would automatically pause Minikube when it's not in use. This add-on is close to my heart because I care about battery life and stuff like that. At that release, we also support custom images for add-ons. And 1.18 is when we improved WSL and Windows for Docker driver. And this release, 1.16, is when multi-node went GA. 1.17 is auditing flag. So if you have not tried this flag called dash-dash-user, you probably don't need to use it, but if you are multiple users or multiple, and you could see the audit of Minikube logs. If you just run Minikube logs, anything is dash-dash-audit. It will show you who run Wet Command and Minikube if you use the dash-dash-user. So try that audit. It's fun to see. Actually, this was about the same time that we automated GitHub logs issues. So when we used how to create the issue in Minikube, we asked them a million questions. Can you give me the log of this? Can you give me the log of that? Can you give me the log of this? And most people will give up. And I understand that. So we created a command that will give us everything we need in one file, Minikube logs dash-dash-file. They would get logs for everything. And put it in one file, you could attach it to GitHub issue. Minikube 19 is when we introduced the Minikube CP or Minikube copy, which you can copy a file into Minikube very easily. That's much more intuitive than Minikube mount. Most people now CP command or copy command as opposed to mounting a folder into your Minikube. Another interesting thing that happened in 119 was the Minikube image command. Minikube image command helps you to build images without Docker. So if you, let's say, don't have Docker, and we use buildkit underneath, which is open source tool, you could just say Minikube image build just like the same syntax as Docker build. Minikube image build dash-t, tag it with your name, and then the path to your file. That was a very interesting release, actually. Anders worked a lot on that. Let's see what is interesting here. Here we add Polish translations. This is Europe, hopefully. Any Polish people here? Okay. It also added some interesting command called Minikube version dash-components. You know Minikube consists of not million components, but a lot, close to a million, I'm joking. But you can see the version of all of those things with Minikube version dash-components. Minikube-124, I remember I was talking about when the Docker desktop license change happened. That's when the Minikube dash-no Kubernetes happened. What else is important here? We fixed the GPU. I guess you might not know. Minikube has an add-on for GPU, so you can run Minikube on your NVIDIA GPU. I never use it myself, but I assume maybe Bitcoin miners or something. I don't know really who uses that. Are you a Bitcoin miner? You're laughing. Okay. 124 is when we synced with Alibaba that helped our Chinese users. Chinese users were in misery before, honestly. They went through a lot of difficulties. I realized there was a project that was forking Minikube code base and was changing where we push our code and it was changing into Alibaba. And they were maintaining that, this guy, for Chinese users. Well, we could just do it ourselves, just sync our images to Alibaba so they could be not in misery. 125 also added binary mirror. Minikube downloads a bunch of binaries. If your binary is in a firewall, that's also helpful not only for Chinese users, but for corporations under a very strict networking that they cannot use anything, public, whatever. Sorry for saying whatever. I don't mean undermined your company's policies. We all comply. I comply. Google, listen to me. What is important here? There's so many of them. One of the saddest thing to do, the hardest thing to do here was generating this chart, honestly. I wish we had less releases in Minikube because it was taking so much time to make this timeline. 126 added the ABPF support. So you could do all kind of EBPF things in Minikube. This is actually a talk in the last KubeCon. Francis did a talk on using EBPF with Minikube. 128 is, of course, the log4j when it was affecting everybody. It was affecting a couple of Minikube add-ons because Minikube uses Golang, doesn't use Java, but a couple of our add-ons were using log4j and was affected by that. We also had a new add-on where they're called Cloud Spanner that you could emulate Cloud Spanner in Minikube. So imagine you don't want to pay for Cloud Spanner. You go and just, oh, I want to build an application using Cloud Spanner. You could just download the new add-on called Cloud Spanner. 130 is when we had the QME driver as experimental on Windows and we have support for CSI volumes and improve the cry-dockerty. Oh, I forgot the 126.1. That's a very important one. That's when Kubernetes broke Minikube. Kubernetes actually broke many times and Minikube identified Kubernetes breaking. It's one of our road maps to add Kubernetes testing with Minikube. This is the last slide. I promise I'll give you all of the links if you want to check out all of the open source projects that I talked about that Minikube created as a part of maintaining Minikube. This is it. And with that, thank you very much for coming to this talk.