 Well, hello, KubeCon and hello, Chicago. Well, if you are here for the Minikube talk, this is the right room. And also, this is the talk after the lunch. So if you're here to also take a nap, this is also the right room. No judgment. All right, let's get started. Before that, let me introduce myself. Who am I? My name is Medya Ghazi-Zadeh. I'm a senior software engineer at Google slash manager. And I'm part of the Kubernetes kernel team. I've also been maintaining Minikube since 2019. So about four years, and I've released 90 versions of Minikube out of the 140 versions of Minikube. By a show of hands, I would like to know who here has used Minikube. I think that's almost 90% of the people. We have surveys in Minikube that we ask people why they use Minikube. And according to our surveys, 42% of the users use Minikube to develop applications on Kubernetes. And the rest use it for learning. And followed by testing is the last category that people use Minikube for. Minikube, when I talk to people in CubeCon, I say I maintain Minikube, they recognize Minikube with the emojis. And behind these emojis stand 800 individual contributors. But notably, two other maintainers, Stephen Powell and also Anders Buchlan, they've been with us for years. And this is the result of their hard work as well. In this talk, I want to, as I've been the maintainer for four years, I would like to give you an overview of Minikube through the years. In each year, what happened. And in the end of the presentation, I'll also talk about some interesting news you have for 2023. So Minikube started in 2016 with a GitHub issue. That was a year before that Google open source and everybody was so hyped about Kubernetes. But it was so hard to actually have a Kubernetes cluster. This was the age before Cube ADM, it was the age before local Cube. You had to do everything the hard way to start a Kubernetes cluster. So the GitHub issue was like, can we make this easy? So Dan Loren picked up that GitHub issue and says, I have some physical Windows and Mac machines. I will try to set something up. So the idea is if you have a Windows or a Mac OS, you're just a general laptop, you should be able to start Kubernetes cluster. And actually these are the pictures of those computers under the Dan Loren's desk that we kept in San Francisco office till COVID hit. And because they required our manual intervention and manual updating, unfortunately, they passed out due to COVID. They're on one of the few victims of COVID in the computer world. But rest assured we replaced these physical machines with a cloud-based testing that we're gonna see in the future slides. Fast forward, 2023, Minicube has about 60,000 line of code of the native code and also 1.8 million line of code, including the vendors. When I started working on Minicube, I developed a set of principles that I have been following to lead this project. And I learned these principles from other projects or from the people before me. Some of them are very familiar. You might have heard from the UNIX principles, XP and Yagney. Yagney is short for You're Not Gonna Need It and XP Extreme Programming. Basically following that as a true principle of Minicube. So another principle has been backward compatible. So if users rely on something, as you expected in Linux world or UNIX world, that you don't wanna change that because to you that might sound like a bug, but that bug became a feature to others. So that was another principle for Minicube that. And also empathy and inclusion to the core really. And my other principle that I've been following was data-driven decisions. Everywhere in Minicube you're gonna see in this talk we've used metrics and benchmarking to make every single decision. One of the inspiring talks in 2018 that really inspired me when I started working on Minicube was this KubeCon talk called Bringing Kubernetes to the Next Billion Users. In this talk, Thomas talked about the next billion of humans who are gonna have internet. They don't speak English and they do not have fast internet connections. And that inspired us to make something for Minicube. First of all, we started adding other languages for Minicube. So nowadays Minicube could speak multiple languages, French, Spanish, two Polish and German, Chinese. But also we make Minicube work with slow internets. That's one thing that we just first started working on it in 2019. By the way, if you speak a language that Minicube is not supported by, you could easily contribute that language to Minicube without any knowledge of coding. You can just go to our website, click on search for translations and contribute your language to Minicube. Another inclusiveness for Minicube is not just about the language or the cultural inclusiveness, but the technical inclusiveness. So Minicube is one of those tools that supports you wherever you are. If you're a Docker user, you wanna start a Kubernetes cluster with Docker, let's do that. If you're a VirtualBox user, if you're a KBM user, whatever technology you have, we will start a Kubernetes cluster for you. We don't have an opinionated way of saying, oh yeah, let's pin our soft onto this or that technology. And in the hypervisor world, we have all these hypervisors that we work with. The green ones are the ones that we have integration tests with. So you can have it with Podman, we can have it with VMware, VirtualBox, Hyper-V. You also have SSH and bare metal. These are the two that I suspect most people have not heard about it that much because based on the GitHub issues that we receive, but this also exists. You can install Minicube and a remote host using SSH driver. Also, Minicube is one of the few, I mean, it's the only way to try all of the container run times supported by CRI interface. So the Kubernetes has this idea that anybody should be able to implement a container run time for Kubernetes, but how can we try it? Today if you wanna try cryo, container-y or Docker, you could use Minicube. And the end of this talk, we're gonna talk about the NVIDIA Docker, which is a new add-on to the container run times of Minicube. Of course, you could try all of the CNIs, that's just basically Minicube start-dash-cni, and you decide which CNI you wanna use. Top of that, the architecture, ARM64, 8x86, and so on. One of the things I'd like to share with you guys is Minicube is not one project. Minicubes from the beginning has been multiple projects. I'll tell you some examples, we're not gonna go into detail of all of them. Since Minicube started, we created our own Linux, kernel module by kernel module, just enough Linux for Kubernetes, and that's the Minicube's ISO. And quite frankly, that one could be its own separate project, and we are hoping to make that a separate project, so we just have everybody use this ISO as something that we've been maintaining for more than seven years, like just enough Linux for Kubernetes. Another one is the Machine Driver, which we recently moved it out to Minicube Machine Organization and GitHub, which means this is the code that knows how to create a machine. Does not matter what machine you wanna create. You wanna create Hyper-V, you wanna create KVM, Docker, whatever type of machine you wanna create, the Minicube Machine knows how to create that. If the name sounds familiar because it is, the Minicube Machine initially was Docker Machine, that Docker Machine, they stopped developing that in somewhere in 2018. Officially, it has been read only since 2019, but Minicube has kept that project alive. Now it's called Minicube Machine. It's basically a fork of that with all the good things up to date. Other things that I'm not gonna go through, Minicube has Minicube GUI, which is like a graphical interface. It's a separate project. Minicube website is our thing. Setup Minicube, which is like a GitHub action that you can use Minicube with. And Minicube was the project that unfortunately we could not use prow, so we have been maintaining our own infrastructure for testing. And the reason for that is, Minicube could not run, Minicube needed testing with virtual machines and nested virtualization because we wanted to test all of these different hypervisors that we support. So we have our own set of infrastructure that we manage as well. Other than the upstream, I mean, the other projects around Minicube, Minicube also has dependencies that we rely on. For example, upstream Kubernetes. Whenever Kubernetes does something, it affects us. It has happened before. If the Cubelet does something, it broke this or that. And basically, if any of these projects do something that breaks, it also breaks Minicube. So I am not gonna go through all of them. Cube, ADM, Cridocardy, Mobi, Bellroot, and so on. You guys remember that this is our test machine from Don Loren's desk that we transformed it to a multi-cloud testing. So we have testing in GCP, AWS, Equinox, Azure, and Max Stadium, and so on. So we have testing everywhere, a combination of hypervisors and a combination of drivers and container runtimes. And during my time working on open source projects, I never seen any project with this level of combination of tests in different environment. So back in 2019, the state of the Minicube, both in CPU usage and the start time was really, really awful. To the extent that I remember when my manager was talking like my laptop is running hot, we would have to ask her, are you running Minicube? And it was not a joke, it was really, a lot of the time it was a problem. And we have this joke now in the Minicube meetings, burning the legs of developers since 2016, but we had to do something about it in 2016. So we started benchmarking, we developed tools that we open sourced, and we created the flame graph for every function in Minicube that is being invoked. We measure every CPU usage of every function. And actually that's a topic of another Q-Con talk in 2020, if you like, you could watch it. In Minicube website, we have a section for all of our talks, you could refer to that if you wanna know in depth about that. This is before after, you could see how Minicube CPU usage went down, and the left, you could see the Minicube CPU usage over time. And on the right side of the chart, I'm sorry that's very small to see, but you could see it larger on our website under the benchmarking section. You could see the Minicube compared to other tools in the industry, like K3D or Kind or Docker desktop, uses the least amount of CPU based on, this open source tools that we developed to measure the CPU usage as well. Same as a Minicube start, it used to take almost three minutes to start a Kubernetes cluster. So in 2020, we made it 21, and nowadays you could even get away with 17 seconds if you have a good internet connection and a good enough laptop. And part of that was also VM-free and preload images, which is its own project as well. We basically invented a new mechanism to preload images for VM drivers and Docker drivers and Poglan. Another tool that we developed called Time2Keytes, this goes back to the principle of Minicube being data-driven, Time2Keytes can measure you from zero to when your Kubernetes cluster is ready. So, and you can measure that when the API server is running, when the SED is running, when the cluster is ready to start running your app. So both of these tools, SlowJam and Time2Keytes, is also open-source, then we use that to make Minicube better. We also have set up automatic benchmarking using these tools in our website. So every day we have daily benchmarking, using all of these benchmarking tools we have in different categories so we can have better idea if we are making Minicube worse or better for the users over the time. Year 2021 was the year of embedded use. This was the year that Minicube was used by scaffold, cloud code, other tools and they wanted features that was good for embedded use. So, for example, auditing, watch flags, schedule stop, JSON output, and so on. 2021 also, it was the year of Minicube CI. We started based on the request we got. We made Minicube work in all of the CI's that we could find, Proud, CloudBuild, Travis, and so on. And also Minicube GitHub Action. We have an official GitHub Action for Minicube. If you're curious how to use it, it's super easy. You can just add that step to your GitHub Action. We also, Minicube did not have a website when I started. It was basically marked on files in our GitHub repo and we overhauled that documentation to our website and we have sections for everything. Tutorials, handbooks, add-ons, frequently asked questions, a contribution guides, and so on. We also created Twitter bots and Slack channels that to communicate with the users better. Also, on Wednesday, we started this new thing called Triage Party that we invite the community to come triage the GitHub issues with us because Minicube, being the front-end project that users go to to start a Kubernetes cluster, we get a lot of the newcomers GitHub issues. And triaging Minicube would be beneficial if we use the community in that because we get a lot of GitHub issues from the absolute beginner users. And by the way, this tool, Triage Party, came out of Minicube and now multiple SIGs and repos in GitHub, SI in Kubernetes world are using that. The link to all of that is gonna be in the end of the slides. Another thing we did was investing in the developer velocity. This tool we built called Gopok, which converts the GoLang integration test to a parsable, foldable, searchable way. So the left is just the raw test logs. Unfortunately, Minicube has to gather a lot of logs because we have embedded systems, we have embedded Linux, we have hypervisors, we have to gather a lot of logs. So every failed test could have up to 10,000 lines of logs. So if you have 300 test cases, that's a lot. So Gopok can make the test logs sortable for you individually, you can link to them. And you can, on the top of that, Gopok can help us to identify the flake tests. So this is an example of comment on the Minicube PR that tells the maintainer, helps the maintainer to know which test is not a flaky test and is failing on this PR. So this test is flaking 0% of the time, so it's highly likely that this PR broke this test. This is also another chart. As I said, one of the principles of Minicube was like gather as much data as we can. This shows different test cases in Minicube, how much they are flaking over time. You can see some tests are flaking more or less over time. That helped us increase our developer velocity. The year 2021, we also created automations for everything. Basically, everything we could do to delegate that to a machine. We did that. So today, Minicube Bot has created more than 500 PRs. There was tasks that we had to do it manually and some of them are very, very long tasks, such as building an ISO or an image that would take three, four hours. And we made the automation to build those images when we are sleeping at midnight and the test would be run on them. And then we come to office at 8 a.m. in the morning. The tests are ready for us. We're just ready to merge it if you want to. The year 2022 was an interesting year. This was the year that Docker Desktop announced that they are gonna charge on macOS and Windows. And I saw a lot of users organically making blog posts that they're using Minicube as a Docker Desktop replacement because basically Minicube has a VM and has the free mobby Docker, the open source Docker in a Linux, which is free and legal to use. So they would just say, let's just use Minicube instead of Docker Desktop. And they were asking us, can you give us a Minicube without Kubernetes? I don't care about Kubernetes. I just want to use it. And ironically, we delivered that feature so you can actually today run Minicube without Kubernetes. This is KubeCon, but this was one of the interesting things happened in 2022. 2022 was also the year of ARM64 with the M1 laptops from Mac, Apple became popular. So it was a year of delivering the KMU driver, also ARM64 for Docker, and so on. Year 2023, this was the Minicube GUI. I'm curious, who here has used or heard of the Minicube GUI? Oh, two people. Okay, I'm surprised that actually people know about it. Not that many people know about it. And this project's actually ready to use. You could use that today. And if you like. Year 2023 was also the year of AI. So today actually we released Minicube just before this talk and you can use Minicube with NVIDIA GPUs today on Docker driver on Linux. So if you have a GPU on your machine and you wanna try AI stuff on it, you could start Minicube with Minicube start dash dash GPUs and NVIDIA. We have a doc on how to use that. We are also working on creating code labs for it. And I hope that this helped the users to get their hands on AI easier. Because in the 2016 when Kubernetes was brand new and fresh, a lot of people wanted to get their hands on the Kubernetes and Minicube helped them to onboard them to Kubernetes and helped with adoption of Kubernetes. And to this year, a lot of people talk about AI but not that many people actually have their hands dirty with the AI. So maybe this will be a feature that would be helping them again in 2023. Also we have another add-on called Kubeflow add-on. You could enable it today. It's called Minicube add-ons enable Kubeflow which you could basically run any AI workflow, Jupyter networks and anything you wanna try. Again, we are working on code labs to have like a walkthrough of trying all of these new AI features. So stay tuned for that. But our documentation, our website, if you just go to the Minicube website and search for NVIDIA, we have enough to get you started. Well, now it is linked to all of the Minicube GitHub repos that I talked about here. You could see the Minicube GUI, the machine driver, setup Minicube GitHub action, slow jam, triage party, GoPoke, which was the one that converts the integration test results to a more possible one. Time2Kids, which was the one that helps you to measure performance of a local Kubernetes from the start to the time it gives you a Kubernetes cluster. We also have a bunch of other things such as pull sheets that helps you to generate leaderboards on your GitHub issues. So one thing that we noticed, a lot of people will help us in triaging GitHub issues, but they would not get recognized in the release notes. So we said, let's change that. So we built a tool actually to measure contributions other than coding so we could quantify that and measure that and give recognition to the people who worked on it. That was the end of my talk, so thank you very much that if you have any questions, I'm here to answer them and if you like, you could also leave your feedback for this session. So thank you very much. Do you have any questions? Yeah. I am very lucky for the past few years that Google has been paying me to do this and the things that I really like. Of course, there are a lot of things that you do other than that, but this has been main focus of that and I'm grateful for Google sponsoring this. Yeah. We are lucky that we have maintainers who are non-Googlers as well. In the beginning of the slide, actually I, let's go to the first slide. Anders Bjorkland, he's the other maintainer. He has been around also for four years. I should have gone up, where is it? Here, yeah. Steven Powell and Anders Bjorkland. They're the other maintainers. To answer your questions better, there's flows of incoming contributors and it goes up and down just like a sinus wave. I have seen days of 15 people coming to the office hour and it's like, give me tasks, give me tasks. And there have been days I've been only one person who is like non-Googlers. So it's like, it comes in waves. And I think Minikube has been lucky or fortunate that more contributors came to Minikube than Kubernetes because I think Kubernetes is kind of like a scale-looking big thing and a lot of new contributors would not want to go touch Kubernetes itself directly so they feel more welcome to start with Minikube first. So we see a lot of people who start their first contribution to the Kubernetes world through Minikube. Yeah, go ahead. What are you asking? I just saw your office hours. Oh yeah. Yeah, yeah, yeah. I'm warning for contributors. Yeah, I'm glad you asked actually. So in our website, if you go to the Minikube website, you have a section called Contributing Guides. So let's see, go to Minikube. So here if you go to here, at the very end there's a contributing section that basically have a step-by-step how to contribute in different sections of Minikube. You're welcome. Any other questions? Oh, go ahead. So this is a very good question. And I think there are many ways you can compare them from performance-wise, from usage-wise. So Minikube has some special features that has some loyal customers that they cannot live without Minikube. And one of them is fast image build. So I didn't talk about the image build benchmarking. I'm gonna show you guys here in our website. This is how fast you can build an image if you're a developer. This is versus Minikube versus kind and versus K3D and versus micro-case. So the orange is micro-case, the green is K3D, the lighter yellow, I mean, this one is kind. And this red and blue are the Minikube. So it is about 36 times faster to build an image using Minikube's Docker and feature. And interestingly, we have the Minikube's Docker and feature for container D as well. So we can call it container D at this point, but for the legacy and don't break me purposes, we kept it called. So we have some loyal customers that they cannot live without. And other loyal customers are the embedded users. So they use Minikube as embedded. So Minikube has JSON output for all of the commands, has auditing, so you have multiple embedded tool, let's say scaffold and cloud code and tilt, whatever. They all use Minikube at the same time. We have auditing feature that, so who did what on the cluster? They could use those features. And we have the slide that I went through. I was scared that I'm gonna run out of time, but I skipped on it very quickly. So there was a slide on embedded use. Yeah, this is some of the features that embedded users use. So Minikube is more tuned toward people who wanna develop applications on a Kubernetes cluster and they wanna do fast and easy. Other tools have their own special things. K3D, we have learned a lot from them. They have done an amazing job. They have a very small image. They have a very low CPU footprint, but they don't have the features that some traditional Minikube users would want. So I think everything can be compared for based on who you are, what kind of use case you have. Good question. Am I gonna create a what? The K2 add-on. Oh, actually, I'm glad you asked. You can create an add-on for Minikube very easily and we have the step-by-step documentation how to create an add-on in Minikube. You just go to Minikube website search for add-on. Here. No, no, not this one. Come on, demo gods don't disappoint me. How to develop a Minikube add-on? Actually, it was the first link. I didn't look at the first one, this one. So this is detailed how you can create an add-on. A lot of people are developing add-ons and contribute to Minikube and we accept them. Yeah, as long as they are open source, they're not like anything proprietary that we cannot include. Yeah, why not? Oh, I posted it right before the thing. Yeah, it has to be there, hopefully. Yeah, yeah. Any other question? At the Minikube website, I hope everybody knows how to get there, just minikube.sigs.kset.doc. If you want to try the NVIDIA thing, just search for NVIDIA here. Using NVIDIA with GPUs at Minikube and it tells you based on what driver and what technology you want to use, it will tell you how to use that. If you want to use the Minikube GUI, I think it's a very cute project. I really like this Minikube GUI. So try out this Minikube GUI thing, it's kind of cool. Well, we have one minute left. Any last question? Thank you very much and have a great day.