 Hello there! I'm delighted to be giving this talk at KubeCon today. I will be sharing with you my experience on building Linux distributions and then I will try to distill some lessons learned and look into the future. But before jumping into the contents of the talk, let me introduce myself. My name is Marga Materola. I'm a long-time open source developer. I work at Kinfolk, a startup company based in Berlin that focuses on Linux and Kubernetes. This talk is the result of my experiences working with Linux over the past 20 years. It's been an interesting and exciting journey and I'm thrilled to be sharing it with you today. I'll first tell you about how my career with Linux developed working at small and big companies, as well as being a Debian developer for 15 years. Then I will try to summarize the lessons I've learned throughout this time. These are my experiences and my opinions, but I'm sharing them with you today. Hoping that you will find them interesting and perhaps learn something new. So, let's start at the beginning. I grew up in Buenos Aires, Argentina. I installed my first Linux distro back in the year 2000. Back then, it was common to boot our Linux boxes in just text mode and only start X when needed. For a couple of years, I worked as a Shapa developer at a consulting company. I managed to convince them to let me run Linux on my workstation, but I was building proprietary software and this made me unhappy. So eventually, I decided I didn't want to keep doing that anymore. In 2003, I quit this job and joined the IT department at my family's company. This was my first job doing Linux. At this company, we were deploying Linux installations to all users, replacing legacy proprietary software with open-source software. Our budget was pretty tight, so we had to make sure we'd spend as little as possible on new hardware. To do this, we decided to deploy thin clients. These were old, used machines that would be considered useless by most people. But we gave them a new life by running a very small Linux image on them that would just control mouse, keyboard, video, and network and run a local X server. This X server then connected to another machine, where all the X client applications were actually running. That way, we only needed one powerful machine per 10 or so thin clients. This allowed us to quickly expand our network to more computers, reaching more users. After a few years of this, we had deployed some 350 computers, and at the time, this felt like a lot to me. I was really proud of how I was adding more Linux users to the world. And I really cared about each and every one of those computers. We had started this team with two people, and with time, it grew up to five of us. So we had less than a hundred computers per team member. I knew all of these computers by name. I knew who used which host, which type of keyboard was connected to it, if the mouse had had issues in the past two months, and so on. I even knew the internal IP addresses for most of them. I didn't feel like we had our own distro. We were just using Debian. But looking back, we had basically all the trades of a Debian derivative, except for a cool name. We had our own internal repo, where we kept additional packages that we needed. This could be packages that were not yet in Debian, or fork packages with special patches applied to them, or packages that we had developed specifically for our needs. We also maintained the configuration of the machines in the fleet using Puppet, keeping our customizations in sync through version control. So yeah, we were pretty much maintaining our own derivative, even if we didn't own it at the time. I called that earlier that I'm a Debian developer, so you can imagine Debian has played an important role in my Linux journey. My first step with Debian were at the end of the year 2000, and as the years passed, I became more and more involved with the distro. By 2003, I considered myself a Debian bug reporter. I took great pride in creating good bug reports with clear reproduction instructions, logs, etc. And for a while, that was enough for me. But this changed in 2004, when I attended the Debian conference in Brazil. Up until this DevCon 4, I considered Debian developers to be some sort of elite geeks that were kind of superhuman, completely out of my league. I felt that maintaining packages was such a difficult and complicated task that I couldn't be able to do it. I was already sending patches to bugs, but I felt that that was the maximum of what I could do for Debian. But then, I met all these awesome people. They made me feel welcome, they valued my input, they explained how things were not so complicated. They became my friends. And so, when I came home, I was a different person. I wanted to become a Debian developer, and I no longer thought that maintaining packages was reserved for elite hackers. So a few months later, I started maintaining my first Debian package, and the following year, I officially became a Debian developer. Throughout the years, I've done a lot of different things for Debian, not just maintaining packages. I particularly enjoy fixing bugs, so for a while, I spent a significant amount of my free time fixing release-critical bugs on other people's packages. I also helped organize a few DevComs. In particular, I was part of the main Orga team for DevCon 8 in Argentina and DevCon 15 in Germany. I'm currently the chair of the Debian Technical Committee, the body that helps resolve conflicts among developers when they can't agree on their own. Sometimes, I think back to those days back at my first Debian conference, when I felt like the only human being among a bunch of superheroes, and it's pretty amazing how my journey has been. Anyway, back to my professional life. I worked at my family's company in Argentina until 2012, when I moved to Munich, to work for the team that maintains the internal Linux distribution used at Google. There's actually more than one internal Linux distribution at Google. My team was in charge of the one used by humans in their workstations, not by containers running in servers in data centers. Obviously, moving to Google was a huge change. Coming from my scrappy company dealing with a tight budget, I was quite shocked by the brand new beefy workstations that were deployed to all our Linux users. But of course, more mind blowing was the size difference. At Google, my team of 12 people was in charge of tens of thousands of computers. And while the team grew a bit over time, the amount of hosts on our fleet grew a lot faster. At any point in time, we had more than 5,000 computers per team member to care for. This meant that it was no longer possible to know the hosts by name, and more importantly, that everything needed to be automated. Not just the obvious, like managing configuration with Puppet and automatically upgrading packages to their latest version, but also other things, like automatically checking the health of the machines and reporting back when there were problems, or automatically creating backups for our users and giving them tools to restore data when they needed it. And not less important was our battery of automatic tests that helped us make sure our releases were healthy enough to be deployed to the fleet every week. Our team was small and a user base was huge. We couldn't possibly deal with all incoming user requests. So we provided a bunch of tools and documentation to help users deal with issues themselves without the need to contact us, including being able to reinstall their computers and install their computer without having to feel a single prompt. Some of this automation was already in place before I arrived. Some of it I helped build myself. Part of what my team did was keep on top of developing new automation tools as needed. At Google, there's a lot of different teams maintaining different parts of the stack. As I've said, my team was in charge of the Linux platform. But there were a lot of other teams providing software on that platform, and an important part of what we did was enabling them to do that. This meant that some changes need to be coordinated across many different people, across many different time zones. And I'll admit, this could sometimes be quite challenging. Finding agreement across stakeholders is not always easy. But what helped me was remembering that everyone was a capable engineer looking to find the best possible solution. Putting aside these differences, there were many things that were the same. When I asked what I did before working at Google, I would reply that mostly the same, just at a much, much smaller scale. In both cases, we were keeping a rep of packages on top of what the distro provided, managing the configuration with Puppet, keeping all our changes and configurations in version control, with the goal of enabling our users to do their jobs. And all of it, using a distribution that was based in Debian. Well, sort of. The team I joined in 2012 was called Goobuntu. It was a Linux distribution based on Goobuntu that followed long-term support releases. Every two years, we would migrate from the previous LTS to the new one. As time passed and the fleet grew, this meant more and more work for my team. Too many things changed between releases, which meant lots of issues needed to be fixed before we could get the new version out. We spent so much effort getting the fleet updated, only to have to do the whole thing again two years later. So, after going through this painful migration twice, I convinced my teammates and other interested parties that we didn't want to keep doing that anymore. Instead, we should switch to a rolling release model where packages got updated. We would no longer have a huge shump every two years, but rather small increases every week. And while there might have been some disagreement regarding which target to follow, I convinced the powers that be that we should track Debian testing. So, in 2017, we renamed our team and our product to G-Linux. We migrated the fleet from Goobuntu trustee to Debian testing, which is itself a new product. The migration took several months, and our target kept moving while we were migrating. And I'm not going to lie, there was a lot of work involved here, and not all of it was easy or fun. But after years of working on the necessary tooling to make this possible, we managed to keep the fleet track in Debian testing in a rolling release fashion. All of this was pretty awesome. I learned a lot, and I'm grateful for that. This was pretty awesome. I learned a lot, and I grew a lot in the almost eight years that I worked at Google, but eventually it was time for a change. So, earlier this year, I took on a position at GIMFolk. As I mentioned, GIMFolk is a startup based in Berlin. It's a small company dedicated to working with open-source software with a special focus on Linux and Kubernetes. In my case, I've been spending most of my time on the development of Flatcar container Linux. Flatcar is a container optimized OS based on the now-deprecated KoroS, which was the first container optimized OS. KoroS was itself based on Chrome OS, which is based on Shentoo. So, yeah, I'm not currently working on a Debian derivative, for now. Moving to GIMFolk was again a big change for me. The thing that impressed me the most was the speed at which one could get things done. At Google, whenever I started a new project, there was a long ramp up time of getting used to new technologies, putting all the complex pieces together, getting by in from all stakeholders, and so on. A medium-sized project could easily take a year to get launched. Working at a startup, there's a lot less red tape, less complexity, less stakeholders to consider. So, things just move faster. Projects usually take a number of weeks, maybe a couple of months, if they are large. And sure, timelines might slip a bit, but everything just goes faster. The other big change was that while I was still doing Linux, I moved from working on a distribution used by humans to one used by containers. There are a lot of differences there. A big one is how security weighs against usability. With any software out there, there's usually a struggle between security and usability. You need things as locked down as possible while still allowing you to get things done in a reasonable amount of time. And when security and usability clash, you need to make some hard choices. I found that the choices made for G Linux were different than the ones made for Flatcar. One example of this is the read-only user partition. In Flatcar, as in CoreOS and ChromeOS, the slash user partition is read-only. You can't install new software in it. There's no package management of any kind. The OS image stays exactly as it was shipped. You can still customize things in Etsy and sideload software via containers. But you can't install new programs in slash user, nor can you tamper with the programs that are already installed. For a human user, this might sound really limiting. But for an OS that's used to run containers on it, as long as it has the necessary tools to run these containers, this might be exactly what you want. All right, so here we are. I've taken you on a super quick journey through my career as a Linux engineer. I started as a young, inexperienced techie, and through years of fun and effort, I became an old-timer. Now that you know how I got here, I'd like to share with you some of the lessons I've learned through all of this. And then, perhaps, speak into what the future might bring. One thing that I've realized is that Linux distributions come and go. The very first destroy I installed was called Mandrake. It was intended to be a user-friendly distro, and although it was not for me, it did have a number of users. Years later, it got renamed to Mandriva, when the development of Mandrake merged with the one from Connectiva. But eventually, most users move away, to other distros. It stopped being relevant, and stopped releasing new versions. Back in my first DevConf in 2004, I was there for the talk by Mark Shatelworth, where he presented the idea of what would become Ubuntu. He explained how the distro would work, and how it could integrate with Debian. Back then, it sounded a bit like Black Mashaic. Many people were skeptical. Nobody could imagine how this would turn out, and how successful Ubuntu was going to be. So, some distributions have succeeded in staying around. Others have died, or become a lot less popular. Why? What lets a Linux distribution stay relevant, or makes another one become obsolete? There's of course not a clear answer, but we can look at some signs that popular distributions carry. Different users have different needs. It's impossible to make a distro that will be exactly what everybody wants, because people just want different things. The most successful distributions are those that are easy to customize by other users, sysadmins, or both. And also to create derivatives from. Derivative distributions are a good thing, because they can target a very specific use case, making it easy for their users to achieve their goals, and at the same time, benefiting the whole distro ecosystem by integrating their patches upstream. All right, so it's important for a distro that wants to stay relevant to be easy to customize, but what about the actual contents of the distro? We should keep in mind that having good defaults helps users do the right thing. This is particularly important with everything security related. In general, people will go with whatever is the default, except for the specific things they care about. And sadly, very few people care about security. So if your distro has good security decisions baked in, most users will then apply those defaults and have these good security practices in their devices. If they need to take action to have those security measures applied, they will likely be left vulnerable. Similarly, if there's something that users want to do, but you don't think it's a good idea, like say, install some cool plugins from random places on the internet, making it easy to do the right thing will have a lot more impact than making it hard to do what users want. Telling users that they are doing it wrong won't take you very far. Giving them a way to easily do it right will make a much larger difference. Another interesting lesson that I've learned is that bickering over trivial rivalries would only cause division and end up in a loss for everyone. If a program needs to be patched, it doesn't matter whether the patch was created by a maintainer from Ubuntu, Fedora, Arch, or Chrome OS. What matters is that everybody gets the patch code as soon as possible. The less duplicated code, the better. Depending on your own history, this may or may not sound obvious, but I do think it's important to say it. We all have different tastes, different goals, different preferences, and that's okay. Diversity of implementations is good because it allows more innovation, but that doesn't mean that we can't help each other, that we can't collaborate across these boundaries. We are all doing open source in the end. We can all work to try to make a better world together. All right, so I've talked about the popularity and relevance of distrust. I mentioned a few factors that might help make or break a distro and what we can do to help each other succeed. Unfortunately, there's some bad news coming for those of us building Linux distributions. The current tendency seems to indicate that Linux distributions are becoming less and less relevant. Over the past 10 years or so, installing libraries and modules outside the main OS became the norm, rather than the exception. Python has PyPy, Ruby has the same distribution, Node.js ships separate packages for everything, including to check if a number is even. And it's not just about programming languages. This tendency to ignore the underlying distro means that a growing number of programs come with their own package management system for extensions and plugins. This includes a wide array of programs like Firefox, Cinnamon, and even BIM, which has like four of them. Including their own way of installing add-ons has two main advantages. First, it avoids the bottleneck of packages getting ingested by the distribution. New plugins will just be available when they are published. And second, it abstracts installing these add-ons from the underlying distro. So it doesn't matter whether you're running Debian or Gentoo, you'll install your KDE plugins the same way. Of course, there's a reason why Linux distributions exist in the first place. So bypassing them also comes with disadvantages. Less or no quality control, less or no security updates, in particular for stable versions, and no integration with the rest of the system. In other words, by making it super easy for users to install the latest flashy thing, it also makes it super easy for them to install random, buggy, insecure, or even malicious software. But users want the flashy things, so they will mostly ignore these concerns. The next step on this trend towards making the distributions less and less important is running some form of contained applications on top of a thin layer of OS. The contained applications could be docker containers, snaps, flat packs, or what have you. I remember back in 2015, I attended a talk by a docker developer who showed how she ran each of her graphical applications inside a separate container. Back then, this sounded rather over the top. Today, this sounds normal. I do this myself for applications that I don't trust. Running everything inside containers can also lead to some bad consequences. The thing that worries me the most is security because many people forget to ensure that all the software inside those boxes is up to date. And most systems are not configured to take care of that automatically. So in this world, what's even the meaning of a Linux distributions? Will distributions matter in a couple of years? In the server space, it's looking like distributions will matter less and less. If you're running containers on a VM or a Kubernetes cluster on a bunch of nodes, you care very little about which destroy is running on these machines. What you do care about is that this destroy is as secure as possible and that it allows you to run your workloads on top. The less software you have running in the base OS, the less you'll have to worry about possible security issues. And if you don't need to think about keeping the small base installation up to date, that's even better. So distributions with a minimum footprint and automatic updates are clearly going to be favored. Of course, we will still need to get the software installed on those containers, so it's not like distributions will stop existing, at least not in the immediate future. But as more workloads become containers or containerized applications, the underlying OS will matter less and less. In summary, for server-side applications, distributions will either be a minimal layer to run containers or top or a repository of packages to create the content of those containers. Now, what about desktop and laptops? For end-user devices, a general-purpose distribution still makes sense today. We don't yet have the tooling to properly manage running each application in some container, to keep all those programs up to date, or to avoid wasting tons of bandwidth and disk space on re-downloading the same libraries and frameworks for each application we run. This is the tooling that will be the most relevant coming up, something that helps us make sense of all these different origins of software, things like keeping our PIPI packages up to date, managing our own Node.js libraries, understanding what's installed in the system, and what's shipped via a container, ensuring that our containers have their security issues patched, and so on. Some of these utilities are being built today. Others are yet to come. When these tools are mature enough, it's likely that Linux distros will stop being relevant in the desktop and laptop space. They will become a thin layer on top of which we will run containers. If you're part of a community that's building a distribution, you might not like this perspective. I understand. As a devian developer, I'm also not a fan, but I don't believe that closing our eyes to reality is the way to go. Instead, we need to accept that the world of Linux distributions has changed and built the tools that we need to make sense of this brave new world. Now, what about Kubernetes? As you almost certainly know, Google created Kubernetes based on a similar piece of internal software called Borg. A lot of the design decisions taken for Kubernetes were based on the lessons learned by the many, many years that Borg had been in use at Google. The good things were replicated, the bad things were corrected. This is of course good because it means that the design was quite polished. But something to note is that when creating Kubernetes based on Borg, only the container orchestration part was included into the new product. At Google, Borg is fully integrated with the internal load balancer, the internal file storage, the internal networking layer, and so on. In Kubernetes instead, all of these are integration points where we need to provide our own load balancer, file storage, bot to bot connectivity, etc. The upside of this is that we can experiment with different implementations. We can try out different behaviors and find out which one best fits our needs. The downside is that we end up with the overwhelming image of the cloud native landscape that once in a while goes viral on Twitter. Some cloud providers have some or even all of these integrations solved for us, which is nice. We can just use the systems that are part of the provider's offerings and we don't need to care about configuring them or making them work with our Kubernetes installation. Awesome. Except, of course, this means that we can't always easily move our workloads to a different cloud provider if it turns out that we get their value somewhere else. And this is even harder if we want to move our deployment to bare metal. This means that if we want to be truly independent of the provider, we need to run our own cluster with our own integrations and this is hard. To make it easier for users to run platform-independent clusters, a few Kubernetes distributions have appeared in the past years. When we use one of these distros, choices have already been made for us so that we can concentrate on running our workloads. It's good to have variety, not everyone will want the same things, so having different Kubernetes distributions for different targets makes sense. What's not so good is to have so much duplication of work, but in the end, that's just part of the reality in the world of open source. It's likely that in the long run, a few of these distros will emerge as the most popular and there will be derivative distributions from those, but that's yet to come. As I said, Linux distributions are becoming less and less relevant. The OS used in the Kubernetes nodes mostly doesn't matter because we just run Kubernetes on top. A consequence of this is that some of the problems that were already solved for Linux distributions, like handling installation and configuration of software, need to be solved again at the Kubernetes level. And we don't really have good and established solutions at this level yet. Say there's a security problem in one of the libraries that our software uses. When using software from a Linux distro, we would just rely on the maintainers updating the necessary packages. We would upgrade our machines to the latest version as part of our routine automated upgrade. What do we need to do for our containerized applications running on Kubernetes? We will need to manually check which applications are affected and then upgrade those that need upgrading. Maybe some of them take longer than others to apply the necessary patches and so we either need to keep hauling until they are available or patch our apps ourselves if it's something really critical. The package management provided by Helm can help us with that but it still has a way to go to really solve this problem. Today when you're managing a large fleet it can get really hard to keep track of what's installed in your clusters and whether some stuff needs urgent upgrades. Right now we are at the dawn of the era of Kubernetes distributions. They are not well established but I do believe that's to come. Partly it's a matter of time until using a Kubernetes distro instead of doing a deployment from scratch gets more adoption and partly we still need tools that don't exist or require more work. What can we do as a community to help move this forward? I think it helps to look at the lessons learned from Linux distributions and then try to apply the things that worked and avoid those that didn't. Have good defaults but make it easy for users and derivative distributions to customize. Have relevant security decisions and practices baked in. Collaborate with others building similar tools and even competing distributions. The future ahead of us is exciting. We can all make a difference if we work together and with that we come to the end. This was my presentation on building Linux distributions for fun and profit. I hope you enjoyed it and that you learned something new. Thanks for listening.