 Ladies and gentlemen, I'm glad to introduce our next speaker, Daniel Wieck, talking about how complexity on clouds and the software stack impacts the freedoms of GNU. Welcome. Thank you very much. See if you can hear me. Can everyone hear me? Awesome. It's a great honor to be here at FOSTA. I've been at FOSTA a lot. I've never spoken, and this is slightly intimidating, so I hope you bear with me and don't throw anything at me. My name is Daniel Wieck. I've been in GNU Linux for quite some time, started back in the 90s in Germany, moved to Rated eventually, and I spend a lot of time on figuring out things like software packaging distribution. I was a product manager for Rated Enterprise Linux for a good part of my career. I was involved in driving the container transformation at Rated towards Docker and later Podman and OpenShift. Right now, I'm in the city office working on open source AI, which I'm not talking about, but maybe next time after the last talk, I'm really intrigued. That would be an interesting topic as well. Disclaimer, I work for Rated. I will use the Rated stack as an example, nothing I say specific to Rated. Nothing I say can be misconstrued in the official position of any of my former, current or future employers such as me. Rated is now part of IBM, we are subsidiary, which is kind of ironic because IBM originally got me into free software. It's an anecdote, I was using OS2 at the time to do a programming project for school, and TurboC didn't work anymore on OS2 when I had been using on DOS before, and my father was riding the train to work with someone from IBM, so he asked, like, can my son get some software development tools for OS2? I said, yeah, sure, tell me what he wants. So I gave them a list and they came back, yeah, that's about $20,000 worth of software licenses. I didn't get them. And instead I started using GCC in the port by, this is the OS2 port, DJ did that, right? So who also works at Rated, it's just kind of funny, and IBM now. So that's how I got to free software, and since I've been in the space, you'll find that I'm a little bit opinionated at times, and my clicker just gave up, awesome. It's just slow. Okay, so the title, like, I count myself in that category now, right? Graviate Fight Bell Rocks, they hate SystemD and their fork distributions. This was started by a well-known debate around SystemD, and my claim is that this is going to be worse than SystemD. Well, I'll just use this one. So what's the traditional role of the GNU Linux operating system, right? And you can have a lot of these debates. One view is you look at it from an infrastructure up point, right? It's a kernel that lights up hardware, makes things good, you enable that, right? And that's a strong view that many GNU Linux developers hold, right? We often have a very kernel-centric and low-level view on things. But the other view is that you look at it from the application down, right? What the user space does, you know, G-Lib C and everything above that, gives you an application interface, right? The kernel with the syscalls, obviously, as well, gives you an application interface that makes an application run on hardware without having to change because of small changes in the hardware. Sure, if you have a different hardware architecture, then there's change. But in general, if there are minor changes, you don't have to change your application. And that's how GNU Linux has become the predominant binary abstraction layer for clouds, right? Today, you compile a binary and you run it in any of the clouds in the VM or on bare metal, and you don't have to change the binary, right? So it's an application interface. And it's really important to have both views, right? The role is really to bring the two together, let applications run on hardware. You know, a general theme we are seeing is that software stacks are growing in complexity, right? I found this on the internet, so it must be true. But, well, you know, the general trend definitely is true. We can all observe that, that crazy line is NPM. There might all be forks of the same project, for all I know. But still, you know, people develop for us nowadays use software, you know, from all kinds of those, and stacks get more and more complex. And that's changing how we look at things, right? Because you have to keep up with that if you want to keep your developers happy. You know, in, historically, what we've done with GNU Linux is that we commoditize the access to software and the choice of hardware, right? If you go back and, you know, I'm skipping mini-computers, which, like us, just small mainframes, if you ask me. So mainframes, right? This was a model where you had a very virtually integrated proprietary approach, right? You had hardware that often you leased, right? You didn't necessarily even own the hardware. You had a controlled operating system, a very controlled ISV ecosystem, right? Often you, like the vendor of the mainframe could control what software you could deploy, right? So basically closed, black box applications on a closed platform, often leased hardware. We had UNIX and Open Systems, things like that in enterprise software. We had very controlled, in most cases, very controlled hardware and operating system environment, right? With controls of which tool chains you get, often with certification problems or even controlled environments on who can get the tool chain to put software on top, right? That depended by vendor. But at the end, it was still a very, very confined model where your choice of what you could do was very limited. At a minimum, you couldn't choose hardware freely, right? In most cases, you had constraints around the software ecosystem. And as the GNU came in, right, when I was dealing with that kind of stuff in university, many people would, by standard, install the GNU tools on their proprietary UNIX to liberate it to some degree, at least, right? Now with what Linux, and I'm using reted enterprise example because I work there, and I was a product manager, right? So intimate knowledge, but that's true for Linux in general. What Linux did, Linux took away those control points, right? One funny way to say it is that we allowed people who could only afford PC servers to run their IT like they could afford Sun servers. But in reality, what it did, it opened the ecosystem allowing you to choose from any kind of ISV, any ISV could have access to all the toolchain. It's a fully modular system, right? You can pick and choose at any level and you could run it on any hardware you like, right? And if the hardware you want isn't supported, you can contribute to any layer of the stack and make your hardware work, right? It's a fundamental change and completely democratizes access to software. That is what free software did. Now how did we manage this stack back then, right? I talked about software complexity. Back then, it was already fairly complex, right? I got to GNU Linux through Slackware, which was a distribution that basically had pre-compiled binaries packaged in tar with an installer that actually enabled me to bootstrap a system, right? And then you would go in and you would start compiling things in user local, right? Because the distribution didn't have everything you wanted or not the version or whatever, and then you use something like Sto to create symbolic links into your user bin tree or in user local bin and things like that. What we did back then, and I remember that feature request actually for RHEL, we actually created binary repositories that we mounted in a way where we are NFS from one central machine where we built it to run on the individual machines, right? And that was all trying to deal with scale, right? If you had big multi-user sun servers, it was fine to treat them as a pet, right? You had multiple admins per server. Then we started with a Linux-based PC server. Suddenly you had multiple servers per admin and you needed to somehow deal with the complexity. The problem is that the state of the machine, the state of the software stack on the machine defines the behavior of the software you compile on it, right? So reproducibility was a real problem and it just didn't scale. It was really like a source-level-late binding we did. And if anyone ever dealt with things like automaker and so on and wondered why, right? That's why, because you had to compile software based on different environments you found. The next innovation in software packaging was pre-built binary packaging, right? So I said Slackware had tar files but they were not really good, right? They didn't know much about the lifecycle of an application. You just untar it. It was pre-built but it didn't really manage dependencies well. It was a pain, right? And it wasn't consistent. But things like RPM or Debian packaging came in and they changed how we consume open-source software, right? Instead of having something that just bootstraps and we compile ourselves, we move to model where things usually are pre-compiled. And remember, there was a time where everyone would install a Linux distribution and it would go in and compile your own kernel, right? It was a time when the kernel still could be compiled in a day on a regular machine. Over time, these kind of habits moved away, right? Nowadays, modules helped, of course, but so there was technology progress and change in what people focus on, how they consume this. And ultimately, we had this massive standardization on a few number of distributions with their pre-built binaries that people use. Now that comes at the cost, right? Both RPM and Debian packaging implicitly move you to one version per namespace or monolithic user space. There's one version of everything installed at one time by default. There are workarounds with relocated packages and so on, but they're painful, right? So back in the day with user local, you could have your own version of some tool, right? If you like Emacs, right? You could have your own version of Emacs, right? And you could have multiple of them and you could consume them on the same machine. We now move to a model where by default you have one version of everything unless you go through Hoops or the distributor goes through Hoops to provide relocation. And that really changed how we use these machines, right? It's a big fundamental change. And for example, like a long debate we had at RETA, it's like we had the Java department, right? And you know Java is like this abstraction layer. In a way, it's a virtual machine, right? On top of something. And we always said, you have to package this as RPMs. I said, no, we don't. I had to leave RETA and come back to understand why, because I was so in the Linux monolithic user space, standardized everything model there. I didn't understand that those people actually wanted every software in their home directory, right? And each user had their own version of something because they optimized from the application down. It was a perspective, right? I looked at it from the bottom up. They looked at it from the top down. And they went, wow, my application works with this version of JBoss, right? And I don't want to change that version of JBoss just because someone in product management at RETA decides that you need to run a new version, right? So there was this inherent conflict that we had where the one version per machine just didn't fly anymore, right? So the problem of scale, we moved from one big service multiple admins to more and more service per admin, replication, more and more service being used by users, right? Sprawl of IT. And the problem we have, one problem is dependency hell, right? If you have ever tried to install something from Pi Pi on a Linux machine, you know what I mean. You probably spend as much time finding out what the native dependencies are that you need in which version as you spend working on the Python code. I'm exaggerating a little bit. But it's a big problem, right? And we have conflicts and versions. It's actually, it's hard work to keep everything acceptable at the same time. That's the assertion that the distribution gives you, but they can't keep up with the complexity of these stacks. You cannot package everything out of these millions of software packages that I showed in the earlier chart. And so the other problem is just scale, just number of machines you have to deal with. So we had tools, we developed tools that deal with that, right? You had automated installation, you had configuration management like CF engine and later implementations like Puppet and things like that. And Ansible, it's a different concept. There's a whole fight of concepts around like whether it's declarative or directive and whatever. But that solved all the scale problem in how many machines you can manage. You can manage huge numbers of machines, but it doesn't solve that the software stack complexity is holding you back, right? So we got a little bit of freedom in that space with virtual machines. And the problem, virtual machines basically emulating hardware liberate you from having to use a machine for exactly one thing, right? Which was a Windows problem to begin with, right? You couldn't install multiple things on one Windows server, so I had to use virtual machines to use hardware for multiple things. In New Linux, we didn't have that problem per se, right? It was totally safe to run multiple services. We have a huge amount of how to do that. The problem we have is dependency complexity, right? If you go far enough in your software stack and create enough dependency on some of your applications, changes in the underlying software stack become problematic, right? Because they have cross effects. And I remember that even though a lot of work went into maintaining ABI stability in an enterprise software distribution, there's always something you break, right? And if it's only because someone uses an interface, they're not supposed to be used. You didn't know they were using it, right? And you didn't pay attention, ABI breakage, something breaks. And our operational model up to this point is actually like fairly crazy, right? We have this idea that we can deploy a machine in production, and then in production, right? So we test something somewhere, right? Then we deploy it in production, which we do by basically forgetting everything we did and recreating the machine on a component level from all the RPM or Debian packages and what else you're putting into that from scratch, right? So you don't actually test the same thing you're putting into production because you're recreating it from a dependency tree that in itself is volatile, right? The problem is that depending on how long your testing takes, you know, the cycle between... If you have a fairly complex set between your testing and you're moving into production, you probably have seen some security update. You go back and retest that, right? Like, it gets fairly complex. And then we run this in production and we think that at any given time we can update a random component and it will just keep running, right? Which, like, it worked for a long time, but with the complexity, it gets harder and harder. And like, if any of you... Like, if you want to try that, I'll just run Firefox on your laptop, right? And then do an up-get update or a DNF upgrade underneath while it's running and see what happens. It can be actually funny because, like, some underlying library gets changed and when it loads the dynamic library in the new version, you have unpredictable behavior, right? And there is no standard way of dealing with that in the existing packaging system, right? They don't have a standard way of telling an application. Oh, yeah, we just pulled the rug out under you, put a new one in, it has a different color and higher, like, you know, there's a step now in there. We also changed the flooring. Yeah, please restart. No, we can't do that, right? So there's a problem. VM started to help with that because they gave you smaller units, right? You weren't bound to a piece of hardware anymore. The problem with that is they are pretty, like, they're machines, right? They're virtualized hardware, so you have to manage them as individual things. In theory, you could create a model there where you create images and then you rebuild images and you deploy images, you update images. At the end, no one did that, right? There's a lot of deployment as image, but then a virtual machine always becomes a pet, right? So the idea of, like, oh, I have reproducibility across my instances didn't really take off. So, you know, we call that VM sprawl. There's a lot of inconsistency and management is expensive because, you know, they are so complex and self-confined you can't easily look into them from the host. You know, there's a lot of... You know, there's a lot of problems in the life cycle, right? You know, I said, in theory, you could create an image and hand that over to another one and then you have organizational structure and delegation, right? And then someone on IT defines a golden image and then you go in and give it to a line of business application owner and they, in theory, just install their user space application. Well, in reality, what most people do is they give them the root password to the VM, which means they have no... like, IT, when they get it back, have no idea what's actually in the VM, right? Things like that. So you start virus scanning them and it's a total mess because in reality, right, the concept, it's too heavyweight and it doesn't go far enough. Now, after enterprise virtualization, which becomes a predominant operational model for IT in data centers, like I would say by the end of the 2010... No, by the end of the 2000s, right? And by 2015, right? Like, I think it's 97% of data centers are virtualized or something like that. I just made up that number. But, you know, it's a huge number. That's the point. So, you know, there was another... The next trend was cloud, right? Like, and infrastructure as a service cloud. So what data centers with virtualization are much more flexible than the data center where you have to order hardware and connect it to do things, right? You can share hardware. But cloud, you know, and Amazon came up with this idea, AWS, that, you know, basically took the concept of server hosting which was used for web servers, right? We had that for a long time, but moved that into core IT, right? And they started a little bit out with... It's called Amazon Web Services, right? You know, really not the primary use case anymore. But they created this idea that you can basically, instead of doing a CapEx model, a capital expense model of buying hardware, you can just rent hardware from them. They manage the infrastructure and you get virtual machine storage, so compute storage and networking, and they take care of that. That basically took the virtualization model and moved it to scale. They also improved a lot on the image-based workflow, even though there's still a lot of pets going on, but they automated a lot around that. It doesn't fundamentally change how you do things compared to virtual machines in the traditional sense, but it gives you elasticity and it gives you a cost efficiency where you can scale your cost with your actual use, right? So you get rid of hardware. That's ultimately what it did. Along that path, we had a bigger change of paradigm, which it's hard to actually separate the impact of free software, clouds, and other cultural changes and software eating the world, I would say. I kind of hate that term, but on the other hand, it's also very, everything we do is software nowadays, right? A bunch of things. We're putting software into our bodies, right? You know, there was an example that Steph had about somebody who has a... what is it called? A pacemaker that is software driven, right? Obviously, right? So you put software everywhere. So software, everything, every product depends on software. Somewhere every business depends on software. And it changed kind of who has power, right? Traditionally, the IT department could standardize things. Now the IT department cannot standardize things because the line of business and their business objectives will override any IT decision. And, you know, at the same time, we had more aggregation of services. We had open source free software becoming the default, winning every software stack that's eating the world is in some way based on free software nowadays. I don't think you can find any... In the general modern IT, you probably can't find any pure proprietor software stack. You would be hard-pressed to do that. The cloud itself is based on free software, right? It's purely free software technology. It's a basic, you could call it, you know, a new Linux distribution as a service if you want to. And we have a change in behaviors driven by that in things like DevOps, agility, right? The way we do software. The key point here is that the liberation with free software liberated the developers in the line of business to do whatever they want, picking from this huge complex set of software. And then that had to be managed. So the modern cloud changed, right, from the infrastructure as a service cloud. So, you know, back then we defined cloud as like, oh, it's elasticity and VMs, right? Today, the cloud is actually an operational paradigm that's based on elasticity. It means you can scale up and down and your costs correlate to that. Develop a velocity through service abstraction, integration availability, right? Which means that a developer never has to stop. They can just keep going and pick services, right? And encapsulate operational excellence. What that means is, right, in the traditional software model, whether that's proprietary or free. Now, when I do a Linux distribution, I give it to you, I guarantee that I build the stuff from the source code I know that I tested these things together and, you know, to some degree that I will provide updates to it when there's a security issue. However, it's your problem to figure out how to deploy it on, you know, figure out the architecture of the deployment and then maintain it and manage the lifecycle. Yeah. If you take a modern stack that has multiple services, right, that's actually pretty hard. So you need an expert, if you have a modern application, you need an expert in web scale, you need a performance expert, you need an expert in databases, you need a storage expert, and you need a network expert, right? Just, like, to get going, right? And then you want them, of course, 24 hours, like, it goes crazy. Now, if you go to the modern cloud, you press a button and you get a database. And as long as you are within the 80% of, like, what everyone does with the database, you don't need to do anything. Your developer needs to know how to do a data model and to talk to the API, but you're just talking to an API and someone else is maintaining that operationally for you. So what the modern cloud does is it operationalizes software, right? And it changed how people see software. While Free Software democratized access to software, now they are focusing, they're taking Free Software and they're operationalizing it as a service for other people so that the line of business can focus on their core business. They can always go deep on whatever component is differentiating for them, but ultimately, you don't have to be an expert in anything, expert in anything other than things you really care about, right? And that comes with a certain amount of problematic behavior, right? They use Free Software, they're not contributing back to the same degree. There's this concept of strip mining, right? You go into the cloud as a startup, free or non, right? It doesn't matter. You have an idea when it's successful, they will create a competing service with their velocity and just take over your business. It's a well-known problem right now that actually has led to a deeper problem for Free Software where certain companies that wear Free Software companies changed into an open core model, right, to try to defend with that. It's futile, from my point of view, the open core approach, it's not defending, it's not futile, but the open core approach is futile, but that's a different discussion. So there is this problem where this operational paradigm, which is really useful for end users, right? It liberates the line of business, liberates anyone from having to carry the burden of complexity outside the spaces they really care about, but it's dominated by proprietary, vertically integrated clouds that even have problematic behavior towards open source and Free Software, right? On top of that, you know, I said their core business is not access to software that's already solved, right? Their core business is operationalizing software so that an average person can use the software, which is too complex at scale, they're just not enough experts available. So they're aggregating a lot of operational excellence, right? I'm using a little bit in extreme picture here, but that's actually true. We had a point where in some areas there are not many companies left who can actually securely operate software, right? I've run into that problem that, you know, well, honestly, I gave up running my own mail server because I just don't, like, the time available against the risk factors, I just can't keep up with it, right? And this is pretty close to, like, what I do for a living, right? So just conservative, like, the average company or the average individual, even worse, cannot even operate software securely on their own anymore to a large degree because of the complexity of these software stacks. You can do a simpler stack, if you're competitive to direct competitors, or, you know, you can try to outsource it to the cloud. So it really, from access to software, the problem moves to operational capability, right? And that comes at a high cost. I mentioned, you know, it's vertical lock-in, right? It has data gravity, right? You put data and it's really hard to get out. It's really aggregation. And, you know, if you look at it, like, it has moved us forward, but we are back into a situation similar to where we started with mainframe and vertically integrated Unix systems, right? In practice, if you go beyond the legal situation for practical consumption. So what's the alternative? What can we do, right? I think, like, what we are doing today with this packaging and Linux distributions is not sufficient anymore, right? We have to move beyond that, because we have to somehow get to a better way to manage the complexity, get out of dependency hell, and manage the operational aspects, right? We have to basically get to the service abstraction, get the application portability across these things, and get to the operational excellence. And just as a proof, right, this is just, I threw it in after the AI talk. This is one of these stacks that my team is running inside RATAD, right? So just a random collection of free software components that, you know, if you want to do, if you want to do real open-source, or real AI work, data science work, you know, from ingesting your data to cleaning your data, training your models, operationalizing that, you need something like this at the end. And that's really a pretty steep hurdle to just do on your own, right? Somehow we have to make that easier, and not only from a packaging point of view, right? And the traditional distribution is challenged because of the complexity, right? We cannot package everything. We cannot make everything work together in a monolithic user space. Most developers today go to native packaging, right? There are some attempts, like, for example, for Python, right? We have PyPy with native packaging in PyPy. So you do something, you do the lower levels of your user space in RPM or Debian, right? And then you create a virtual environment of some kind with Word and or maybe PIP-N. And then you basically do everything else in there. And you have conflicts and dependencies between the two because, you know, there are native codependencies in your Python packages and they're not always wheels pre-built for everything you need. And so you deal with that. There are other kinds of, like, you know, condo, for example, is trying to create a distribution, a user space distribution on top of that that exists across different underlying user spaces, which creates its own set of problems because then they pick their own GCC and their own lip standard C++ and become ABI in compatibility with the underlying operating systems, things like that, right? Complexity is just painful in that and it's becoming too volatile, updating all of that in runtime at the granular level is too complex. So what do we do? Well, containers luckily came around, right? Which gives you the separation of the application user space from the host, right? So containers are probably most of you know that. They're not magic, they're just a bunch of kernel features, right? Namespace, C-groups, and in some cases, SeLinux, in the latter cases, SeLinux for security isolation. And that gives you basically the ability to run an independent user space, somewhat independent user space for your application and you don't have a monolithic user space anymore. So you switch, it's changed route on steroids, technically, but it's really useful. And if you want to try that out, I don't know if that's in Debbie in your toolbox, there should be, but that would be awesome. But the toolbox container in Fedora is a great example how that actually is practical, even for interactive work, right? You create that and it creates a separate environment in which you have, you can install tools in an unprivileged mode without touching your host, right? Really useful for development work. And the difference to VMs is that they're lightweight, right? They don't include the kernel, they don't necessarily include their own networking, right? They are just what you need. If you did write in containers, just what you need for your application. And they have a layering model that can be reused. Layers get reused. So you have some level of space operation, right? The VM always uses full memory, unless, well, there's some crazy things you can do for page sharing, but containers by default reuse memory if they are built from the same underlying layer. So it's a highly efficient way, much more lightweight, and the OCI containers that came out of Docker, the Docker project, OCI is a standard open container initiative. They also include, and that's critical, they include the life as a pull-push model, so the publishing and the transport model for these things, which ultimately is what helped move past the problem that we had with VMs where we never moved beyond the pet model, right? With containers, what you can run them as a pet, right? And if you do this toolbox thing, I mentioned, you will probably run a pet, which means you have a container that you customize for your specific work. But in most cases, what you do is you move, you move, you separate building and testing and operation in a life cycle model, ties it into your CI model, and the thing you are building, testing and operationalizing is exactly the same binary, right? And like one way to look at containers are they are fancy emulation of static binaries on top of a dynamic stack, right? That's really what they do. So you move from the late binding model where dependencies get resolved when you deploy in production and get re-resolved when you update in production to model where your dependencies get hard bound before you test. And you create automation around that to make that move seamless and fast. At the end, it gives you better quality. It separates different applications from most of the cross-effects of dependency things, right? So it's not your web server won't break because your database got updated, right? And some dependency got updated because in separate namespaces. It also moves like when you create... So there's a lot of containers, most of the containers you're going to deal with, you're going to create somehow yourself when you use off IT. However, a lot of the components and prepackaged services, you will get from a vendor, right? And it creates a separation between kind of what I call download to build, like things that you build into your own applications in your own containers, either on a package or in the source code level and things that are prepackaged services that become easy to consume. You just deploy your database and it's basically everything from the kernel interface up to the database. And that means that a lot of the burden of operationalizing and validating that it works is moved from the end user to the vendor, right? We've seen that at Reddit, right? It was actually a pretty hard move, right? Our QE concept used to be, we have all these RPMs and then we change one and then we retest that RPM, right? With containers, we pre-build containers for services, we ship the underlying infrastructure and now we move that, we move the full stack validation into our QE, which improves quality because now we are always retesting whenever anything is changed. So for our customers, things that at the end we left them to figure out, now moved into our internal QE by just changing the model inherently. And that will make your life easier if you consume pre-build containers because it moves a very complex interface, all these dependencies between binaries into a fairly simple container to kernel interface, which is fairly stable but can have issues, of course, so you still need to test that one. Beyond containers, of course, most applications are not just one container. The container is just one. You can do multi-service containers, right? You can install system D in a container or another in-it system and then run multiple services inside your container. You want to limit that. It depends what you're doing, but in most cases you would have one, basically one service per container, you know, to limit interactivity. So in a container, anything that's in the linker dependency trees in one container, if it's just an API dependency you put in a separate container and you orchestrate them together, right? So in reality, most are multi-container applications, most applications, and that's where the next interesting challenge comes in, right? How do you orchestrate them together? Now, the answer is Kubernetes, from my point of view, right? And that... So Kubernetes, look at Kubernetes as... It's two things. It's a cluster manager, right? It implements the concept of the cluster as a computer and it's an application definition language and orchestrator, right? So what it does, among other things, but it clusters your machine and manages scheduling of services across machines, but it also defines the dependencies between the individual containers and how to put them together in application and make sure that, for example, all services are running, that they get brought up during... So it's kind of, you know, a simplified version is kind of a... a clustered system, D, right? The interesting piece here is that if you put this back into the cloud concept, right, it allows you to have portability for the full application, right? You build your applications containers, which gives them, within certain constraints, portability independent of the underlying kernel version and the host, isolates them from the host, depends on what's on the host. You have an application definition in Kubernetes and it itself is independent of the underlying fabric, right? You define an abstraction. So the same application definition with exactly the same binary containers can run independent of where you're running them, right? You can take them to a public cloud, you can take them to a private cloud, you can run them on bare metal, you don't have to change your application. So it's basically taking what Linux does takes that to the more complex stacks, liberates you from dependency hell and gives you the abstraction to describe a multi-service application. That's really what Kubernetes does here. And, you know, you get... you get, of course... We call... Technically it's a meta system, D. Some people call it a meta kernel because it's easier for people to consume. It's, of course, not true, right? I don't know. You get into some of the same discussions you have with Linux distributions. I repeat, oh, I want the same... exactly the same behavior from every implementation of Kubernetes, right? Upstream distribution versus, oh, we have diversity in the implementation and in competition and they're not necessarily compatible, which is what we have between Linux distribution, right? You cannot take a binary from Fedora and run it on Debian and expect it to always work, right? Sometimes it works, sometimes it won't. Sometimes things are named differently. I think that's a good thing, but, you know, that's a debate that's raging right now and you can follow me on Twitter and I'm in a debate about that right now. So the next problem, and I heartened that earlier, the whole operational model, right? Kubernetes itself doesn't solve that, right? It solves... Containers solve stack complexity. Kubernetes solves orchestrating an application and having the broader concept of the multi-service application be something you can describe and port. But now we need something that manages the life cycle, right? It's a simple problem, right? Your database needs to be installed, right? And probably the first time you want to internet it. We had ways to do that. We can do that in container with scripting, but that's, like, very static. You want a more dynamic way. Then you need it to be updated. That gets a bit harder. You want it to be backed up, right? One problem that always drove me crazy is that... We never really solved that problem in a standard way, like backup, right? You know, if you know Windows, right, you're running Microsoft SQL on Windows on VMware, right? You press a button in VMware. It goes to Windows called a service called VSS, a volume shadow service. It says, you're going to be snapshotted. Then that goes to Microsoft SQL and says, oh, you're going to be snapshotted. Flash your buffers. Go say, stop. They take a snapshot. They say, you know, I'm ready. They take a snapshot and then restart. And it's just one button. As soon as you introduce Linux in any of that, that goes away and you're installing agents as a root that have custom scripts for the application. So we don't have a standard facility for managing things like that. Operators promise to solve that. So an operator is an active component that represents the runtime logic for your application. And it implements five levels. It implements install, deployment, upgrades, life cycle management including backups. Then operational insights and then ultimately level five would be an autopilot. And the promise here is, so the idea is that we take free software beyond access to the software, beyond packaging and move into operational excellence. And we encapsulate the operational excellence with the application. Which is the only way you can really make that scale. The idea is that you have an external management system that knows how to run every application. It didn't scale. It didn't work. We tried that too. Everyone tried that. These universes of config scripts, they break too much. The only way you bundle the operational knowledge with the application, maintained by the people who really understand the application, and then deploy it through a consistent model into a controlled runtime environment. And it's the last thing that the combination of operators and Kubernetes give you. They give you a controlled runtime environment with a standard set of levers you can hit. And a way to encapsulate the knowledge in that. So that you get to a similar model like what I described earlier. If you go to a public cloud, you press a button, you get a database. As long as you're in the 80% use case, you don't have to know anything about the database. We can get to the same thing. So if you there's a vendor called CrunchyDB who is based on Postgres and I'm not going to go into details how much they open source open quite, I actually don't know. But it's an example just of someone who created an operator and they're progressing in creating the logic to manage standard database behavior for the 80% use case on a cluster so that you can consume the software in any way you want wherever you want on any footprint and still get that level of encapsulated operational excellence that otherwise you can only get from the public cloud. And that's how we enable everyone to really run their software themselves beyond the pure access and security. There is a project operator where you can find a lot of operators they work on general Kubernetes and basically that changes the platform into a new concept it's the app centric platform where you have this additional cluster model and you basically split the new Linux model into a host layer that brings up the hardware an application layer that's inside the container and it's your binary runtime like it always was and then an orchestration around it that brings all of this together and makes it scale out and operationalizes it. And that's what Kubernetes does for you. So my conclusions a little bit text heavy and you know I'm an artist as you can see in my slides. So Linux continues to be relevant and the historic role of democratizing access Linux is a leading example of free software that democratizes access to technology and enables you to do things that otherwise you can do. I couldn't afford the IBM compiler stack on OS too and when I got my hands on Slack when in 94 that went away and I solved my problem. The cloud changes what people see and how people consume software and you know cloud is in a way it's a symptom of the complexity and all these trends but we have to keep up with that the downside of cloud is the concentration, the strip mining the concentration of knowledge it's a simple problem of available capability in the market and we think that containers and Kubernetes are the alternative for that that enable us with things like operators to provide to democratize the cloud concept it also enables like Linux did right it continues a tractor of enabling an open ecosystem on your choice of hardware and we need to take free and open source software access beyond packaging beyond making the code available to operational excellence two thoughts I'd like to recommend one the first one is just because it's my team doing it it goes into software complexity Frida Prakrani is going to talk about that later today and then Scott McCarty is going to go deep on containers and he actually knows something about it I'm just an imposter so I really recommend that talk tomorrow morning other than that I'll close with this outlining picture and before you despair that because this is all fiction unicorns are real thank you thank you for the great talk we have a few minutes left for questions so are you running your mail server again packaged as a Kubernetes operator so I'm not doing so will I run my own mail server again so my goal is yes I will run my own mail server again now I think it's unrealistic for me to run my own mail server on my own so I think what we need is a community model for operational excellence there and maybe I can talk about that next year at FOSSTEM I have an idea I actually shared it with you Steph once three years ago I have the domain so I need to do the work hi can you tell me a bit about the toolbox which doesn't seem packaged yet but what does it do with the network name space because that's where I've stubbed my toes with Kubernetes a lot the v6 support is absent to hostile and I can't run a mail server without v6 alright so I think there are two questions in there so one is like toolbox I mentioned toolbox so toolbox is not a Kubernetes thing it's just an interactive container the initial motivation is you're running an immutable host like CoreOS or previously atomic or whatever equivalent you can't change the host and you need an unprivileged user space or whatever you want so that's where toolbox is good by the way quick call out if you're looking for good linux laptop starlabs awesome it's a British company doing linux optimized laptops unfortunately they're sold out right now because I want to get one for my daughter but awesome there was another company mentioned earlier I always recommend starlabs toolbox just interactively now the problem with Kubernetes and IPv6 is a problem it's this thing it makes me despair that we have to recreate some problem over and over again because we don't learn hopefully it's going to get fixed now because there's a lot of demand because we ran out of IP space this time but toolbox doesn't solve that problem please excuse you need IPv6 I understand the Greek gentlemen thank you thank you everyone