 Our next speaker is Dirk Handel, someone I've known for quite a long time who's been in the open source community for many, many years, almost as long as Lienis. He spent 15 years at Intel as the chief Linux and open source technologist, is now at VMware. Please welcome Dirk Handel. Thank you. So let's leave Jim's microphone on a little longer so we know what's happening. You forgot one announcement, so with the shift of location, the Linux Foundation also found a date that gives them cheaper aid. So the next conference will be December 24th, 25th, and 26th. Exactly. So Angela has a perfect track record of scheduling conferences over every birthday in my family, my wedding anniversary, Halloween, and now Valentine's Day. Fantastic. Thank you. Alright, just to be clear, next year is going to be not on Valentine's Day, it'll be later in February, so you're safe. And with that, I'm going to turn off my mic. Good morning, everyone. Wow, this is bright. Hi. So yeah, my name is Dirk Handel. I'm VMware's chief open source officer. I've been around this for quite a while. I've been doing Linux for about 25 years in open source, which wasn't called open source back then, a little longer. I now work for VMware. We are a fairly active contributor in open source, whether it's Linux or OpenStack or Kubernetes or many smaller projects. And we have a lot of our own projects that we drive. Clarity is Jihad, are you in the room? We have a fantastic software design system that we open sourced a few months ago. We have a number of projects in the cloud native space, Harbor and Admiral that are actually integrated in a bunch of these projects. So we do a lot of work in open source to help move along this industry and be more successful. But this talk actually is not about VMware. I want to do a little bit of what Linus talked about. I don't want to give you an inspirational visionary talk about all the wonderful things that maybe we'll do with great exponential charts behind me. Instead, I want to talk about what really is the hard part of open source and what really is something that a lot of people love to ignore. So this is a very familiar topic and it feels like I've given this presentation on and off for 20 plus years. It's one of those things in the English language that is very weird. The English language uses the word free for a number of different things that in other languages are different. So in German you say something is free as in free of like speech or it's umsonst as in gratis or whatever. You don't pay for it. And a lot of people seem to think that open source free software is actually free. Who has heard this argument from someone who says oh I use open source software because it's free as in dollars. That's actually surprisingly few hands. Maybe you're all tired. I did notice that the coffee break starts about now according to the schedule. So this is something that I encounter a lot in conversations where people say oh I'm going to shift to the open source version because that is free. And then I ask them so who runs straight upstream from GitHub in production again show of hands. Okay, I want your names. That's fantastic. No one does that and anybody who claims that they do that they have a very narrow definition of what in production means. But at scale in an enterprise environment no one uses straight upstream. Linus runs his upstream kernel on his laptop. That's production. Okay, but in an enterprise environment what you run comes from somewhere and how you get from an upstream project into production is what I want to spend. The rest of my time today talking about so what one of the ways to do this of course traditionally you go to an open source vendor. You go to your the local company down the street who does that or to one of the global players and and you get a enterprise ready version of an open source product. I used to work for SUSE a very long time ago. We were the first company with an enterprise Linux distributions less in 2000 1999 actually and and redhead of course does that so do many others. And as you go up the stack you will find enterprise ready products of a lot of open source projects. Even more companies look at this and say I can do this myself I can put a few engineers on this and I can productize this open source project. And so I'll take a look at this a little bit from from a schizophrenic point of view from a point of view inside of the project what does it take as the project to do things right. And the previous conversation the previous talk I think that an excellent job explaining many of the processes that it takes inside a project to get this right to be ready to to be used by an audience in a production environment. But I also talk about this from the perspective of a company that wants to use open source and wants to think about what does it take if I just take a project and try to run my business on top of it. And I guess that reflects a little bit back to the conversation that we had a little earlier from AT&T. So from a project perspective it starts really really simple really basic stuff. Do you have a source code repository? Do you have a bug tracking system? Do you have a way for people to to submit code to you and in many ways this is one of the nice things about GitHub. GitHub provides you all this and it may not scale depending on the size of the project as some projects are finding out. But in most cases the infrastructure exists and it's not rocket science to do this right. The harder part is actually to take it seriously. The harder part is to really every day look at the bug reports really every day respond to them help the people out there engage with your users. Because again if we take this into an enterprise environment the only way you can do this is if you have the ear of the developers if the developers are willing to talk to you. But this is kind of where we get to the point where we have this split between the people thinking of production and the people who are the open source project. And again this came out in the last talk a little bit because an open source project by definition by its nature is focused forward. It's focused on the next cool feature the next version the next release the next alpha the next better the next thing it's always this innovation engine that we talk about. But if you're trying to get something to production you kind of need to say OK here this is the version that I'm taking here is when I'm taking a break and now we're getting ready with these features with these things. And we take them to the place where we can actually run this in an enterprise environment. So silly things. Is there a good change lock. Whenever I get an update anybody here on Twitter by agents. OK so whenever I get an update to the Twitter app and I look at the change lock in the Android store and it says bug fixes. That's like incredibly helpful because now I know bugs were fixed. I it's one of those things it seems so simple but can you please tell me when you do in your release what actually changed. If you want to to to use something in production you want to see OK I switch to the next release and you don't know what's different from the last release. It's really hard. And overall the release strategy needs to be understood. Now the restrategy the Kubernetes team talked about. So they have about two weeks where you can post features in this feature repository. And then after that they don't don't allow more feature postings anymore and then they spend a total of about two and a half months 10 weeks. That sounds awfully familiar. I've heard a rhythm with the two weeks merge window and then a tent never mind. I expected someone to pick up on that. But now it goes beyond that when you when you look at the release cycle. So what does that mean. How how are stable releases done. How are security releases done. How can someone who uses this in production understand the flow of a security fix into something that they can actually use. Now let's take a step back and look at this from a enterprise production point of view. You have this one project I'm talking about. But really you need to look at your whole stack. You need to look at all of these projects. You need to look at all of the dependencies. What is the flow for security updates into those. How do you get new releases of each of the components. How do you synchronize these. How do you get to something where you decide yes this is what I'm going to run. And here is what I'm going to do if any of these projects now gives me a security update. And here is how I'm going to be able to continue to run this in production. And you really need to do this. You need to do this regularly. You need to do this reliably. And you need to do this in a reproducible fashion. Because at any point of time in your stack there could be a catastrophic problem. A catastrophic problem that exposes you to a risk of an attack to a risk of a crash to risk of downtime that you don't want to deal with. Now automated tests are great and CICD pipelines are a wonderful thing. But they can't just be unit tests of each of the projects that you use individually. You need to think about your whole stack. You need to think about the production infrastructure. So for example in your test bed are you using the same type of database. Are you using the same type of distributed environment? Or is it all for convenience on your laptop? And you run all on the same system. You don't deal with network latency, anything like this. And then you need to look at production scale. One of my favorite things when I see startups get successful or a random web page suddenly being hit by a Twitter post that went viral. And then you suddenly see all the testing that they've done with, oh we might get 200 users at the same time. Let's make sure we scale to 200 users. And then 20,000 show up. My wife goes to a conference every year and this conference has a registration website. About 4,000 people attend. It seems to be pretty reasonable that you assume when you open registration that most likely roughly about 4,000 people will connect to the website. Yet every year that website goes down. And I ask myself, who tests that? How are they testing this? What are the assumptions that they're making? And this is what you need to ask yourself when you test the software that you're putting in production. What assumptions am I making? What if these assumptions are wrong by an order of magnitude? What if for some reason it turns out Michelle Obama will be at that conference and now 200,000 people want to try and register. How do you deal with that? Another question that a lot of people don't seem to look at when they talk about taking an open source project into deployment is how do I do an upgrade? Even worse, once I've done that upgrade and I've realized that in my testing I didn't find out that these two components no longer talk to each other, how do I do a rollback? How do I do this with persistent configuration so that as I change my versions, my configuration doesn't get erased by the next update because somebody brilliantly included the Etsy directory in the package. No one has ever seen that. How do I do a data backup? Anybody from GitLab in the room? Data backups are a good thing, by the way. And all that should be automated, obvious, right? It should be tested. It should be integrated into your whole tooling. And it should be done in a way that you can document best practices so that when you are out on the slopes somebody else in your team can also deal with this when that goes wrong. Now, many of these things that I'm telling you, I hope everyone is nodding and saying, sure, of course, we do that. Of course, all the time. Yet I run into this so frequently that companies start from a very small test bed and they, you know, in development they tried something and they went into the test environment and they go into production and everything comes to a screeching halt. Things that no one seems to be willing to do in test is geo scale. So if you have a enterprise product today that is used globally, you have to have an idea how you scale that across multiple geographies, multiple data centers. How do you migrate workloads? How do you deal with load balancing on a global scale? How do you manage all the assets? All the pieces that are part of this network that you then put out into production. How do you deal with silly things like network latency, network segmentation, suddenly some link goes down and A doesn't see B anymore. How do you deal with denial of service attack on one of the data centers? How do you deal with the whole threats that come by having a distributed system that now needs to share critical configuration data across the internet, basically, across long distances? How do you deal with distributed storage? Now, of course, somebody, I'm actually surprised no one raised their hand yet, somebody will say, oh, but that's covered somewhere else in the infrastructure. I'm using this other tool that does that because open source is great and we have all these other tools that do that for us. And then whenever you actually do this, you find out that while you were developing, you made a lot of assumptions how this will actually work. And these assumptions may not be correct. I ran into this just in a very small way when you suddenly realize, oh, it is nice if your networking system assumes a certain architecture between the addresses of all members. And once you go into a global environment, that little assumption that everything is in the same class A network suddenly isn't true anymore. And everything that you design fails and the world gets to watch. It's awesome. Let's see. I'm sure I forgot about 20 of the things that you should look at. I'm sure I forgot a lot of the problems that people here in the room have run into. I wasn't trying to create a cookbook of how to do this right. I was trying to get you to think about what it takes to get from, I have this really cool open source project where a lot of innovation happens, where we all are excited about the new feature to the other end where someone is trying to run this in an enterprise. And I want you to leave this presentation with a few very simple thought models. Think big. Regardless how little you think the target audience may be today, it may change very quickly, very dramatically. Think context. Think what is around me, what else is happening in this data center, what else is happening in this stack, what else will people be doing who use my software? Think security. I didn't spend a lot of time talking about security. I mentioned this in the context of upgrades. But it's frightening to me how few people think about the security implications of the architectures that they built. Think automation. If in your regular deployment process there is anything that needs to be done manually, whether it's part of the backups, whether it's part of scaling, whether it's part of deploying a new release, if there's anything that's manual, I promise it will fail. And do this for every release. This is not something that you do once and then you're done and everything will magically work for you. You need to do this regularly. You need to do this reliably. You need to do this reproducibly. And you need to really take the time to think up front about the process that will take you from a beautiful open source project to something that you can run in production. Thank you.