 Sorry for starting a little bit late folks, but the other folks ran a little bit over. I work for a company called Percona. We had a booth down in the hall. If you don't know us, we're database consultants. We also have our own forks of most of the major open source databases out there. And this talk is open sources winning, but we can still lose. I need to get hold of me. There's my address. The slides I've updated to the website. I put this in here about the stuff I put in for the talk that goes into the schedule just because later if someone comes back and finds a slide deck, they can find out what I'm actually talking about. So open sources winning, but we can still lose. Once again, my name is Dave Stokes. I am a technology evangelist for Percona. Worked for many years on MySQL. I'm an author of a book called MySQL and JSON, a Practical Programming Guide. I have another book that I got to finish on MySQL. Indexes, how they really work. You'll hear a little bit more about my background in a little bit. And right now, here's where you hear about my background. We're taking a little trip back in time. There's a couple of people in here with gray hair who probably know all this stuff. But I run into a lot of folks these days who do not know anything about where we came from before the days of open source. I started with computers in what I now call an 80-column world. This was when you did things with Hollerith cards. These little pieces of card stock here actually presaged the computer for many, many, many years. They might look harmless, but they actually sent over 10 million people to death in Nazi concentration camps. Ironically, you would use a device like this one down here to write your Fortran or cobalt programs together and went through the one computer your school or your company had. And if you got real lucky, you transitioned up here to a CRT monochrome. You had one color. Or if you were like where I started, you actually had this thing looks like a combination of a jack camera and a typewriter called a teletype or an ADSR33. The 80-column world kind of still haunts us a little bit just like the way railroad tracks are actually off of metrics that they use in the Roman army for creating roads. So when I came along, this was about it for a programmer. You were lucky if you had a step debugger, a CRT, and maybe some extra file space to store your files. So this was it for a long time. Now the old adage was you'll never get fired for buying an IBM computer. Other vendors were out there, but IBM was kind of the beyond dominant player in the area. And ironically, the thing kind of started edging that out in the days where companies, entire companies, Fortune 500 companies would have one computer or maybe two, was the little advice called the IBM PC. Suddenly, your financial guys were able to use a program like VisiCal can do what if estimates. Like if sales went up 20%, what would our taxes look like? That sort of work. So it was kind of exciting to see the transition between mainframes down to PCs. But the trouble was you still had one vendor for everything. If you bought a big computer from a vendor, you bought their hardware, you bought the support contract, you bought the software, you bought the training. There might be some third party applications, but not a whole lot of them. And the user group, if any, was kind of dominated by that company. And you'll see on that list of vendors a lot of companies that are now long gone that were out there and playing big and large 30, 40 years ago. Now the trouble with this is if I wrote a COBOL program for an IBM machine and I sent it over to you when you had an NCR, it might work, it probably wouldn't. If you had a DEC or data general, it definitely wouldn't. Everything did not interoperate. Now about this time, I started on Top 10's machine because the school I went to had KL 1091, a nice machine. You could have up to 50 people working on it at a time, which was then revolutionary. I went to another school where they had Top 10, Ristus E, Vax VMS, and another operating system called Altrix, all from digital. The only trouble was that digital decided they wanted to sell more Vax VMS and killed off every other operating system. So you found your investment in whatever you had was suddenly wiped out by some senior VP's decision to come up with a different plan to raise revenue. Now about the same time, some folks at Bell Labs, people who you probably have met who come to these shows for a couple of years, started working on a operating system called UNIX, which was a plan on something called Multix, which was an early multi-user operating system. Now the only trouble with Bell Labs is they were part of AT&T and those of you old enough for number AT&T, they were always a pain in the rear to deal with. They were a nasty company, very much a monopoly but supported by the government. Now they gave copies of their operating system to UC Berkeley. So kind of jointly they developed the UNIX operating system. Now the great thing here is that you could actually run UNIX on several different types of hardware, not a whole lot, but a handful of different hardware. So if something I could write that ran on my Vax might run on an AT&T 3B2, it was amazing because suddenly you could interoperate. The only trouble was once again dealing with AT&T, they had some weird licensing issues and once again, they were never really fun to deal with. And about this time, other people started making copies of UNIX and HPUX, SunOS, Solaris, Zenix, and they all claimed to be POSIX compliant which was an open standard for if you wrote something that ran on my Sun microsystems that might run on my Apollo domain box. Not all the time, but it was a little bit better chance of having interoperability. And then around 1991, a miracle happened. We started hearing reports of this crazy guy from Finland named Linus Torvalds, giving away, giving away, not charging you, not making you sign like that, giving away a UNIX-like operating system. And I like to refer to him as St. Linus Torvalds. By the way, when I saw this picture on the internet, I'm going, oh yeah, I now have a theme for my talk. Okay, confession, I'm a database person. I see everything through the lens of databases. I find them fascinating. I know a lot of you would rather crawl naked through shards of broken glass with lighter fluids sprayed over you and then ignited than talk about databases, but I find them interesting. Now, a big revolution kind of happened a couple years ago. Open-source databases became more popular than commercial databases. Let me put it this way. This means DB2, the big Oracle database, the stuff from SAP was less popular than Postgres, MySQL, MongoDB, and several others. Now, if you extrapolate the trends, it gets even worse for a lot of the big databases. This here, I think, is the top 15 databases. But if you zero in on the top handful, you'll see that the only one who has a rising line there is Postgres. MySQL's tailing off a little bit. Oracle's tailing off a lot. Mongo is really disappearing. We'll talk a little bit more about that. And access is kind of holding steady. So databases are doing well, just not in the open-source world, but overall in general. Now, Synopsys was here. I don't know if you've gone through to their booth or what was their booth, but they published recently a open-source security and risk analysis report for this year. Now, they scanned 1,700 code bases the previous year and they found that 96% of them contained open-source software. So if you're an open-source fan, which you probably are, they're so great news, right? Well, 87% of those include security and operational risk assessments that came up with some bad news. Over half the software in use has licensed conflicts. 31% of the code bases contain open-source with no license or customizable licenses. We can talk about that a little bit later. Almost 90% of code bases contain open-source that was more than four years out of date. And 91% of code bases contain components that had no new development in the past two years. So we're building skyscrapers on a sandy beach. Now, the average number of open-source components in a given application was almost 600. So all the little bits and figures from the DNS software up to probably your user interface, generating software, open-source is great. Now, a lot of these have at least one known open-source vulnerability. It means we know there's holes there, we know there's people exploding them and they're still out there. And 48% of code bases had high risk vulnerabilities. So no people are exploding those problems and doing a lot of damage with them. I hear some percentages of various code bases. You probably all know about jQuery and Lodash and some of the other ones out there. But it shows you that a lot of things that we rely on daily or stuff that we don't know that we rely on daily is vulnerable. Gartner says that almost half of us will have some sort of attack on our code base in the next couple of years. Now, in this report, they were talking a lot about DoD, Department of Defense businesses. And if you've not worked in a defense contractor, there's a lot of technical debt there, there's a lot of old stuff you're dragging for support that might be years or decades old. And they're kind of looking at open-source as a way to get around that. And you'll hear people in the DoD saying, well, we're being more open, we're using more industry standard, but they're actually using open-source software as a life ring. They're drowning. A lot of other industries are drowning. And if they see that someone else is running software that they can take advantage of, they're jumping on the open-source bandwagon. So basically, we're winning, right? This is great, yeah. That's why we're all in Vancouver of open-source software. Well, now that databases are winning and a lot of other stuff is doing really well, it's not time to sit back and relax we've done great things, we got to keep progressing. And I call this natching defeat from the jaws of victory. In 1995, I moved to the Dallas-Fort Worth area. This is right after the Dallas Cowboys won their last Super Bowl World Champion. I'm not a big football fan. I know too many ex-players are bad at batting beat up and I have a hard time watching the games. But every year, about this time, you start hearing Cowboys are gonna do it. They're gonna go to the Super Bowl this time. And they never quite do. So we have some great ideas that popped up in the last few years like the software bill of materials. Now, this is great, it goes through all your code and tells you where the sources are come. And in some cases, it will tell you how badly out of revision it is and things like that. But a lot of folks I see who are employing me with this run the SBOM, run it once, look at it and never look at it again. Also, a lot of our efforts are duplicated. Do we really need 117 JavaScript packages in NPM for JSON encoding? What could we have done with that talent and that code if we put it on another thing? But we spent a lot of time on something like that. Also, the learning curve for beginners is ever-steepening. As I mentioned earlier, my first programming job, I had a CRT, I had a step debugger, I had a text editor, and that was about it. You look at the modern thing where you have CI CD processes, you have linters, you have all these other things. New people coming in are facing an ever-steepening wall. And some of them don't understand all the steps or the reasons behind the steps that make shortcuts. And by the way, politicians have noticed us. This open-source stuff sounds great to them. How do they benefit from it? How do they increase their own power and reputation with it? You might have heard of the Linux Foundation. They were worried now about a new EU bill that could fragment open-source. Politicians coming to your aid are not always there to rescue you, they're there to rescue them. So a lot of the new laws that are coming in, thank God the Linux Foundation are looking at them, could actually put the brakes on what we're doing with open-source. The other problem we have is open-source licenses. There are over 1,400 licenses according to Wikipedia. If you go to open-source.org, they're recognized over 100. Why is this a problem? Well, the biggest problem we have right now is that new people are running lots of stuff on GitHub and GitLab. And the most popular license is none. Well, what does this mean? Well, one of these days we're gonna have some sticky lawyers that are gonna come through and they're gonna take advantage of this license and they'll reassign it. They'll make a derivative work that's just slightly changed and be able to lock us out of what we've been doing. A lot of the stuff that I see, especially coming from licensed changes and the legal areas is basically gonna pay for a lot of lawyers. Unfortunately, it's gonna come out of our pockets. When people start doing customized versions of licenses, like we'll talk about Mongo in a little bit, we're all gonna end up paying lawyers to come back and tell us exactly what they think it means and until the court decides exactly what it means, we're spending a lot of money on that. Now, example here is the JSON license. The software shall be used for good, not evil. Depends what your definition of good and evil is. Is it going to be gluten-free software only? What is it gonna mean? So gotta be careful. And whenever you have ambiguity and you have it in a document that a lawyer is gonna use, they will beat you over the head without mercy. And a lot of our software is aging like milk. 91% of projects looked at in a Black Duck audit contained open source software that had no development in the past two years. Is that because it's perfected and has no bugs, no problems, nothing can go wrong? Has it been abandoned? Is it good enough? How do you make sure? Another problem we have is a lack of incentive for contributors to perform maintenance activities. You're fresh out of college, you're living on breadsticks from Olive Garden, and you're spending all your free time from your normal job working on a project. Then you get married, have kids, you don't have the spare time. And that software project that you did as a little hobby or a resume pluffer is suddenly something someone else is depending on and they have no idea what's going on. You might have seen this cartoon where we have an example of this wonderful infrastructure setup and Jenga-like it's all resting on one little peg by a project that may have not been modified since 2003. So one of the big things you hear about open source is it has to be better software because of all these eyes looking at their software. How many people here regularly read open source software on a monthly basis? You actually spend a couple of minutes going through the code. A couple out of the room. A lot of us just trust that someone else is looking at it. And if they find something, they know what it is and they know how to report it. We are very, very trusting. The MPM debacle of, what was it, 18 months ago now is a good example of this. Outdated. As I mentioned, 91% of projects contain outdated versions of open source components. So there might be a better version out there. There might be a version with CVEs that are patched that you don't know about, but we're not running them. So in this case, we know there's a patch out there. We may not know there's a patch, but it has not been applied. Those in the dev sec area, especially the ops area, know that you're fighting fires constantly. You are juggling on the unicycle going downhill with a tailwind and you may not have time to go back and check all your software and be able to patch them and upgrade them. Now, organizers that are often familiar with commercial software for which they get the updates on a regular basis from their vendor because they have a support contract, they switched to open source software, there's no one to do that. So they get caught off guard. And unfortunately, the end user's responsibility to be aware of this stuff. And if you're doing your normal job, are you reading all the mailing list you should find out what's changing? Now, past couple of years, S-bombs have become very popular. And knowing where all your components are sourced from is valuable. But like this drawing here, we should see all the pieces of the airplane, but do we really know how they go together? How to assemble them? How do we make sure the library we think we're using is really the library we're using? Also, we have issues on how do we audit report updated software? Do you have a programmer who runs a little script and goes through and collects all the information, sends off a memo every week? Do we need something that actually does report cards for everything? Do we need something that sweeps through all the GitHub and Git Labs repos and come back and tell us, hey, this is great, but the SSL library is two revisions old. And once again, if we have million of eyes looking at the code, it's still easy to lose playing three card Monty. How do we make sure that what we're seeing and what we think is out there is actually what we're running? Most of us these days use a package manager to do binaries. And we're just assuming that the source code that we look at matches up to the binaries. So, three card Monty there. Also, do we need a program to adopt homeless software? A lot of our developers who are around from the early days of Linux are now coming up to that age where they want to go off and do something else for the rest of their life. Maybe you get a different job and you leave your project behind. So, maybe we need a college or university program that takes people learning software engineering and has them run sort of management and maintenance through all this. And of course, we'll need Sarah McLaughlin with a sad ad and pictures of kitties. Database world a couple of years ago. Had a big shock. Some folks decided they weren't making enough money. They didn't like what AUWS or the other cloud vendors were doing, so they changed their license to keep them from doing that. MongoDB changed their license to a server-side software license. And by the way, if you're friendly with a lawyer who won't charge you money for advice, have them read this paragraph on the left-hand side. A service that must release the software it uses to offer their software. Lawyer, I know, read that and said, well, that basically tells Amazon, Azure, OCI that all your software has to be made available to Mongo. Is that the exact legal ruling? We don't know yet. No one's actually come through. But that's the type of thing that a good lawyer will tell you this is not something you wanna touch. Also, Maria changed one of their pieces of software. To a business software license, which starts off as closed where you have to pay royalties, but then after four or so years, it becomes open source. Four years in a software project is a eternity. And if you're depending on software that's four years old and you know there's better stuff out there, you might wanna look for an alternative. So what do you do after your software choice changes its license from something that's open source to something that is definitely not open source? Well, you can run the version before the license change forever, which is incurring a lot of technical debt. And it's not something you really wanna do willingly. You can do it like we did. We forked MongoDB, we have branches of MySQL and Postgres, we add our own stuff on top of it and Perconi gives it away for free. So if you want an enterprise version of those Mongo or MySQL, and you don't wanna pay Oracle or Mongo for the stuff, we give it away for free, but you're now dependent on Perconi to keep doing that forever and ever. Find an alternative, don't like Mongo? Okay, I'll go to some other NoSQL database. Good, but you're gonna need to rewrite. You're also gonna need to retrain your staff. It's not gonna be a lift and drop situation. Now, each of these options have their own associated risks and costs, just not financial costs, opportunity costs, time costs. And unfortunately, the more customers I see, the more intricate projects going on, there's very, very few simple changes go on anymore when you try to transition to something else. And unfortunately for most of us, when these announcements comes out, they're a surprise. And my role as a manager was, I always told my employees, no surprises. If something comes out of the blue, we'll take it on the chin, but if you know something bad is about to happen, give me a warning. Unfortunately, with these type of license changes, you don't get warnings. And of course, there's no guarantee that it's new software, if you do go to it, also doesn't change to something else in the future. It's equally unpalatable. So, what are we supposed to do? Well, what happens if you don't notice the license change? You have a lot of exposure. Part of it's financial, the rules that the company set up for your fines are out there. You're also gonna have some legal costs. The other thing is, if you're not paying attention to this one package, what other package you're using have also changed. The public's gonna see you if you're not complying with the new license rule and skipping licensing costs, they're gonna see you as a cheap SOB. And in the future, you're gonna have other incompatibilities that you may not notice that are gonna put you even deeper in the hole. Now, what happens if you have a product that changes this license and you do nothing, just hoping that you're gonna skip under the radar? Well, many companies I know will fire you for doing such a thing. It's your responsibility to know what your tools are. And of course, on top of all that are the financial legal and public risks. This is getting, this might be a sensitive topic for some folks. I used to hit just about every Linux Fest in the Northern Hemisphere for several years. And what was interesting, while this shows an example of professionalism, a lot of other Linux shows are definitely not. One of the problems we have in the open source world is that we trust the people who are running the software are as smart as the people who wrote it. So you just tell them to go RTFM, go read the manual. Now, other, our competition in the commercial side have the ability to do a lot of hand-holding. I don't know if you've bought a commercial database for an Oracle or a DB2 or something like that. There's a lot of sales engineers and sales people there to help guide you and walk you through and help setting up stuff. A lot of the stuff in the open source world there's no hand-holding. Support is by a mailing list which they may not be able to read easily and get the information from the volume of material that comes through. Sometimes our communities are not exactly supportive. Another problem we have is education. I mean, sure you can watch a video on YouTube. You might be able to catch a couple things on Hacker News, but a lot of the commercial pieces of software have well-developed educational programs that most of open source does not. A lot of projects have a hobbyist mentality and this is a barrier to acceptance for a lot of folks. The lack of turnkey options for many projects, pieces of software make commercial software seem more attractive. Ikea is great if you like little Allen wrenches and can read diagrams. If you don't have any spatial logic, Ikea furniture is not for you. Troubles integrating with other packages that are old and familiar with the corporate environment. I've actually had people ask me, how does your database work with the spreadsheet that the Cyprian Department works with? So it doesn't. So how do we integrate those two? One of the key tenants of open source are not highly high on the desire list of features or from the general public. You know, the great thing is, hey, that software, I don't have to pay the big licensing fee to the big corporation, but I have payroll tomorrow. How do I run that? Also, we need to stress functionality and adaptability over a hobbyist to lead a stance. Our system might be better, but what is the bottom line cost? A lot of folks here are open source and here's this free software and they figure everything's free. So they get very surprised when they find out that they end up with support contracts or engineers specializing in the area and suddenly that idea that it's free is ruined. This is an interesting quote I picked up last week from register when it comes to Linux distros, one man's molehills and another man's mountain. Since you're here at a Linux show, I have a feeling that you all have your favorite distro or two and absolutely loathe other distros. A lot of people who are new to the open source area have no idea of those reasons. About a month ago, I bought a piece of software off the internet for mixing music from a guitar into a digital audio workstation. I paid 20 bucks for this piece of software. Did exactly what I wanted, I didn't mind paying the money. However, the first four days I had the software, I had five phone calls from the author's team and they were telling me that, hey, we're just checking in, we're making sure you're doing okay. Do you have any questions? Do you have it running? What are you doing? Do you want to share or something? We listened to it and it would give you some critiques and it was amazing for 20 bucks, whatever profit they would have made, they blew in phone calls on me. Now, the great thing here is I honestly think somehow for a lot of open source software, we need to find a way to find this type of interface with the customers. I don't know how to do it, but the handholding was really great for $20 piece of software. Personal bugbear, error messages are usually written to be, well, the lifeline for your user of your product, they tell them what's going on, why can't I do what I want to do to see an error message and that's the lifeline they're going to grab onto. Usually error messages are written by junior developers and they might make sense to the junior developer, might make sense to the junior developer's manager and absolutely nobody else. Error message just makes sense to those who know, but they don't. Example, values must be numeric for phone number is better than a stark bad value pop-up. My wife, God bless her, is a science teacher, has a very clinical mind, but when she gets an error message that's incomprehensible, turns her off the package. Also, we tend to use a lot of science fiction terms at GROC. Also, North Americans tend to use a lot of baseball metaphors, but Europeans find very charming but are incomprehensible. And also, it helps if you refine it from clarity. I honestly wish we taught junior programmers how to write a good error message and then go back and review them. And one of the things that someone I know who wrote technical manuals for living said, write as if your grandma was using your code and her life depended on it and you want to make it easy for her to use. So, how do we win? And by the way, we will not win them all, nor do we really want to. We are in competition with commercial software. Those of you who've ever had open source software project that was going strongly and some of you had a senior VP comes in saying, hey, my buddy at Microsoft says I can do everything I need to do with Microsoft software. It's a rough thing. So, we need to do better promotion of open source software on ourselves. Sadly, a lot of folks see marketing as a dirty word, but we need to promote what we're doing and give people exposure to it. We're gonna have to get tighter coordination along the entire process, which means your CI CD process is gonna need an SBOM check and it's gonna need someone to eyeball what the SBOM is telling them to make sure that everything is coming good and we generally need to communicate better. Also, I think we're gonna have to come up with a standard for upgrades. One of the pieces of software I deal with regularly upgrades the SSL library every quarter, which is good because the software for the SSL library updates every quarter. But some of the downstream packages from there update that SSL library maybe once every two years, once every three years. We probably need some sort of rule of thumb that says if library X that you depend on has changed in the past six months and it's a minor change and it doesn't have any big implications, you need to upgrade. Big example here is the last time the show was here in Vancouver, I ended up talking to the developers of EXT3 file system. Their big bugbear was that they were doing all this really neat stuff with the EXT3. They got rid of a lot of problems that we are suffering in Linux with the EXT3. But the distros were not scheduled to pick up their code for another two years. So what good does it do to paint the moon of Elisa but the museum's not gonna be open for two years? Project list, we're gonna need some way to have a list of projects that are abandoned or need help. We're gonna have to do some mentoring and we're gonna have to keep from reinventing the wheel. My early example there was the MPM libraries for JSON encoding. And with that, I wanna thank you all for coming out. I know this has been a long show and when I saw it was the last slot of the last day, I was hoping it wouldn't be just me and the AV guy, although the AV guy is kind of a nice dude. If you wanna copy the slides, they're uploaded on the conference website or you can get it off my speaker deck. Any questions, comments, concerns? Yes ma'am, you? Oh, hi. Thanks, it was a good talk. I used to go to the open source convention when it was like in Portland, the O'Reilly one and there was a kind of informal meeting of folks who sit on open source boards that would like meet in person. I'm in the neighborhood, I work on open standards so they're like, okay, you can come. And I was noticing back then, this is, I started going between 2005 and 2010. I don't know what we call that anyway, that era. And I was noticing that leadership training might be a good idea for this community. And I even talked to folks who I know do leadership training in the nonprofit sector and said like, because they don't really wanna do the corporate stuff, but they're kind of also neighbors to open source that might be culturally appropriate to these communities, or at least adaptable as a starting point, but it's still not happening. And a lot of what you're talking about in the list of problems could be solved by upskilling the soft skill side of things in this community broadly. So where is that happening? And should I, now that folks like you are making these types of presentations, come back and say, where's the LF leadership training department so that the soft skills get better? Because those things aren't, all the things you named aren't gonna get solved until those are also developed in this community. Yeah, well, one of my hopes is someone from the Linux Foundation hears about this talk and starts going through the slides and yeah, we can do that, we can do this. A big example that I used to love was the PHP community. Love it or hate it, PHP is the dominant scripting language on the web. They used to get together once a year at a show called PHP Tech and they'd all go out for beer and they'd hash out, this is what our new version's gonna do at, here's where the rules are. COVID killed that off, so that communication's been stifled for a couple of years. They're meeting again next week, so hopefully a lot of the face-to-face stuff will do, but we definitely need some guidance from Linux Foundation and other folks to say, let's get these folks together, let's kind of raise the bar. You may not, yeah, it's not about, like these communities know how to go have beer with each other, that's not what we're talking about. It's like how to actually gain skills in how they conduct themselves, how they lead, which is slightly different than more. Yeah, how do we make the folks in the head of the committee can actually run a committee? Yeah. Yeah. You just turned it off. No, I didn't. How do the people have the skills to do many things that it's not about the functionality of running a meeting, it's how do they have the inner capacity to do these things well consistently over time, not follow the recipe of this thing? Yeah, how do you get a leader rather than an administrator? Yes. Okay, that's a good question. Hopefully we'll get something. Anyone else have any comments or? Well, if you have anything on your two side, please hit me in my email or my Twitter. And with that, I wanna thank you and it's been a long week and I know we're one of the last sessions, but hope you had a great time and hopefully see you later. Take care.