 Test, test, okay. Lou? Hi, good morning everyone. I'm here to introduce your next speaker, who's gonna be talking about Scales Open WRT Conference Network. So please give a round of applause for the man, the myth, the legend, Robert Hernandez. Woo! Good morning everyone. Thanks, Ashton. So if you guys didn't know, Ashton's my girlfriend, so I don't know how she got talked into I didn't want to be in a marriage with you. Yeah, sure. So anyway, yeah, the conference Wi-Fi at Scale, the network you can connect to right now was built by the tech team. You can see us in the traffic cone orange t-shirts. Not our choice this year, but they were the ones that were delivered. So we're working with it. So I'm Robert Hernandez. I'm the CTO at Nebula Works. You can find me on Twitter at sarcasticadmin. And yeah, let's get into it. So the tech team I mentioned, you'll see our kind of logo in various places. All my shirt right now are my little lab coat thing. But it's kind of our little marker for the team. But this team manages the entire infrastructure and the network for Scale. And that happens for the week prior to Scale, leading up to Scale. We get here like Sunday, Monday, and work through the week, getting everything ready for you, all the attendees. And then there's also after Scale, we do a couple of work parties throughout the year. So that is just kind of a weekend or a Saturday event where we work on various projects to try to prepare for the next show. So really a lot of what I'm covering here is a big team effort. It's not just me, it's everyone that is wearing orange. So if you see them around, feel free to talk to them, thank them for the hard work. There's a lot that goes into making this whole thing work and it again, it's a big volunteer effort. So conference attendees need Wi-Fi. I don't think any of us have ever gone to a conference where Wi-Fi wasn't provided. And at Scale, we have our own network that's actually provided using open source technologies. And we've had other options available to us. We could have used the hotel Wi-Fi. We could have done a number of different things but we thought it was appropriate to actually use open source tooling to provide the open source conference that we're all here and attending to learn more about. We had, and I first got started on this. This was back in January 2017. I didn't know much about the tech team. I was just kind of threw my name in the hat and said, hey, I could volunteer. And this question came up on one of the first calls I was on is, does anyone know how to build open WRT? And I wasn't doing anything at the time on a team. I didn't have really any responsibilities. So I said, well, I mean, I've built other pieces of software, Linux, kernel, FreeBSD, those types of things. Shouldn't be that hard to build open WRT. So yeah, I said, sure, I'll give it a go. I'm trying to just be a team player and pick up and help out where I could. And then we got that whole build working and I just mean working in the sense that it compiled and kind of ran on hardware. And then three months later, it was Scale 15x. So we were going to Scale 15x with a new team. At the time the team was in the transition period. There wasn't a lot of people that were coming from the prior team onto this team. So we were kind of coming into this blind and we ran with open WRT and made it work. I don't know if anyone was there for 15x. Maybe you can tell me how well it worked. I don't know. Yeah, maybe a little bit, yeah. It was a little touch and go for most of those days. Yeah, yeah. Yeah, so there was a number of different things going on. But yeah, so we learned a lot during that whole experience and after 15x we had a post-mortem and a couple of different discussions. But the decision was made to stick with open WRT. I and a couple other folks felt very strongly that we should keep the conference running on open source technologies where we can do that. We don't get the luxury of being able to do that for certain things. Like switches is kind of a big thing where we don't have a lot of options. It's really, we're just kind of, we don't have a lot of things out there that are open source switches that are cost effective. So that's just kind of a side note. But for open WRT and the access points and the wireless we could definitely keep running with this. So then fast forward to today, 19x, five years later, you know, we've learned a lot. We've done a fair amount of conferences since then. The processes and the techniques that we're using to build what amounts to a bunch of these netgears has been a lot. So really it's about 120 consumer routers. We just turn them into access points, right? They don't do any kind of routing. They don't do anything like that. They're just really dumb access points at the end of the day. We, for what we're using at the show today, it's mainly, it's called three models, but they're really all the same model more or less. The hardware's the same. I think all that's different is some of the, the like the flash and like the offsets and things like that when you're loading the image. But it's just netgear 37 and 3800. And you'll notice that it doesn't have a whole lot of features that you would see in maybe your newer, more advanced wireless access points or home routers. But really what we found with this model is a lot of stability. And it's been a very good model for us to continue to build upon because it just works. And it works fairly well. And we just need to provide, we have a lot of them around. You'll see them in various corners of the conference. But that's really the hardware that we're working with for the conference today. And just a little side note, we've tried another model I didn't include it here, but we've since tried to kind of have an alternative to this particular netgear, but those haven't necessarily panned out. I'll talk a little bit more about where we're going in the future with the hardware, but for now it's really just this netgear, the set of models. So that's the hardware. And then we've got the software, and really I just wanted to call it first and foremost, just as we're going through all of this, that what I'm gonna be covering, the various things and scripts and all of this stuff, it's coming out of our repository, the scale network repository. This is our single source of truth for anything that we're doing. It's gonna be in there, so anyone's welcome to look at that. It's completely open, it's completely public. So feel free to check it out and see kind of how the sausage is made, if you will. So the image that we use for scale, it's not the stock image that you download from the nightly builds and those types of things. It's actually a completely custom build that we pull directly from master on OpenWRT, just periodically. Like we'll kind of do it every quarter to six months, just kind of depends on where we are in our development cycle. But we try to get a pretty new image for the show and then make sure that all of our configuration and testing passes that and we can bake that in for a good couple of months before getting here. But that's happening throughout the year. And so the current image, we're on the 510 kernel and you'll notice that the packages that we have wouldn't be found in your normal OpenWRT configuration out of the box. You could install those after the fact with O package, the package manager. But we actually bake all of this in and this is just the main highlights of the packages. We've got Bash, Python, OpenSSH, LODP, TCP dump, RSSLog, A-Pinger, and Xabix. I think most of those are pretty self-explanatory. And then what we have in our, as far as the files and configuration that we're putting on there, there are like the service specific configurations, the SSH keys, those types of things. And then a lot of that is actually templatized with a tool called Gomplit. And it's a go templating tool that just allows us to do some conditional logic in the configs before we load them. We might wanna turn on or off, like SSH password authentication for testing initially, we'll leave that on and then we'll turn it off, come to show. So there's just kind of some couple of knobs that we turn, as well as things like the network configuration we'll do. We have a lot of different VLANs and a lot of things that we're serving up on the wireless network. So we actually have a pretty sophisticated network configuration for the access points. So that's put populated there as well. What we've removed that you might find in the normal image is Lucy, so like the gooey thing. We don't need that, it's just not something we're gonna be able to log into a hundred of these things and click around. We have the Drop Bear, which is the SSH service that you usually get, DNS mask and log D. All that stuff we just didn't find was very useful. Yeah, or it was not functionally gonna fit for what we were doing, thanks David. Yeah, there was definitely some things like in busybox, we had to kind of be like, okay, yeah, the busybox one kind of works, but we need to go actually to the full version of whatever tool that you might have normally had, just because we didn't have all the options available. So once we had the image built, then it kind of comes down to the flashing process. I think, how many of you have flashed open WRT or just anything onto a router? Yeah, it sucks, it is horrible. So imagine doing that for 120 of these things, which is what we used to do. So anytime we wanted to load the entire fleet up, we ended up doing what I call just the traditional open WRT flash, which was holding the reset pin and waiting for the LEDs to show you that it's in reset mode and it can accept a TFTP flash and that the ping, huh? Yeah, yeah, yeah, so after a while, so this is very error prone obviously, like somebody flashes it, thinks they loaded it, it puts it in the done pile, then you find out that you put it out there, it doesn't work and you gotta kind of come back and redeploy it. So this was the early days, that 15x time when we were doing this traditional flash, we then evolved into what I call solo flash. So this was still having to hold the pin down, but once the router got into its flashing mode, we could actually detect that via a malformed ping packet and then go ahead and then say, okay, cool, it's available, let's flash it, so we would have an expect script go and flash it and then say, hey, cool, I've added the MAC address to our list of known flashed open WRT APs and we'll move on. So we had this solo flash which served us well and still serving us well to this day because there are times where we load something that's wrong on one or two APs when we still have to use this just to get it back into a known good state. But this year, and leading up to the conference, we had a work party and we actually went ahead and did what we call the mass flash. I'll say I have mass there, because this is mass flash, but, and this was, because we had open WRT loaded on all of the access points prior from 18X, and we knew how to get into them via an SSH key, what we could do then is actually say, okay, let's put these things on the network, they get a DHCP lease, and once they get a DHCP lease, we can SSH into them and flash them with the like the sys upgrade mechanism within open WRT. So we tried that out and actually built an expect script to do it, and then this is what ended up happening during our work party, here's some pictures, which is, it's about 20 access points all lined up in a row, as many ports as we had on the switch and as many power bricks as we had, we were flashing that many access points. So we went from doing it in like really a day to two hours flashing this stuff, and it was much more consistent, because after the flash, we could then test to make sure that they got the image we expected them to get. So that was a huge improvement, and here's another picture, and I just kind of included this too, if you notice on the back here, that was my laptop, that was just running the script, and that was like the DHCP server for us as we were kind of going through stuff. I just thought it was a good, it captured pretty well what was going on the time, and you can see all the power bricks here for what's going on, but again, that's now our new way of flashing this stuff, and it got us out of having to deal with the hold the reset button and doing all of that, so that's been really cool, and a huge improvement for getting these things loaded up prior to the show starting, and even when the show's actually starting, we've been able to leverage this script and this technique to flash the access points while they're on the show floor, so the only thing that we have to be mindful of is not doing it while people are present in the room, and there's a lot of activity. Obviously, it's not gonna be a good experience for anyone. And the last one here is the auto flash. I'll get into that a little bit later in a couple of slides, but I just felt like it was good to call it out here as part of the flashing process. This is something that we would typically use for testing, and it's actually this device that I have in front of me, but we'll come back to that in a sec. So now that we got all these things flashed, what do we do to manage them? Because we don't have a central controller or anything like that outside of really the DHCP server. So what we actually do is we use custom DHCP options to set things like the channels for the 2.4 and five gigahertz networks, and we'll also do a third option that will actually change the entire network config. So we pre-populate a bunch of network configs on the APs for the known scenarios that we know we'll have to put things in during the show, depending on where these things get deployed. Yeah, yeah, we use SNMP for a little bit. I can circle back with you on that, but yeah, SNMP is an option. We had an issue too with the SNMP daemon. It actually ate a ton of resources on these devices, so we ended up having to go a separate way. I don't know if that was a momentary problem with the package or what have you, but that was kind of, we had SNMP probably about two or three years ago. We just didn't end up really utilizing it, and as David had mentioned, Xabix was, yeah, yeah. And then so really the whole thing is, we'll reflash the APs for the major system updates. So we're not gonna do too much on these things, ad hoc, that just kind of gets a little unwieldy to manage. We do have SSH, so we can get in there and start to debug things if things are looking weird, but we're really not trying to push a lot of configuration over SSH. That's really what just the flashing process is gonna be for, from our perspective, and at the most we'll leverage the DHCP options. And then we have R-SysLog and Xabix, as David was mentioning, we got that for observability, and that's been working pretty well for us for the last couple of years. Oh yeah, yeah, just to kind of back up on that, the, like a lot of the wireless utilities, we're just kind of leveraging R-SysLog as the vehicle to get some of this stuff out, like who's connected, what clients and those types of things. So that's constantly running on a couple different custom daemons that's running on one of our virtual machines back at the knock. So the next section I wanted to get into was the automated testing. So we've got all of this stuff built out and everything, but doing it once is really easy, but doing it consistently and repeatedly, as someone who's developing this stuff, like I found it really hard for a long time because there was so much set up, there was so many different things I had to configure and watch out for that we've implemented a few different testing frameworks, if you will, for the access points. The first one here is a golden test. So I mentioned Goplet as the templating engine that we're passing our configuration templates through. So the golden test leveraged the same templating, but we actually track like the known inputs and the known outputs, so that anytime we make a change, the golden tests will be run and either if we have a change that we know need to make, update the golden tests, we're just doing a comparison. So I have a snippet here from the script which is generate the template and then we just diff. So we keep a known good copy and then we just diff it. So that way, if I rewrote the templating logic and I was like, oh, this should do the same thing, does it really? Because there are many times where that template, the conditional, something's up that I wouldn't have caught and then we go through the entire two hour build process for an image as much about what it takes and then I find out after loading that image, another 30 minutes or in our case now like five minutes, still it's about a two hour time. Oh shoot, the template was off, the service didn't start. So this has saved me a lot of time and the team a lot of time in having this type of testing in place and it's very simple. Again, it's available in the repo, you can take a look at it. But it is just diffing a known good output against the output of updates. Then we have a weekly image build. So the OpenWRT image that we use for the show is built weekly just to make sure that we don't run into weird drift kind of problems because we're using external endpoints and things, grabbing different packages. A lot of stuff can break. It's not really something we can control. So we build it weekly just to make sure that it's still usable. And if there's something that breaks like an endpoint changes or like sometimes drivers move around, weird things like that. Someone deprecates a version of a driver somewhere for the radios, those types of things. We just at least know that we have to update it. So this has been running probably for the last couple of years and it's been really nice because it just kind of gives anyone on our team the confidence that this stuff is gonna work if you make a change and you're gonna rebuild that rebuild process is gonna work. And so this is what it kind of looks like. It's just using GitHub Actions, which is a CI tool made available by GitHub. The repository I linked to earlier was GitHub repo. So we have Actions available to us. And we just run this, I guess, weekly. So that's been the last, I don't know, was that last 10 weeks or so. And then here's the artifacts that get produced. So you've got the image itself and for the three different models they're all inside of this archive, the build log. And then we've got actually a different architecture here, which is the, this is the TP-Link architecture that we were attempting to move to. That's currently still being built and that's only for one model, but that's being included as well in the build. The nice thing about this is we know that what we've built actually works for multiple architectures. So we've broken it out enough towards not, we're not totally assuming the Netgear type of access point that we're building, which was an effort in and of itself, but that's why we keep it there right now, even though we're not using the hardware. Cause eventually we do wanna support another model. So this is just building just to give us that confidence that it works. The next type of automated tests that we have are server spec tests. So what this is, is this is a Ruby library that will actually leverage SSH to go on to a machine and then check it based on things that you've described that you want checks made for. So we've got things like, in this case, this is just checking our network config and it should be a sim link or it should exist, should be a sim link, should be on by root and should be also have the group as root. And for the network, that makes sense because for us with all those different network configs, we just sim link to different ones based on what we get from DHCP. But this is actually just a snippet from a lot of tests. We have about 50 right now, but you can just imagine, you can do pretty much anything you want and anything you can describe or do on the shell you could do with server spec. So it's been really, really nice. It's actually caught us a few times. We had bad packages that didn't actually be installed but they didn't really work. So it's really something that's gonna be helpful for us going forward as we continue to build out more and more of the image and getting more sophisticated. And then, so I mentioned the auto flash earlier. Bringing that back, so our auto flash process kind of wraps up a lot of what I just covered in the automated testing. But it's a way of taking any image you have, loading it up on the OpenWRT hardware and then running the server spec tests against it. And that's actually using this piece of hardware right here. This is the actual, if I cracked open this access point. This is the board underneath. We've got our various serial connections here. We've also soldered to the reset button so we can toggle that via a relay and the Raspberry Pi that's facing me. You guys can't really see it. I have a couple pictures I'll bring up after this slide. But this actually will just do a flash for you. So I can actually trigger this from a pull request and just say, hey, flash this image and it could be an image I built. It could be an image that came from the weekly build, whatever it is. But this helps to where I don't have to have this setup sitting on a desk somewhere. I just have this thing. And this just takes an ethernet, jack or ethernet cable and some power. And I'm done. As long as I can get to the internet, it's good. So like I mentioned, the hardware is the Raspberry Pi. It's got a relay. It has an ethernet adapter and has the AP board. And then the software running on the Pi is actually FreeBSD 13. It's got the GitLab Runner running on top of it. And it's got Magic Wormhole and Expect. So I don't know if folks are familiar with Magic Wormhole but this is just a way to transfer files kind of from any machine to any other machine. It's really cool little Python project. Check it out. That's what we use so I can give this machine an image I built on my laptop. I can really do kind of anything. And the GitLab Runner, the reason we didn't use Git Hubs Runner was just because it's written and go, it runs pretty much anywhere. The C-sharp one from Git Hub just seems to be a little bit more work. So yeah, that Runner is just listening for events that are gonna come in via comments and pull requests and then it will trigger, it will run through the whole thing and it will return back whether it was successful or not. So here's some more pictures for folks. And you guys come check it out after the talk and take a look at it. That would be true. Yeah, questions that get rid of Wormhole if I used V6. Yes, most likely. So again, the board on the top there, the relay in the middle and then the pie on the bottom, that's just my workbench. And then you can actually see here, this was the way it was for a little while we were taping these points that we soldered to the reset button. I sense hot glue them because they were they were still moving quite a bit and breaking but the hot glue has been pretty stable ever since. And then that's just the relay and some of the wiring up. The relay is used to actually power cycle the thing versus touch the reset button. It's actually the AC power to the whole board. So that's the auto flasher and that's been really, really helpful for us going forward. We talked about it for a long time. If any of you guys ever talked to us about dealing with this whole access points and building all this stuff, it was like, oh, it would be so nice if and we finally got around to building this out. So it's been pretty cool. Yeah, these devices don't support POE. Just being kind of consumer grade, it'd be nice. But then there's also kind of matching it to the switches. Like we're on a Juniper 42 or the EX-4200s. So I think we just need to sometimes there's, I don't know if there was a standard for POE but sometimes that's not necessarily followed. So we would have to deal with the bricks. It would be nice for runs though. You don't have to find power, put them next to power. Sometimes that can be difficult. Yeah. Yeah. So yeah, I mean, I don't know how realistic it's gonna be to get POE but it would be nice. So I just wanted to talk real quick about the radio testing. So we talk about building the image, automating the iterating on the image that we build. What do we do for actually testing the radios? How does that work? Since we are providing Wi-Fi, we should be testing the Wi-Fi that's being broadcasted. So a lot of that right now is, it's not as automated as it should be. It's still kind of working in progress for working our way up to this point but we do have an iPerf test. So we'll run iPerf for an extended period of time on an access point that we build. So I'll just usually do it from either the auto flasher. I'll leave that running or I'll just load it onto an access point and then just kind of do like a burn in for a while and then track the results, make sure that it's within the expected throughput and speeds that we're expecting. Then the second one is dog fooding. So anyone here even is, if you wanna run this image, you totally can. I run it at home. So it's my main access point or access points at home. So yeah, I'm definitely, if it's not working, I'll know about it. So then the last one here is surveying. So it's different being different buildings, even here, Hilton to Pasadena, there's different things we have to contend with the way the rooms are laid out, the way what they're built out of, what propagation we get depends on where we are. So we do do surveying during the conference to try to make sure, hey, we've got the coverage we think we need in these various areas. But it's like that's always kind of dependent on also who's in the room. And actually there was a talk given by David Lang, who's part of our tech team, and about kind of the how wireless works with people in the room and how it's transferring through our bodies and those types of things. So it's an interesting talk, check out from years ago, but it's Lisa 2012, thanks David. Yeah, but that's really what we're doing for radio testing. I'll talk more a little bit about that after a couple more slides, but those are the three things for the radios. And then the impact, so like what's done all this stuff, like how's it really panned out? And I'd say pretty well compared to 15x. I mean, I'm not staying up doing an all nighter trying to get things going. So really it's a stable platform for the conference. We get here this year, we got here, we didn't touch any access points flashing them. We'd already done that at the work party about a couple months prior. So that was a huge benefit to freeing us up to accommodate this new venue, new, just a change in venue for the year. Then we had the auto flash decreased the feedback loop from like 20 plus minutes to five minutes per test. So anytime I'm developing or anyone on the team is developing something, it doesn't take, in 20 minutes is like generous. I probably should have put really like two hours if you're building an entire image and you don't have the cache yet. You have to like build once right to get all the cache. So it's usually 20 if you've got the cache already for the build. But anyway, that feedback loop and loading it onto this thing makes it way, way more consistent. There's 52 server spec checks that are being checked on all of our images. And then this is enabling us to be able to support more than one model with limited overhead. Cause if you can imagine anytime anyone makes one change to what goes in this build, we should be testing it on both platforms. And we really couldn't for a long time. And that was just kind of a, I saw that as a pretty sizable risk. So we now can build out more of these and just swap the board out and then we'll be good. So the goal is to have that for whatever we get, whatever we end up adopting going forward. So I just want to cover some, some Marata for 19X and some things that we did at the show that we had to deal with and fix. So it was kind of interesting. You know, we're here. We just fixed this stuff like a couple of days ago. Maybe a day ago, depending on what it was. But we had the Wi-Fi radios remained up if the initial boot had no connectivity. That was solved long ago, but we had changed some things, some scripts around and we didn't account for not ever getting a DHCP lease or even a DHCP response from the server. And so that actually, our automation never ran. So the radios remained up even though there was no network and that's always not a great experience for the individual who's using the wireless network. So that's since been fixed and we're rolling that out. It's just, we gotta do it strategically on where we're rolling that out too. But by and large, all the network has been set up so we don't have random networks that are just not going anywhere. Good question? I was just for the, for mass flashing, yeah. No, there's a DHCP server. So sorry, this assumption I guess I might have made. There's a DHCP server and DNS, a buy-in server. They're all running virtual machines coming out of the knock. So all that stuff is not running on my laptop. The rest assured, that sounds crazy. It was only for the mass flash process for, yeah, yeah, we've done some other things. But then we also had some DHCP reservation mismatches in range overlap. So with us, usually being in Pasadena, we have two buildings and we made some assumptions in our configuration where we had, okay, we're operating in two buildings. So when we kind of mashed all of this stuff together, some things got missed and the DHCP reservations and the ranges kind of got mangled. So we definitely had to deal with that when we were scratching our head looking at it going, well, it should be getting the radio channel information. It should be getting its host name, what's going on? So we had to deal with that earlier in the week and then we had some DNS ACLs with again, the assumptions from the Pasadena Convention Center. So it's a lot to kind of reshuffle this whole thing but we're learning where all of the assumptions have been made so we're getting better around that but I thought it would be interesting to clue you guys into what's going on on that front. So the future, so where are we going with the wireless? What's next on our list of things to do? And probably the top one here is, I have just kind of this Belkin RT3200. So this is just another consumer grade router but it's Wi-Fi 6 enabled and it has a nice form factor. You can see there's no radios protruding out of it. It's just a box so it can be beat up pretty good, moved around so it's kind of a win because we really liked this form factor of the Netgear. Again, it's just a box. So this one looks promising to you as far as support from OpenWRT. That TP link that I had mentioned kind of briefly, it went up and down in its support. So it would work for a while and then the driver would become unstable and then we would have to figure out, okay, can we really use it for the show? And more often than not, we ended up having to just say, no, not this year. And that happened like two times in a row and we really lost a lot of confidence. And then some of the people that maintain that driver don't care to support the radios that are, let's say, have lower memory available to them on the hardware. The next one, there are, there's another, yeah, they're the same model. So there's a linksus and a Belkin, yeah, they're the same. So yeah, if you see either of those, I can't remember the linksus one, it's E something. It's not as nice to say as RT3200, 3600, yeah. So yeah, there's two models of that hopefully will also be easier for us to get them in bulk and find them. Some of this stuff becomes a little hard to find at times. So we'll see how that goes. If we end up choosing to adopt this, we were hoping to actually plan on playing around this weekend and just see how well it works with a build and all of that stuff. Then we had, we have Dawn. This is, can you remember what it stands for? I mean, it might not stand for anything, but it's really a way for us to have a decentralized, decentralized controller, if you will, so that we can do things like 802.11. I think it's K and V. I can't remember what the actual, the protocol is, but we can support roaming better and actually handoffs. Was it? Yeah, so yeah, we can do a little bit better on the roaming and actually letting clients know, hey, don't stay connected to me. You have a better access point over there, and then kind of suggesting they move if they support that. It is somewhat dependent on the client, but that's an interesting project from this Berlin Open Wireless Labs. So we'll see how well that works, but it does look promising. We just haven't had a chance to play with it. We wanted to do some more automated testing and automate the client Wi-Fi tests. That's totally possible. Just, we haven't got around to doing it, but we should be able to run those tests pretty regularly with any flash. They might just take a little more time because we might wanna run them for like 24 hours or things like that. So we might maybe run them on a weekly basis. We haven't really decided, but we just need to configure some clients and test that stuff out. And then advanced surveying process. So we don't have too much as for surveying. Like we do a decent amount, just like walking around with our laptops and kind of testing various points, but really advancing that surveying process and seeing what we can do there. There are some other open source projects that allow for that. We just need to see which ones are gonna fit our needs. And just another shout out for the repo for folks. So the scale network repo, do check it out. Do come in. You can report issues if you notice something. We're pretty active in that repo, done a lot of work. But everything that I've covered here is inside that repo, it's our single source of truth. And then this talk, you can find it at github.com, sarcastic admin talks. And yeah, I think that's really about it. I actually can bring up my browser if you guys wanna see the auto flasher and what it looks like. I was hoping to run it for you live. But Murphy's law, it ended up, it looks like github, or I say githlab, I'm having some problems with the endpoint that was triggering the build. So I don't know what, that's never happened before. But that was happening this morning. Yeah, so I don't know what's going on. If you wanna come by the knock and see how we're doing, I'll be looking at that probably later this afternoon. But that's just kind of a little work in progress. But I do promise you it does work. It's not just a bunch of wires and stuff. But let me bring up the webpage real quick and I'll show everybody what it looks like. Yeah, that's one thing I'll mention is there's no way I can cover all of the things that have been done. This is just to kind of a quick overview of the last five years. So we just felt that we needed to get out there and talk to all the attendees about what's going on with the network and what we're doing to continue to make it better. So what I have up here is just real quick, I have a pull request. I was updating something about, this is the dynamic APinger gateway check that broke the wireless radios being up when no network was available. So I didn't do all of my testing like I thought I did. But this comment here is really what triggers our builds. So on that auto flash or piece of hardware. So I just say we have a keyword slash tux and sub command of openwrt flash and then the identifier for magic wormhole just it gives you this like little identifier to know where to grab your file. And then it will respond with, hey, you know, your thing's running and there's the GitLab pipeline. So then if you go and click in there, you know, it'll open up on GitLab and here's our test and let me scroll, I'll start to the top. But can everybody see that okay? Should I zoom in a little bit? That doesn't do, it kind of just gets narrow, but at least a bit better now. All right, so, so yeah, you can see starting at line like 22 here. We're getting the image and then we're starting our auto flash process here which is a script that comes from the scale network repo and then we're toggling some of the GPIO pins kind of setting everything like resetting everything because I don't know what might have happened during the run. So just kind of getting everything reset here is and then about line 37 here, we're gonna start toggling the pins to what we want them to be and then start pinging away. So we're gonna ping to see if we get a response from the access point. And once it gets this malformed ping packet, you see this kind of like weird wrong bytes. We know it's ready for flashing. So then the expect script will then spawn TFTP connection to the access point and transfer the image to it and then once that's ready to go, we create a new interface which is on the 503 VLAN because that's where we, that's like the one of the management VLANs for us. It gets loaded on there and then we just wait to see when it's coming back up. So at this point, it's flashing, it's flashing and we're just gonna wait for it to come back around and we keep waiting, waiting, waiting and then finally when we get a response, we know, okay, cool, it's up and the AP has been flashed and then at which point now we can do our server spec tests. There's some kind of weird deprecation errors. Kind of looking to that. But I run our server spec tests and there's just some arguments or some environment variables passed to it. But then you can see everything running here which is like, you know, command, you know, which APinger, so do we have APinger installed? Do we have awk? Do we have bash? So this kind of runs through, it was like 52 checks and checking a bunch of other stuff. You have like the config version scripts making sure that's present. That's what changes our configs. So given a certain config number, like we know this one doesn't exist 99 so it should return one if it can't find a config. Those types of things are all running and then if we get zero failures, you know, awesome, we ran through our 52 checks and then that's it. Then our tests are complete and the job succeeded and we're good and that image is functional. So yeah, that's really about it. Thanks everybody for hanging out. Yeah, so does anyone have any questions? Yes. Yeah, that was probably the second one, like the range overlap. Okay. Yeah, there was some addresses that could hand it out that shouldn't have been handed out and had to go track down some APs that thought they were other things. It was a good exercise. Yeah, I mean, we knew it was gonna be interesting. Owen could probably do a talk alone on the switch configuration and it's apology that we had to kind of rework and rebuild throughout the week, so. Yeah. Yeah, this is for, I think people in the recording, just if the question was if you could come by the knock and just kind of hang out and ask questions, absolutely. Yes, where do we do the testing? So a lot of it is actually just at either our work parties or I have it at my house. At my house? Yeah, that's why I like the dog fooding. I'm running those access points the entire time, the entire year, so. What? No, no, workspace. We have a couple of hacker spaces that we get some time against. Some people are members. David's a member of Null Space. Where's Null Space? Burbank. Burbank, yeah, and Burbank. Yeah, it's okay. All right, get people quiet down real quick. That's okay. Thank you everybody, good. Yeah. It's, yeah, well the open source controller that we did look at for a while, I can't remember the name, but it ended up kind of, yeah, open-wisp, I don't think was something that was at the time we looked at it, something that was well-maintained, or it was gonna fit for what we wanted to do. I don't know the state of it right now, but that's certainly a good question. The AP flashing of the image process, I like, I'm used to that. We just need to get a little bit better about how sophisticated we can be at targeting different access points. But that would get us a lot of consistency versus any kind of weird drift. Because if you think about it, if you're pushing out from a central controller, if someone were to log into that AP, and you could disable this, right, you wouldn't let people just log in. But what happens if it doesn't track all of the config? Some of it was just, we were trying to get by with just decentralizing it. Maybe, yeah, yeah. But the other thing with some of the controllers is that the controller goes down. Then do the access points work? I don't know. Some of the architectures. Yeah, sure. Sure. It's just, there's been some other things that we have to factor in like Owen's mentioning. The central controllers, they could work. Just be kind of a paradigm shift from what we're doing right now. That was something that kind of came up around, I'd say like 16X. So that was maybe 2018, something like that. And yeah, we just didn't feel like it was something that we could rely on. And we had most of what we needed. So we just kind of kept going with what we've got. I don't think any of the problems that we've ran into would have been solved by the central controller, by and large anyway. Because even deploying an access point that can't talk to the network, it's gonna be online. And it's gonna try to broadcast radios if it's misconfigured. So it's just some certain things that we just didn't see a need for the central controller. Yeah, good question. I'm special. Yeah, it's like the LADP information for us is just making sure we know what it is. So like if Owen goes into the switches and wants to know who's gonna do what ports, right, physically, the LADP information because it comes from the hostname that there's a reservation for in DHCP, then Owen knows, okay, that's this access point that we have, you know, it's Santa Monica A1. I don't know whatever it is. But it's really that channel information and the network config. That's it. Just where it is. Yeah, yeah, just that information and maybe the channel information. And that's something that maybe, yeah, yeah. Uh-huh. Yeah. Anyone else? Yes, yeah. That's something that Don Project might be able to help with. Again, we haven't looked too much into it, but it does have some pretty cool techniques for dealing with multiple, access points deployed in an area and having them actually communicate and coordinate clients. It's probably, yeah, yeah. The transmit power's lower than usual. You usually expect by design, yeah. Yeah. Yeah, that's what the advanced surveying comes in. Yeah. Yeah, so we'd like them not to have 100, but they do seem to be stable with having like 100 different clients, but they don't, you know, we wanna distribute that. And that's again, part of looking at Don. Yeah, they are pretty stable, though. These things have been through hell and back and... So that's actually even just a problem regardless, right? Like Wi-Fi or not, because you're hammering, everybody's going out and downloading like whatever ISO for the install fest or multiple ISOs. And it's like, oh yeah, you download, you're downloading, you know, five gigabytes of whatever, it's 100 people. So that's actually like another caching discussion that there have been caches that have existed at scale. Pre-dates me, so I don't really know what was done. Sure. Yeah, yeah, yeah. Yeah, yeah, yeah. Says the guy who's on the wireless right now. I got it, yeah, I got dog fooded. Different talk. Okay. Yeah. Yeah. Anyway, I think all the questions were good. Cool. All right, thanks everybody for hanging out. Appreciate it. Thanks for the questions. Okay, let's get started. Oh, that's loud. Sorry for that. Let's get started. Thanks a lot for coming to my talk today. This is a bit of a smaller group. So I suggest that we do this interactive. I have lots of slides here, but I prefer to have a bit of a conversation, especially with a topic like that, which I think it's a bit more interesting. So please always interrupt me if you have any comment, questions, anything. If you disagree, I'm sure there's a lot of stuff to disagree with. Just interrupt me, please. Yeah, the topic is why the GPL is great for business. This topic, of course, as you might imagine, is was triggered because of some license changes that happened in our community lately. I will cover that later, where some projects decided to switch from GPL-style licenses or open source licenses to proper terry ones to fight some cloud providers, but I will cover that later. About myself, I'm a free software and open source developed for a long, long time, over 20 years, involved in different projects. KDE, for example, the open desktop network at a time, use data manifest to W3C many things. I'm probably most well known to be the founder of own cloud and successor, NextCloud. Who in the room knows NextCloud? Okay, most, that's good. But this talk, as I said, will not be about NextCloud. It's more about a bigger question with open source and business models. So before I jump into the question why the GPL is great for business, in my opinion, I want to do two other things first to talk a little bit about free software licenses. Probably won't tell anyone any news here, especially in this group, but I think it's very important as the basics for this argument that come later. And then the second part also about open source business models a bit because it is also important for the topic. Okay, so in the 80s, this guy, you all know him. He came up, Richard Storm, and obviously he came up with four freedoms at a time where he postulated and that it is our requirement for a real free software, for a real open source. The first is that you should be able to run any program as you wish, there should be no restrictions. Second is that you should be able to study how the software works to look inside, to learn from it. This obviously requires that the source code is available. The third freedom to redistribute the copy, to give it to others. It says here to help your neighbor. Yeah, this obviously was written before the internet became, was big. So software was distributed on disk to your neighbor by hand. And the fourth most important one that you also have the right to modify the software and also then to redistribute a modified version. So that's the four freedoms. Little bit later in the 90s, the open source initiative came up with the open source definition. They use a little bit of different words but at the end it's a similar idea that you should have certain freedoms connected to your software so that you have free software and open source. These principles, obviously, they're resulted in some licenses being written. The different kinds of licenses were covered as in a second. Why is a license important? Isn't like software licenses the most boring thing on the planet? I'm not a lawyer, so not. But yeah, there are, of course, important because whenever you give something like software from one person to another person, you give it with certain conditions attached to that. That's called a license. They basically give the software to someone and tell the person what they're allowed to do with it. And there are different kinds of licenses. I will try to explain them with this scale here. It's from one side with very little requirements, basically, to the most restrictive on the other side. I'll give you some examples. The first one on the far left here is called public domain. It basically means that you give up ownership over the software. You just say this is now owned by everybody. Very popular in the 60s, 70s, and 80s. This is difficult in some countries. I think it is possible to have something as public domain in the US. In Germany, for example, in other countries, it's not possible to give up ownership or it's not possible to give up basically the control about the thing that you created to license. So that's not really possible in some countries. But what's possible then is in the next block here. These are usually called permissive licenses. Examples are MBSD, MIT, Apache licenses. I think you can summarize them mostly as here's some software, do what you want with it. Just don't sue me. That's basically the summary of those licenses from a bigger perspective. You can take it, you can turn it into proprietary commercial software and then building a product on top of that. So that's a special thing. You can really do whatever you want with it. While in the middle block here, you have these copy left licenses. These are typically the GPL, LGPL, HGPL and so on. With these licenses, this is not possible. It's basically, it has this built-in guarantee that this software will always stay open source. You cannot, because you have to reach and distribute it under the same license. That's basically the idea here. And this was one of the big innovations, obviously, from the Free Software Foundation and other organizations at the time. Then go to the orange block, proprietary licenses. They usually come with some stronger restrictions like you need to be paid to be able to use the software, obviously. And then trade secret is the most extreme case where you cannot even see it or use it or whatever. Okay, so typically, those three blocks here, these are considered open source and free software licenses. So why is this important here for this talk? Because obviously, the title is great for business. Business usually means earning some money. So how do you earn money if the licenses basically say that you should be able to give it a wave to everybody? Yeah, it's free software. So you cannot charge for a license here, because the license says that it's basically free for everybody to use. Okay, so this obviously triggers the question, how do you then earn some money? How do you, how do other open source business models? If I'm saying GPL is great for business, then obviously there needs to be a way to make some money here. I went through Wikipedia, there's actually a Wikipedia page about open source business models, and it's actually quite funny and surprising to see the full list because there are quite some interesting ideas in there. I will go through the most important ones here, how you can make money with open source. The first option is selling branded merchandise. That's actually happening. There are some open source projects in the organization who actually do that. They're selling merchandise for their software. I don't think you can really pay like your full development team with just selling t-shirts. So this is something that is theoretically possible, but I don't really think it's a considered open source business model. Next idea in the list from Wikipedia is selling certificates or trademark use. There are actually some projects who use that. I think Moodle, this e-learning software, where you can really, yeah, it's free software. You can do whatever you want with it, but if you want to use the official Moodle logo on your service that you build, you need to pay for that, basically. That's quite unusual, but seems to work for some. Then we have donations. I mean, as I said in my slides, I came from KDE, 10 years ago, I was involved in the KDE project. There we really collected donations and we were able to pay a few people with that, but obviously only a handful. Like, I don't know, maybe five people were able to be paid by the donations, but there were thousands of volunteers unpaid. So this is also more theoretical business model, I think. The next one is that really exists a bit more is bounty-driven development. So you have this idea that you go to a website and you can request a feature and you can say I want to pay a little bit of money and someone else also pays some money and then there's a little bit more money than in the bucket, and then someone decides, okay, I implement this feature. And then the feature is implemented and the developer gets the money. The problem here is that this only works for new features. People are only paying for new features. No one is paying for the maintenance of the software. So this is also, I don't think you can really sustain a full development team with this kind of business model. Next one is advertising. You might think, well, that's horrible advertising. I don't want advertising in my software, but it's actually quite common. For example, Firefox, I would say is advertising. It has a preconfigured Google search engine, for example, built in and some other places where it shows information or advertising. So this is quite successful as a business model. Then again, a bit more theoretically, is this delayed open source? I think it existed with GhostScript. I don't know if this is still the case where the open source version is a bit older. If you want to use the latest version, you have to pay for it, and that's proprietary. And like the proprietary paid for version becomes open source after, I have no idea, a year or two. So basically pay for after latest software, if you wait a little bit longer, it becomes open source automatically. Again, I don't think super successful. Then there's another one, open source end of life. This basically means, hey, it's proprietary software, pay us to use it. But if we discontinue the software or if the company disappears or something else, then it automatically becomes open source. So the user has the guarantee that the software doesn't disappear, but of course, at the end it's proprietary software. So now it becomes a little bit more, yeah, realistic. So there is software, open source projects or open source companies who are offering a hosted version of their software. So the software is for free. You can download, you can install it, do whatever you want with it. But they also offer an official service from the company. This is, I don't know, GitLab, for example, does that. Some other organizations too. This is popular. Of course, it becomes a bit tricky once this all competes with the big cloud providers like Amazon and Microsoft and so on. But more about this later. Then it becomes even more realistic. So now I would say we enter the area of real, realistic open source business models. The first is selling professional services. This basically means we have an organization building software, releasing it under open source, and then a company meets a feature and they come to the organization and say, hey, I want to have this feature and then thereby some consulting days into the features developed. This exists, but very similar to the crowdsourced feature development. No one pays for the overall maintenance of the software. Everybody just pays for features or bug fixes, workshops, trainings, but not really for the overall maintenance of the software. So this is a bit of a problem. So all examples of open source companies or project with this business model are usually a bit smaller. I think with this model, you can sustain like a small project with like 10 people, maybe 50 people, but I don't think you can really become the next redhead with this model. Next one, quite popular. It's the so-called open core model. It means that the center of the software is open source, the core, but it's usually very limited from a functionality perspective. If you really want to do a little bit more things with it, you need to pay for the enterprise edition and then this is like proprietary and then you have all the features. So open source is basically used as a marketing tool. I am not a big fan of that because well, it's not really open source, but exists and it can be successful. Examples for that, what are examples? Elastic, yes, exactly, elastic. Onclouds too, yes, exactly, yeah. Yeah, this was one of the experiments we did at the Oncloud times. Bad experiment in my opinion. No one really liked it. The customers didn't like it, developer didn't like it. I also didn't like it, but yeah. MongoDB too, yeah. They also have this licensing thing now, they also have the enterprise features, exactly. Next example is dual licensing. It means that the software is available under two different licenses. One is usually an open source one for free and the other one is a proper terry license for money. Why is this useful? This is for example used by the QT library. It basically means QT is available under open source license, GPL I think, or LGPL nowadays. And you can use it to build other open source software obviously, but if you want to build software that's not compatible with the GPL or the LGPL, then of course you have a problem and it's not possible, but you can pay the company and buy like the enterprise version of it which comes with a different license without this restriction and then you can use it. That's basically a trick and because, for example, in automotive, QT is used a lot in automotive, you usually have so many different components in your car that you usually need like a license for your software that's really compatible with most other licenses. This might not be the case with this copy left licenses so this is a bit of that trick to earn money. But I think last but not least, come to the last business model and this is from my perspective the most successful one. This is basically selling support subscriptions or enterprise subscriptions. That's used by Red Hat, by Souza, by MariaDB, by Next Cloud too, many others. From my perspective, this is the best or most usable open source business model. This basically means the software is completely open, no tricks with licenses, completely open source and everybody can just download it from the website and use it. In the case of Next Cloud, for example, or Souza, Red Hat, they also have their community projects Fedora, Open Souza, and so on. Canonical, the same. But the assumption is that if you're a real professional organization, if you're like a big service provider, if you're a government, if you're a big company, then you probably want a little bit more than just the zip file downloaded from the internet. You really want to have a contract with someone and can provide you with an SLA, can provide security updates in time, can do trainings, certificates, certifications, and so on. And that's then basically usually part of like Red Hat Enterprise Linux, Souza Enterprise Linux, and so on. And from my perspective, this is the most successful business model. A little bit insights into the Next Cloud one here. We advertise this that you get like four benefits, basically, first is expertise. This means you have support contracts. You have direct access to the developers who wrote the software. So if you really have a problem with something really complicated, some setup doesn't work, I don't know, maybe you found a bug. You can really talk directly with the people who wrote the software. 24-7 Enterprise-grade SLAs, of course, helping with migrating the software and so on. Then you have influence on the roadmap, obviously. So if as a customer you can say what's missing and then we might develop it. Third one is stability. So you have more access to special knowledge base, more information around scaling, security, and so on, and then compliance with certifications. So these are basically what you're paying for. You're not paying for the software license. Again, software license is for free, but as a company, you want a little bit more than that. And I think this is the most successful business model. And of course, we saw it with Rathat. Rathat was sold or was bought by IBM for 34 billion. And Rathat didn't own Linux, right? That didn't really need to have ownership about any of their code. I mean, there might be some pieces of Rathat Enterprise Linux that's actually written by Rathat engineers. But I mean, this is the minority because while Linux distribution is the combination of a ton of different software pieces, like developed by someone. So it can become a really successful big company without really owning the code. That's not really important. So again, this works very well. There are many examples for that. But of course, nowadays the world is changing a bit. You now have this thing, the cloud. Cloud computing obviously means that people, companies specifically use more and more infrastructure that's not really hosted by themselves but by a cloud provider. You just buy some storage or some compute or some database transactions, whatever, just as a service from a bigger cloud company. These means that also the cloud provides themselves, they are provided like the support for their software. You just like sign up, giving you credit card number, whatever, and then you have access to, I don't know, an object storage or whatever you bought as a service. So there's no real need to buy support from the developer of the software. That's the tricky part. And this is why quite some companies lately decided to change their licenses. So the third example is Redis. Was really a nice open source company. Super popular, a lot of people use it. But they are lately added the common clause which sounds just like whatever, small modification. But in fact, they added this common clause which means that it's forbidden to sell the software of a hosted or consulting around the software without a commercial license from them. So this is a restriction they introduced in their license which conflicts with the four freedoms I showed at the beginning from Richard Stallman and also conflicts with the OSI open source definition. Because the OSI definition also says there should be no restrictions around use of the software. So this common clause here makes Redis proprietary and no longer open source. Which is a bit tricky, obviously. Because we all like your open source and there's no longer open source. Second example, MongoDB, they did something similar. They called it service site public license. At the end it's the same, just adding additional restriction to the license which makes it proprietary. Cockroach, DB, same, in that case it's called business source license. No commercial services are allowed without a commercial license from them. Okay, so why is all of this happening? Because these companies think that they are no longer able to sell a lot of support subscriptions in the future because everybody will just use this software as a service from the big cloud providers and no one needs them to support subscription anymore. Yeah, yeah, I think I should repeat it. The question was that MySQL had a source available, MariaDB, MariaDB had a source available license. I mean, there are many creative ideas how a software license could look like. This is an example where you say, hey, our customers can see the source code and have certain rights. Maybe another right to change it, I don't know. But it's still conflicts with the open source principles and free software principles, so it's no longer open source. I'm sure that this source available license also says that you lose the right once you stop paying, I guess. So yeah, this is not open source, that's not free software. How can I go? Okay, yeah, I'm interested in experiments. I like experiments, but the fact is still that this is no longer open source. There was this article here on TechCrunch, which triggered me a bit, I have to say, because it says that the common clause stops open source abuse. Written by this journalist, which is a bit weird. I come to that later. Yeah. So this is a bit of a weird headline because as I said earlier, common clause is no longer open source. So how I should turn something into not open source, stop open source abuse. So this is basically, hey, we save open source by killing it, which is a bit weird. But of course the topic of my talk today is why copy left licenses like the GPL are actually great for businesses. Again, they had some examples, the biggest open source companies in the world, like Red Hat or Suze, actually don't need license changes. They're making billions with support contracts without owning the code. But there are some people who say, no, no, that's no longer work in the cloud age. You really need to have these license changes. I disagree. So what are the benefits of having a GPL software as a startup, as a company, as a business? So from my perspective, the first big reason is that it creates the best communities. Next cloud, for example, is, and this is the same for any big successful open source project, is really written by thousands of volunteers. In the next cloud case, we have over 2,000 volunteers adding code to this product. And they're only doing that because our contributors think, I don't really think, they know that they're not second-class citizens, that they're real contributors and they keep the ownership of the thing they did if they implement a feature, it stays their feature, and everybody can use it and no one ever can take it away from them. So we as a company, for example, we cannot do any license change or something. We have shared ownership of the code, same as with the Linux kernel, or like a Linux distribution, it's shared ownership of the code by thousands of people, and it's not really owned by the company itself. And this creates a happy and a good community, in my opinion. And that's only possible with the GPL. If you would do some other things like, hey, you can sign this contributor agreement and you give over the ownership of your code to the company and other weird constructs, it doesn't really feel fair. So from my perspective, the GPL, as it exists with the Linux kernel, with next cloud, and the other cases is like the fairest for everybody. Second is, if the current maintainers really do a bad job, then someone can always fork it. So if, I don't know, we, for example, as an excellent company, we do the releases. For example, we maintain the website and stuff like that. If people think that we are doing a bad job, someone can just take it, rename it, and start a fork. And the fork is then, of course, under the same license, again, GPL. There's no special rights. If the fork is better, then the fork survives. But that's something that's really a good thing about the GPL. And it just enforces the best results. Another very important thing is that if you use standard open source licenses, like the GPL, then this global upstream, downstream network works. But as you know, the open source infrastructure nowadays is super broad. If you just look at GitLab, GitHub, somewhere, you have so many tiny projects, so many people who do amazing things. And if you use compatible licenses, then you can take code from there. And someone else can take your code and build something else on top of it. It's really standing on the shoulders of giants, and this only works if you have proper GPL or proper open source software. If you come up with your own license, or it's proprietary, then this all doesn't work anymore. Next argument is working with the best developers. So if I do job interviews, for example, I hear a lot that developers like to work, like for NextCloud in this example, because they like open source, and they like to write open source software as a job. They know that what they're doing as a job has a meaning because millions of people all over the world use it already, and they can even use it themselves at home if they want to, or in the soccer club or whatever. If they just write proprietary software for a proprietary company, then who knows what happened with it. And we saw this many, many times then. People write some software for years and years, and then it's just discontinued or something. So it's open source really has a bigger meaning. Yeah, that's not a great example. Yeah, exactly. I mean, your name isn't the license header, for example. It's in the credits. You can show it to people. You can say, look, this is what I did. It's under your name. You can be proud of it. If it's just work for some company, then you might not have that. Yes, trade, yeah, that's not a great example. Yes, yeah, you have your reputation because you're doing good work and you have a bad reputation if you don't do good work. And it doesn't matter who your employer is. It's very important. Next example is building an ecosystem. That's, of course, also very, very important nowadays. So with Next Cloud, you can build apps on top of it like plugins or applications on top of the framework. People only do that because they know that, well, Next Cloud itself is GPL, which means you cannot go away. The licenses are clear for everybody. It's like a good foundation for your work. It's not, you're not building on top of some, I don't know if you're building an application for, I don't know, iOS, for example, you never know how Apple will change their terms like next month. If you're building on top of a GPL stack, then this cannot happen. Obviously, then you know that your investment, your time is useful and will survive. Next example is that GPL enforces good community governance. It's so fascinating how many projects actually need like complicated foundation and steering committees and stuff like that, where the GPL just enforces some of those good behaviors directly in the software license, which I think is also a very important point. That also from a customer perspective, they're like it. I mean, we have many customers because they know that there's no vendor login, so they can keep on using it even if they stop the support contract. That's very, very important. Which leads to fair relations and to happy customers. You hear a lot in the business world that this is a bad thing because you really want to force your customers to renew and to pay for it. I don't know, for me, this is a bit weird. Usually you have customers because they like the product and the company and the contract and everything. If you need to force them to pay, then that's usually not a sign of a good product in my perspective. Then the whole topic around owning the code. So from my perspective, owning the code itself is not important at all. Like, exactly, we as next slide don't own the code. It's distributed ownership. Same with Linux, with many open source projects. Owning the code itself is sometimes needed for weird business models like dual licensing, which I think is a bad idea anyways. And owning the code has some other benefits. For example, if you're an investor and you want to have a big exit at some point, then you want to have lots of money for your startup, then you need to sell something. And usually then it's a good idea that you can sell the IP of the product that increases the value of your startup. But yeah, as an open source community, you don't care about that. As a company which is more built for sustainability, you also don't care about that. So my perspective, owning the code is not important nowadays. Customers also like that they have the flexibility to take the code and do whatever they want with it, deploy it, change it, patch it. That's something they also like a lot. Contributions flow in and out of the companies. That's also a very, very good thing. We as NextCloud, I talk too much about NextCloud here, but we have customers who develop themselves additional features in NextCloud and then contribute it back upstream so it flows directly into the product. The same happened with universities or students. We have a lot of people just change it and then basically contribute it back. And this is so nice. This is how it should be. This is where basically everybody works together on the same software component. And this is another thing that JPGPL really encourages. Other licenses like MIT really can lead to more fragmentation because you don't have to really open source your change. Another aspect that's also not mentioned enough in my opinion is that a copy left licenses, all these open source licenses, MIT, Apache, BSD, GPL, HGPL and so on, they're really well understood. They're really discussed by lots of lawyers in court for like 25, 30 years and everybody really understands what I mean. Try to understand the terms of service of a cloud service you use that changes like every week. I would say this is actually a risk because you really need to pay a lot of lawyers to actually check all the contracts you sign while at GPL, everybody knows what the results are. For a second. You don't have a lot. Yeah, exactly, that's another point, yeah. Yeah, a copy left software can never disappear. Even if the current maintainer or company disappears, the software still sticks around. This happens a lot. And now I want to give you some real world examples how this really all, what influences we really had from the own cloud and next cloud experience. So the first example I want to give you is the Contributor License Agreement because as discussed earlier, the own cloud times, we had a business model based on dual licensing and also open core. I think I made it pretty clear already today that I don't like it, but this was a decision at a time. You can only do dual licensing and open core if you have the ownership of the code, which means everybody who wanted to contribute something to own cloud really needs to sign this contract to give the company ownership over the code the person wrote. Obviously this was a big blocker because a lot of people just want to quickly fix a bug or do something here and there. And then you say, okay, thanks a lot. Please sign this contract first. This is irritating and people don't like that. Also, it feels a bit weird. So why do I have to suddenly give this company ownership over the code? I only want to do a small change here. Yeah, and it just creates irritation. You look at it and you don't know what's happening here. And I think the other model, again, as we do this next cloud, Leung's kernel and other examples are a lot better where everybody just keeps the ownership of the code that contribute. Just everybody has this shared understanding about the license. We are working together under, I don't know, GPL v3 plus here or something. And everybody agrees on that and that's all. There's no contract needed, no lawyers are needed. I think it's a lot better model. And we changed that in next cloud. Next cloud has this shared ownership. We no longer need any contracts. I need license agreements, anything anymore. Thank you. And of course this directly resulted in that we have more contributors, more happy contributors, better community. Second example is that a lot of employees were also quite unhappy with the situation because the roots of that own cloud comes from this open source community and a lot of students and just enthusiasts there. And then lots of them were then hired as employees then because hey, you want to do what you do as a volunteer, you want to do it as a job, you get paid for it, great. So we had like lots of those engineers that came from the community. Yeah, and they were quite unhappy when they realized that they really had to write proprietary software sometimes, enterprise features that are not an open source product. So yeah, again we changed this next cloud everything. Employees to a next cloud is open source and they like this a lot better than we have a better team in my opinion. Another example is also that the customers were quite surprised because at the time we really advertised hey, own cloud is open source, use it, use it and then the customers using it getting a support contract and everything is great and then deploy it and then we had several cases where then the customer decided to stop paying, cancel the contract and then they get a nice letter from the lawyer which says like yeah, great. You can cancel that but then please downgrade your enterprise addition to the community addition and by the way you will lose all those features and the branded clients we built for you they're also proprietary, the source code you can use open source if you build it on your own cross-platform that's fine. If you want to use our binaries then they're proprietary and so on and so on. So they really made it really hard for the customers and again unhappy customers usually a bad idea or something we changed it next cloud. Okay, so I now brought you all these examples. Why I think that GPL is a better license for a software company. So why is this then happening? Why is then Radis and Mongo and all the others why did that change this of their license? What's the explanation for that? And again, coming back to this slide my surprise was already revealed earlier because the author of this article is actually as mentioned earlier actually a venture capital person who invests in all this open source companies here and want to have the maximum return of investment and of course the maximum return of investment you have if you sell like software licenses for example or if you own the code or you have all those other like hockey stick gross effects that venture capital like but that's not necessarily the interest of an open source community of an open source company who cares more about sustainability than the big exit. Okay, which comes, leads me to the end. I think a lot of open source software including next cloud only exists because of the GPL and because of open source. This is nothing we should fight or change or question because of the opinions of some investors in my opinion. Open source and the GPL are a great thing and they can really work for businesses and I don't think that we need to give up all the four freedoms for some fast money or some fast IPO. I don't know what I want to do because GPL actually works if you use it right. Thanks a lot. Test. So the four fundamental freedoms of free software are zero indexed. So it should be zero, one, two, and three. Oh really? Yes. Of course, of course, yeah, obviously. It seems interesting to consider the topics described in this talk and late of the keynote address this morning. I don't know if you had a chance to hear it. Basically it was talking about the problems particularly that smaller projects have in this era where anyone can incorporate your freely, your open source software into their larger project and then it becomes part of the bill of materials of some critical infrastructure and does that incur new obligations on people who are maintainers of small projects who don't even know what other people are doing with their code? So who knows who's using NextCloud but do you ever worry that you might have liability or responsibility for people who aren't using it for purposes that you don't even know about? Okay, big question. So the main question at the end was if I worry about liabilities, if some people use it for something I don't want or basically that's what you mean, right? Yeah, yeah, I mean this is an interesting question. I mean the four freedoms and also the open source definition that clearly say there should be no usage restrictions. So from a license perspective it's pretty clear. We cannot do any restrictions that who can use the software and who not, great. But of course the world is a bit more complicated obviously. So first example was lately, like two months ago or something there was discussion in our translator community because NextCloud is available in over 90 languages and they're all translated by volunteers and there were some people who basically came up with the idea that we should now delete the Russian translations from NextCloud because of the current war which led to a lot of discussions what to do. At the end the consensus in the community was that like the enemy here who is attacking Ukraine are this are not, this are the government. Government is the problem, not everybody who speaks Russian because that must doesn't make any sense to remove the Russian translations which you put in wouldn't mind. That's not the thing. And yeah, we don't want to hurt the people it's basically about the government. But this was an interesting question. A lot of people in our community didn't want that NextCloud is no longer used in Russia which you cannot really restrict it easily. Another example was that I was contacted by a journalist from the BBC like two years ago. He wanted to do an interview. I didn't really know at the beginning what the topic was. During the interview it then became clear that she really wanted to talk with me because in some research I figured that ISIS was using NextCloud. Yeah, not a very interesting situation. Yeah, I'm trying to. So again I didn't do this research this was done by this journalist but it seems that I found some server somewhere on the internet who was somehow connected to ISIS and was running NextCloud on top of it. And the journalist didn't really understand the concept of open source at all. She somehow thought that this is a customer of us or something which is obviously not. Yeah, from my perspective it's a similar situation as if they would drive a Toyota probably not really the fault of Toyota or eat some more drink, some specific drink not for the fault of the manufacturer of the drink. But it's an interesting question, right? So if I would be in Richard Storm's position in the 80s with his four principles it's possible that I would have added a clause which says something like you cannot use it for, I don't know, dollar bet. Of course obviously the problem is to what, how bad this should be defined. But yeah, it's an interesting question. At the end you cannot really restrict it. Yeah, that's another, that's a good point, yeah. If you would add to our license that whatever dollar ISIS is not allowed to use the software obviously they would ignore it, right, a good point. Yeah. And of course with other software that's more popular I don't know Linux or Android or others. I mean, everybody assumes that bad people use the software too. I think no one complains to Google that Android is used by criminals. But yeah, it's an interesting question. This is an old topic I think when we first met had you on a panel and we were talking about data encryption and privacy and open source. So when you're selling to, I'm not selling but convincing governments to use open cloud, data sovereignty and all that. And then companies that have proprietary technology say that they are more secure because X, Y and Z. How do you, you know, change their minds? Yeah, that's another big question. It's so interesting that if you talk about security that a lot of people think they have different opinions what security is. We for example, we really care a lot that we have a proper patch process that we have a security bug bounty program. They'll get like, I think 10K at a moment if you find like a serious problem in X cloud. Then we have a way to report it. We roll out patches, we do CVEs and so on. But of course that's only the part that we can influence. There are some things we cannot influence. For example, if you run it on a server that is insecure or if you don't have the latest patch version installed and so on and so on. So to have a secure next cloud you need to have no problems in the software but you also need to run it in the right way which makes it tricky. An argument that I heard before was that so you're saying this is more secure than Dropbox but Dropbox employs 1,000 firewall admins and we are not so they're more secure. So this is a complicated question. It really depends how you define security. But the question pops up, still pops up, it opens up, it's more secure, less secure, it's still a question. I thought that it's consensus nowadays and that transparency is good for security so it's better to open the code and everybody can review it than to hide it but it's still a question all the time. Yeah we try to make it as easy as possible to make it possible to run next cloud in a secure way. We have all these checks on the admin page which is like hey this component is missing or this hard disk is running low or whatever like all the tips to run it. But yeah, any other question? So I'm actually hosting a next cloud through a provider up in Canada. Aside from asking them and needing to learn a little French to talk to their support. How do I find out whether or not they're paying upstream, right? So they're obviously running and supporting the community and that they're using it and providing a service but how do I find out if they're working with you and working with the company or the community in general as good players and hopefully also a little financially? That's a very good question, that's a very good question. We are planning to introduce something that you can I don't know maybe use a certain logo only if you basically have a support contract or something like that but an interesting challenge. At first I'd like to thank you for your contributions to open source. I guess not necessarily a question for you but I'd like your take on it is I'm developing a deep learning model using GPL tools. What does that then mean for my model? I have no idea. Ask a lawyer. I don't think it has an effect to be honest. I think the GPL talks about derived work which I think is the code. Not the data you process with the code but I'm not sure. Yeah, that's kind of GitHub co-pilot is proprietary but it's based on open source code is input right? And they haven't gotten sued yet I guess. They what? Yeah, maybe we'll have a GPL v4 with this explicit thing about that. But my question was about like activist licenses. I think there's one called anti-996 that's trying to prevent like awful working conditions in China and I guess it's not open source because it puts some restriction on the people using it. Is that your, do you agree on that? It's so hot. I mean as you said it's not open source. These restrictions make it not open source which is a problem because then suddenly you're incompatible with all your other parts of the stack and has other problems. Like morally I can fully understand this how to do that. I'm quite an idealist and there are many things I don't like in the work, in the world. And if they use just I don't know the creation of Next.Community is something better, I hate that of course. I would love to put in some restrictions for evil stuff. But yeah, it's not really practical for many reasons. Cannot enforce it, you cannot really check it easily and it just creates just a bunch of other problems because you're suddenly not open source anymore. But yeah, it's a tricky one. Next Cloud will have its own long list of upstream projects on which Next Cloud is dependent. Do you have any special thoughts or practices or how Next Cloud is dealing with its upstreams? I'm not sure I understood. I mean Next Cloud depends perhaps on QT or G-Lib or things like that. Do you have ideas about good business practices for a company like Next Cloud in dealing with its upstream dependencies? This question is very big, I'm not sure where to start. Recommendations for good business practices regarding upstream components. Yeah, I mean, we for Next Cloud depend on many things. I mean, the server is written, like most of the course is written in PHP, so we rely on the PHP community. We need the database like Hostgres, MariaDB, MySQL. We can use Redis for caching, lots of components we rely on. What are many things to say? First of all, it's important that they are stable and reliable, they do not pick any dependency that it's like fragile or might disappear or it's not well maintained. So this is a better thing, this example is all fine. Sometimes we have a challenge regarding support contracts because if we sell support to our customers, we only sell support for Next Cloud, not for dependencies. They're not all GPL, that's correct. Yeah, I mean, the GPL is only, the usual understanding is that GPL is only binding if it runs in the same process. So for example, databases or, yeah, Redis for example, right? I just showed on the slide that Redis is no longer open source because I changed the license. We can use Redis, but we use Redis via like a socket. Yeah, it's of course a consideration. I mean, the Redis support is like older before the license change. It probably would have used different like NoSQL database if the license change would have happened before. So I'm not, we don't have any like strong proprietary dependencies. I think this would be a better idea because then the whole stack would become proprietary. Then it's useless that your free software on top if the stack below you is proprietary, but it's not the case. Something else? I think we're out of time. Thanks a lot. Testing, okay, cool. Alrighty, I think it's starting time, so I'll go ahead and start. Welcome everyone, I'm Ben Cotton. I am the Fedora program manager at Red Hat. This talk is your bug tracker and you, if this is not the talk you expected, there are other rooms where your talk you're expecting might be. I like to start off every talk with a little bit of just basic housekeeping. So I use he, him, pronouns. The usual disclaimers apply. I just introduce myself as being a Red Hat employee who works on the Fedora project, but this is all my opinions. It doesn't actually reflect what Red Hat or Fedora thinks. Unlike other talks I've given, I agree with all of my opinions this time, which is kind of cool. This talk is under the Creative Commons Attribution Share Like 4.0 license. Please feel free to do with it what you like, subject to the terms of that license. If you have kind things to say, here's my Twitter handle. It's also in the bottom of every slide. I'd love to hear your feedback or the funny lines that you thought were the best. And if you have unkind things, you are of course invited to keep those to yourself because nobody's gonna be harder on this talk than I will. And I would like to keep questions and not a question, but really more of a comments at the end. I will probably talk a lot faster than I did when I rehearsed this, so we'll have plenty of time. I do wanna put in one shameless plug real quick. Much of the content that you will see in the next hour is also in this book that I just published. I happen to think it's pretty great, but I will not talk about it anymore. I'm not here as a sales pitch. What I am here to talk about, I'm gonna talk about why you want to have a bug tracker. That will be a very short section because hopefully it's gonna be pretty obvious to all of us. We'll talk about what you want to track, how you can pick a tracker, and then how to triage, close, and analyze bugs. What we won't be talking about are fixing bugs. This is very much not how you interact with the bugs and you fix bugs in your software talk. This is about the project or program management level sort of tracking of bugs in the aggregates. We really won't be talking about individual bugs almost at all. We won't be talking about specific bug trackers. I have opinions about some of them, but mostly all bug trackers are terrible in their own way. What we'll be talking about is how you can pick one that's appropriate for your community. So with that, why do you have a bug tracker? To track bugs. I mean, that's on the label. But realistically, bug trackers are the cornerstone of building an open source community. You can have a lot of things, mailing lists, chat programs, forums, all kinds of stuff, but you need to have a bug tracker. Because a bug tracker is the way that you communicate with your users at the most basic level. People tell you, here are the things that are broken in your software and you're like, oh gee, thanks. But it's also how you coordinate with contributors. If the project is more than trivially sized, you probably have multiple people working on it and you have multiple bugs, and so you don't wanna all be accidentally fixing the same bug. That's not a super productive use of your time. It also, your bug tracker is a historical record, so it builds a source of information about the project. It's not a particularly consumable historical record, but the raw data is there. I kind of liken it to like an archeological dig. You have the experts that go out and actually dig up the artifacts and analyze them and then turn it into consumable history, and the bug tracker is the same way. You're not gonna learn something by having everyone go through and read all the historical bugs, but for the people who are wanting to do the bug archeology, they can put together a useful history of the project. So I think we're hopefully all in agreement now that we should have a bug tracker. So what is it that you want to track in your bug tracker? There are some kind of obvious things like the type. Is it a bug? Is it a request for enhancement? Is it a question? Is it spam? You want to be able to know what these things are. You want to be able to track the status or state. Is it open? Is it closed? Is it in progress? Is it on testing? There's lots of different things that you might use, but at the basic level, is it open or closed? Is a really important thing to be able to track. You want to be able to track the type of closure. Did you fix it? Did you say, nope, we can't fix it? Did it go end of life? Is it invalid? Things like that. We want to be able to track the impact. And the impact is sort of how important it is to the user. Like, is it something where it's like, this is kind of annoying, but I took the time to report the bug. Or is it like, hey, this is gonna delete all of my videos of grandma's 95th birthday party. Those are useful things that we'll talk about more in a little bit on how you would use that, but that's an important field to have. And the priority is, you know, basically the internal, you know, how quickly are the developers gonna work on it? And you can be really tempted to have priority be like, you know, a very strict ranking of this bug and this bug and this bug, but that falls apart pretty quickly at scale. When you have a hundred or a thousand bugs, you're not gonna sit there and individually rank them. But having categories is important. You want to think about, I track the version. This isn't applicable to all software projects, but most of them have some sort of release system where there are versions. And you might want to know, is it in version two or version three because, you know, how you deal with that might vary. Some projects like Linux distributions, for example, are very large and have many components. Or it might be an application where you have like a web layer and a database layer. So if you have those different components, you want to know, you know, be able to track which part it's in because that helps you kind of route it to the right place. And also, you know, the platform might matter. So, you know, operating system and architecture, sometimes bugs are very specific to one of those. And even when it's not, as we'll talk about later, just knowing that information can be really useful in terms of what you learn about your community. You also want to track time stamps. And the nice thing is time stamps usually just sort of happen automatically in almost every bug tracker that still exists in these days. But, you know, it is useful to know how old is this bug? You know, is that a five year old bug? Maybe it doesn't actually, the bug report isn't valid anymore because it got fixed and nobody updated it. Things like that. You want to know who reported it. And again, this feels kind of obvious, but like you need to be able to reply to that person. And so the way you do that is to know who it came from. And then an assignee who's working on it. And there are probably more things that you can think to track. We've gone through a lot. A lot of them seem very obvious. But there might be other things that are sort of specific to your community that you think about. The really important thing here is that you want all of those things we talked about in the last three or four slides or whatever to be separate fields. Not like in the content of the bug. Because it turns out that if you try to get people to write machine parsable, free form text, they're going to be really bad at it. You just can't, even with some natural language processing, you're going to have to really work if you have to extract it. So having each one as being a separate field really makes it a lot easier for you to do things with that metadata later on. And if you're lucky, the tracker provides all these fields or at least the ones you need already. So let's talk about picking a tracker. These are some things, like I said, I'm not going to talk about specific bug trackers, but these are things that you want to think about as you're deciding what to use. If your community already has a bug tracker that you're using, cool, just keep using that because the cost of switching is probably going to be way more than the benefit you get from switching. But if you're starting a new project or something is forcing you off your old bug tracker, these are things to think about. So you want to think about authentication and accounts. This is both, do you want to be able to use your project's own authentication system? Large projects often have their own account system for email addresses and various internal tooling and stuff like that. So it'd be nice if you can use that in your bug tracker too. It's just one less account for people to manage. On the other hand, a lot of your users might not have a project account because they're not really involved in contributing to the project. So you might want to make it so they can log in with a local account or use GitHub or GitLab or Facebook or Google or some other authentication provider to be able to log in. You want to think about if they have API libraries, so almost every bug, I didn't even put it on here because it feels super obvious, but being able to interact with an API is great. If there's already a library for it, then that makes it a lot easier for you because you don't have to write it, you just import Python, Bugzilla, and you can do things with Bugzilla in your Python without having to reimplement everything. You want to think about its status as free and open source software. And this is a question for you and your community to decide. Some communities absolutely have to have free and open source software, they will not use anything else. And if that's the position of your community, that's fine. Other communities don't care or maybe have a weak preference for FOS software, but whatever bug trackers you're looking at, you want to make sure that whatever your community values, your bug tracker reflects that. Similar to it with hosting, most projects these days, when they're starting, they don't want to host their own infrastructure anymore. And I'm a big fan of not hosting your own infrastructure unless you have to because generally, that's not the expertise of your community. You're not here to write your own version control system and then develop your cool widget. You're here to develop the cool widget. But some projects do have a preference or a technical need for being able to self-host. And if that's the case, again, you want to pick one that allows you to do that. There are some things regarding safety and confidentiality that you want to consider. Hopefully your project has a code of conduct and part of having a code of conduct is being able to enforce that, which means hiding or deleting comments that are abusive or otherwise violate the terms of precipitating in your community. And moderating or blocking the accounts of people who are frequent violators. But there's also, there might be a need to submit a private bug or a comment or attachment because like I say, it's, you're attaching some logs or a core dump to help the developers figure it out. You want to make sure that the tracker that you pick supports that, you know, big fan of all bugs should be publicly viewable, except for the rare exception when they shouldn't be. So we talked about a whole bunch of different things that you would want to track. And the odds that a tracker has all of them built in are pretty low. So you think about, do you need customizable fields? You know, can you create like, hey, add this field to be this? And set like, you know, have it be a date or a Boolean or whatever. You want to think about the access controls on the metadata. So basically all of the things we were talking about in the previous section, who can change those? Some bug trackers, basically any authenticated user can change the version or, you know, change, you know, certain other metadata, but not everyone can change the subject or the bug report or comments or things like that. Again, this is something where there is not a right answer, there is your community's answer. There are things that, you know, you might want to make it so that people can change some of the basic metadata because you have 20 different people all experiencing the same problem. And that way one of them can say, oh, it's still in this latest version too, when you don't have to wait for the reporter or an assignee to make that change. But there are things you might not want anyone to change unless they're sort of inside the project or things like that. Things like linking to version control or other bugs. You know, if you make a git commit, can you reference the bug in your bug tracker and have it automatically make a comment? That's super useful. But a lot of times, you know, projects may not care or, you know, the systems may be separate enough that you don't want to plumb that together. But, you know, for things like Fedora, we use Red Hat Bugzilla and we can, a lot of times, we add links to like an upstream tracker because it turns out the bug isn't in the Fedora package. It's in the upstream project. So we want to be able to track that but also refer to the upstream bug because that's where it's probably gonna get fixed. Bulk updates. Those are a thing that, you know, we'll talk about closing bugs. If you have to do it at one at a time, it gets super painful. So having the ability to, you know, either through a scripted API or ideally through a web interface makes your life a lot easier when you have to make bulk changes. Things like milestones and epics can be really useful if that's how your community does development. Being able to have that in the bug tracker and assign bugs to a particular milestone or helpful if your community doesn't use it, you may not need it and you don't care. You also want to think about the reporting capabilities of your bug tracker. We'll talk about this more a little bit but being able to just generate some tables of like bug count by version. Being able to do that really quickly is useful. Being able to, you know, graph bug counts or whatever over time can be really helpful. You'll probably end up doing your own custom reporting but if it has some stuff built in that saves you some time. More things to think about. Does it have a default assignee? So you might want that for particular components, it assigns it to a specific person and then that person is the assignee until they pass it off or somebody else claims it. Or you might be like, we don't want a default assignee, we want it to be explicitly unassigned because that communicates that, hey, nobody's looked at this yet. Again, this is not the right answer just your community's answer. Does it support multiple assignees? One of the cool things about open source software development is that the person who is assigned to a bug is often not the person who actually comes in and fixes it and most people just don't bother to update the assignee because they just rather fix the bug and move on, which is cool. But also if your project does pair programming or things like that, you might have multiple people working on it or you might have a development assignee and a QA assignee or a documentation assignee so if that's the way your community works you wanna make sure your bug tracker can support that. Being able to mention other users is super helpful. You might be like, hey, so and so, can you take a look at this? Or you might be like, we don't want that because that's a vector for harassment and it's not worth the trouble. So again, think about that. And then being able to watch or subscribe to bugs or to components or you don't necessarily want to interact with the bug, you just wanna get updates. You wanna see what progress is being made or see if somebody commented on it. Sometimes you might be able to, you might want to subscribe other people to a bug without their permission or you might want to not allow that. We do that a lot in Fedora and it's actually really helpful to be able to add somebody but sometimes it does get abused and people tend to delete Bugzilla email a lot more than I wish they would. So one other thing to keep in mind is that whatever you end up picking as your bug tracker, you're not gonna just use it as a bug tracker and that's not just the request for enhancement that you get but it's also, you might just use that as like a Q&A system so people have questions about how to use your software and they come to the bug tracker because that's what's there and so you use that to interact. You might use it for like internal governance to conduct votes on who gets the commit bit or how you make decisions on spending money in the project or whatever it is. So just sort of keep in mind that no bug tracker ever gets just used as a bug tracker sort of in the same way that spreadsheet software is used for way more than just spreadsheets and presentation software is often used for like poster design and all sorts of graphical design stuff that really shouldn't be. Your bug tracker is gonna be the same way. All right, so you've picked out your bug tracker, it's cool, people are now submitting bugs, now you need to figure out what you're gonna do with these bugs. So we're gonna triage your bugs and on the first pass really what we're doing is we're just sort of sorting them into making sure they're going in the right place. So there's some questions that you ask as you go through each bug. The first question is, is this a bug? If it's not, does it belong in your bug tracker? If you're using your bug tracker as also a way to track requests for enhancement, then cool, these tag it the right way. But if it belongs somewhere else, then you either move it there or you tell the other person, hey, this isn't where it belongs, please file or whatever in this system. You wanna ask, is it a duplicate? This duplicate bug reports never accurately reflect the number of duplicate bugs because it's just at a certain scale, it's really hard to know the state of every bug. So you hopefully can like search and maybe it'll even suggest, hey, this bug looks like that one, but otherwise you just kinda do your best, like market is a duplicate if you know it's a duplicate. You can ask, is it a security bug? You, we'll talk more in like how we prioritize on the next slide, but you might do certain things with security bugs, tag in your security expert or just add a particular label to it or something so that you can track those sort of things later. Is it filed in the right place? So if you're developing Python and somebody comes and reports a bug in some application that's written in Python, it's not actually a Python bug, so you send it to the app, it's an appropriate project. Similarly, if it's a bug filed against your project, but it's actually, the bug actually is in the upstream library that you're using, you probably wanna direct people towards that because you're not gonna be able to fix it yourself. Then you ask, is the bug reproducible? If it's not reproducible, that's okay potentially, but it's definitely a lot easier to fix reproducible bugs because then you can tell that you fixed it. And if it's not, you might ask for more information and we'll talk a little bit more about like potentially closing those later. And then does it say what happened? It broke, not helpful. But if it actually says what happened, then people can start working on it. And if not, you can go back to the reporter and say, okay, so what was it that actually happened? What did you expect to happen? Those kinds of questions. So now we've got everything sorted into the right place. So now we're gonna start figuring out which ones to work on first. And no matter what rules you have in your community, people are gonna work on the bugs that are interesting to them and within their skills. But you can at least sort of generally set a priority that indicates to the community what it is, what should be worked on first. And so like I said earlier, it's really tempting to try and assign a numerical score and have a very strict ranking. That's not how people work. So low, medium, high is a good priority scheme. So basically you wanna go through these questions and the sooner you get to a yes, the higher the priority. So the first question you want to ask is can physical harm result? And you might think, well, this is software. What physical harm could result? But it turns out that software usually runs on hardware and hardware can do things like get really hot and cause fires or there was an example a few weeks ago where a chess robot grabbed a seven-year-old's finger and broke it. So if you have a bug that causes something like that, that's a really important thing to fix. But it's also, software is everywhere in our lives now. And so if you have an application that's used to get support for domestic violence or make other reports where if the presence of that report gets into the wrong hands, it can cause real world harm to people. So these kinds of bugs are always the ones you have to fix first. So if you answered no, hopefully, then you move on to the next question. Does it cause data loss or corruption? People will be very reluctant to forgive your project if you have deleted the video of baby's first steps or pictures from somebody's wedding or something like that. If a company is using your software and they lose their customer records, they're not gonna keep using your software. And if you answer no, we move on to, does it permit unauthorized access? So this can be sort of a duplicate of the invisible harm result, but it also just, does it turn your laptop into a Bitcoin miner or things like that? These are, again, important bugs to fix. And then, does it reduce accessibility? And I think, and I'm just as guilty of this as anyone else, but in open source, we do a really bad job, generally, of making our work accessible and our community's accessible. And so anytime we make this worse, that's bad. So really, these first four questions I would kind of put in the, these are the high priority things. If you answered yes to any one of those, it's a high priority bug. So the rest of them are kind of maybe medium priority and again, it's very fuzzy, but is the default configuration affected? Because most people aren't gonna touch the defaults or they're gonna lightly touch the defaults. And so if the bug only appears in a very heavily customized installation of your software, it's not that it's not important, but it's going to affect a lot fewer people. Similarly, if there's a reasonable workaround, then you might be less likely, I guess, this one really should be, is there not a reasonable workaround? And reasonable is an italics here, because if the software is designed for casual users, telling them to go download this patch out of our Git repo and recompile while standing on your head and make sure you're holding down the enter key at the same time, that's not really a reasonable workaround. So if it's something that your average user isn't going to be able to do on their own, that's, again, more likely to affect more people and you need to fix it sooner. Does the bug affect all platforms? So if it's only on Windows 3.1 running on I-286, probably not a big issue for you in 2022. But if it's all of 64-bit hardware, that could be a problem. If it's all Linux distributions or just any operating system, it increases the number of people that are probably affected and the sooner you want to fix it. And then the last question I have here is does it embarrass you? And this is not in the sense of, boy, I get embarrassed anytime I write a bug in my software, but it's, like, do you have a splash screen that has your project's name spelled wrong? Like, it doesn't actually affect people's ability to use it at all, but when the tech press does a review, they're gonna be like, wow, these guys can't even spell their own project. So that's kind of embarrassing. If you answered no to all of these, then it's probably a low priority bug. So now that the bugs have been prioritized, your community can go through and get all of them fixed and everything is great and we'll talk about closing bugs. So this part is a little contentious, I think. There are definitely varying opinions on how to close and when to close bugs. My take is you totally should. If you disagree, I look forward to your talk, but for now we're gonna go with this. So the reason that people will disagree about it is closing bugs sends a clear signal of intent. This bug is not going to get fixed. For whatever reason, don't expect this fix. And clear communication is very important. It also makes the bug tracker easier to scan because if you have 1,000 bugs and 90 or 900 of them aren't gonna ever get fixed, it makes it really hard to scroll through and see the ones that you actually care about. On the other hand, closing bugs when the bug report isn't fixed makes people angry. And I know that because every six months I close 5,000 to 7,000 Fedora Linux bugs and I see people on social media and I'm a mailing list and in the bug say, ah, you never fixed my bugs, I'm giving up and sorry, but we do have to do that at some point and it's okay for people to be angry but some people would rather avoid that conflict entirely. You might also have contributors in the community say, well, I was gonna work on that someday. Like, I will get to it, I promise. But they can reopen it in that case if they were gonna get to it. And there's also people that say that having open bug reports represents a catalog of known issues and that asterisk is there because it's known and also reported. Not every known bug is actually reported in your bug tracker because sometimes people just don't bother. And I do have a lot of sympathy for that argument but I think listing the known issues for major things is put it in the release notes. We don't need to keep the bugs open forever. So now that you all agree with me that we should totally be closing bugs and it's a great, how do we do that? Well, obviously you can close it when it's fixed. That's the happy case. Everyone loves that. If it's a duplicate or invalid or spam, as soon as you make that determination, close it. No need to keep spam bugs open forever. If the bug is missing information so you don't know what the bug report actually is, it can be closed. If you can't reproduce it, you might close it. You might leave it open and see if somebody else comes up with a reproducer. If the version that it's reported against reaches end of life, you say, well, we're not gonna fix bugs in this release anymore so why bother keeping the bug open at that point? And as a more general case of that, if you know that you can't or won't fix it for whatever reason, and this could be, just you've come to rely on this buggy behavior in other parts of the software and fixing it's too complicated, or if it turns out to be the space bar holding that down heats up the CPU like in the XKCD comic. But once you've made this determination, again, close it. But the important thing is that you want to have that communication ahead of time. People don't want to be surprised by this. You can't be like, oh, you know what? We decided that we're gonna close all the bugs. That's not a great way to engage with your community. Now you can have the policy and clearly communicate it in a bunch of different ways and people will still miss it. You can't show up at everyone's house and be like, here by the way, here's our bug closure policy. But you can at least make a good faith effort and do your best job of getting that out. When you close the bug, you want to explain why. Don't just close it and walk away. Say we're closing this because the version is end of life. We're closing it because we can't reproduce it, whatever. And then similarly, you say, if you think it should be reopened because it's still valid against a supported version, reopen it with the new version. Or hey, if you can try to away-frust-reproduce it, give it that. And ignore the complaints, because like I said, people will be upset when you close bugs that you don't fix. And it's not that you don't care, but you can't take it personally because this is what you have to do to keep your bug tracker manageable. So the process is gonna vary by the closure type. So I talked earlier about somewhere, once you make that determination, there's no value in leaving the bug open. Just close it. But you don't wanna be too aggressive with things like missing info or can't reproduce because just in the same way that all or most of your contributors are doing this on a volunteer basis, your users are possibly using the software on a volunteer basis. Certainly they're under no obligation to ever report bugs to you. And as much as it doesn't feel like it sometimes, they are doing your and your project a favor when they report bugs. So give them the same grace that you would want. If somebody reports a bug and the next day they're like, why isn't this fixed? You're gonna be mad. So if you go and say, I can't reproduce it closed, they're gonna be mad. So give them some time, understand that they may not be waiting patiently begging their email to have a notification that you said you worked on it. A week sort of feels like a minimum. Two to four weeks seems a little more generous because we all like to disconnect from our computers sometimes or sometimes life just happens. So give people time, maybe mark it with a missing info tag and then do a search for anything that's missing info and hasn't been updated in the last four weeks. Okay, that's when you go through and close it. All right, so we've gone through, we've closed all the bugs because they're either fixed or they're never gonna get fixed. And so now we wanna do a little bit of analysis on them. And why do we want to do analysis on them? Because it allows us to learn about the user community. It's easier than a well-constructed survey because it turns out doing a good survey is really, really hard. Especially when you don't have, like you don't know who's using your software because people can download it from wherever and they might not even get it directly from you. And so how do you know to get the survey in front of them in the first place? And most open source projects don't want to put in a bunch of phone home reporting because that's sort of antithetical to a lot of people's reasons for choosing open source software is they want to have that control. They don't wanna be phoning everything home to the upstream project. On the other hand, because it's easier, it's less scientifically valid, but for the most part, you're not going to present this at a scientific conference where you have to have a lot of rigor. We're just trying to learn some basic things so we know something about our user community. And so what we can do is we can learn some approximate things like what percentage of people are using this on Windows versus Linux versus BSD, which components are being used, how quickly are our new versions being adopted? And all of this depends on bugs being equally distributed throughout the project, which is probably not true. But as a first approximation, it can be really useful. If you see that 75% of your bug reports are coming from people running on ARM architecture, then either that's really popular in your community or there's something about the ARM architecture that is particularly buggy for your software. Either way, that's really useful to know. And then knowing how quickly people adopt new versions of your software might inform how you decide to do your release cycle. So in Fedora Linux, it's every six months and we see pretty rapid ramp up in the number of checks to the repo on release day. It shoots up pretty quickly and then it holds pretty steady until it sort of trails off at end of release. With enterprise Linuxes or Linuxese, you see a much slower adoption rate. And so people, they're more conservative, they're taking more time to test things and stuff before they really start deploying it. So depending on if you're targeting like community desktops or enterprise systems, you'll see different rates and you'll want to adjust accordingly. You don't want to have your software reach end of life when people are finally starting to use it. That's not super helpful. You can also learn things about your project. So you can learn about like bugs over time. Are we getting more bugs? Are they, we're getting more high priority bugs. We're getting more security bugs. And this can be really useful because you can, you know, bugs, bug reports aren't bad. Bug reports are a sign that people are using and engaging with your software. So an increase in bug reports could just be that more people are using it and that's actually a really good thing. But if you see that your number of bug reports is increasing faster than your contributor community is, that can be a sign that it's, you need to try and recruit more contributors so you're not overwhelming people. You can track things like time to resolution. So basically delta between when the bug was closed and when it was opened. Ideally that's pretty low and, you know, on a downward trend. And if it's not, you can look at, all right, do we need to do more to try and like triage bugs so we can close the obvious closures early? Do we want to, you know, try and have more testing so we're not introducing bugs in the first place, things like that? And you look at things like bug severity because if you're seeing a big increase in the high priority bugs, that could be an indication that something is going wrong in your development or your testing process and you want to try and catch those before they get out. But there are things you can't learn. One of the things that I would really like to know is how quickly are people getting a response and how quickly are we actually acting on the bug? It turns out that's really hard to do unless you sit there and like manually go through for each bug and say, all right, this is the actual like first wheel comment on it and that was three days after it was opened. Okay, we'll mark that. In a previous job I did that for our support engineering team and we had like a dozen or so tickets come in a week so it was pretty easy to do. When you're getting thousands of bug reports you can't manually do that. So it'd be really nice to know that but you really can't reliably calculate that. You also can't figure out who the best or worst bug fixers in your community are because remember we talked about the person who's assigned to the bug might not be the person who actually fixed it. And frankly, I don't think there's a lot of value in trying to do that because what's gonna happen is if somebody shows up as the worst bug fixer or just not the best, it's gonna be really discouraging to them as a contributor and it could just be that they're mostly doing other things that are valuable to your community like writing documentation or doing design or going give conference talks. So if you're trying to do that, think about your motivation and maybe find a different way to reward people for fixing bugs. You also can't learn affiliations. You might really want to know, hey, what companies are filing bugs and using our software because maybe we can go hit them up for some sponsorship money. And you can tell that some, but it's not reliable. There are a lot of people at Red Hat who work on Fedora and they do their Fedora interactions with a personal email address in Bugzilla because they wanna keep their Fedora contributions and their day job contributions separate, which I totally respect. But so like people sometimes ask me, how many people at Red Hat work on Fedora? And I say, I have no idea. And I'm certainly not gonna be able to tell from the Bugzilla email addresses. A lot of people who are participating on behalf of their employer probably don't use their employer email address because they maybe have been doing other stuff in Fedora and they don't wanna have a separate Bugzilla account for every different hat they wear. There are other ways that you can learn this sort of information, but the Bug Tracker is not a reliable one. So this is another fun one. You can challenge your assumptions because at one point I was looking through some bugs and I was like, man, everybody just marks the severity as urgent because of course it's urgent. It's the most important thing to them. That's why they came and filed the Bug Report. And then I looked at the data for about 15 releases. Turns out most people don't even set the severity. These are from Fedora Linux 19 through 33 bugs. Almost 75% of people didn't even set the severity when they filed the Bug Report. And of those that did, only a small percentage said it was urgent, which is like, this is actually about the distribution you'd expect. But I was very uncharitably assumed that people did that the wrong way. So you can think you know something and then you analyze the bugs and turns out you don't know anything at all. So when you're analyzing bugs, you wanna look for things like trends that will cause a change in behavior. If it's not going to change something that you're doing about how you develop or test or support the release, then why bother? It can be useful, like fun to just know cool facts, but it's not particularly useful. When you're analyzing bugs, you usually wanna ignore duplicates unless you're like looking at how many duplicates do we get? Because that can sort of be a clue of how hard is it for people to find existing bug reports. If you're seeing a big increase in duplicates, that probably means you need a better way to make it so people can tack on to an existing bug report. And basically you always want to ignore spam because you're really not ever gonna do anything like, oh, this is how many spam bug reports we got. Unless you're just gonna completely disable the bug tracker, you're gonna get spam, it's on the internet. When you're thinking about it though, you think about percentages or raw numbers and which one tells the story that you're actually trying to figure out. Going from 10 to 100 bugs between releases, that's a big number. Going from 1,000 to 1,090, that's the same difference, but percentage-wise it's a much different change. On the other hand, it's like say, you have one security person on your team who needs to sign off on every security bug. Going from 10 to 100 is a really big deal and going from 1,000 to 1,090 is also a really big deal because that's still 90 more bugs that one person has to deal with. I generally try and go with percentages for most things just because one release might have 10,000 bug reports and the next one we have 17 and so it's really hard to make comparisons between the two unless you're dealing in percentages. But there are certain times when you do want to be able to look at the raw numbers. All right, so how do you actually do this? A Jupyter notebook and pandas, those great tools. I really like, that's what I do for Fedora Linux bugs because I can check it all into a Git repo, I can share it with the community, they can make pull requests to improve things. Like when I recently discovered that I was calculating percentages wrong because basic math is hard, that's something that somebody else could have noticed and fixed for me. R is another great tool, you can use a spreadsheet, we've talked about how spreadsheets get abused. I don't think spreadsheets typically have enough power to really do some of the cool filtering and things that you want to do. But that's an option if that's what you have available. But being able to make tables and graphs is really helpful because then you can really get a feel for the trends and what the bugs are telling you. So I have like a half hour talk of exploring Fedora Linux bugs that I gave at Nest with Fedora last year. So that's available if you wanna watch it. I'll be doing an updated version next week at Nest with Fedora. So if you wanna sign up for that, it's free and it's online, it's our annual contributor conference. And it'll be a lot of the same stuff as in this one. I'd actually love to go over it with you now because I think it's really neat. But it's a 25 minute talk at very fast talking and we have about 14 minutes left. So I don't see how that would work. But as promised, we do have plenty of time for questions or more of a comment than a question really. So if you have either of those, I'd love to spend the next 15 minutes talking with you about it. So the question is, is there a plan to move Fedora Linux bugs to something other than Red Hat Bugzilla? That's a very loaded question. So, REL Engineering has gone on the record as saying that they will be moving REL bug trackers to Jira instance that Red Hat has. Candidly, I have no interest in trying to convince the Fedora community to move to Jira. One, I'm not super familiar with it anyway. I don't feel particularly fond about it. But also there's a lot of people in the Fedora community who would be very hard pressed to accept a proprietary solution. That said, as far as anyone has told me, Bugzilla will continue to exist indefinitely. I don't always find out about things ahead of time. You know, I said earlier that, you know, keep using the bug tracker that you have. Bugzilla is really awesome in a lot of ways. It's also very painful in a lot of ways. If we were starting Fedora Linux today, I do not think I would pick it. But we have 36 plus releases of history in there. And so just being able to go through and do some of this historical analysis of release by release is really helpful. There will be some discussion in the Fedora community about potentially moving at some point, but it's more of a what is it that we're looking for? And if Bugzilla meets what we need, then I think we'll stay there. So if I can summarize this correctly, I think the question is basically, why are there not more open source bug trackers that can be easily customized to fit the exact needs? Is that a fair character, is it okay? So I think one, bug trackers just aren't an interesting thing for most people to work on. You know, a lot of open source is people trying to scratch their own itch. And yes, everyone has a bug tracker, but nobody wants to use the bug. Like fundamentally, nobody really wants to use the bug tracker. Like we understand that it's a necessary thing, but I have not talked to anyone who's really excited about it. And then there's also just, like you mentioned, Arch is a very flexible and customizable distro. That's also why it's not necessarily the best for newcomers, right? Because with the flexibility and customization comes a lot of complexity. So there are a lot of open source bug trackers out there. There are a lot of proprietary ones. I think there's just a general trend towards using things like GitHub or GitLab, where you have both your code repository and your issue tracking all in one place, which simplifies a lot of things. And especially for small projects just getting started, being able to use software as a service is really big benefit to them. Just from the being able to get started on the thing they actually want to work on standpoint. There are obviously reasons that a community might choose to not use proprietary or open core software as a service. Sure. And just to summarize the question for the recording, or the comment, it was basically that a lot of open source projects like GNOME have moved to GitLab in recent years and been able to migrate their historical data. And I know GNOME is very happy with their move to GitLab. And I've heard the same from other communities. Any other questions or more of a comment than a questions? So the comment is GNOME saw an increase in contributors after moving to GitLab. And I think that's one thing that's really powerful. I don't know how much of that is because it's GitLab versus Bugzilla or whatever other systems they were using. But using a tool, and I think that's why GitHub in particular has become sort of the default place for a lot of projects is that there are a lot of people who already have accounts there and they're already familiar with using it. And so the barrier to new contributors is pretty low. And that's something in Fedora that we've tried to work with people on is lowering that initial barrier of contribution because as a 20 year old distribution we have a lot of historical things that aren't necessarily the way we would design it today but are very difficult to switch away from. And so new people have to get used to a lot of different tools or different workflows that they might not be useful to or used to. And so having that additional complexity and unfamiliarity does make it harder to get new contributors started. All right, Sherry, go ahead. Thank you. One of the things to point out to when you're constructing statuses for bug statuses GNOME is an interesting project because it really interfaces with the enthusiastic Linux crowd which can have very strong and sometimes toxic opinions. And so when we get, because there's always a difference of opinion between what the community does and what the project does. And a lot of times, I had actually worked on this because the bug statuses is important because they actually trigger people. So in Bugzilla, we have something called won't fix. And if you just imagine what that means, won't fix, screw you, you know, it's really what it, so a lot of times we have to change things into it. So instead of won't fix, out of scope, right? So that's something really important is you don't put things when you close them that trigger people and create emotional responses because they go back and then because of the addition of social media, they will use that to hammer their project on a regular basis because they're looking to, you know, punish the project for one reason or another. So that's a, it's one of those really crazy things that we had to deal with is people who get obsessed with their contributions not being accepted because they think it's so important to them. So that's another bit. And one last comment, emojis. So in GitLab, when we move to emojis, it's used for racist comments. So instead of racist comments, they put an emoji of a monkey or they do plus things to say, look, 500 plus is on this bug report. It must be important. And it's like it's out of scope. So those are other things to think about it, especially when it comes from a racial or equity kind of thing is it can be used to punish maintainers and other commenters. Thank you. All right, we have time for one or two more questions or comments or completely unrelated asides. I'll entertain whatever, it's fine. So maybe this one is also kind of opinionated question, but like if closing bugs is controversial and the counting bugs also is almost meaningless cause they mean different things. So why can't we leave them open? Like why is there a need to close a bug that's actually not addressed? The user is still unhappy, leave it open forever and you can manage expectations by saying if someone wants to work on this, Patch is welcome. Yeah, what do you think about it? So that's certainly a viable approach. I do think that at a certain stage like you know you're not gonna fix it and I think sometimes it's easier to rip the proverbial band-aid off and just say, this isn't gonna get fixed. Stop holding out hope that it's going to. It also does make bug analysis a little easier. One thing I didn't mention in the slides is that I really don't do any analysis on the Fedora bugs until after release has reached end of life because at that point everything is definitively closed and we can say sort of this is an unchanging set. And that said, if your bug tracker is useful to your community or if you have some way to like hide or deprioritize ones that you really don't expect to get fixed, it's not always time-based. Like you might, the bug that came in yesterday, we're probably not gonna fix that. Let's move it sort of down the pile. If you have a way to indicate that, that could be a valid thing for keeping bugs open longer-term or indefinitely. Another corollary question is like, how do you handle bugs that are present in upstream and you think it's responsibility of upstream to fix? So bugzilla, like the Fedora bugzilla doesn't have a good way to link bugs to other bug trackers? Bugzilla, Red Hat Bugzilla does now have the ability to link to some upstream bug trackers that it specifically knows about. The approach I would like to see is that that bug gets referred to, you know, sent upstream whether it's the maintainer opening a bug on behalf of the user or the user going and directly doing it. This is sort of my dream world that everyone would have sort of a concierge to do that. And then being able to track that until upstream closes it and we pull that new release in. In practice, a lot of people will just market closed upstream and, you know, say, you know, go to the upstream tracker for it. I don't know that there's necessarily, neither way is particularly the right way. You know, there's arguments to be made in both cases, but yeah, a lot of bugs that are filed against Fedora Linux are entirely upstream bugs. Like, you know, there are things that the maintainer might fix, but a lot of times, like I maintain a few software packages. I'm not a developer. I probably can't fix most of the bugs that would be reported against them. The best I could do is refer them to the upstream project. When it's fixed upstream, you can close the bug after you create a package with that fix. So that's when it's up to you. So you have to create a package, new release of the package with the fix, and then you said, it's fixed in this package, but you don't close if it's fixed upstream, you shouldn't close it. You should close it when you created a new package with that fix. And also, you close bugs also because triaging with a bunch of open bugs that you're not going to fix is harder. And it has happened to me that I'm triaging and I've seen that bug and I said, that bug is not going to get fixed. So just close it and say something nice because you can get trashed. We had one hand up in the back of the room and I think we're at times. I know there's a half hour between sessions, but I do want to be respectful of everyone's time here. Hopefully quick. I'm curious if Fedora has a processor. Do you have thoughts on rolling over bugs to new versions as they come out? So the bug comes in on a version and then new versions are getting created maybe before the bugs are getting fixed. So what is the process you guys use for that, especially with regards to the analysis that you do? Okay, yeah, so what we do is we close the bugs when the Fedora Linux release goes end of life. So we don't automatically roll them forward. There's a case to be made for that. I think in a smaller project that might make sense. And a lot of times you do have people after we close it end of life reopen and say, oh, this is still a problem in the latest release or whatever. But at the number of bugs we have, if the individual maintainers don't stay on top of it and move that bug forward to another release, it's just really hard to do that at scale. One thing we do is several weeks before the end of life closure, I run the script and it puts a comment on every bug says, hey, this is gonna be end of life on this date. And so if you're still seeing this on one of the newer releases, please update it and then it won't get closed. In an ideal world, we'd have a lot of people who could go through and like triage and double check all those bugs. I would love to have a team of like a dozen people who just went through Bugzilla all the time and did that stuff. That's really boring and thankless work. And it's hard to get people to do that, unfortunately. Okay, well, thank you, everyone. If you do have more questions, I can talk to you later. I'll hang around for a few minutes and then I will be in the Fedora booth, probably a good portion of tomorrow. And also my Twitter handle is there or you can find my emails, you know, search for Ben Cotton Fedora and it'll show up pretty quickly. So I'd be happy to have more conversations with you later but thank you so much for your attention and enjoy the rest of scale. Can you hear me? Oh yeah. Okay, everyone, this room's not sponsored. Yeah, no, this room's not sponsored. Okay, so good afternoon, everyone. I hope you are having a fantastic time at scale this year. We are back. It's exciting. A lot of faces I've seen before many times and a lot of new ones. So this is a face actually that I get to introduce that we actually love because she's gonna speak on a topic that is great because my daughter is a speaker and my daughter spoke the first time at 11 and she saw her talk the same day she spoke and she was so inspired and she's a conference speaker and she's speaking around the world. So wherever we are, whatever city we're in, we gotta find this lady right here. So I'll leave it to you, Ms. Dawn. Take the floor with the fantastic purple hair. Inside the crowd, give it to the lady again. Thanks so much for the fabulous introduction. I'm a big fan of the Banks family if you hadn't guessed that already. Oh here, thanks. So I am gonna talk about how to become a conference speaker. The slides are already posted on fast wonder blogs. If you go to the speaking page, the slides are already there so you don't have to worry about taking pictures, remembering links, that sort of thing. I will start by telling you just a teeny, tiny bit about me. I have been in the technology industry for well over 20 years working mostly on open source projects from within companies like Intel, Puppet, and now at VMware, I'm responsible for our open source community strategy within the open source program office. I'm also on the steering committee for the Linux Foundations to Do group. I'm a board member of Open UK. I'm a governing board member and maintainer for the Linux Foundations chaos metrics project and I'm co-chair of the CNCF contributor strategy technical advisory group. I also have a PhD that I got by researching how people collaborate within the Linux kernel. Now when I started being publicly visible by giving talks at conferences is when my career actually really started taking off. Now, public speaking is not something that came naturally to me. Early in my career, I was absolutely terrified of speaking up in front of other people, even just like small internal company meetings. But I knew that it would be really hard to be successful if I didn't get over those fears. So I started accepting every internal opportunity to speak or give presentations that I could find in the companies where I worked. I also took advantage of loads of corporate training classes to help me learn more about how to give presentations, speak public speaking, that sort of thing. And then my career really started taking off in 2006, 2007 when I started blogging and speaking at events. I was lucky to have a well-known colleague, Denise Cooper, who recruited me into several of my first panels and lightning talks at events like Oskon, O'Reilly's Oskon. And Alyssa Camahort Page of early blog her fame. She was a regular reader of my blog. At the time, I didn't actually know her, but she invited me to be on a panel about open source at South by Southwest, which was sort of my first like really big speaking opportunity. But this gave me the confidence to start doing my own talks at conferences. And I've done over a hundred talks at a wide variety of conferences. And around the same time, I also started organizing conferences and being on program committees, which gave me a whole new perspective on the talk selection process and why certain talks get accepted and others don't make the cut, which I'll talk more about during the session. We'll start by talking about the benefits and why we're here. The real work starts by actually selecting a topic and a conference, so we'll talk more about that. I'll also talk about writing titles, abstracts, bios, and other supporting information to increase your chances of getting your talk accepted. I'll give you some insights into the talk selection process, along with a few tips for writing and preparing your talk. Then we'll wrap it up with links to a few resources and some final thoughts. And I will have probably about 30 minutes at the end for questions, so keep that in mind. There are a lot of benefits of speaking at conferences that people tend to underestimate. First, most conferences, not all of them, but most of them allow speakers to attend for free. So it's way easier to convince your boss to let you go to that conference if they don't have to pay a huge conference fee. And second, speakers usually get special access to other speakers and maybe some VIPs as well. You get to use the speaker lounge if they have one and maybe even attend some VIP events for speakers, which gives you opportunities to network with other experts. And third, speaking at events will help you become more well known as an expert in your field, which can help you gain credibility at your current job and make it easier to advance your career. And by speaking at conferences, I have been able to travel the world and meet all kinds of really interesting people. And now I can travel the globe and meet up with friends that have met at conferences. And conference speaking has been amazing for my career, but it's also been just a really positive personal experience for me. But speaking at conferences doesn't just benefit us as individuals, it also has a lot of benefits for your employer. Speaking at industry conferences provides visibility for your employer and showcases your expertise and your technologies. In my case, our customers, ecosystem partners, peers, and potential future employees see that VMware has smart and talented people who are respected for their expertise. And there's a lot of visibility that comes from conference talks. In the months and weeks leading up to the event, you'll typically see posts from the event on Twitter, LinkedIn, maybe other social sites to promote sometimes even individual talks. And your company will often promote talks on their social channels to make sure that you're getting some visibility for the talks that your employees are giving. And many talks are also like these recorded and posted online after the event. So that allows you to gain additional visibility for the talks well after you've actually done the event in person. And it's also a great recruiting tool. Potential candidates see us giving presentations and they get excited about the work that we're doing which can make them want to come work with us. So the next time they're looking for a new role, it's great if they remember your talk and think about applying to work at your company. And it's important to remember that you are representing your employer when speaking at conferences. And what you say will reflect positively or negatively on your employer so be professional and kind when interacting with other people at events. Now different companies have different processes for getting approval to speak. And even within a single company, the process can vary depending on lots of things like your business unit, your team, your experience, maybe the topic, the time of the year and lots of other things. So the best place to start is always by talking to your manager. And your manager should be able to help you understand the processes you'll need to follow and your manager will need to provide approval. It is very important to get your manager's approval and feedback before submitting your talk to the conference. It can be super disruptive for conference organizers when someone has a talk accepted that they didn't actually have permission to deliver because then the organizers need to replace a speaker and it tends to reflect really poorly on both the individual and on their employer, the company. So just get approval from your manager to avoid in writing a proposal is indeed to come up with a topic. And this is the first place where imposter syndrome can really get in the way of your success. Coming up with a topic is hard for a lot of people because they feel like they need to be the expert in a technology or know absolutely everything about a topic before submitting a talk. But this just is not the case. You only need to know enough to provide useful content for people who aren't as experienced as you are. At most conferences, the audience probably has more people who are just getting started and are eager to learn. And in particular, attendance at sessions tends to be skewed towards new people. And there are different tracks. You can submit something into a beginner track. And a good way to find a topic is to think about why people come to you and ask for help. Think of the technologies that you love to work with and talk about those. Many conferences are also filled with talks about processes, culture, community, management and loads of other topics that lots of people can give a talk about. So think about something you've done, love doing or know a lot about and talk about that. My biggest caution about selecting a topic is that you should be really careful not to propose a topic that's really a product pitch. Most conference organizers will reject any talk that even seems like maybe you're trying to sell something or pitch a vendor's technology. So this could be a really fine line to walk, right? Especially for those of us working in a software vendor like VMware, in my case. If you work for a software vendor and propose a talk related in any way to what your company sells, even if it's an open source project, it's less likely to be accepted and the selection committee will take a more critical look at it. Because conference organizers tend to be very skeptical about talks from vendors. And it's hard to get some of those talks accepted. And keep in mind, there are loads of ways to promote your company and your products at conferences with various types of sponsorships. And sometimes those sponsorships even come with sponsored talks. But even for the sponsored talks, I would encourage you to find a creative way to talk about your technologies that doesn't seem like a product pitch. Now the reality is that pitchy talks are declined for a reason. People don't actually like them. People do like talks that are compelling and interesting. So when selecting a topic, think about how to make your talk unique. The topic itself doesn't necessarily need to be completely unique. But it can help to find ways to make your approach to the topic unique and innovative. And you can do this by bringing your unique perspective to the topic. The topic should be important and relevant for the conference and for their audience. And it should be focused enough that you can tell a compelling story within the time available for your talk. And I'll talk more about some of these areas when we get into the section about writing your abstract. Before we talk about picking a conference for your topic, let's briefly talk about CFPs. The most common way to get your talk into a conference is by submitting a proposal into the CFP for that event. The catch is that most CFPs are only open for a short amount of time, maybe a month or two. And the CFP usually closes many, many months before the conference actually happens. So this actually requires careful planning in advance if you want to speak in an event. In August or September every year, I put together a document listing all the conferences that I want to attend the next year. And once a month, I look at that list and I see which conferences have actually published their CFP deadlines. And the ones with open CFPs or CFP deadlines go into my to-do list so that I don't miss the deadline to submit. And this is really, really important because if you miss the deadline, then you're just not going to be able to speak at that event. It's also important to remember that every CFP is a little bit different. They may require different information. They'll have slightly different word limits for things like abstracts. So you should very carefully read the CFP instructions for proposing a talk to make sure that you follow the requested process and include the required information. Once you have a topic, you need to find the right conference. Spend some time researching conferences and looking at open calls for proposals. You wanna make sure that you submit a talk that's relevant and likely to be accepted at this particular conference. So please, please don't just blast your talk out to any conference with an open call for proposals. Find a couple of conferences that have had similar talks in the past and have an audience that you think would be interested in your topic and then tailor your submission to that event. For each conference, review the titles, abstracts, and bios for talks that they have accepted for previous years. And use those previous talks to get a feel for the types of talks accepted, the length, the amount of detail in the abstract and the types of speakers they've had in the past. And keep this in mind when writing your proposal and then tailor your title abstract and speaker bio to the specific conference. Now, you can also do this the other way around by finding a conference that you want to attend and then picking a topic, which to be fair is actually how I tend to do it. But looking at the CFP for past events is a good way to get ideas for a topic that might be a good fit for a conference that you want to attend anyways. If you're new to conference speaking, local meetups are almost always trying to find talks for upcoming events. And they can be a really nice, friendly place to do your first talk. Having a presentation in front of a smaller audience, especially a local one, where maybe you can get some friends or colleagues to attend and that can help take the pressure off while you get some practice. And these local events are a great place to try out new material before taking it to a bigger event, whether you're a new speaker or an experienced one. And then there are loads of very friendly, more regional open-source events like Seagull in the Pacific Northwest, Texas Linux Fest, Ohio Linux Fest, Scale here in California. Great one. Writing the title in abstract is probably the most important part of preparing your submission. And this is another area where I see imposter syndrome rearing its ugly head. This is not a place to downplay your topic or be mysterious. You want your topic to shine and you need to be clear about why this is an important topic for this particular event. Your abstract should be clear, descriptive, and detailed. It should give the organizers enough detail to understand what you plan to cover. And since it often becomes part of the conference guide, it should also provide a clear description for potential attendees. Now the abstract should be long enough to describe your talk, but not so long that organizers feel like it's a burden to read it. So keep it as concise as possible. The scope also needs to be realistic. Find out how long the conference sessions are likely to be and make sure that you think about how much you can reasonably cover in that amount of time, while still leaving time for questions. And use the abstract to list or describe the key points that you want to make during your talk, while also being clear about why the talk is important and what people will learn if they attend. Now keep in mind that you don't actually need to write the presentation until your proposal is accepted. In fact, I would discourage you from trying to write the presentation before the talk's accepted. You run the risk of spending time on a topic that for whatever reason just isn't interesting for conference organizers. And sometimes organizers, and I've done this, will respond with special requests to make maybe slight changes to your topic. So you want to have the flexibility to incorporate this type of feedback. Now this doesn't mean that you should be submitting things that you haven't actually done yet. So the text should be there, you should be something that you could reasonably write a presentation about. I actually have a bit of a formula that I use when writing my abstracts. The level of detail varies by topic and conference, but if you look at my past conferences, most of my abstracts look something like this. I always start with a paragraph that tries to draw in the audience and generate some interest by focusing on why the topic is important and relevant for the audience. The goal with this first paragraph is to help attendees understand why they should attend my talk and make sure that the program selection committee knows why my talk is important and relevant for their conference. The next section, usually three to five bullet points with some details about what I plan to include in my talk. This helps attendees and the selection committee understand exactly what I plan to talk about. And I also try to end my abstracts with a summary of what the audience can expect to learn. Now, this is definitely not the only way to write an abstract, but it's worked well for me in the past, so I thought it might help people who are new to writing talk proposals as a way to get started. Titles for talks can be tricky. You want a title that sounds interesting enough to stand out. It also must accurately describe your topic and it should do all of this in maybe three to seven words. Now, I am super guilty of writing really long talk titles. And usually this just makes it hard for me to tell people about my talk because the title just becomes too much of a mouthful and then I can't remember the whole thing and it just gets embarrassing. And from a practical perspective, long titles can be hard to fit on conference programs. Also, try to avoid buzzwords in your talk titles. Those of us that regularly select talks for conferences can get super jaded and tired of the same buzzwords that make every single talk and every title sound exactly the same. And keep in mind that many people will decide to come to your talk based mostly on your title because that's what shows up first on the schedules. And you don't want people to be disappointed if your title doesn't match the content of your presentation. A lot of people don't understand how critical it is that you have a well-written bio to go along with your talk proposal. To be honest, when I'm selecting talks, I actually start with the person's bio before I read the abstract because I wanna know what their relevant experience is. And this is another area where imposter syndrome can get in the way. You should use your bio as a place to brag about yourself and help the organizers and attendees see that you have the expertise to give this talk. This is not a place to be modest or downplay your experience. As I mentioned, your speaker bio is actually a critical part of the acceptance process. And you should customize it for different types of talks or conferences to emphasize your experience in a particular topic or technology. Conference organizers want to know that you have expertise that's relevant to your topic. And your bio is probably your only opportunity to demonstrate why they should select you to speak about a topic. You might wanna cut some less relevant information from your bio to give you a little more room to be specific about the expertise you have in your topic. But like with the abstract, your bio should be short and concise. It's not a dissertation. And also remember that this, again, is not a time for modesty. You will need to brag about your accomplishments and make sure that your bio puts your expertise in the spotlight. And one of the tricks to writing a bio if you don't already have one can be to have a couple of friends or colleagues write it for you or at least the first draft of it because a lot of us have a hard time sort of talking about ourselves and that can really help you get over some of those barriers. So that's a good option if you don't have one. Many CFPs ask for links to supporting materials like presentations, blog posts, videos of previous talks. Again, this is not a place to be modest, but a place for you to show off and demonstrate your experience and your expertise. If members of the selection committee haven't seen you talk before, this is what they'll use to learn more about whether you have the expertise and the presentation skills to give the talk. If you don't have something ready to share, you can always write a blog post on a related topic and contribute it to one of the many, many blogs who accept guest content like opensource.com is a good one or maybe to one of your company's external blogs. If you need a video, you can prepare a five minute presentation, record it and put it on YouTube. For supporting material, I do recommend maybe focusing on quality over quantity by using your best examples and not necessarily everything you've ever done. Now this is the Here Be Dragons section of the talk to provide a few warnings about what not to do based on my experience being on program selection committees. A lot of people don't want to give away the point of the talk in the abstract or maybe they want some mystery or some big reveal, some surprise. And in my experience, this is a mistake. When I'm selecting talks, I need enough detail to know if it's going to be a fit for my event. And I need to make sure that it's not going to be too similar to some of the other talks that I might accept. And the only way to make these decisions is by having a detailed abstract. And honestly, I've often rejected talks simply because the abstract was just too short or lacking enough details for me to make a decision about the talk. When I see poorly written abstracts or bios that look like maybe they were thrown together at the last minute without any proofreading, I worry that the presentation will be of similar quality. If someone doesn't take the time to proofread their proposal, what will their presentation look like? As I mentioned before, I've presented loads of conferences, but I still try to make sure that someone else reads my proposals before I submit them. I've had great feedback and suggestions from friends and colleagues that have helped me make my submissions and my talks stronger than they would have been if I'd just done them by myself. It's important to remember that your talk is competing with a whole bunch of other talks and proposals are declined for lots of reasons. Yours might have been a great proposal, but maybe they had another very similar proposal that was accepted instead. Or maybe your talk wouldn't have been a great fit for this conference, but would be a great talk for a different event. And I know it's easy to get discouraged when your talks are declined. Some of my talks are declined. Even very popular speakers don't get all of their talks accepted at every event. So this is just part of the process. Now, after you get your talk accepted, you need to start getting your little ducks in a row to present an awesome talk. I know that some people like to write their talks on the plane on the way to the conference. I am not one of those people and it's not something I'd recommend. If I'm preparing a brand new talk that I've never given before, I will usually start working on it at least a month in advance. And I know that sounds like a really long time, but this gives you plenty of time to think about what you want to say. And if you run into a bit of writer's block, you have plenty of time. Just put it aside for a few days and come back to it fresh when you feel like you have more ideas. And I also find that when I'm working on other things, I'll get ideas for what I've forgotten to add to the talk. So starting early gives you loads of time to think about additional material or improvements. By starting early, you have plenty of time to figure out what you want to say and organize your talk. If you use something like my abstract formula, those Kipola points are likely to be the main sections of your talk. In general, most people would recommend organizing your talk by starting to, before starting to create your slides by using an outline of some sort. Now I have to admit that I actually organize my talks using my presentation software, but I don't start right in by creating detailed slides from beginning to end. I create slides with just titles, maybe a couple of notes about what I want to talk about, which I can then rearrange and make sure that I've got some logical groupings and then a good flow for the presentation before I start adding all of the details. I also recommend having plenty of speaker notes, especially when you're taking time and preparing early. It's easy to forget the key points that you wanted to make on your slide, especially when a whole bunch of people are sitting in a room watching you. You don't want to get nervous and forget what you wanted to say. And I find that having speaker notes just makes me a lot less nervous because I know that they're there when I need them. Do not worry about over-rehearsing. I see way more people come into talks who are under-prepared rather than over-prepared. I practice my talks out loud just like I plan to give the talk and I use these as dry runs for the actual presentation. And this helps cut down on the nerves because you know what you're gonna say and how long it's going to take to say it. Now this is why it is really important to practice by actually speaking with your mouth out loud because if you just think the words rather than actually saying them out loud, that will almost always tend to be faster because we think a lot faster than we actually speak. And if you base your timing for the talk on this, you will run out of time when you do the real thing. It can also help to do a practice run with your teammates or maybe some friends to get feedback. And they're likely to be interested in your topic anyways, so this is a win-win for everybody. You can also record your talk and watch the video. It can be super painful to watch yourself present, but I almost always see something I can improve when I do it. It's also important to speak slowly during presentations. Many of us, myself included, tend to talk faster when we're nervous or when we're excited. It's important to remember that we also need to slow down and pause between topics to give the audience a bit of time to process what we're saying. And this will feel uncomfortable at first since pauses will almost always seem longer to you than to your audience. And again, this comes with practice. Practicing your talk, again, out loud while speaking slowly and inserting pauses will help you get used to the slower cadence before you do it in front of an audience. You'll also want to promote your talk both before and after you give it. You can talk about it on your favorite social media sites like LinkedIn and Twitter before the talk to encourage people to attend. And then again, after the talk with links to your slides and maybe video if the recordings were published, you should do the same via your internal channels within your company like Slack or other internal updates. And I also recommend reaching out to your marketing folks within your company because they are almost always happy to find ways to provide even more visibility for your talk. I also keep a list of every public talk I give on my website there, which honestly helps me remember which talks I've given at which conferences. But more importantly, it also gives me a single place that I can use to promote my talks and it becomes kind of a speaking resume. If you don't have a website, you can keep them in a Google Doc or any other format that you can easily share with other people. And as a reminder, you are representing your employer when you speak at a conference. So they probably want you to be as prepared as possible. In addition to the tips I just talked about, you probably have other company resources to help you prepare. There might be training from your PR public relations team to help you answer questions and avoid maybe disclosing things that you shouldn't. There might also be options for training on public speaking, writing presentations and related topics. And finally, as I mentioned earlier, doing a practice presentation in front of your colleagues is a great way to get feedback in a safe space so that you can improve your talk before you present it to a broader audience. And hopefully you work with lots of smart people so don't hesitate to get their feedback and advice. Before I wrap up this talk, I wanna leave you with just a few resources you might find useful. I did write a two-part blog post series that covers most of the topics from this presentation if you want to refresher afterwards or to share with some colleagues or friends. A couple of people who work with me in VMware's open source program office are on the open source summit program committee and they wrote a really great article about how to craft a better talk proposal that's more likely to be accepted. So that's a good resource as well. And I also ran across a really nice article on Dzone about how to become a conference speaker and why you might want to do that. Let me leave you with just a few final thoughts. I talked about imposter syndrome in a few places during this talk. It's easy to default to doubting your abilities and feeling like you don't know enough to give a talk at a conference, but it's just not true. We all have expertise and knowledge that we can share with people who are just getting started. Again, you do not need to be the world's leading expert on a topic to give a presentation about it. You just need to know a few things that can help other people learn. And by bringing your authentic voice and unique perspective to the topic, people will walk away from your talk with new insights that they wouldn't get from another speaker. With that, thank you for coming to my talk and we can't open it up for questions. Questions. I have a microphone for questions. Oh, okay, but there's a recording. There's a recording. I'll repeat your question. Yeah, that's a really good question. The question was if you get a talk that was not accepted, do you usually get feedback or can you ask for that feedback? And the answer to that, like lots of things in life, is it depends a lot on the conference. So a lot of conferences, I don't know of any conference that proactively provides that feedback when they don't accept a talk. A lot of conferences are happy if you ask to provide some feedback from the selection process. Other talks are unlikely or other conferences are unlikely to do that because especially like the big ones, like I don't think KubeCon will do that, but because they get thousands of talks proposed and only accept like 10% of them. So they can't possibly provide feedback on that many of the talks that didn't get accepted. So it depends a lot on the conference itself. I would say one of the best ways to increase your chances of getting it accepted is really to have some people review it who are experienced speakers and preferably people who are on talk selection committees. Because I can tell you that I occasionally, I don't do this as much anymore, but I used to review a lot of the talks people proposed at places like KubeCon from within my company. And some of the original proposals were terrible. Like that's just never gonna get accepted to anything. But you work with them and you're like, this is how you write a talk proposal and you kind of help them get it up in the quality. And we went, when I worked at Pivotal, we went from having like one or two talks at KubeCon to having like a dozen in the year that we had like there were three or four of us who were reviewing all of the talks to try to increase our chances of getting talks accepted. And it helps a lot. The other thing you can do, even if the selection committee doesn't provide feedback, if you happen to know someone who's deeply involved in the project or maybe someone who's on the selection committee, maybe not someone who reviewed your talk, sometimes you can ask them to just, hey, can you have a look at this and tell me what I might have done wrong? And that can help. Yeah, other questions? Yes. Yeah, so the question is, when you were just starting, how did you get over the anxiety and push yourself to do it? The biggest way that I did that was I really took a whole bunch of training courses on how to become a speaker. And then I just accepted anything that anyone would give me and internally within my company. So I did like, I was a sysadmins, I did some training courses for people using some of our UNIX systems. I used to speak at like the New Hire Orientations, which was like a big auditorium of a couple hundred people. And I just sort of made myself do it. And it takes a while to get over that anxiety and you really just get over that by doing it. So the very first talk that I did was a, it was a training class about how to use this new fancy tape backup system that we had with the Robotic Tape Library. This was in the 90s. And so I was giving this presentation about how you can use it. We had this software that made it really easy. And I talked so fast that I was incomprehensible. So yeah, I was excited, I was nervous. My voice was shaking like the whole thing. And you know, afterwards my colleagues were like, Don, people really didn't get that. They were like, you gotta slow down. You can't just keep doing that. So, you know, that's how you learn, right? And so yeah, so I just kept doing it and tried to get over the nerves as much as I could. But you know, when I first did it, I would just be like sick and just, I would just feel terrible. And I just kind of kept making myself do it. It was a question in the back. Yeah, yeah. So the comment was that Toastmasters is a great organization for helping to get over the public speaking hurdles. I'll be honest, like the Toastmasters tend to vary a bit by group. I actually tried to that. And the group that I was in rang a bell every time you said um, which only made me say um more. And I got so frustrated, I finally sat down and I never went back. I was so, I was angry. I was frustrated. It was not the right approach for me. And so like find a really good Toastmasters group. If you find one like the one I was in, don't just walk away forever because I've heard really great things about Toastmasters. So just like find a different group that's actually good. Yeah. Oh, question is did I ever find an instance where the nervousness assisted me? I don't know. I find that nerves does not tend to assist me. So I try to avoid the nerves as much as possible by being as prepared as I possibly can. Yeah, I don't know. I don't think I've run across that myself. That's a good question, though. Other questions. I've got loads of time. Yeah. What techniques do I use to slow down my speech? That is practice. And sometimes again it can help to record yourself because then you'll see how fast you've talked. And it's something I, you can tell this by like my answering my questions. I tend to naturally speak very fast. And it's something that I have tried really hard to learn over a period of very long time because I work with a lot of colleagues in other countries. In particular when I worked at Intel we had a lot of people in Asia and they really just don't understand me when I speak too fast. Which is true, well frankly, of people in English as their first language probably don't understand me when I speak too fast. So it's something that I've been trying to learn over time is just to slow myself down and pause. But I don't have any like specific tricks that I use to do that. Yeah, so the question was, everyone has a day job and where do I find time to prepare these talks? I'll be honest, that is kind of hard. This is also one of the reasons that I start preparing talks like a month in advance, especially for new talks. Because then I can kind of fit it in between things. And so in between I've got a little time between meetings I can write some speaker notes on that one slide or I can find a picture to go with that thing. And so that really helps because you can kind of break it down and you don't, especially you need some focus time to create the outline and figure out what you're gonna say and kind of the general flow and theme. But what actually takes me more time is finding pictures and writing my speaker notes. And that can be done piecemeal. Once you've got a flow for the presentation and an overall structure, then you can tackle those individual things and you can do it in relatively short periods of time. But it is hard. There are times where I just tell my boss, I've got all these speaking things coming up and I'm not gonna get a lot of other things done. You still do the urgent stuff, but it's hard. Other questions? Yeah, so the question was, how do you handle questions and the interruptions that they provide and how do you get back on track and how do you deal with questions that you don't know the answer to? You really need to learn not to be afraid to say I just don't know the answer to that. I say that a lot. I give a lot of talks about various things in open source and people for some reason like to really ask me a lot of compliance questions. And my expertise is on the community side. I don't get involved in licensed compliance because I find it incredibly dull and it doesn't interest me. So when I get questions about that, I'm honest, I'm like, you know, compliance isn't my thing, I really can't answer that. Or if it's something that you really probably should know and just don't have off the top of your head. There's nothing wrong with saying, oh, hey, let me find a link for that or let me find some details and I'll send those to you after the meeting or whatever. And it can be hard to handle those interruptions. So some talks, I'll take questions like throughout the presentation. I used to do that a lot more. Some talks, it's easier just to wait until the end because of the way the presentation's structured because like this one, I talk about bits and pieces of things the whole way through. And so it's easier to do all the questions at the end. But you do really have to learn how to manage the time. I will often cut off questions if people are asking just too many questions about a topic, I'm like, hey, you know, I've got some other things I need to get through here. So I'm gonna put a pause on the questions and then if we have more time at the end, we'll do more questions. But then you do have to sometimes like shorten other things on the fly because you've spent too much time answering questions. And that takes a bit of learning how to do that. And it is disruptive because you've got like speaker notes and you've got the things that you wanna do. And so while you're trying to talk, you're also trying to think, well, you know, which part of this can I just not talk about because I don't have time. And so that's super challenging for sure. It's not just you. Other questions. We still have loads of time. Happy to answer questions. Yeah, in the back. Oh, okay. So the comment in the back was, if you're doing a highly technical question, a month may not be enough time to prep. Yes and no. So a month is plenty of time to prep the actual presentation itself, but you shouldn't probably submit it into a CFP if you don't already have all of the tech. Like all of the knowledge that you need to actually do it and have done all of the things before. So where I've seen this go off the rails is, and I saw this happen to someone not that long ago where they proposed a talk to a conference where they were gonna build all this stuff and there was like Terraform and just like all this stuff. And they hadn't actually done it before and they couldn't pull it off in time for the conference. And so they ended up giving a talk about part of that but they couldn't actually give the whole talk that they had proposed. So yes, highly technical talks can in some cases take more time to prepare, but if you've already done all of the technical things, so I'm giving a talk about the GitHub API at all things open. I'll still probably work on it about a month in advance, but I've got scripts that do all of the things I'm gonna talk about. So I've got things that I can pull from to make it a little bit easier. Other questions? Yeah, so the question was, how do you learn more about the audience to decide if it would be to what the audience is like and how do you determine that? One good way is to actually attend a conference before you speak at it because that gives you a really good way to have some insight into the people who attend. The other place to look is look at the sponsor prospectus because the sponsor prospectus almost always has really detailed information about the audience because the sponsors want that information and those are the people that fund the whole conference. So they're usually very detailed about the audience and the sponsor prospectus. But you can also, you can look at the conference website and maybe get some details there. If the conference has happened recently, you could even look on Twitter and see what people were saying about it or you can go back and see if people wrote some blog posts about the last one and see what they were saying about it and you can probably glean a bit about the audience from that. Other questions? Going once, twice. Okay, thanks everybody for coming to my talk. For everybody who didn't know that's a phone. Yeah. Just, you know. They're like, please, it's not a phone. No, no, it's a phone. For all of you under, you know, Right, right. Yes, where's my mouse? Uh-oh, except I can't get it to, uh-oh. I started now, I can't finish it. I can't get it to stop. Sorry about that. Let's try this again. Okay, good afternoon everyone. Good evening, whatever time it is. Thank you guys for coming. I hope this is getting recorded or something. My name is Patrick Schwartz. I'm a pre-sales engineer for SUSE. I've been there for about four years now. Before that, I was a systems admin, system architect for the bulk of my career. Have taught, you know, Linux certification classes and all the way down to climbing telephone poles with spikes. So I've done a lot of things in my career. Also, a friend of mine helped me, a good friend. His name is Moose, he would have been here but wasn't able to for a variety of reasons. But he helped me with a bulk of this. And he is my guru, a mentee that I've been working with from Ethiopia. And we're gonna talk about some Ethiopia stuff in here tonight, so I thought it was appropriate to have him on the slide deck. So what I wanted to cover tonight, this evening, whatever time it is, is this project called Keywix or KiwiX, however you wanna pronounce it. And what is it? Why do we care? You know, all that. And what we did with it in containerizing it and then how do we did that via Rancher, okay? And then how we containerized KiwiX. And then I had planned for a short demo but I'm having serious networking issues. Because my demo machines was behind my VPN at home and between hotspots and wifi it's just not working. So we'll try. But that's the beauty of slides. I have it all on the slides anyway so we can talk through it. Sound good? And at any time you have questions, don't, you know, feel free to interrupt me. I don't have an hours worth of content and I know I'm sending in the way of game night. So, you know, I get it. But we can answer questions and if I don't repeat the question, remind me to repeat the questions that gets on audio. Sound good? Okay. So what is KiwiX? It's an offline reader so that we can take content like Wikipedia, Khan Academy, TED Talks and bring it all into a isolated server basically. And what I'm gonna talk about is what I've done with this project already. It's already in use around the world and these pockets of places that don't have internet access, okay? That you can't, it's really, really difficult to take books into. It's really difficult to, you know, they have limited power, you know, to launch, you know, full scale desktop PCs, you know, that kind of thing. So KiwiX gives me the ability to take all that content with me onto a site and stand up a small box and then anything like a Chrome browser, a cell phone on an isolated network can then access all of that data and just like if it were on the line. And so that's what KiwiX does for us. And so in the why, well, as the slide says, basically half of the world's population don't have access to the internet. I mean, think about this is 2022 and we still don't, you know, half of the world's population don't have access to the internet. And, or if they, you know, or don't have power or enough power, right? And so the lack of infrastructure is a huge piece of this. And then the cost, when I'm on my trips to Ethiopia, you know, sometimes I have to go to the local store and buy whatever's available there. And generally it's 10 to 15 years older technology than what I could buy here today. And generally three times the price of what I would spend for the newer stuff here. So really old technology, very high price. And it's very hard to import things into the country or like, you pick on Ethiopia because I spend a lot of time there. And speaking of, here's an example of, this is a small university, believe it or not, that actually hosts about 3,200 students. Okay, 3,200 students. It's a seminary school and a nursing school. I teach midwifery. And that this isn't the only location, but this is very typical of a, this is one of the newer buildings that they've been standing up. And this classroom down here, as you can see, is the computer lab that I stood up in 2019 using Raspberry Pis. And that was, that's that room right here. Is that one down here? They had administration here, classroom, classrooms, and then another row of classrooms down there. So you can see it's not a high tech. You know, you think university, I mean, what do you think of? You know, you see Barclay and UCLA and you think of that kind of infrastructure. This is not atypical for a university setting or a school setting in Ethiopia or the orphanages that I support are very much similar to this. And so they, right behind this tree, there's, you can't see it because of the tree. There's two little pieces of solar panels that come down into a battery pack inverter that provides then the power we have to power up our Raspberry Pis and our little 1U micro Dell box that hosts my KiwiX server. So that's the why, you know. So now this university and the orphanage that we support can support the kids can get a full education. You know, Khan Academy classes are huge and not just Khan, but there's a lot of other open source learning environments that we can then take and put into ZIM files. And KiwiX has a ton and there are all kinds of languages that most of which I've never heard of before, right? But I'm able to download it in English or New Air, which is the South Sudanese language or Amharic, which is the Ethiopian primary language. And so they can take classes in their native languages, whether it be elementary school, you know, all the way up to, you know, generally, you know, middle, lower college, right? And so that's the why of KiwiX. It gives us the ability to take, because they don't have internet at this school. They have very limited power. And so with Raspberry Pis and KiwiX, I'm able to then provide them access to education. And so now they're able to continue with their education without having to carry over, you know, literally tons of paper material that would then get, you know, torn up or wet or because it's a very human environment. This continues to just work, okay? That's, and this is an example of the why. And I'm not just talking Ethiopia or remote Africa places. We have places right here in the United States that are just as remote and just as disconnected. Some of the rural South, some of the rural Indian tribes, you know, some of the reservations don't have access. And so we can provide that with the, as I called it, the magic of open source right here in the US. And so here's some of the tools that I was able to use with KiwiX. We have the KiwiX Manage, which creates this XML file, because like when I took to Ethiopia, you know, I had a two terabyte drive and it had several hundred ZIM files. And so by using an XML file, then I wouldn't have to manually list out all 200 of them on the command line. I could just point to a library file instead. KiwiX read gives me the voted read search. And of course KiwiX serve is what I then put on a small little box that can then re-broadcast via Apache or Nginx for to actually serve those ZIM files. I can build, you know, the KiwiLibraries, there's over 3,900 books already in the library. 110 or 1100 in English. You know, so it gives you an idea that there's another 2,000 or more that's in other languages. You can even create your own ZIM files. If you've got copyright permissions for the website that you want to then reshare, you can do that by using the ZIM tools at the two locations that I have on the screen. And it's really, it's very simple. You give it the URL, it goes out and sucks down the website, creates a ZIM file that you then can put in your directory and add it to your library file. Okay. Any questions so far? Okay. Very good. So, Rancher, we'll talk a little bit about Rancher. This is, because this was a combined idea here that I had, myself and Musse. We wanted to take this standalone application Kiwix as an example, okay, that we use. And how do we go about putting it into containers? And being a Zeus employee and being Rancher's an open source platform, I thought it would be a great segue to talk about Rancher and we're gonna look at that. But what is Rancher? Anybody here not know Rancher? A couple? Okay. So Rancher allows us to manage Kubernetes clusters. It's a complete stack to manage all aspects of your Kubernetes clusters. Whether they're, no matter what CNCF Kubernetes engine you desire to use, one of the clouds, one of ours within a Rancher RKE or RKE2 or K3S or one of the clouds, like I say. Any certified, CNCF certified Kubernetes engine can be ran or used inside of Rancher, managed by Rancher. And so it gives you that ability to do a full DevOps idea of managing your environment through to production, okay? Okay. So how do we go about then bringing this all together? KiwiX and Rancher, how do we pull it all into? How do we go about containerizing this application? Well, initially, when I first started, KiwiX does have a Docker image. It's a Docker hub image. But again, I threw my young friend a challenge. I said, I don't wanna use that one. I don't wanna use that image. Let's make this a little bit more fun. Let's use the SUSE base container image instead to host this application, okay? And so he went out and created this Docker hub using the SUSE BCI to host our KiwiX application to actually compile this KiwiX application using the SUSE base container image. If you have questions about SUSE base container, you can always go to the registry.susa.com and you can read more about SUSE BCI and what it can do for you. But it was kind of that fun little challenge. Do we always have to, and I always try to push the envelope. What can we do with open source, right? Do we just have to take, I mean, I could have just stuck with Kiwi Docker, right? I could have just stuck with it and just ran. But I wanna push the envelope. What else could we do? Well, SUSE BCI was sitting there going, use me. All right, let's try it. Well, and behold, we were able to, wasn't that hard actually, from what he tells me because I didn't do it. So we pushed the envelope, we created this. So he's got his Docker hub directory on hub.docker.com with the SUSE BCI KiwiX serve image, okay? And so again, it was all about an exercise of why not? There was nothing wrong with the official Docker KiwiX image, but let's try to push that envelope. Let's just see what else we can do, right? And so we did it with the SUSE base container. In my lab, for what I was gonna show in the demo, we used the Metal LB load balancer and we needed it back in storage to host all of those two terabytes where the ZIMPOWS that I was using, for example, right? So we needed, so I had some prerequisites that I had built in, we needed a load balancer, we needed some backend storage, okay? But then we went out and created a Helm chart for this application because Rancher gives me the ability, I can point to this repository, this application repository, and say this is where my next app is from. And all my Helm chart data is sitting there. And I'll show you that in just a second. And so I've got this Helm chart now on my GitHub repository. I can add this application, this Helm chart repository to Rancher and say this is where my Helm chart is. And in a couple you'll see, it's actually very simple. Once I do that, then it's just a matter of installing this app, okay? And so the Rancher steps were really, really simple. Add the GitHub Helm repo, create our persistent volumes, which like I said on our prereqs, we used NFS, and then launch the install. My friend Musse created a form that allowed you to answer specific questions like which library file you're gonna use, I mean, because you don't have to call it library XML, you can call it whatever you want, right? But you needed to be able to tell the command line what that answer was, what file are you using. So he created a simple form inside the Helm chart, then it answers some things like the namespace, where's the storage, and the keywix options, for example. Okay, and so because I've got the screenshots here, I know this is really hard to see, and I've got a couple of screenshots, I was hoping to show this to you in live. Again, and we can try again here in a second, but I wasn't having great success. But I was able then, first steps, create my persistent volume, you can see I created the ZIM files, I told it where my NFS server was, I gave it a name, I'm using the plugin, NFS share inside Rancher, and then down here I created the next step beyond create, step two here was a couple more options that I needed to select, like read only, or read writes, or whatnot. So that was kind of step one, creating my persistent volume. My step two then was creating the repo, and again, I told it I gave it a name, I gave it the URL, there's my PHS for SUSE GitHub, what branch was I using, any authentication that I needed to use. And so I was able to then create my repo, and then simply click on install, and then all of this data here comes out of my Helm chart. Some of it's from the Helm read me, some of it's from the Kiwix read me that we created, and it gives you some examples, and I just get a, there's more to this, I just cut it off for screen sake. But then I just click on the word install up in the right hand corner inside of Rancher, and then it gives me a couple more pages, you can see I've got step one, step two, I tell it what namespace that I'm gonna use, where I'm gonna deploy it, I've got storage and Kiwix configuration. This is that menu that I was telling you that Musse created for me with a template inside Helm chart. And then last but not least, here's my storage data, where's it at, NFS share, where's my Kiwix configuration, so there's my, this is actually a typo, it should be dash dash library, library XML, when I'm passing the command line arguments to Kiwix, Kiwix, and once I do that, it goes off and installs that application in my Kubernetes cluster, and I could have told it, I wanna make sure there's always three instances running since I've got metal LB, that metal LB will be the one IP address I get to go to, to see that interface. And then of course, within Rancher, I can also see a lot of, I can just look right at the values, the YAML file for the values, and see there's my dash dash library and library XML, I'm telling it what namespace, my volume name, pieces about that, where's it going, LeapMac at my house. And so, voila, then boom, here's the end results, I was able to create an IP address for me within my metal LB space, and provided a interface, and I only had like a six books loaded in this particular example, just to keep it small and portable. But I've got some Wiki, I've got some Khan Academy stuff there, which then allowed me to, the audio works, there's audio in any of these, they work, video in any of those, those work. And so, that's what, that's the process of getting, just taking this idea of, you know, this Docker, this standalone image, pulling it into what else can I do with it, pulling it into Docker, or pulling it into Rancher using the SUSE-based container image. And so, that was where I was going. So let's try one more time here, see if I can get the VPN to work. Any questions why I bring this up? Thoughts, suggestions, comments? No? You guys are too easy. Sorry, yes sir. Well then, yes, yes, it is actually, very much so. And again, it goes back to standard, you know, load, depending on how big you make your KiwiX serve machine, or in this case, how much resource do we give it in the container world on our, on our, and I did, by the way, get this, this is working to some degree. Can I make it a little bigger? Oops, I didn't mean to do that. Wrong button. You get this right and left thing going on here. The KiwiX server, it depends on how much, how many ZIM files am I going to share? They, a couple years ago, they had a couple of bugs that didn't let you share more than a couple hundred ZIM files on a Raspberry Pi, but I think they've fixed that now. But again, it's a matter of resources. How many clients are going to hit it? You know, and how, you know, is there a lot of video behind those clients, you know, typical kind of stuff? Yes sir. Exactly, so I work with, you know, I asked the, the, so the question is, who helps curate the product or the, the, what I'm taking there, right? Who's answering the questions of, which ZIM files do I take? Right? And so for me and my project, I asked the, both the orphanage director, I also asked the, the college president what's important to you? Cause I'm not gonna, I don't wanna load a bunch of content that doesn't mean anything to you, right? And so I worked with both and each one had specific curated content that they needed. The orphanage obviously needed more elementary, middle school, some high school content, you know, a lot of Khan Academy was really important to that. The university, however, needed more of the typical higher grade content, you know, a lot more TED talk-ish kind of ideas, you know, kind of expand the mind on what is out there, right? And inside of, outside their little world. So you work with where you're going to see what they need. Yes sir. Constantly, right? So yes, that is one. So the question is, you know, how often do I update the content? And to some degree, I mean, yes, everything, as soon as I download it, it's static, right? It's not a live update, you know, it's not like, you know, any updates that are gonna happen that happen tomorrow, you know, for a Khan Academy class, for example, is going to be, it's gonna show up in my ZIM file. So, but at the same time, how often does teaching, you know, algebra change, you know? So I don't have to worry about too much, you know, of that. Now, like, let's talk about world history, you know, for example, yes, when I went in 20, and I first set this up in 2019, obviously there's been three years of updates in world history, right? Some very dramatic changes in world history in the last three years. So that's, they're not gonna have that access with what they have today. So I go back about once a year, and when I go back, I take a new SSD with me that has a lot more, you know, maybe newer updated data, newer ZIM files on it. And with that, then I can, you know, just refresh their, what their infrastructure is. So yeah, there's lots of, you know, ways to, and that requires two different infrastructures. Yeah, exactly. So, but that's all a matter of, you know, like the nursing school would need more medical, you know, information versus the theology school would need, you know, their needs are very different, even though it's the same school. So, so there's my rancher node. If you look real closely at the top, it says Leap Mac. It was because this is open source of Leap running on an old iMac. So, yay, open source of Leap. And so it's my master rancher node, but I've got a couple of clusters down here. You know, you can see I've got K3S, RKE2, and for example, the one we were just looking at, from the demos, from the slides, see if this comes up here. Gives me the dashboard, and if I go into service discovery, services I think, is it here? And see, I've got this, oh no, here we go. And right here, if you'll notice, really tiny at the bottom of the screen, I've hovered over that, it shows me the IP address it's going to load balance to. And just for grants, let's just do that. It's gonna open up a new tab, and there's my live keywix server. And this, for example, my friend took the, this Ethiopian English readers website as an example. He created a ZIM file for it. And so I can look through, you know, like I spend a lot of time in Gambela. I can look at their stories, you know, about Gambela. And if they have any, you know, audio to them, you know, they'll play. So, it gives me, you know, so he was able to take this as an example of using, then I can come back up here and hit home, and get back to my main page. You know, there's my Khan Academy, for example. This was the geometry one for Khan Academy. It's a little bigger to, takes a little bit more to read. That's, especially through a VPN. But you know, I can, you know, watch the class. Okay? And so you can imagine, you know, a group of kids sitting out on a dirt floor with a small little battery powered projector, pushing it up against a wall, watching how to do geometry. Okay? And so, in, you know, that's what KiwiX does for us. By adding it to the rancher and, you know, learning how to do all this, and to be quite honest, this was kind of a project for me to get deeper into what's possible, right? And some of that, you know, like the base container image, I hadn't played with yet. And so it forced me to present, to learn more about it, right? The same with our rancher writing Helm charts, and speaking of Helm charts, if I can find my mouse again. Oh, I gotta go the wrong way. Okay. So here is my Helm chart. Make it a little bigger here. For this project that we just kind of put together. And, you know, like, again, like a lot of things, it's a learning project, right? This is by no means finished, but there's our steps, there's some examples, the chart info that you're gonna look at, some examples, you know, of course, typical left things we needed to do, you know, to read me for the app, read me for the main top, we've got our questions YAML and our values YAMLs. But this was, you know, just our simple chart data that I would then, you know, add it to Rancher to make it an app that I could deploy via Rancher to my Kubernetes clusters. And so that's, you know, one idea, you know, for the idea of having different spaces because it's a Kubernetes cluster, I could then have, I don't have to have duplicate hardware, you know, I could have different, I could have one NFS share with multiple libraries files and each one is using a different library file with all the ZIM files in the same directory in the same NFS share. And I could, because I can set up, I need three nodes for a minimum, for example, I could update the ZIM file library and then redeploy very quickly via Rancher. So, does that make sense a little bit? I hope it's still waiting, it's still spinning. That's not good. This is what I was afraid of. So that's those pieces. Yeah, yeah. Seems like I can get to my home network, I can't get to other sites. So anyway. And so that's just kind of, you know, an example of what we were able to do, you know, very simple, taking a very simple little project like KiwiX with a real world need, but then seeing what I could do with it by pulling it into Kubernetes and how to create the Helm chart, how to tie it all together, how to then redeploy it, how to write Helm charts using this. I know we've had a couple of sessions that we had a whole track this week on Kubernetes. I hope you guys were able to make some of those. And so, you know, I think one, we had one track today or yesterday, it was talking about writing Helm charts. So I look forward to seeing the recording of that one. So any questions so far too? I'm kind of closing, you know, near the end. So as a repository, sure, sure. Yeah, I don't know. I would imagine so. You know, if it's meant to be let's see where's it at. So we have, you know, out of the box in Rancho, we have, you know, hours that we provide, that we support directly. We have those that are provided to us by partners, you know, that, you know, are known to work. And then we have things like mine, for example, that is, you know, a third party repository that I just added to this repository, added to my repository hubs, so that I could, you know, if I needed to, you know, go to, and add a job, is it under workloads? Forget, I'll add an appointment. And so I would add a name here. I would tell it, you know, what image am I using, you know, that was where I would put in my KiwiX, for example. So, you know, I would imagine so. If it's, you know, a container, a bowl, is that a word? Then, yeah, I would imagine so. Any other questions? Okay, well, I won't belabor the point. You kind of get an idea of what I'm doing with KiwiX and the far regions of the world. Get an idea of what we can do in the Kubernetes world, you know, taking simple apps and then containerizing them, using SUSE-based container image, even, you know, that's even something it wasn't originally designed for, or with, rather, I shouldn't say for. And just continuously pushing that envelope. What can we do, you know? Thinking outside the box, you know, not just taking the defaults. I hate taking the defaults. You know, the next, next, next idea, right? So, I like playing what-ifs. All right, well, thank you all. Again, I appreciate your time. Look forward to game night and get some dinner before then. All right, thanks all.