 So today's talk is called your services not open source and in many ways it's the longer version of what we talked about at the keynote yesterday. There's more room for discussion, maybe some of you have had time to think about aspects of this and we can go into more detail on some of the more interesting areas here. I see Tomasz is in the chat room Tomasz Tomaszek. He's put together some of these slides and he's of course welcome to join me. You can also sometimes direct questions at him. And let's get started. So, why am I talking about this topic. This topic is relevant because I care a lot about open source. I've worked in open source for the last 20 years and been just like many of you. One of the contributors to this movement that has changed the world. One of the things that I did was contribute to 100 different projects all sorts of different ones. A lot of GNOME projects, but also tons of different infrastructure. Sometimes the FreeBSD kernel, I used to contribute there, PostgreSQL, OpenLDAP, Kerberos. In many cases you'll find that my contribution to a project or another is a small one. A few patches here, a few patches there. And this was all done with the goal of making things work together really well. Something that I care a lot about. And making different projects with different goals and different timelines and different people look to the user like they're part of one bigger whole. And so fixing these small issues that allow things to work seamlessly together has been my passion. In many ways, I also was one of the main contributors to Cockpit. We worked on that and started up that project and it brought together a whole bunch of different APIs. A whole bunch of different components. This is an old list, but I think it's over 90. And talks to all these different APIs and components and files and commands and so on directly. This meant going into many of these projects, either myself or one of my team members or one of the project members and contributing something there. And it's safe to say that because we were fluent and unable to go and become contributors in all of these different projects, we were able to be successful. Without that capability Cockpit would not exist. Each project is different. Each project has different requirements, different timeline, different culture, different languages, different standards, different quality. It's a vast collection of different ways to contribute. But one thing is consistent. Open source projects allow users of those projects to become contributors, and they all allow this in one way or another. Open source thrives because it converts users into contributors a small, small percentage of the people who can use a project who run it, who experience in some way, decide, hey, I want to change something and they figure out that they can and they become a contributor. If you take away this function, open source starves. And so not only have we been able to build such amazing things on the fact that you can contribute to all these projects, fact that we can bring things together the fact that we can bring thousands of projects together into a single Linux distribution is an amazing fact and it's powered by the fact that we can contribute all these things. But on the other hand, if we start to see this trend reverse. I am concerned that this part of all of our legacies what where we have changed the world in this way will start to star. So here's an example. I'm a small time PostgreSQL contributor a while back I added a feature and I made a few bug fixes for because I was experiencing some problem when running it. And this community was an amazing example of open source community that not only was very welcoming. Not only was very competent, but was also made me successful as a contributor. I started through documentation, their process for evaluating my change, reviewing it, giving me feedback on it, and making sure it was on their list every week in their discussions and kept me up to date on how to participate and everything turned me into a successful contributor of PostgreSQL. If you look at Amazon RDS you'll notice that PostgreSQL is being run there. And you can you can actually look up different reasons why you would use PostgreSQL on RDS. And you can get all these amazing lists of the things that you get when you do that things like obviously backups backup and restore high availability replication infrastructure performance tuning monitoring. I mean you name it all these things you would have to figure out if you ran PostgreSQL on your own. And that's very compelling. I totally get why a bunch of people want to consume PostgreSQL as RDS makes total sense. Someone else can deal with all of that. And I can focus on what's important to me, which is using PostgreSQL, putting my data there, structuring the data appropriately, and using all of its features. However, when run as a service like this as RDS, there is no mechanism for a user or some small portion of the users to become contributors. And as more and more people consume software in the in a as a service in this way. This has me worried as makes me feel threatened makes me feel threatened that open source is going to starve in relation to how many people use software as a service use some of these same projects as software as a service. And for a long time this concerned me, and I was very uneasy about services. And despite the fact that I and my teams, different people I know run services. This was this thing that was, it seemed very hard to reconcile in my head the fact that they seem to be a threat for open source. And if we look at this is just a fictitious path of someone of a user of open source software, figuring taking the various steps, the various stages that you need to get from user to contributor. And on the left hand side we have open source software on the right hand side we have software as a service. So, you know at the beginning you see the user would use the software or happily, and then figures out hey wait a second. I can do better this this this thing over here needs to be changed it needs a fix. The first big obstacle is realizing that you can contribute if it's open source. A lot of people will just walk away. They don't even, it doesn't even make sense to them that they can contribute to this thing. The second is figuring out which code to change. If you've used a complex project like like no or anything that's running a bunch of microservice or kubernetes or, or anything really figuring out which code to change is a tough, a tough task, being able to tweak something change something and understand that your changes are having an effect on what's running. So, figuring out how to run it how to build it how to make it actually work. How to make your change take effect how to experience it. Then, usually you break the thing and a lot of people give up at every one of these stages they give up. They're like oh it's not worth trying to contribute. And then you go down further with less and less people sticking, sticking with you. So, to get the change fixed make it functional and hey it works it's all my problem. Now how can other people use my change, everyone should have this, and buy and buy it each of these stages is an obstacle, but eventually you get some contributors out of this, some contributors are like me might come by the project. I don't quite work out with it for a little while and then move on but some people become also routine contributors to a certain project and become its maintainer and so on. So, this should seem obvious to you. On the right hand side we have software as a service and you can see that many of the things that we many of the techniques that are typically used to overcome these obstacles do not exist. We lose the users who could be contributors very rapidly and it windows almost nothing. We solve this problem. And why is this a problem in the first place let's look at some of the fundamentals behind the some of the first principles that we would need to think about the one of the first principles is that open source thrives, because it can can move some percentage of users to contributors. But let's look at others. So what is the service, a service is more than just the code, more than just software. Obviously very prominently in the service is the software that's that's running, and even the operational code, if the people running the service have any muster they they're they're encoding their operations in various forms and various techniques and various playbooks or Kubernetes operators or what have you actually automating a lot of the operations that turns into code, but beyond that code there's a lot of other things going on here. There's infrastructure it's running on there's other users, there's backups there's scaling there's disaster recovery process there's legal agreements about data. There's interconnected services that provide perhaps authentication or other aspects to the service. There's tools that are integrated logs there's performance tuning high availability you name it there's a whole bunch of stuff that probably missed a bunch of stuff here as well. So a service is much more than just an instance of the code that's running. Once you bring in all of this, it has sufficient complexity that it's much more like an organism. And leads leads to this claim open sourcing your code is insufficient to make an open source service is a services way more than just code. Open sourcing the operations. That's operational code is insufficient to make an open source service. It's necessary. Both of these steps are necessary, you would not call your service open source if you didn't open source your software, or your operational code and, and, and you see services for example that only open source their code and example is Travis has all of its code on on GitHub. It's hard to say that Travis is open source. No one knows how that all comes together it's very difficult. Other projects have open source part of their operations like get lab for example. It shares the operational, it lets you spawn up its own instance. Basically the setup and deployment scripts and a whole bunch of different other aspects of the operational code, so that you can run it, but even this is insufficient I claim to make an open source service. And this is because the users of the service. They explicitly chose not to be involved in the operations. When they chose to use software as a service. And so forcing them to operate the service as a way of contributing to run their own is explicitly counter to the choice that they made. So that RDS example with Postgres user of RDS explicitly chose not to be involved in the operations of Postgres and therefore, if in order to convert that user to a contributor they have to go back on that decision you're going to have a very low success rate. And what's more is that the service is sufficiently complex all of those various attributes coming together that it's not reproducible. Here we have two twins, both from the same source code the same DNA, identical at some point. And yet you can see how rapidly they diverge. At one point to these twins may well become enemies, who knows, but all of the other attributes that come on top of the source code on top of what defines it makes these into different entities different characters. So if you own a service fully and bring in all those other aspects again, we saw in that word cloud. You will have a different service, and much of the value of the service comes not from the software itself, but from all comes from the operations and then everything that goes along with it all of its interconnected web with everything that's running. So, thus we reach this conclusion. This is a challenge. The challenge is that a user of an open source service can discover which component to contribute to in that running service, make a nonsensical change to a service that is running. So you're going to enter style log statement or change the spelling or something, and experience that change when using the existing service, or when it's accessing their data, or when, when, when they're sending workload to it, I mean they're a user. So they experienced the change that they've made iterate on that change that's useful obviously we're not going to merge a change of spelling just for fun, and then propose it for merging. So there should be a way that communities interacting with an open source service groups of users groups of contributors can share changes can stabilize new behavior, and can work together on different aspects different functions, functionality in the service before everyone else experiences that new functionality, those new changes. This might seem like quite a quite a crazy goal. It really implies that hey, we should be running people's random changes or their their pull requests in the service. And I mean the mind boggles with all the all possible things that could could go wrong here how would you do such a thing. The thing is when we sat down and thought about this, it became clear that all of the techniques that we need in order to pull this off. We already have today. There's already someone doing each aspect of what's necessary to pull this off. What's more is that there is a simple iterative path that gets you from the comfort where you are today running your service. The standard techniques, you're familiar with to this goal. It's possible. And so, as an example, we will take you on such a journey. There are some projects that are going on this journey, some of them have taken first steps of this journey. And then they have, then they have stopped some, some are explicitly deciding to go all the way through this journey towards having a contributable open source service. So, this is all available in text form by the way. Let's take a look. I will click a link here. At the end of my, my slides, and we can bring it up in the, in the, in the tab as well. So just a second. There we go. And actually I'll paste this if you'd like to follow along here in the chat. So, and, and you know, this is this is an imaginary playbook, some projects are doing various steps here, but there's certain parts that we're not familiar with. It goes from the comfortable to the uncomfortable from the known to the unknown. So let's start. So the first thing that you need to do you have a, you have a standard, a decent project, a decent service that's using CI as documented that's using pull request that's using all the standard techniques that you have in modern software development. And one of the, one of the most important things and this is something that's often neglected but many of you are familiar with already is you have to document your deployment workflow. Document it, make sure that that it's understandable that other people can do it have someone else sit down, for example, and walk through, maybe, maybe someone who's just joined your team, or, or a new contributor to your service, and walk through this, this, this documentation and actually deploy another instance of your service. Next, automate that make sure it happens continuously. You would, you would run that automation against the staging infrastructure. Again, there's nothing new here many of you, I hope, in this room running services already do this. You have a staging infrastructure where you take new batches of changes perhaps before they're released into production, and you push them out into staging. And, and contributors or, or users are working in that area and making sure that those, those changes are, you know, saying, you give them some testing you give them a run around before you deploy them into production. If you get stuck here it's a very awkward place to get stuck, because there's that big risk factor, when you deploy something from staging into production. Can it all go bump in the night because it's a big batch of changes that have piled up and they haven't really seen the light of day yet they've been having some fake workload hit them, they fall past CI and so on. And you've reviewed them well, but it's still like a scary moment. Every time you batch this stuff from staging into production, and the longer you wait to make that transition, the scarier it gets. So, the continuous aspect of this is quite vital. But again, we're on familiar ground here, nothing revolutionary, nothing magical. Let's move on. So, what many projects do next and I think this is important is to take some traffic, have some people actually use your staging environment for real. This implies some steps, such as making sure that your, your production and staging can access the same data that they can have the same experience when using staging except for only the changes in behavior that that people have contributed or the team has worked on. And the project has worked on they're not yet available in production. In fact, there's there's many, many examples of people doing this. This is not that amazing or insane but it's important of course to make sure that the people using staging, perhaps going to a different URL for that are getting the real experience they're not using they're not working with fake data they're not working with fake behavior, but they're actually getting the proper experience they're authenticated as real users and so on and so forth. So this is typically the next step we go through. And off we go. Now, here's a key part is turn your staging more into a canary or AV deployment, where you root some of your traffic you charge your workload in some way, and send part of it to staging, you might have a very intelligent map that says certain users requests go to staging to experience that that different behavior. Or you might have it based on workload you might have it based on a certain other criteria, if the contributors, oftentimes this is a team can can update that definition of what goes to staging they're able to switch stuff between production and staging produce their workloads which their behavior between the two. And moving on ahead. We can see that this gives us the capability to have end deployments and deployments that each have different functionality we see our production are staging there where a lot of traffic goes, but we also start to see the ability to have deployments for pull requests that are not yet merged for features where groups of contributors are working on something together and getting it ready and and putting their real workload through there to ensure that this feature works the way they intended to and stabilizing it. And so far we've we've we've kind of focused on many of the deployment mechanisms and how you might, how you might pull this off but we're getting kind of suspiciously close to the ability for an outside contributor to come in and actually propose a PR. Now, there are several steps after this, they are somewhat unfamiliar. We need to apply techniques that that do some code review before a contributor can randomly, you know, root some traffic to their PR. We would need to have some interaction on making sure that it's about that the code is evaluated that's not leaking data, or it's not doing destructive tasks and so on, but many of these things are already available. I mean, these techniques are already used in one way or another in projects, many projects, for example, will do a code review before passing a test through CI to ensure that it's not a Bitcoin miner or it doesn't, you know, break the CI infrastructure. And that's an example there are also other places where certain deployment certain DevOps or SRE techniques require rigor around data separation of persistence. And there's, there's, there's many places where when confronted with a problem we can either use an open source technique, or an existing operational technique to solve that problem. And so far as we discussed this and looked at these kinds of things. We haven't found a particular obstacle that someone hasn't already figured out an answer for. They just haven't used them all in this way to enable contributions from people outside, or in combination together. But I'm reasonably certain at least so far the data is showing that we can actually use existing techniques to pull this off. So, again, the challenge, the challenge is you run an open source service. How do you get to the point where you can have a contributor that maybe you're running part of the services in fedora, or you're running services. Using using a project that that had that that other people can run themselves, but how do you make it so that a user can discover that your service is contributable. Hey, and that, hey, here's the pieces of it here's the map of the microservices that that they're involved and and how to find the place where they need to, where you know they could contribute they can make a change. If you make that nonsensical change. That's often the first step to a contributor is just understanding that, hey, I can change something. It might be just figuring out anything like we said log or spelling or anything like that here. And then have them experience that change. Have them experience it. When they're accessing the service or by a staging environment like we talked about these are all different steps towards this goal. Be able to make more changes there. And at the end when it's ready propose it for merging. And then of course for it advanced that you know advanced part of this challenge. How do people share changes that are not yet merged into everyone's behavior this is fundamental we've depended in open source on this very staged approach where I make a change I run on my own computer doesn't have a load that I share with others. And they tried out and they review it and they try to work on it then it might go into a project that that other people are tracking then it goes into a distribution. Fedora as it said, as they like to call them as the Fedora likes to call itself it's a cutting edge distribution so trying to get as many recent changes there as possible, then it goes to more stable places and so on that entire thing is missing. So those steps are missing when working with services and this, this is kind of the end goal of that where you have communities that are able to work together like this. There are examples of people projects pursuing this I packet has talked about this that packet is on this journey of making sure that the packet service is contributable. It's not all the way there but it is taking those steps along. There are many other pager does it to I hear from Neil awesome. I saw a sig or like a proposal for for for implementing some of this and pager that someone linked the other day. So, and like I said, many projects are on this road or services are on this road at some point. And so it's about taking that next step towards this destination where. Open where users can become contributors without becoming operators, they can contribute to a running service and actually work in that way. So, here's two links. What the first one is about. Operate first. It's an effort to make sure that we codify and share an open source, all of our operational code operational expertise and the second one is more of a scratch pad that I put up with this challenge. Some of the playbooks related to some of the steps you might take, and I'm hoping that we can, we can work together in places like that. And maybe the playbook that's written there so far is, you know, it's, it's off. There's something, you know, there might be different models for how we do this different techniques and tools and so on. And so I'm looking forward to all of the different projects that are interested in working together on this, and many are to see how we can share our knowledge, share what worked and didn't work, and actually come up with some, with some best practices here. All right. So, we're ready for Q&A in the last 12 minutes. And so if I understand this correctly. Oh, hi, Justin. Hey Steph. Would you like me to read the questions or do you, can you see them. I can see them. I see two questions so far. The first one is, GitLab has parts, certain parts of their operational stuff open source, but the code that is that their service runs on is open, is, I guess, not open source. Is that, is that what you meant, Neil? Is it really workable with the open core model that they use? Yeah, GitLab is a bit tricky. So they have, they are open core, and that is a ding. And they are, they are effectively, at least more effectively than other service is accepting contributions. What they have, even though they don't let you run, they don't give you the entire operational side of their service. What they do give you is this thing called the GDK, the GitLab development kit that allows you to start up pretty easily, GitLab on your own, essentially have your own development environment. So if you, Matt, if you plot them on this, on this model or in this playbook, they definitely are somewhere along this road. And for the people, for the developers, given that GitLab is something that a lot of companies or a lot of projects run themselves. And so a lot of the people who are touching it are operationally minded. It does seem to be working. It does seem to be working that that some of those are turning into contributors. They have, they have numbers that show that, and we can't deny that part. I do believe though, that if you do go down this road, and have real open source services that people can contribute to actively, without forcing them to fork or clone this entire and run and operationalize this entire service, that you will have a community center around the running service in the center of gravity that, that, that removes the need to jump to crutches like open core. Once you have that kind of center of gravity around your service, you won't need to go to open core you won't need to be afraid of holding some part of it back so that other people can't run it because you have a whole bunch of contributors there actively participating, actively working together on different behavior and moving the service forward. The clone ability of it you can't clone that kind of activity, and the clone ability of it as we said before with the service is not really possible. You get a facsimile or a different behavior in the service when you call that. So the second question is how you can make business with open source software libraries. That's a, that's a complex question. And I guess it's sort of outside the scope of what we're talking about here. I did touch on it a bit with the open core model. But there is another aspect that often comes up. So if I may, I may answer a business question that comes up when talking about this topic of open source services and that is, how do you pay for all of those extra deployments? How do you pay for the fact that other people can come and use the service and contribute to the service. And this goes back in my mind to that simple function that open source thrives by taking users and turning them into contributors. In other words, you had those users in the first place, and you were paying. You were somehow making a business, your service was either worth it, was funded, or your infrastructure was funded and you were somehow, in some cases, running, if it's an open source service, you might have been, someone may have been donating the infrastructure, or there may be maybe other aspects in play, but somehow you were servicing those users already. Now the question, the challenge is how do you turn them into contributors? Not how do you, you know, it's not a wasteful activity there. And of course, as we do that, we must be mindful about deploying those deployments in such a way that they scale, that they don't each double the cost of running the service. But there are many techniques, whether it's running projects in Kubernetes and pods, or whether it's using different tiers of infrastructure that allow this to be a nonlinear explosion of costs. Question from David Duncan. Ah, so constraining the legal and terms of service problems is something you keep in the scope of the outsider. I'm looking at 10.3 and 10.5 on the services. You keep the activity human. What do you mean by 10.3 and 10.5? Are those slide numbers? You see, you could answer in the chat if we want to have a dialogue. So as you can see this playbook goes first focuses on people who you trust, usually team members and achieving much of the advantages here for easy contribution for those people first. And then it punts towards the end, some of, okay, how do you make everyone be able to contribute. These are things that that we do need to look at. But the legal and terms of service related to other people contributing other people, for example, hosting content that that is not legal in your jurisdiction. Or and then being if they do being banned from being contributor. Many projects have this, for example, if you go to GitHub and sign up, you know, you'll have to sign a terms of service and so on. So there's those aspects there that definitely need to be, we need to have a common best practice for for some of them in order to in order to complete this playbook. There are some other aspects to to to legal aspects or, for example, interconnections with other services agreements with other services and so on that are out of scope of this because you're running your you're contributing to an existing running service. It's not necessary for every contributor to involve themselves in every agreement that perhaps for example a service has with the with another service has authenticating that's providing authenticating occasion to it or providing backups to it or providing some portion of the service that's interconnected. Those are out of scope because the contributor doesn't need to launch an entire service all over again and doesn't need to be involved in all of those aspects. So we'll be I'll be putting up together with some of my co conspirators like to wash hey we're putting up some more content on these pages and for example tracking where different projects are in charity model or maybe they're taking a different approach to this by correlating information here and I really hope that as we do that we will find the models that most projects can use to achieve this challenge and that we can we can actually pull off effective contributions. On services and thus have true open sources services.