 So let me first introduce our keynote speakers. Our very first person is Marshall Hilt, the guy with the headset or without the headset now. Marshall is an open source enthusiast. He's been in the open source communities for some time already, but his favorite topic in the last decade already is talking about AI ops and AI and machine learning. Of course, Marshall today is working at Red Hat with an office of CTO. CTO is the incubator for us for new technologies and new types of things. So he's always excited and exploring new open source technologies. Then we have Steph here. Steph, I know you for some time as well already. You've been in the open source community for 20 years or plus already, and you've been involved in many different upstream projects, but your passion is to make sure that different upstream projects actually work with each other. And then we have a healthy ecosystem of great Linux components. And that's what I like about you. And then we have Tomasz Tomecek. And also I know Tomasz for almost a decade now. Tomasz is a great example of a person who approached me at one of the DEF CONS nine or eight years ago, asking me, hey, Radek, I love this open source thing. How can I start participating in it? And today Tomasz is a principal software engineer who's working on some of the core projects within Red Hat Enterprise Linux or recently on CentOS Streams. So again, one of the great engineers that we have here at Red Hat. And today we're gonna be talking about open source operations and open source services. And first I wanna ask Marcel, what the hell are we talking about? Yeah, that's a really good question. And it starts like every fairy tale with once upon a time, but it's this one is based on a true story. So once upon a time, open source, next slide, please. Open source entered the world. Actually, it was code that entered the world and code was made happy by open source. So we revolutionized the code development, how we develop software basically. And I don't wanna talk about this part yet because we're at DEF CONS. Everybody should know what's why open source and why it's so great to have open source communities. And we had a lot of projects spawning and communities evolved. And you had a healthy competition against other and amongst others. And some companies would differentiate on the top of the common code. I mean, Red Hat ships open source products based on open source code. But the code was the value, right? So code was there to solve the problems. You would have features to differentiate yourself from other projects. And Ops more or less was an afterthought. And then in 2006 or so, Cloud came along and it embraced code. And I mean, it was hugging code really, really, really tightly. Right? And then suddenly Ops became important. So you wanna get your code running in a reliable fashion because in the end, if you're not operating, if you're not available, then your business would lose money. So you could not really have outages by any means. I mean, we take it really, really for granted nowadays that our phones and the internet always works. So a new persona emerged. It's the implementation of the DevOps community. It's the service reliability engineer, the SRE persona. But now is it really balanced? So is code really as valuable as Ops? So there's this person, Matt Essay, he's working at AWS for open source. And he tweeted just recently about this story about Yagabyte, a company essentially switching gears from an open core model where they had parts of their code not open to open sourcing everything. And he said, yeah, that was the right decision because in the end, the code was not the compelling value. It was the usage of the software. So if the customer can't use the software, then it has no value. So it's really an operationalizing it so that the customer can be productive. So everything is not just the code, but everything is also the Ops platform. So I would go even one step further and say Ops is even more important. And kind of by definition, it's always proprietary. I mean, we use open source tools to run our operations for monitoring and stuff like this. But in the end, every implementation of a data center, of an Ops service of your workload that you're running, that's private and the snowflake by default because of very good reasons, because of privacy and access permission. So you can't just let everybody access your backend and data center, that wouldn't make sense. So my core proposition here, or my core thesis is if the value in IT is an Ops and Ops are proprietary, then open source has a problem. And in the Kubernetes community, we see that movement from where we try to operationalize the knowledge and the excellence of operators as in the person into a bit of code, basically codifying operational excellence in something that we can ship with the product. We call that operators. And I think if the developers who will implement those operators and the SRE guys who are running in these operational issues and basically experiencing what it means to operate the software, if they can't meet and if it's just another throw my stuff over the wall thing, then we're really, really missing out some great opportunities. So I think what we need is a cloud installation with full visibility into the Ops center, where you can really look at everything completely transparent down to the stack, so that we have that collaboration going on. All right, Marzo, thanks, I see where we're heading, but honestly guys, I don't know what the solution for the problem is. Steph, I'm pretty sure you got something. So I guess, as I understand it, Marcel, open-source in the code is not sufficient to make an open-source service. We see a lot of services that throw code over the wall. Some of them aren't Red Hat, some of them are not and share their code, but it's insufficient to run it, it's insufficient to contribute, it's insufficient to actually participate. But I would say that open-source in your operations, whether it be via operators or other mechanisms, is also insufficient to make an open-source service. It's necessary, so I like that, I'm buying what you're selling, but I think it's insufficient and here's why. So a service that's running is really a whole bunch of different interconnected things. I mean, source code is there, it's very prominent both the source code for the service itself and for operating it, but there's a crazy amount of other things that contribute to the value of the service. All of these things are really hard to replicate and all of these things define the service itself. If you have a service, perhaps like GitLab, where it's intended for other people to run, it's a product and yes, you can run it, you can consume it as a managed service as well, then yeah, I get it, sharing your operations might be good enough, but for most services, well-developed services, that doesn't play. Open-source thrives because it converts users into contributors. A small, small portion of the users of an open-source project become the contributors and that's what open-source builds off of. This is how it continues. Conversely, open-source starves when it cannot convert users into contributors and we see that with services, with software as a service so much, that there are users who are consuming it and some of them notice that, hey, we need a fix and then even just figuring out, hey, can I contribute to this thing? All of a sudden we lose a bunch of participants, people who would have contributed. Here in this diagram, we have open-source software on the left and we have software service on the right and you can see that even walking through the process of a user turning into a contributor on typical open-source software, there's so many steps where we lose people, we lose users and a few of them filter out and become real contributors. Many of you here have gone through some of these steps or other obstacles, but when you look over at the software as a service side, there's no way to overcome many of these obstacles and we pretty much lose everyone and that is the challenge that we're facing. What's more is it's not just about these mechanisms here and all the obstacles, but the users of a service explicitly chose not to get into the operations. Why are they consuming that service in the first place? Why are they using software in that way? Because they don't want to operate it. So if operating is necessary, operating your own copy of it or spawning up your own instance or all of that kind of forking it in some way is necessary to contribute, we lose them right away. And the other dynamic is that as we noted before, these two folks here, these two twins started from the same source code, the same DNA and yet they're growing up very quickly to be different people. In fact, who knows, they might even be enemies at this rate, look at what's going on here. But if you fork a service and all of the other stuff around the source code, the infrastructure, the interconnected services, the legal agreements, the data, the other users and so on, essentially your copy of the service is a different service. And so we have to overcome this challenge that a service itself, a well-developed service is not a reproducible thing. We have to, so the threat that's facing us, let's use Postgres as an example. On the left, we have PostgresQL, which is one of the most awesome communities that I've ever contributed to. I've only contributed a feature and a few fixes, but not only did they make me feel welcome, they made me successful in contributing and made it non-surprising, expected, treated me like a peer and this was amazing. On the right-hand side, we have Amazon RDS which runs Postgres, runs all of the operations for you, the backups, the monitoring, the scaling, the high availability, the infrastructure, everything that you could possibly need makes it much easier to consume Postgres. However, if everyone started consuming it that way, using Postgres that way, the pool of contributors would dry up. There's no mechanism for people to convert from a user to a contributor. So here's what I would say is the challenge that we face and the challenge that I put before you. This is the target, the goal we need to get to is that a user of an open source service can discover which component of that service, which microservice, for example, to contribute to, can make a nonsensical change, perhaps adding a printf statement or changing the spelling of something like so many of you have done, can then experience that change when using the existing service, when it acts on their own data or perhaps is operating on their own workload and then iterate on that change until it's useful. Until they can propose it for merging. And another part of this is that we, the communities will form around this and we need to enable them so that these communities can share changes, can stabilize new behavior and work together on certain features before every user of the service stabilizes those changes. We're familiar with that here. We work on Fedora. We stabilize things before everyone else gets to them. So. All right, Steph, you scared me a bit because this really sounds like anyone will be able to contribute to my favorite service and take it down and break it. Is this thing even real? Is it even doable to make sure that people can contribute to services that are just running online? Tomás, I'm pretty sure you have some answer for that. Yeah, okay, Radku, I'll try to answer that. So I would like to take you here on a journey from the traditional development process to the ultimate contribution and operation workflow. So let's start with like this first stage and I'm pretty sure that all, like all of you are familiar with this. Like this is a well-established open source project, like in our case, open source service. There are full requests flowing in. There's continuous integration, which means that all the changes are being tested. And then if you want to run your service yourself, you just open read me or like the documentation and deploy it locally. And ideally, this would be running in containers so that it's really easy to do. Like we all know this. So if we go to the step two, like the next step would be to running all the operation tasks, like as soon as they are merged, like either it's code or either it's the operation like playbooks or scripts so that we would have added like another deployment. In our case, it would be stage. So on the left side, we have the production service which is generating us money and that's what users are using. But at the same time, we would have stage where we would have the all the latest code available. And again, this is pretty common and pretty like around all this. So let's go number three. And number three, like the biggest difference between two and three is that in two there was no traffic for stage. Like we would just make sure that like our latest code is good. But in three, we actually want to expose stage to users. In this case, that would be probably contributors. So here you can see that our customers are using the production deployment which is like stable and scalable secure. But we also have this stage which is different probably not so secure and hopefully matching production, but maybe not. And that's where the value is like our contributors if they use stage, like we are getting ton of feedback like if it works, like if it breaks for different circumstances and we probably don't want to expose this to the customers and like get them yell at us. And at the same time, we want contributors to still use production. Like so it's really up to them to pick like what they want to choose. And that's a manual step to be honest. So here in number four, we would like to introduce a new thing like a router or a proxy which would be the unified point for everyone to interact with our service. Like whatever deployment it is, like this router would route the traffic to the appropriate deployment. So that's for example, our contributors would still be able to use stage and customers would be exposed to production. And if we wanted, we could start routing some other production traffic to stage and make sure that our customers like it works for them and stage actually works with their workloads. And if we got here, I mean, we can even go like some like infinite, right? We can create as many environments as possible and test features as they are being developed. So let's say that I'm going to work on my mega feature and I won't have it deployed and I want all my contributors or like all my teammates to try it as soon as possible. And this way, if we have the traffic router automated, like it should be very easy to set up. And if you look at this picture, you might be asking like, what's with the kittens? And what Steph said, like Steph said that we would like to turn users to contributors and I would like to step it, like go one way further and I would like to turn our users and customers into acute kittens. And this is number five, like this is what I personally call the ultimate contributor and operation workflow. Well, you are able to very like iterate on your service very well, deploy it, you have automated most of the tasks and it's very easy for people or for the community to try changes as they are being developed. And here we would like to invite you to work on this, like work on this with us because not everything is solved. Like number six, what's the step number six? Like how do we treat data, which might be like embargoed customer data? How do we treat security or even like what Radix said in the beginning, how do we make sure that production doesn't break? And that's what we are trying to figure out and we'll have links for you in the end of the presentation. And in the meantime, I'd like to show you some examples. Yeah, Tomas, I like all this but give me some real projects or this actually works or there are some snippets where it's already working. Oh, okay, yeah, yeah. Okay, let's go for it. So in first example, like here we have a screenshot of a pull request and you can see that the maintainers of the project are able to request a testing version of the project. And then there is a bot which is which deploys the change like somewhere and makes it available. And this is exactly what we are looking for with these open source services that I as a maintainer can request it to be deployed and then I can interact with the service and try to change myself or like anyone in the community. Like this is the perfect example of what we are aiming for. And you can do this, like this is for example, cook with podman, you can do it with other projects like, okay, do your Kubernetes as well. And on the next slide, I wanted to show you how we are doing it in our project in Becky. Like our project has many features and one of the features is that it's a GitHub app. And we have two separate deployments and up to our users to pick, like if they want to be on the one which is latest which sometime might be not so stable or if they want to be on production, which is stable and that's like that's the main thing. And they'll just pick it by like picking different GitHub apps. And you can see that we don't have the router thing here because users still need to opt for the specific version. And if we had the router, there would be just one and we would have some different logic to do it. And in my final example, this is open data. And I would say this is an excellent example of the like stage one that you can contribute like changes to their container images or to the manifests how they are being deployed in OpenShift. And you don't need to know how to deploy them. The CI system will take care of it but there is no way to like perceive your changes. Like you still need to deploy it locally if you want to test it out but the contribution workflow allows you to like so that it's tested for you. You don't need to know how to do it. All right, great Tamashi, thank you. This was a lot of interesting stuff I have to say and I'm pretty sure there are some questions on the audience. So guys, if you want to ask something right now please use one of the chats the stage chat here in Hopin or ask through the Q&A section. Before you do so, I've already collected some questions from our Twitter and from our Discord chat. So I'm gonna ask a few of those here right away. One thing that I'm really worried about is data privacy and user data. How do we ensure that the data are not leaking from some of these services or how do we make sure that for these stage environments I can use the same data as I would use for the production environment? Steph. Yeah, so there's a couple of things we can use here. One of the things you'll notice when Tamashi went through that is many of the techniques he shared were existing techniques used towards this goal, this challenge that he wasn't inventing anything new and I believe that there's existing techniques we can use to solve the data challenge. One is the fact that in open source we regularly run code against our data that other people have changed. But we do have a step that happens of review of other people vetting that code before it touches our data. So it's important for us to work on this and make sure that in this contribution model to services the contributions can only touch the data of the user that they originated from. You'll also see many projects have an okay to test flag or some other indication in their workflow that says they're ready to allow their CI infrastructure to run on a contribution. That means a maintainer may have reviewed it sufficiently to know it isn't a Bitcoin mining bot or some other thing that does crazy stuff. Similarly, these contributions would certainly need a review step of some sort before a contributor or even a team member we can deploy their code against real data even the user's own data. So by bringing these kind of techniques together I mean there's probably more here. I know that in the SRE space there are techniques for ensuring that only the software can touch the data and never the operations code for example or other things like that. And we can work together to figure this part out it's not completely done but I'm pretty confident we can figure this out. And I would add to that that in security one thing to solve this challenge never really worked security by obscurity. So instead of just don't touching it and making everything close I think it's a real problem and we don't have answers to that. So let's open it up for research. There's a lot of research going on in data privacy data governance just because data is also one of my main concerns these days data bias and a lot of things are going on there also in the AI domain. So that's something that needs to be actively solved and not only in regards to data that you are producing in your deployment but also in how we exchange data. So I think we're beyond a world where I'm just sitting on my pile of data and I'm trying to do my best there but I'm also working on data that is coming from my competitors from people that I'm working with from my customers and such. So how do we exchange that data and make it still secure? How do we make it private? I think that's one of the main challenges ahead and we made that good for face recognition voice recognition, but we really suck at doing this with the data that the computers are generating. So Marta, I get that and I get the contribution model for services but what about the operational side? How would we contribute to the underlying services that power the whole application, the whole service and are there any specifics for that? So the how is, I think that's pretty straightforward. We just in air quotes need to replicate what we did with software for operations and that doesn't exist yet. So if I want to jump into a software project I'm looking at on GitHub on beginner labeled issues and I can just contribute there. How do I get into operations? I can apply for a job, yes but can I set up a reproduction environment somewhere? No, because there's money involved. So opening this up for a community approach where people can actively contribute to running services, running productions. I think that's critical for also for the educational piece, for growing new talent that is willing to go into that SRE route. Obviously you still will have some deployments that you can't completely open up. So I would say there's a sort of an open cloud that is running workloads that are not super mission critical where we can reduce the SLAs and we can break things and we can get PostgreSQL developers running and production great service of their PostgreSQL database always with a caveat that it might go down that you shouldn't be storing private data in there that you could lose, right? But then building on top of this operational excellence that's captured there, we can ship a baseline to the actual production great running PostgreSQL databases and train on that data and train really, really, yeah, turn it into code. So that's the business side of it. Great. Question for Tamayesh, probably. Do you see any other services that would benefit from this approach? You gave us some examples where pieces of this audit work but what other things would be the next step for you? Well, all of them, I mean, Ratko, you tell me. I would love to have my Gitlub running. How would I do that? Well, I would say first the Gitlub as the upstream project would start need to participate in, like would need to enable this kind of contribution workflow that you contribute to Gitlub and then you are able to experience it, let's say one hour later when it gets deployed and there is a bot in your merge request would give you a link, like how you can try to change out. But I mean, if we were at real DevConf inside the Chateau, I would actually ask the question to the audience, like what services you would like to see that they would use this kind of contribution and operation model? Yeah, what services would you like to contribute to? Put it in the chat. But I will say Gitlub is already pretty far along on enabling contributions. I mean, obviously they've doubled down on making sure that you're able to run your own instance, which is called the Gitlub development kit, I think. And it's fun to get started with. Do it in a VM because it does all sorts of crazy stuff to your system. But also they have a staging environment where recent merge requests before they're deployed to everyone get launched and you can go to a specific URL and experience those. So they're not all the way over where people would experience a change necessarily before it's merged in the hosted service. But as far as services walking down this road, they have taken those initial steps and do have quite a few contributions even when you look at the numbers in the data. I would say it's every service that you really care about. So that's how open source software got big, right? You have software and something that bothers you. So you dive in and you try to fix it. You contribute because you had some itch scratching you. So why not tweet to Twitter and say, look guys, I'm missing this feature. How can I contribute to you? Go out and raise your voice. So you just made me think about one thing. This sounds like we're gonna end up with many clones of different services. And my question is, is that okay? Or do we wanna prevent these clones to be spinning up? I guess that same thing will apply to open source. So I guess fundamentally we'd like, of course, I would make the claim that many services when you clone them, they are no longer the same service. But in such an implementation model, like Tamar said, there might be infinite behaviors depending on who configures the load balancer to connect to what pull requests in the background and so on. And how do we keep that from getting out of control? Well, we do that pretty well in open source where if you fork or you make a pull request of a project or change it, suddenly you stop getting everyone else's contributions. Everyone else's changes. And this incentivizes you to get your stuff merged if you think it's worthwhile because then it becomes part of the default behavior of everyone and you can continue to get this stream of changes. If you just keep your change to yourself and say, hey, this is good enough for me, then you give up on the rest of them. And I think we can apply that same model here where such an outstanding contribution doesn't get everyone else's changes during that time. And it incentivizes you to complete the work sufficiently to bring it back together and merge that brand, try to then continue to have the fork. And I wouldn't even consider this as a bad thing if we have multiple instances and flavors of your service running. So the question is why did we in the first place stripped down the features and say, okay, I'm just serving out the smallest common denominator of features to the wide audience because I only have a small team that is capable of running the service and we must make sure that it's really running reliable. So if we solve this problem that we can really scale out horizontally also on the feature level and let everybody contribute. So I wouldn't mind having my own version of Twitter available with say GPG signatures for tweets that I can see whether they're coming from a bot or not. Maybe nobody else cares about it but I would like to see this feature. So if Twitter would be able to run different flavors of their service without being impacted on their commercial model, I don't think it's necessarily a bad thing. Great. Mirek brought up an interesting point in the chat and GitLab is not a great example because it's not a true open source project. And this made a thing about licensing. Have you guys thought about how licenses would actually work in this ecosystem? So I believe the licenses are fundamental to sharing the code. The licenses operate on copyright and obviously we need to continue to use real open source licenses to share both the source code for the service and the operational code that is open sourced. And so that's a foundational part of this open sourcing the work. However, by and of itself it's insufficient to incentivize contributions to a service. Yeah. I just recently heard that the license that you choose also depends on your business model. So I'm not really a lawyer and I usually don't care about the license too much which is a shame to say here now in the public but I usually take code and I tweak and tinker with it if I have the source code. Somebody else should care about whether the license is okay but that's nothing I should say in public. But it's not only about the source code but it's also about the data. So thinking about the data that is being produced from your ops team, how do we slap a license on this? And the Linux Foundation just came up with a license for data two years ago, LFA from the Linux Foundation AI. So also think about how you could, if you run a service, put a portion of your operational data, of your services, of your tickets and such under a certain license so that it's for posterity and for research available. I just read a comment in the chat from Rado, he's talking about open source alternatives to existing services. But one thing that made me think here is that some of these services, you're not able to run them locally. You actually require a huge infrastructure to run them on. How would you make sure that a community has access to such infrastructure where they can run the services? If it's running on a public close, someone has to pay for it, right? Yeah, so there's a couple of things going on here. First of all, not everything, I can't run locally everything that Fedora produces. I cannot run many of the IoT builds. I cannot run the S390 artifacts and so on. Open source regularly produces things that not everyone can run. So there are barriers, I don't have access. I mean, I could get access maybe if I figured it out, but I don't have access to many of the architectures and systems that are necessary to run Fedora. That said, as we described, making sure that everyone can operate your service is necessary. Making sure that you bring operators into the code. In this case, we're kind of hinting at Kubernetes and OpenShift operators, but there's other ways to do this. And making sure that much of the logic, much of what's proprietary or much of the things that ops teams do are encoded in code, that's what Marcel was going after in the beginning. And this really opens up the amount of people who are able to operate a service. Those people, of course, in many cases will have a slimmed down behavior of the service. Why? Because the service connects to other services to do things like authentication, connects to other services in order to call for a certain function or connects elsewhere. It's an entire constellation, really. And so if you clone it and run that, the one service yourself, you may be able to, you get all the code for running it, but you're running something different than the original one. So in addition to enabling everyone to run that themselves, we also need to enable contributions to the service as it stands, or else we're putting up too high of a barrier to contributions. And I'm not super worried about the money factor here. If we produce something that has value to companies that can monetize on the community that produces great artifacts, then they will happily throw hardware at it. Remember, if the service is free, then you are the product. And I think something similar will probably happen with some open source services, where the service is free, yes, and you can tweak and tinker with the operations, but the data that you're producing there will be used by the community in return to create better products so that companies can sell better versions of their products. In the end, somebody has to make something out of the money that they're investing there. And we showed that it works with open source software. So why shouldn't it work with services and operations? And it actually already works for some if we look at how NextCloud works. I mean, their product is completely open source. You can go deploy it locally in your home easily. And at the same time, they're a company and they're making money. Works. I'm running out of questions here. So guys, do you have stickers? I would like to talk a little bit more about the open core aspects because it's going on in the chat. And it is true that open core is a crutch and it's unfortunate to see certain companies like GitLab or others rely on that crutch because they seem to believe that there's no alternative. I believe that if we truly enabled open source on a service and contributions to that service directly, that there would be such a community around it and such a center of gravity of people participating that you would not need to rely on crutches like open core in order to sustain it and in order to have a business model around it. Obviously, this is theoretical and unproven but it is a conviction that if you actually go all the way with open source, you start to see all the advantages of it. And if you go halfway, yes, there are risks that you then mitigate in awkward ways such as open core but if you actually enable this completely you get such an advantage from it that it is sufficient to outweigh those risks and you don't need to rely on crutches like open core. So let's see. I think if we pull this off, we will see a viable alternative to open core. Great. We're running out of time so I would encourage folks to continue the discussion. Either on Discord or here on the chat. Fox, it's been really great to have you here. Famous last words from all of you. Yeah. So there's one urloperate-first.cloud. That's actually a URL, type it into your browser and you can join a community where we operate OpenShift and Open Data Hub in a community fashion. And the first talk just starting now shows you how to do this with an AI application on Data Hub. And we'll go into more details on these topics and perhaps have more discussion on them tomorrow at 2.45 and your service is not open source talk. And I would encourage you to also click the other link down below to look at this challenge, some of the playbook that Tomasz described and other ways, user stories and other ways to participate. All right, Steph, Marcel, Tomasz, thanks a lot. You've been great keynote speakers.