 Thanks for joining me for this last session of this year's DEF CON. Oh well, penultimate session, I should say, because there is a win-win-win session today with amazing prices I hear and amazing quizzes. So thank you for bearing with me. I'll try not to make this too long because I know you've had an amazing conference and all your heads are filled with these amazing ideas and the new information that you got. But the thing that I want to talk to you about sort of ties into the overall theme of this conference and that is that nobody knows that your service is open source. So this doesn't work, let's use that. What is the talk all about? First of all, let's start with the challenge of services to open source. Then let's continue to understanding what does it mean to be an open service? What are the basic requirements for any given service to be considered open source? And then let's actually decompose that a little and look into an actual tangible list of things that you can check against to verify that your service is open. Then let's go into one example. Let's look at ImageBuilder and see how open ImageBuilder is. And finally, let's figure out how you can let everyone know that your service is open source, which sort of concludes the talk. With a demo and then finally a quick poll for a logo for all of this. So the challenge of services, yeah, go watch the keynote. This is something that we've talked about at the keynote of this conference. I can say a few words about it, but I don't want to talk about it at length because it would really mean repeating the keynote. So open source is facing a challenge. Services and managed service offerings do not necessarily have the same implications that other open source services have for our way of consuming the software. We're not really copying it onto our drives for our way of inspecting the code because if the code is not here, how can we make sure that we can actually look inside? And also for changing it and then for contributing it back. So our entire flow basically is questionable all of a sudden. And the question is how can we fix this? How can we improve things? And that is a very general and big question that I can't answer here and right now, but there are already services and that was also part of the keynote that are open source. And it's just not really discoverable about how we can use our sort of the workflow that we know from contributing to open source and from inspecting the code. How can we apply that to services? And that's something that I want to look at today with you. So first of all, let's start with the first bit. What is an open source service? Steph already mentioned this at the keynote. We have two very basic requirements that we've defined. All the code and assets have to be open and the public should be able to contribute. I mean, these basic requirements apply to all open source software, I would say. Like generally, that's what it means. You're able to look at the code and to look at everything that makes up the software and that you're able to contribute. And the same applies to services, but as I said, it's in a slightly different way because you don't necessarily download the copy of the service locally and run it so you're not able to necessarily inspect it or to contribute back. So let's decompose that a little bit. Let's look into what could this mean? Like what could these two requirements mean? Because just looking at the two of them makes you think, wow, this is a bit incomplete or at least it's very general, you know? Like sure, the source code could be somewhere but it could be accessible but it's possible that you don't really know about it or that the contributor workflow isn't even described because nobody ever thought that anybody outside the team that is actually building the service is running the service would ever have any interest in contributing. So how open is a service? The first two requirements sound really familiar, you know? You want to have open assets and you want to have an open contribution workflow but at the same time this implies many things around those two requirements that also need to be met. So I will go through all of these in more detail now but just roughly, you know, if you have a contribution workflow that heavily relies on documentation being available, you need to be able to follow instructions, you need to learn about the service to be able to contribute back. You need to be able to submit tickets or read existing tickets in order to be able to understand, you know, which issues have been discovered before, what has been figured out before. You need to have a way to communicate maybe with the authors and maintainers of a service. So open communication is definitely key. And then finally, and that's a bit of a stretch goal of course, open operations. So open SREing of a service. So making that as open as possible. Actually, Michael, who also had a talk today on incident management, he came up with this open source service scorecard system that I based these six points on. I think they make a lot of sense. They may not be complete but I think they give you a very good starting point for understanding how open a service actually is. So let's look into what open assets means because it sounds very obvious in a way but what does that all entail and which parts of a services code are necessary because there you will discover the differences between traditional software and services already. So the first one is obvious, the source code is open and the second one unit tests are open. That's also sort of something that we're familiar with. Like we have unit tests in our software in general. We might even have performance tests and functional tests but these become more and more important when software is long living and running as a service then you need to have these types of tests. Necessary of course is also that all the dependencies are open source because otherwise how would you be able to understand how systems tie into each other and how many dependencies make up the end product. The final service that you're consuming. And then the sixth point is where it really differentiates you want to have the metadata on how the deployment works. Why it says metadata and not just deployment the deployment is open. You will have secrets in your deployment instructions. You might have a YAML file lying around somewhere that's called deployment YAML and it describes how your service is being run I don't know on OpenShift or on any type of cluster. And you of course don't want to expose your secrets but you want to expose the template. You want to give people an idea of how is the service actually being run. And there is really nothing bad about that. I would say there is very little risk involved with telling people how the service is stood up. Then the contribution workflow. The first point is very obvious probably that external contributors can follow the same workflow as team members. It sounds very obvious and it's something from traditionally working on open source software we used to. We download the source as I said and we fiddle with the files and maybe the team was very nice and included some editor config or other sort of instructions that your IDE is set up in the correct way that your contributions will go in smoothly. But overall sort of traditional software is meant for local execution while services are not. So you need to have a way of actually following the same workflow as a team member that means that could mean developing against the staging instance that is publicly available. It could also mean locally bringing up the stack in some way be it through I don't know Docker or actually running a deployment locally if that's feasible of the service. Then of course the second one is also very obvious this workflow needs to be documented. It sort of ties in with the documentation requirements. I'm not repeating it there but this workflow has to be very accessible and readily available. The third one is sort of again opening up the workflow to everyone. Contributors that regularly contribute to a service should be able to run CI. Why is this, I mean why do we have to mention this is whenever we submit a pull request CI runs and that's it. But running CI on services can be expensive. It depends on which kind of testing you're running. Sometimes you're bringing up an entire cluster trying things out. Maybe there are other services that depend on your service and this can be a very expensive operation so you might not want to expose this fully to everyone at all times. That's why it says regular contributors so that you get the mental model of people who regularly contribute to your project get a way of doing this. Be it through a team member approving it or having a separate group, enroll best access control where regular contributors can get in and then their PRs will automatically trigger CI. And then the fourth one is really more of a social thing I would say. I mean you can codify it, you can make a rule in your team or you can really make it very clear but external contributions shouldn't be stale. They shouldn't go unheard and this is something that I mean the more contributions you get, the more challenging this becomes in any type of open source project. But external contributions are basically the future lifeline of any project. If you don't review them at some point this flow of contributions and this interest might dry up and the project might die out and the same is true for services. So this is very important in terms of having an open contribution workflow. Issue tracking and planning that's very straightforward I guess, having an open issue tracker and exposing all the issues to the public and having an open roadmap. I guess the level of detail that you include there or sort of what that roadmap entails that may differ but overall giving the public a way of understanding which direction your project is heading or which direction your service is heading which things you intend to implement next. Documentation is also fairly straightforward. You want to have user documentation and developer documentation but in this case the developer documentation needs to include things like operational instructions. So it needs to be clear what the people that actually built the service expect around the maintenance and the running of that service. Well, communication, sure, you need a public mailing list of sorts or a channel where people can reach you, it could be Matrix or RFC or anything else and ideally there would also be public meetings. This is a little bit of an optional requirement I would say because public meetings really only make sense when you do have an audience of regular contributors that want to engage with you. So I mean offering it and setting it up initially might be nice but if you discover there's no interest there's probably no use in maintaining public meetings when nobody joins ever. But I think it's very good to keep in mind that the entrance barrier sometimes can be very high and very difficult for people to get into a new project. Yeah, and then finally, sort of the stretch goal, having an open status page, having open logging, monitoring and alerting, that of course implies that you have a way of effectively concealing secrets in logging so you have to be very intentional when you're logging you need to really know what you log and don't just print everything to a console and then just open it up to the public that might be actually exposing a service to attacks or vulnerabilities. But I think it's sort of if you do it this way, if you open up your logging then you very intentionally set up your logging. So I think it's just a very good mindset and very good practice. And open incident management, that is really what also Michael talked about I think two hours ago or something. In his talk, open incident management is a very good practice because there is no shame in incidents. They happen, they happen everywhere. There are some really good examples, I think also from companies like GitLab really having very open incident management. And yeah, I've never really heard anybody laugh about their incidents, it's something that of course you're not proud of but the important part is that you learn from it and yeah, that you also give others an opportunity to learn from it. Okay, so let's go into an example. This was a gap analysis that we conducted in our team nine months ago. So we looked at these criteria that I listed for you and we asked ourselves how open is our project? What I need to add, I mean, I don't want to tell you what image builder is because Andre very skillfully did that about 40 minutes ago. So the important bit to know is our project is a hybrid project in that it is offered as an on-premise version that is packaged in Fedora and in Centro Stream and in Well and is also run as a managed service. So let's put it this way, we had probably an advantage to services that only run as a managed solution. Parts of our openness, I think, come from this aspect. You could always run the service on your laptop or on your local deployment. So therefore you were able to download the assets and inspect the code anyway and all of our code was on GitHub at all times. So the first requirement was easy for us to satisfy. The contribution workflow was maybe a little trickier but we luckily had an existing Docker stack that would bring up the entire managed service. Like you could do both. We have basically a way of setting up the development environment for the managed service but also for the local deployment. The next one was already a little more critical. So our team is working at Red Hat and of course the issues that get created on GitHub are public and of course the issues created on Buxilla are to some extent public but we have an internal Jira instance that we use for tracking and planning and where we recreate the tasks and the issues that make up for our plans and for our roadmap going forward and that was still internal at that time. Also documentation, there was a very noticeable gap. I mean our documentation was there. It was all upstream and also available to rail customers but definitely not about the service. So basically there was some developer documentation that was really geared at people running the thing on premise and running it locally on their computer but we never thought that it would make any sense to include service-related documentation because our team is building the service, we're maintaining it, everybody else is just consuming it so all we really need is user documentation but if you really want to enable anybody else to understand the service, to be able to contribute you need to open up that information as well. In terms of communication, again, I guess we were mostly thinking at the time, it's our team that's working on it, we don't really get a lot of external contributions, why would we have much set up there and we basically had an IRC channel and we said if anybody wants to reach out, they can but internally we were using Slack in the team and then I remember I think one year after I joined I hadn't looked at the IRC channel in a while and I looked and then every time a day goes by, there is an ERC at least, there is a line injected, oh the day passed and then I scrolled up and so somebody asked a message and it stays and stays and stays and no response. It was really weird and then I noticed that, yeah that's really not, that's not really the image that we want to convey to any type of external contributor that we don't really care about that. So yeah that was very helpful to realize that actually our IRC channel was orphaned and that nobody would really be encouraged of interacting with us if we don't react. And then finally open-cycled reliability engineering, at that point in time we had none of this, we had no open status page, no open logging, alerting, or incident management. We hadn't really thought of this at all. I mean maybe another piece of information at that time, I think when we ran the gap analysis, our service was I think in production for something like five months, it was publicly available. So it was still very early. And that's I think why it was also very valuable and great that we were able to tackle these things very early on. So how does this look now? Like basically from a couple of days ago, I updated this slide and I look at all of these things again. But first of all, there's no more reds anymore. There are still things that we're not yet doing, but in aggregate in every section we have at least one green check mark. And yeah, green and red, for me makes orange in this case. So if you look at issue tracking and planning, this was a bit of a process. It meant informing the team, informing the business, anybody who interacted with our issue tracker, it is public now. There's of course still a possibility to write sort of hidden comments as there is in Buxilla or in probably in any issue tracker because sometimes you want to put information there that is somehow relates maybe to secrets or to customer information that you don't want to expose but generally speaking, all of our tickets are open. In terms of documentation, we worked really a lot on our service related documentation to make it easy to understand for everyone looking at it, how is the service deployed? What's the architecture? Which components are involved? Sort of where are they deployed? How is everything working? In terms of communication, we started by creating a public mailing list because we figured that if you have to actively look into an IRC channel to get a message, if that's not really part of your routine, then messages may get lost but everybody somehow looks at email. And if the email pops up in a mailing list, there's a big chance that somebody will see it and will respond. And we also registered a matrix channel and several team members were very active and wanted this, so I think it will not be often soon. So one of the remaining gaps in this area is that we still don't have any regular public meetings but as I said for me, this is sort of more a thing that is, well, ideally we would have public meetings but then we'd also have to have an audience and we would have to have regular contributors. We of course have that because within Red Hat we're collaborating with other teams, we have meetings with those other teams but we don't really have an external audience yet. And I would hope that this would become a thing. So yeah, that is the hope. And open site reliability engineering, we have an open status page now so you can look where the image builder is working and running and we still don't have open logging alerting or incident management. I think that we will work on open incident management because I think this would be very valuable and it's very achievable. I'm at this point less sure about open logging and open alerting, maybe alerting but logging and still very uncertain about it. I think that would take a lot of effort from our team. Right, so this is the example. I mean, as I said, our team is working at Reddit and we're not the only ones going through this process. So this is really an effort that goes across many teams in Reddit and this is why I'm particularly proud of the next slide which shows that on console.reddit.com and in the inside services, all of these teams have gone through this gap analysis. So these are both screenshots basically of the analysis that they've done. You can see there's a lot of greens, there's also a lot of reds, but they're working on it and this is something that I am really proud of. You know, that generally speaking, people have really understood that message. So when I approached them, they were very happy and very ready and actually Steph was key in that effort. And so yeah, I hope that we can get to similar results that we got to with ImageBuilder with all of these other services. Right, we haven't really spoken about the actual problem that ImageBuilder is an open source service. We consider it one, but you probably didn't know. You know, when you enter this room, maybe you knew about ImageBuilder from Andre's talk or from Achillea's talk, but you probably didn't know that it was open source or how open source it was. So we've defined four requirements together with Steph, like what, how can you make sure that your service, that anybody recognizes that your service is open? So first of all, there is a visibility requirement. If you know, you should see something, some kind of indicator that lets you know that the service is open. Secondly, there is sort of a criteria of understanding people should be able, from the reusable thing, they should be able to understand what is the service architecture, what are we looking at, you know? What are the components that make up the service? The third one is actually more tricky than it sounds. It's introspection. Somebody should be able to inspect the code, but not any code, not, you know, your latest GitSHA or your experimental branch. They should be able to go to the exact version of the code that is currently running in production, because otherwise, really, it's a little pointless, you know? You might have fixed things in your main branch already, but maybe for some reason, you know, it hasn't really been deployed yet and it would be really misleading, you know? It could also mean that if you run into a problem with an API and you want to look into the source code to verify that maybe something's going wrong there, maybe there is no typing or nothing that will prevent a certain error from occurring, if you don't really have the code that is running in production, you might not really get to the answer you're looking for. And then finally, also contributions. Somebody looking at that, at that reusable pattern should find a way how to contribute. Right, that's it. Doesn't sound very difficult overall, I think. And we've tried to come up with something that I will show you now. And if you go to this URL, you will also be able to experience it. So this is ImageBuilder in reddit console.reddit.com. And at this point in time, and we will be talking about this in the logo poll, this looks very, very integrated. So here you can see a little branch icon that sits up here and that you know, if you're sort of a developer or a software developer, you've probably seen this before and you might think, hmm, what does that mean? And if you click it, it tells you this service is open source and you can explore all the repositories to view and contribute to the source code following this link. I've conveniently opened the link already in the next tab. That's what would happen anyway, but just to make sure that we don't get any network hiccups, let's just jump to that page. So here you can see immediately what is the architecture of the service. You see all of the components that make up ImageBuilder at this point. You see the front end, you see the back end, that both of those are in the console.namespace. You see that ImageBuilder Composer is actually in a different cluster. Some were completely different. And then you also see that there's a work of lead and there are some databases that are running attached to both services. When you want to go into one of these, when you want to click into one of these components to explore the source code, you can of course click and you will see that it does take you to the exact hash that is currently running in production. That works for all of the components, of course. So as you can see, this satisfies the requirement of introspection, of being able to introspect the code that is currently running and the criteria of understanding. You can see which bits and pieces make up the service and maybe from your error or from your interaction error, you will already know, ooh, this was a front end error, whatever. I got some weird pop-up that shouldn't be or the button was three pixels to the right and I really hate that. Let's fix that upstream. That will be my reaction probably. Then of course you'll be wondering how can I fix things upstream and then there is this section that tells you how to contribute. So we have a developer guide. We have actually a development stack that you can set up both for the front end and for the back end. The back end of course includes the entirety of the stack in this case. We have a way how you can reach out to us sort of the matrix channel, the mailing lists, but also the issues and pull requests that your links are still missing here. I should add those in, but they're also publicly accessible. And then finally, you'll see the assessment that I showed you as part of this presentation also here. So you can see which things we're actually doing and which things we're currently not doing. I'll probably try to improve this a bit over time and add more hyperlinks that you see that the source code is open and unit tests and sort of you get a bit of a hint to each of these requirements, how we satisfy them. But at this point in time, I really just wanted to show how open the service is and make sure that everybody who finds a gap or thinks that we don't deserve a green light where we currently have one raises it with us. That's one of the reasons why we want to make this transparent and we want to expose this. Right, so this is how it looks. This is basically the thing, so a badge that you can slap on your service that makes people understand this service is open source, but that also takes you to a page like this that then explains how does the service work, which version of the service is running, how can I fix issues in it. Right, and now, I mean, as I said, this is just a button, but we did try to make this as easy for you to use as possible. It's really just three lines of code. It's a reusable thing. The idea was always to create something reusable also to make it consistent. You want to have the same experience at all times. It's very easy to write the JavaScript thing that does a pop up and stuff and then every service, the wording is different or maybe the icon is different or the placement is different. You really want to have a consistent experience. That's why we started these within our organization to use the frontend components library and just add this new widget there, the open source badge. And as you can see, it only takes one argument and that's the URL to the page that you want to refer people to. We don't want people to change the wording. We don't want people to fiddle with the icon or anything like this. It should always be uniform because that is how we satisfy this first criteria of visibility. It should be recognizable. You should immediately know what to expect and how to get it. And then finally, we're at the interactive part of this session at the end. We're currently thinking of what kind of branding do we want for this? What should, you've seen that I currently have added a branch icon because that was available in pattern flies icon library. So that was easy. But we were really thinking or we were really into minds a little bit. Do we want to create custom branding, something that stands out, something like the OSI logo that after a while and after you've seen it repeated in many, many places, you will understand what it means and it will mean something very specifically. And that would be opens our service. And that would be option two, custom branding. Or should we go for something that we currently have, maybe add some color coding to it just to make it stand out, to not make it look too integrated into the service. That's very technical. It means something to everybody who's a developer. So maybe it already reaches the right audience, but maybe not. So I think we're a little bit into minds here and I would be really interested and after you've all heard my talk, which option you would actually be interested to see or which you think would really solve that problem having an icon, seeing it annoying. I've seen this before. If I click here, this service is open and I can actually inspect the code. So maybe just with a very quick hands up, I know you had very little time to think about this now. But maybe just give me a gut shot. Two of you would go for the option one, basically the known icons, the ForkMe icons, the generic icons. Okay, that is 15. And who would go for option two, custom branding? Okay. The public account provides. Are you holding both of your hands up? Okay, so that was 15 to nine. That sounds like there's a slight preference for the ForkMe direction. We haven't really decided anything at this point. I'm very happy to receive more opinions on this to be honest, but generally speaking, I want to create something very soon that can be reused and that can be shared out through all of redhead services and through all of console.initially, ideally later on also for anybody. It's not really, it's not, it's maybe something that starts at redhead but it shouldn't be something that is restricted to redhead services. Yeah, and with that, I'm at the end of my talk. Thank you for coming in for listening to me. And now it's question time if you have any. Go ahead. Okay. The problem with option two in my mind is that the hope to establish it that's something that people recognize because until you do that the risk is that nobody will understand what it means and then option one is preferable. On the other hand, if you can't establish option two, then it's of course better that the term something was specific. But until you get there there's, I mean the risk that nobody will understand it. So how do you get to establish it to something else? Exactly. Thank you. That is the summary of the conundrum that I was facing where I thought, I'd ask all of you for your opinion. No, it's really, that's exactly the point. You know, how do you wanna get the thing off the ground? How do you wanna get people to click it and to actually look? Control services becomes more mainstream. That's a better time to try to move to something more specific. Yeah. More than people know the basic concepts. I mean, the thing that I'm afraid of a little bit is, you know, there are other projects that do open source services already. There is like GitLab, for instance, GitHub also to some extent, like some of its code base is open. Even at last in shares, their code base, not in a very open way, like they don't have a repository that you can commit to or that you can submit changes to. But you know, sort of this concept, it's not really, it's not new or it's like the main, and that's also the thing I don't wanna necessarily establish a Reddit brand. I would rather have something that works for Reddit but also for the community, something that can be very easily repeated without anybody thinking, ah, I'm slapping a Reddit logo on my service. Why would I do that? So I think the barrier has to be low and it has to be accessible immediately and you have to understand it when you see it. And that's, I think, the tricky part with custom branding. As you say, you need to first establish it a little bit and have several services on board, which is why I want to have their usable pattern because then of course, you know, we can swap it out at some point. Martin. Yeah, I want to buy shit if it's on the cost, no. Okay. So we have some practical experience with actual contributors from outside the team that tried to read the documentation and then set up the server who sent the PR. Yes. Yeah, that was a very good point. So, of course sort of. It's like the difference test. Yeah, yeah, exactly. So this part, this part actually, we should probably make that explicit here. You have to have at least one contribution, you know, that proves that this is all working, you know. So if you don't, like how can you prove that external contributors can follow the same workflow? You need to have a contributor that has done it. So yes, we have had this. In our case, it was somebody from another team in Red Hat, but we've also had many contributions from the public, maybe not necessarily always to, you know, the entire service stack. And, you know, as I've showed you in our architectural diagram, there are lower level components anyway that it's very different to contribute patches to those. You know, also the development setup is very different for the lower level components. You can run them basically on their own. They don't necessarily need to bring up the entire stack that runs in console work. But of course, if you want to fix something in one of our APIs, you have to be able to do that. Otherwise, it doesn't work. And yeah, we have, so we have examples for this. Yeah. And I know this is something that affected us very much. Like I've lived a couple months ago, even inside our team, you won't have different development environments and there will be like several code students come in and try to follow these and come completely stuck. And this is where we set out that maybe they are not as good as we thought, like there's documentation. I think yeah, I think this is, yeah, I think this is why this is overall, you know, even if, you know, the question can always be, what's the business value of this whole thing? Why should your employer or why should anybody go for this? You know, why should anybody support this? And it does make you much better as a team. Like these are the workflows that we're used to, the traditional open source workflows. And they're currently, if they don't work, so for instance, in Reddit at least, I can speak for that environment a little bit. We've broken this paradigm with services. So teams have a very hard time contributing to each other's services because it's not obvious, it's not the same thing anymore, you know? You don't just go simply to the one repository that has the same name as the package and that is in a place where you would expect it and there are all of these links. Sometimes you have to click, you know, through several repositories until you find the thing. First you go maybe to Fedora's package and then you realize, ah, that's the upstream and stuff. But there is a way. You can sort of retrace the steps and that's not necessarily the case with services. It's sort of much more intransparent. And then, yeah, even across teams or even when you onboard somebody new into your team, these are just best practices. Initially when somebody new onwards onto your team, they are like an external contributor. They maybe have no rights because they don't want them to do all of the things already. So I think it's very general and it's just many of these things are general best practices. They're not revolutionary and not new in that sense, but I think overall they make for a very good assessment of how you're running your project. Any more questions? Because we are out of time, so thank you for your attention.