 Good morning Mark Always special So as my colleague said my name is Lydia Schmidt. I work in program management team. I Do well for money more years than I can count And I'm not sure that's good or bad. I'll let's pretend it's good Before I start I would like to know how many red heteros are in the room. Can you raise your hand? Oh Oh Okay guys, so I'll go through how we develop rel with focus on how we Focus on upstream So if you know this a lot and You don't agree with me are there was going to be like at least 10 minutes of questions after this one Feel free to shoot at me. I know I survived worse discussions But let's let's dive into it straight away, right? So what I look at I'll start with what's real with some really, you know basic definitions Then I look on what's how we actually work with upstream So that's actually basically how we touch what's outside of the company how we work with the company's etc Then I'll look at how all of this actually impacts what we do and Then I'll try to wrap up. So Let's get started. So And I was thinking where else to start Defining rail and like rails open source Distribution right so I started to know what's actually an open source software and you know when I was doing the research, you know wikipedia is a great source of definitions, so I use this one and For me, it was a great refresher how actually the open sources is defined and Don't forget that open source software is just a special case of open source There's another term in wikipedia there and it's basically, you know a Way or type of computer software that allows sharing between users The other piece I was looking at is how you can actually make money on something that's That's for free right everybody can download it compile it run it Deploy it etc. And when I was looking through articles, I found four major ways that Can be actually used to run company that produces open source software So the first one is support and services. That's where redhead is right? We provide subscriptions I know we provide consulting services. We provide support and this is the value added that actually customers Buy from us. They don't buy the software They buy all the stuff around it and they pay for it on a new basis You know, we have different type of types of contract So this is what redhead just know the refresher for the others So that's the business we are in there's actually two or three more ways That seems to be sustainable from a long-term point of view. So The the next one is advertisement partnerships And What you can imagine behind that is Mozilla foundation Mozilla corporation Because of what they basically do is they get in contracts with companies to Position searching services high in their list or you know, setting something as a default service Like no yahoo paid Mozilla years ago. Almost 400 million dollars to make yahoo as a default search engine So that's actually another sustainable way how you can run business in open source The third one is paid additional features what I mean by that is you know, and that's a lot of company do You basically Provide core of the software as an open source and Then features that are interesting for enterprises. I know you actually charge for them this for example You know how my sequel gets funded? Know all the all the distributed database stuff, you know You can get that for free and there's some conflict in there because basically, you know by open sourcing Just part of the product you get sometimes to the end of it to the conflict with communities as people You know may contribute features that are part of your proprietary offering And I know there could be pretty bad clashes between the community and the company behind the software and Then so there's a service. That's like a body on that wordpress uses No, you can get wordpress as a prop no open source content management system in style on your server No run its no no charges, right? But if you want to avoid all the administration tasks and just Use the blogging services you go to wordpress.com you pay some annual fee and they maintain the service for you so that's like that's the Last from the Sustainable I found like 40 more No from Kickstarter to set telling the selling t-shirts to pay what you want. I know all of that But you know, I couldn't find any evidence that this is something you can run a company on Because like t-shirts eventually, you know, you sell it to all the users you haven't you are done so This is to know to understand what the what's the open source and what could be the business models behind it and where rel is Then well rel is only loose distribution. So this is one one more definition what the distribution is and again Wikipedia No, you can read through it. But you know where it connects to open source is actually the this paragraph That actually says that typically our loose distribution Includes open source software and free software that are available both as binaries and as a source code. So Based on all of this you now know or refreshed, you know, it's easy to say a redhead enterprise Linux is a Enterprise great Linux distribution with support and service business model, right? That's a very short definition But that's only one way how to look at rel It just gives you the what's the business behind it how rel looks, but it doesn't give you any other Dimension of rel is and rel is like a huge beast what I mean by huge It it is something that's big and when I was thinking how to show scale of rel I was thinking hmm. So one way how to look at it is How many source rpms we had We used to build a server version of the system and the source rpm is basically Like you could make it equal to one upstream project that's being packaged in a way that you know, we can distribute it So this went from like nine hundred in like two thousand six four ten advanced server through 1300 that's 511 To 1900 that's 610 to 2200 in 76 last fall and You know 80 beta had a little bit over that So that shows you What we are dealing with in terms of how many projects we need to integrate together to work as one piece of software because customers don't care about This package or that package they do care about the functionality that the whole thing provides So that actually tells the story of how we were Getting bigger and bigger and bigger in terms of complexity. We have to deal with right But that's just just one dimension and Brandon I can give you link to those data no problems and And there is one more slide I actually created and that's this one And you know the numbers are small. So those are years This is 5,000 10 15 and 20,000 and What I actually looked at is stuff we can measure at redhead and this is how bug how many bugs we closed every single year for real and So you can see that you know it went through relatively small numbers in 2006 2007 to something that's like very close to 10,000 a year in past couple of years and if you look at these spikes This is just a Luck that we made may manage to release two minor versions in one year So that's why it otherwise it would be spread out roughly even you can see that this is basically double of this this one and If you recall I said this is just minor versions. So this doesn't show 506070 or 80 work at all Because that works actually happening mostly in upstream in Fedora. We don't measure it in house And it would skew the numbers anyway, but you know for me, this is terrifying view This is how much we crank through the system every single year And you know the other piece is as you remember how the number of packages grew over time You can actually reflect that in the number of bugs we fixed because this is route for five six Seven this one. That's a software collections. That's an offering on top of rail. So nobody's running away. That's good So if you have all these in mind How actually we work with upstream, right and last definition I promise I Was looking what upstream is so again Wikipedia just to refresh the The memories so it's basically, you know, it refers to a direction towards the original authors or maintainers So when we say, you know, I send patch upstream I know we basically send patch of where I took the software from could be Apache foundation GN work or kernel, etc, right? So having refreshed that I Was thinking about when we work with upstream there are actually two types of interactions The first one is how we actually keep up with upstream and what I mean by that is this redhead heads Like rail has hundreds and hundreds of engineers behind it as a product but think about To 3,000 packages we get into the system, right? It is a power play that doesn't go in our favor at all So we need to have a ways how to keep up with what's happening out there and where we actually Don't have too much control about what's going on and the answer to this is Fedora, right? I mean, it's probably a common knowledge in the room You know, there's like Thousands of upstream projects then, you know, we go that all gets integrated in Fedora and you know Why I say integrated because You know, one of the main goals for distribution is actually making sure it provides a coherent user experience So the integration is the key piece the distribution does integration of all the pieces so it works together and So there's a lot of integration work happening in Fedora project That's being sponsored by redhead and that's actually when you want to look where a rail is heading This is the place. There's no other place. You can have a look and No from time to time for every three four years. We pick one Fedora version and we make a rail out of it so this is really true that, you know, most of the development happens here and and The rail is basically using Fedora's up as upstream and Fedora's using all the projects out there as upstream and The way this goes You know, rel leverages all the all the work that happens here With integration and testing, but you know, this is still not good enough for enterprise customers to deploy in production environment This picture looks simple in real life It's a little bit more complicated because there are projects like atomic host that were initially developed here and then moved into Fedora and then you know the whole workflow was Straighten up to make sure like most of the stuff happens here before getting back to rail There are a couple of basic principles when we work with upstream And the number one one one is upstream first policy What does it mean is when we develop a patch a feature something we need to make sure it gets integrated In an upstream project and there is a really pragmatic reason behind that and That reason is that if we get all the code into upstream The next time we take the source code we get all those patches with it So we don't carry the burden of maintaining everything in house because if you know we couldn't scale it away so we're making sure that as much as we do gets accepted and We can actually then work with that later on The other piece is influencing upstream and I know in my world when you look at the close source of pair Those guys control what they do, right? In open source word that that's not true So we cannot Tell Apache foundation do this because we want you to do it that that wouldn't fly right? now all we can do is actually influence them and Influencing happens through developers So when it matters for redhead, we do make sure that there are developers behind what matters and those developers are talking with upstream Making sure that you know we get our stuff in maybe we don't get what we want but we get as close as we can and You know, I wanted to also mention rebasing versus patching because it goes back to the upstream first policy You know when we deciding How to fix a problem inside very often the fix is already out there and There's a like decision to make if to take just that one piece a couple of lines of code or if to take the whole New version and that's a decision. That's actually happening on the front lines of the engineering Like every now and then like I would say almost every day But that would be a little bit overstatement But you know, it happens pretty frequently and this is also influencing how we work with upstream because Sometimes upstream moves way too fast for us to keep up with it, especially you know when we need to freeze The features so sometimes we backboard stuff and sometimes we keep up with upstream So that was how we keep up how we make sure that you know Whatever happens out there without our control, you know, we can use it. We can actually ship it to our customers The other thing how I know what we deal with is actually how we drive the change and When I was thinking about this Internet is full of controversial federal federalists is just Google it you'll find like articles back from 2011 2010 when federal did something Community didn't like So I was thinking there's actually like free buckets when we drive the The features did upstream so One is it's relatively easy, right? There are upstream communities where we have really good Cooperation with really good relationships with I named couple of those like carano. That's a long like that's evergreen You know, we are very heavily invested there Cubarity there's a little bit different example from the last two years where major companies Including Google us are collaborating together to create orchestration layer for containers and there's many and many others so this is a Not as easy, but relatively easy game to play Well, it could be also hard, right? I picked free Because of those three are probably something you can recall Like a ceiling oaks Most people still turned off. I will even won't ask question because I would see way too many hands. I love that I would like But from our point of view from enterprise point of view. This is a must-have feature To actually be able to sell to government Because this is one of the key elements to get government government certifications Like FIPS or common criteria and without that we cannot hit public sector in North America at all That would be a huge loss from the financial point of view So this is something we do push as a company is not very well accepted It took years and years and years to get at least decent user experience and You know, we still keep pushing that System D. That's another story that collected to 7.0 when You know, there was a huge huge discussion You know who will take this which distribution will ship it which will not, you know, how how it's Sorry work how it sucks, right? And there were like strong people supporting it. There are people who dislike it Like it's a really controversial feature even now four years later And a team did a great job of making sure like it's a really nice piece of software to work with But there are still people like me who don't get the basics because I'm I was born so early that I still have in a Dean my head, right? So the change is hard and I don't pretend that None free is another thing that hit a lot of users and I know there are people, you know loving KDE or other Desktop so this is maybe not that impactful but still like, you know, this was a change where We had to you know, we were pushing a new Behavior to street to users through a software because the GNOME 3 and GNOME 2 Controls completely different now. It's much better now, but you know, there was a huge Degradation of usability for many when going from ground to ground free So those are examples where we struggled we couldn't couldn't give up in some of the cases and You know, it doesn't earn us earn us a many credits but there's actually third bucket and I call it it can be even harder than that and You know, I picked just one example here and this is Docker and this is a great example of Us colliding with business objectives of a different company Because the Docker the company Was actually fighting over patches with us which ones to take in which was to reject Because what redhead wanted from Docker as the software was different thing than what Docker as a company wanted from the same piece of the software Great news is You know, we have forking in open source world. It's pretty commonly used to create, you know a Competition, you know, you can work this work around this in different ways So, you know, we actually saw this as part of real aid via a different Toolchain that actually provides the same functionality even the compatibility layer But, you know, we don't have to fight over patches anymore So that's how I think you know, we work with upstream, you know, as I said We can have a discussion later on where we were we heading Then I was thinking how actually open source impacts what we do, right? No, it's not it almost always goes both ways and we influencing them they influence us and I use word influence because that's the key and you know, there's so many many many sources, but you know, I always use kernels a great source of Showing what we do as redhead because when you look here, you know We are second in like comparison by change set, you know we are in fourth place by Compressing how many lines we changed in the fourth 20 kernel and It's a lot of what we a lot of what we do is a program measure. I Look at the number in a different way So, let's pick this one 7% right. There's the 7% of code we Contributed we had some control over what it does Because of maybe we know, you know, we had to push it through the Approval process of kernel community. Maybe we work with Intel engineers with customers, etc But 93% Went from elsewhere and it went from very good places, right? AMD Intel those are probably the best guys to support their chipset. So that makes a lot of sense but It also means that there was no control over what we do there. So what it does is We influencing what's happening in kernel but we also have to leave sometimes with what we actually get and Product managers and I had to love the term pizza. We didn't order Because sometimes they get stuff they may not like but they have to live with it. So other things that are happening is And I also almost sorry again mean I mentioned the upstream first policy. So When you think about it if we try to work on everything with upstream It also means it influences us internally It can very well happen that if if you are customer of redhead you say I want this fixed that We tell you I'm sorry We cannot fix this for you because it clashes with were up is upstream heading or Maybe, you know, it takes us months and months because first patch gets rejected second pens get patch gets rejected and it takes us some time to actually Make sure that we have a solution that's Satisfiable customers needs and the project we actually interact with so that's that's a pretty significant impact in some cases Especially when we start looking at some controversial features The other piece that that hit us pretty well is Kernel a bi or user space a bi promise what that means, you know, we basically promising our customers If you compile something on 7 1 it will run on 7 2 7 3 for some specific no list of system calls or library calls and There's not too many upstreams that actually think about anything like that Which actually means that we have to put a lot of effort into making sure nothing breaks as we are using newer and newer version of some packages and You know, there are some libraries like jillip see that actually do miracles there You know, there have a mechanism how to work with it, etc. But elsewhere it could be a lot of work Now the other piece that upstream is not thinking about at all is life cycle, right? They don't release new versions when they are ready you know, they They may drop backward comparative compatibility altogether and yet on rail we promised ten years of support so Like when we shipped seven all back in 2014 will have to live with it until 2024 So there's a lot of time that we have to maintain this piece of software and Maybe even now there are versions of Packages that are in there are seven that upstream doesn't care about at all So again, that's extra work. We have to put into all of this to make sure that The product is usable. It doesn't have any any, you know major security to security flaws The other piece is maintenance So very often if there is a bug fix that, you know, we need to develop for a very old version of a software No, it goes out of our pockets right now. We have people who actually have to work on it and like When I was looking how much stuff we actually solve I don't have it like I have a data I didn't want to bombard you with a really, you know, busy graph But we fix roughly to a hundred issues like that every single month There's a lot of work that happens just to make sure that Customers can use well As a stable operating system the other part that actually is impacted by of open source is The program team and what program team is program team is a team that actually drives the planning and development of individual releases and You know at some points we figure out there's You know one team doesn't rule them all it didn't fly at all and you know, we had to split and create about It depends it changes all the time like it was initially 20 now It's almost 40 different subsystem teams that actually handle individual technological area select kernel subsystem kernel networking low-level libraries stuff like that and This actually led to a delegation of responsibilities closer to the code But my second line on the on the slide says distributed decision-making and what I mean by that of course the subsystem team contributed to that because of You know we pushed the decision one level lower those teams are now like handling their own stacks But also developers Have to make decisions on their own when they actually work with upstream projects There's a lot of decisions. They have to make and then not every of them could be bubbled up You know deciding bubbled down So there's a lot of that's happening in the engineering organization where our engineers are actually asked to do partially product management work because of they have to product measurement wouldn't scale at all and The last piece that I think you know impacted rail program really significantly is Speed up of the whole software world What I mean by that it was okay back in 2006 7 to do waterfall right because of like we were releasing every like six to nine months You know, it was an okay pace for customers The upstream wasn't changing that frequently That's not a picture. You can see today so We actually started to adopt agile, but some Pieces of the company can never be agile They Like marketing, you know, they cannot deliver marketing message every week right or every two weeks So we still have to make sure that we work in waterfall fashion and the program level as an interface to other other Organizations in the company or to our partners and yet using agile principles inside to make sure we can keep up with a pace of development and This is my last but one slide because when you think about it The picture I just Painted is there's a lot of issues connected with open source development, right? Like of control people do what they want, you know, you have to influence which is much more time-consuming than actually Ordering people what to do etc. And why do we do it, right? And I think the answer is very simple because open source is superior development model It's much better than closed source because you get collaboration. You could never ever get on the closed source development model Especially with partners like no we interact with Intel AMD Interact with HP, you know a lot of companies you get transparency people can look at your code can contribute Often by no looking code you get peer reviews Are many many open source project actually adopted release early release often? That means like they don't wait until they have a perfect solution for everything And there is a competition like there are most of the cases for one issue one problem There is multiple solutions you can try and you can actually decide which ones actually fits best for you So um, I would like to wrap up with it's not easy to do open source development It's not easy to run business in an open source company, but it's definitely worth it because No, our customers are actually voting by buying the software from us So and with that I would like to wrap up and there's roughly 10 time 10 minutes for questions 15 minutes Okay, who wants to start Brandon smiling. I'm not sure if that means he wants to ask something or I think okay, so thanks for your attention Hopefully it was useful for you. And then if you need anything I'll be in the corridor running to me as a wish. Thanks a lot