 Good up I'm not sure why you're clapping you don't know what I'm going to present yet So so if you say no if I'll see the same thing after I finish, I'll take it as a good job done so No, I'll talk about how we actually make major races Because I had an honor to actually see three of them being done real five real six or seven and I watched real eight, but I cannot talk about that yet and I was thinking that actually There is a very good story to tell Tell about real six and real seven major leases because there's a lot of lessons learned We took from that but before I start I explained how we actually make those things So this is this slide you probably saw seen already like thousands of times, but this basically Oh Explains how we get from upstream projects through Federa into rail and you know It may seem as a simple thing, but there's actually a lot of important stuff is happening in this transition because those two Federa and rail they are distributions and distributions main goals it is to integrate although Packages together right and you know when you walk at scale of operating system This is not about one or two pieces of software This is literally thousands and thousands of packages you need to put together in a way that actually works together, and it you know brings value to users and customers and So when you look at this just like You remember it because you know I was thinking that it's actually good to look at how we made real six So this is schedule I Intentionally didn't put too many dates because I can't right. I mean those are internal information from red head But we started somewhere in 2008 at the beginning and then finished in 2010 so that's you know four years of work The other pieces when you look No, we did three alphas And we did two betas And you know here you can see the linkage between the rail eight Roster in real sticks alpha and Federa And this was just refresh there's actually One thing that's being hidden in this picture, and that's very very good story No, you can see there's federal 12 alpha actually used for alpha 2 and alpha 3 and This actually shows the importance of having Federa as a upstream for L because This actually reflects a problem we had back in 2009 when we couldn't get our power PC and Mainframe as 390s these architectures compiled internally in house and the reason for that was that Those two architectures were lagging By two to three months in Federa already So what actually has happened over there was no we could get x86 64 done no problem No, that was all up to date But those two special architectures that like not everybody has a power PC nine under their desk, right? So those were lagging and There was a significant amount of work that we actually had to do internally to get get that done To get that compiling or compiled to get it actually booting and running So this one piece The other pieces when you look at these These are points Where we actually did something we call mass import what that means is no we have internal source repositories where we actually Store all the source code for L and at that point we just wipe those out We take all the code from Federa and selected packages and import them import them internally and it's another point Where the Federa plays an important role because we did that wipe out four times, right internally So that means that even Federa On source code level contains references to rel if you see in any spec files Those are basically description how to build Piece of software. There's a load of references to rel in Federa And this is the reason because we do use Federa as something that actually is Is at the development ground for rel So this is rel 6.0 story rel 7.0 Now we did three alphas one beta Started 2011 and the 2014 And I still remember the date June 10th because of This was my Reese. This is where I was working on for four years and You can actually already see a couple of lessons learned there And I go back Now we used federal element final alpha throughout well final This release was a little bit out of sync with what was happening in Federa Well 7.0 on the other hand was perfectly in sync Because we were just using the final Federa bits to import them in house So we just went through Federa 16, 17, 18, and 19 as clockwork Just doing them because this is the way how Federa brought the best value for Red Hat Because most of the work happened upstream from rail point of view that means Federa and we put everything together So this is the story of how we actually take all those packages that are somewhere out there Run them through Federa where they get integrated where we do all the major Development changes and then bring them in house and build an operating system from going through this There was a lot of lessons I learned. I mean working on for four years on something That actually learns you a lot through pay mostly, right? so I Was thinking what are the lessons to share? and one lesson I learned very hard in the hard ways the documentation is key and I'm not talking about user recommendation. I'm actually talking about two kinds of documentation one is documentation for teams Imagine four years long project with hundreds of engineers on it Those teams will change over time people will come and leave, right? No, they do something now and then they don't have to touch it for next three months. They forget So having a consistent good documentation where people can actually go to Figure out how to do certain things make a makes a lot of sense the other piece is Documentation that leaves a bread come trail. I would I mean by that? When my colleague was running real sick so She left a lot of project related documentation behind her No meeting minutes face exits retrospectives tons of stuff and When I started with 7.0 all of that was like chest of gold. I found because there was There were things they know went wrong. They were things that went right. There was a lot of stuff to learn and And you know use for 7.0 development So even if this is not for you know my colleague slice She didn't need to work for me like no She just created all the documentation as she's she went and I did the same on 7.0 But you know, it's basically for future generation, right? On our future team that will work on something that that long in next couple of years So that's one thing that was really really hard for me to Realize and you know, it's not easy The other thing is rinse and repeat what I mean by that when you recall 7.0 schedule, you know, we did alpha 1 alpha 2 alpha 3 and beta and I don't it you know, it's four years long project. We actually found a way how to make sure that You know, we repeat certain parts of the project to make sure that you know, we can actually Run retrospective on that piece. No, we did alpha one run retrospective on alpha one. It was so hard to do it No, we improved the process a little bit run alpha to run as a retrospective on it Improve the process a little bit when we were doing alpha 3 as a program. I even didn't notice it happened Because team already knew what to do. They were running like like a clockwork. So this is one example The other example is you know, when we do something we call Snapshot that means like you know on weekly basis We create all the ISO images and everything and test them and it was that's another rinse and repeat approach You know today, it's called DevOps agile back then, you know, it was waterfall and we were doing snapshots so like terminology change, but the benefits of doing something repeatedly as quickly as possible and In a way that it doesn't changes over time as quickly so for people for people. It's Almost a routine that there's a lot of value in it But you know when you do retrospectives There's one important thing, you know asking team What they liked what they didn't and it doesn't matter if you're running waterfall if you're running agile, you know, there's no In scrums, there's piece of it that's forgotten the name It actually says like the team gets together and reuse what has happened. One important thing is Act on actions Doesn't make too much sense if you ask people what they think if you don't do anything about it the next thing I was Dealing with in myself pretty hard is cone of uncertainty if you are project manager program manager, I Know your scrum master And you have something very long ahead of you Don't try to make perfect plans Because and I like that analogy a lot plans don't survive contact with the enemy Right what I mean by that If you have a three years long project plan maybe Six months ahead have a like overall plan, but don't try to like nail every single action for next three years It will change, right? It will change so hard that you know, you'll spend tons and tons of time redoing what you already planned and throwing it away You know being upset so on girls have an old when we were running the team What we did was basically We had a quarterly meat planning meetings where we planned for the next six months and Like you know updated schedule updated, you know processes as we need it, etc So that actually at that time gave us enough room to work with all the teams across Across the company the other pieces are that product is work of many teams and what I mean by that is Yes, engineering is important development and QE do a lot of work But once those two teams have bits ready to ship There's a lot of other stuff that needs to be ready to ship at the same time you need to have support Ready, you need to have marketing ready. You need to have sales organization already so they know how to approach the customer you know Sometimes some kind of certifications user documentation is like a lot a lot of things that needs to happen and even if you Develop something in a job fashion There's always a point in time when all those teams need to know what are you going to ship because they are now some of them I might my favorite examples actually marketing Marketing is not able to deliver on two weeks. It's they need to have something to build yet They are message on and then communicate it to the rest of the world and This is the last piece I actually learned and I cannot stress this out strongly enough Because if you work on something that's four years long You do need to find destruction Just small approaching on site here and there You know make fun do something different just you know it can get at you Really really quickly the other pieces You know In today's world we hear a lot about work life balance. It's not a buzzword No, when when my colleague was doing Real six you remember the alpha free story and you know, we had to like know Do something special in house? She had a vacation plant for two weeks. She wanted to go to Peru and And She was really Hard-thinking should I stay at work and finish this really important piece of work? Or should I go somewhere else no spend my time with my husband and take a rest and she decided to do the later She gave me all the work. I was freaking out at that moment. I mean, it was so important for her to recharge So she knows you could I should carry on in next you know for the next couple of months So that's where I would like to finish left 10 minutes for questions Who wants to start excuse me Biggest challenge for the team big I mean, it's it's hard to say there was one biggest challenge because Like every like the project goes through multiple phases, right at the time I know we were did a little bit of planning Etc. But in case of rail the biggest issue is the size of the project And what I mean by that, you know if you have a team that's hundreds and hundreds of engineers big There's a serious communication issues Because you know, it's so hard to hit every single organ people in person in the organization it's so that's one piece the other is that At that time communication between development and QE wasn't the best it actually refers to the previous talk because QE was seriously lagging behind development and Stuff was literally thrown over the wall to them and developers didn't care much and So these questions about this is if I have a more details how we structure the documentation So if I talk about project documentation and you know meeting minutes stuff like that For that we actually have a place where to store those so like for me there was like, you know a Place where I could find all the meeting minutes like every not every single week that I know what has happened on 6.0 And you know, I did the same for 7.0 for the phase exits stuff like that, you know, we had a folder where we had all those documents Where I could access them. So and at the time it was as easy as using SVN and track where we stored PDFs So it was like, you know, really simple On the other hand, it didn't break over years and years, right? I mean something that was created in 2009 I was able to find it even in 2014 which are like, you know, the simpler solution the better in this case Right because So, you know, the question is about that, you know, we were able to to find those important things and you know, this is because of At that time, like our manager forced us to create Structure of documents. We didn't like it now It's better to create something than throw it away when you don't need it But he really was pushing us to make sure we put it in one common place even as PDFs So they couldn't be edited later on but you know, it was still as I said, like, you know Chest of gold when I needed it The other question So what do we do before the final final federal release? So We basically do development in two places What is in upstream because, you know, eventually when the package gets updated upstream it gets into federal Right. So that's a source code and the other piece of it opens going directly in federal because you know What what what usually happens when we import stuff into rel is we change compile flags You know, we change configuration of kernel stuff like that But you know, usually, you know, we take no bits. No, sorry source code from federal one one to one We just restrict the number of packages we import just to those we need Because federal has like no much higher number of packages than what we actually ship as in rel And it's really like the federal is our development destroy That's why I know it gets break broken from time to time and No, they are actually independent project of rel. So, you know, when we were doing gross, I know we approached them I mean we approached fesco. There are dream steering committee and actually very kindly ask them to make sure they keep hitting the dates Because it was so important in throwing to red hat that we could actually work with those and they were so glad they Almost did that. I mean it's software project, right? Okay, I think we have maybe room for one more if there's any Sure So can I imagine that you know, we'll just keep cranking our major releases like as a continuous deliverable. Did I get it right? There's one thing that stays in the way, and it's called Actually two things. It's a life cycle because rel has a 10 years of Support and During that life cycle we promised to not to break kernel a bi and at some places even the user space a bi And it would be so difficult to keep up to date with all that happens in the upstream Why not breaking those things and like I get that like not everybody Understands why the old stuff is important but like Tokyo stock exchange for example runs rel and They just cannot operate and change the operating system every year or something like that. You know for them it's important to Deploy it in right for years and that's actually one of the values proposition of red head right or rel Why they pay us? Otherwise they could use federally could use something different one more like two minutes I don't see any hands. So thanks a lot