 All right, I think we'll get started. So we are three minutes past the hour. For those of you joining the call, you're gonna see six people on the screen, but if you double click on the slide deck, if this happens to be your first presentation at Nest, you'll get a full screen view of it, just to tell people that. So thank you very much for taking the time to join our little session. Have a good morning, good afternoon, good evening, and good night wherever you are. And let's get started. So this is the CPE year in review. So we wanted to put a session together to one, let you know what your CPE team has been doing for the last year, and we're pretty much going from flock to nest to try and keep it a continuous story, and hopefully we'll get to do this next year in person. Your presenters today are myself, Lee Griffin, and I'm the engineering manager for CPE. Pierre-Yves Chabon, who's our technical lead, and Ifa Maloney, our product owner, and Aleve. Ifa and Pierre introduced themselves when they get to be our relevant sections within it. I want to kind of begin by talking about the scope of the team. But before we do that, let's have a quick reminder of the mission that we have. So the CPE team services both the Fedora and the CentOS communities. So we're actually split between two communities that have a very close relationship and a lot of overlap. So our main goal is to provide systems administration in and around the infrastructure and release engineering that helps go towards producing these outputs, which end up being operating systems, as well as the services that run in that production line, and a lot of other services in between as we're going to see going through the presentation. We also deal with application maintenance. So there's a specific set of apps that we have been designated as the owner of, and we try to provide maintenance there, including upgrades, bug fixes, and so on. And then the last part is we're a gone for hire, or a team for hire, if you want to call it that, where we actually build out initiatives. So this is where the Fedora and the CentOS communities can come to us with a request to utilize the fact that we have full-time and very talented software engineers and infrastructure folks available to try and produce something new and cool that will ultimately help the ecosystems that we work in. So that's pretty much the three pillars that the team is built upon. And you might have noticed on the first slide, and you'll certainly notice on the bottom right of every one of them, there's a logo. So we have a logo since we turned up last year. We were actually a little bit surprised when the Facebook client platform engineering team turned up with your CPE logo. So we were somewhat inspired by that, and we wanted to get a strong identity. And there's a lot of symbolism in this, and I won't spoil it too much, but I'll try and let people guess where some of the symbolism is coming from, and you can hit that up in the chat and we'll reply to it as the call goes on. But I do want to take the time to call out Mo Duffy and her creative ability to pull this together. It's been a fantastic effort. The team absolutely love it, and this is really the first time we're getting to unveil this more publicly. So you're going to see it on a lot of our outputs in the coming months and years. So in Irish, because I know Mo has a love for the Irish language, Gaurav Milamodogov, Maureen. So the scope of responsibility is just something I want to touch on here. As I said in the earlier slide, we do look after services. And last flock, believe it or not, for the team of our size, we maintained over on the hook for a hundred and thirty plus services. That means we're responsible for the lifecycle of them from bulk fixes to upgrades, team enhancements. And we had to do a lot of soul searching to go through those applications and retire some apps that really should have been retired a long time ago. These are applications that were still on Fedora too. And in some cases they'd fallen over and not being restarted and nobody really missed them. And in other cases, we looked at unifying the services that we have on both the CentOS website and on the Fedora site. One example of that was the PACE service, which now redirects to the CentOS PACE. So in the last year, we've managed to reduce our focus, shall we say, our footprint to look after 56 applications that we have designated as run and maintain. By that, we mean we look after the service lifecycle, as well as bulk fixes, upgrades, enhancements and any kind of requests that come in, particularly community contributions, which we always love. We have 35 other applications that are within our infrastructure where we provide power and ping. And that's a really cool way of saying that we'll host them only for the community or for other teams, potentially within Red Hat that work within the community. What does that mean? It means we don't understand the code base. We probably don't even have permissions to go and make changes in that code base. And if it falls over or hiccups, somehow we'll try and restart it. And if we can't, we'll have to tap someone on the shoulder and ask them to get involved. That is still a huge footprint, as you can see. And you'll see in a minute the number of people we have on our team. And one thing we've really struggled with is how do we reduce that number? Because invariably, an application in here is useful to some people. And we're gonna have to have some difficult decisions with the community over the coming year because a big blocker is our friendly data protection laws, which is GDPR. And that means we can't gift an application to someone because if we try and gift it to someone, we lose the data. Now, all of these applications are open source. So anyone in tier, you can take it and run it and so on. But the real value in a lot of these applications is the data that they hold. And we have no way of gifting an application without having to go in and surgically attack the data, which is just so difficult. So we'll give a call to arms later on to see how people can help with this and what that looks like from a conversation. And maybe in the Q&A, people might have questions around this for us. So that's one half of the decline. The other half is infrastructure. Because we look after two sets of infrastructures, we've kind of broke it out here so people can see it. So in Fedora, we have 117 physical and 250 virtual systems. Now, since our COLO move or data center move, we have multiple XVMs that are hopefully getting migrated to OpenShift pods. And we haven't counted them in this number. CentOS is 147 physical servers and CentOS-CI is 278. Now, the CI is a shared service between CentOS and Fedora, which a lot of our community use day in, day out. The obvious scope of responsibility here is regular maintenance, making sure we're up to date with patches. You've seen the boothole vulnerability last week that our team working very busy behind the scenes to make sure all of our servers and all of our infrastructure was up to date. We also look after every single thing from soup to nuts to borrow a well-used American phrase including the networking, the layout to the data center, cooling, backups, hardware upgrades, warranties. You name it to do with the lifecycle of the server. It falls under our team. And the last one before I wanna pass it over to maybe the more meaty section is the team composition. So the team has actually grown over the last year. We've had several new hires. We've expanded our infrastructure team with two new hires and we've replaced some team members in our development side as well. That kind of stayed more or less level. So we're really up maybe two net new infrastructure folks. We created a product owner role and that's IFA who you're gonna hear from in a couple of minutes. And we expanded the management team because people need people management within Red Hat as a company and that's allowed us to expand that out to give those folks good career paths and good career opportunities. So the breakdown is on the slide. I'm not gonna read it out and we'll share the link in the chat if Pierre hasn't done it already to the PDF if you wanna refer back to this. But that's the rough breakdown of people that we have within the team. So it's 27 to 28 people all in and it's not evenly divided. So that means if we have a software engineer and effort we can't throw 28 people at it for example we only have 14. And as you see in a couple of minutes that effort is split across multiple initiatives and multiple things that we own. Okay, I'm gonna pass it over to Pierre. Pierre I'm happy to keep sharing just give me a shout when we wanna change slides. Sure. So I'm Pierre Chibon also known as Pingu and probably better known as Pingu as we've seen yesterday in the pub quiz if you missed that. So I've been in federal since 2007. I've been in Red Hat for about seven years and I'm now the teammate of the CPE team which is basically the only team I've been at since I joined Red Hat. So I'm going to run you through a little bit of what we've been working on over the last year. We'll start with the first one, right getting. So if you remember last year we were in the middle of implementing this and it landed basically late 2019. We already gave a couple of presentations at DEF CONF as well as post them about it about how it works and the process there. Both videos are online if you want to check them out. But basically it was an important initiative for us because I think it was one of the first initiative that we approached as a team that ended up being multi months I could even say multi years but because of the different iteration that it went through but the last iteration was the one that finally succeeded which ended up being approached as a team, multiple people in a team were involved in that. It's also being a very much a multi-system initiative because we had to implement things in the package in Koji, in Bodhi, in Federal CI and then coordinate with GreenWave and WebDB what the core of the getting mechanisms here. So we involve people outside of our team as well as multiple people within our team to be able to reach that. And the end result is that we are able to gate the right packages before they land in the right build routes in a way that is fully automatic and transparent to packages, you know, if everything goes well. We've also added a couple of features which are nice outside of raw height such as the on-demand site tags which can be used on stable releases as well which also makes things like a fit package chain build working nicely in stable releases. So that was the first big one that we started one. Very focused on Federa, very focused on right but it's also gave us one of the first base to how do we want to work as a team or do we want to progress things on that front. And the next initiative, at least that's your clue, thank you, and that's we started basically in January this year is Nugging. So Nugging replaces our FASACON system, FAS2. As a reminder, FAS started in 2008. The first commit I went back to check it yesterday was from Toshio in 2008. It's Python 2 only. It's run Sturberg year one for memory. I don't even think it runs 1.1 or 1.2 and it definitely doesn't run Sturberg 2 who also exist. There is no way to port this to Python 3. We used to run it on Rails 6 and then we tried to migrate it to Rails 7 and we did not manage. So we are now running it in OpenShift on a Rails 6 container. And the nice thing is that Rails 6 is, well, Python 2 is already out of support since the beginning of the year and Rails 6 is getting out of support in fall 2020. So Nugging was basically a very important project for us to work on. So we are moving to a more modern platform. It's basically a free IP on the back end and Nugging is built on the top of it and it's basically a self-service or community portal to free IPA. Adding a lot of the feature that free IPA does not provide such as the possibility of changing your password yourself. But we also rely on a lot of the feature that free IPA provides such as two-factor authentication. We currently have two-factor authentication but it's only ever used or useful for people who are in the CIS admins and actually have shared access to the machines. If you want to log in in any of our web app, we don't have two-factor authentication support. Well, that is something that's going to change on the Nugging front. At least Nugging itself will support the 2FA and then we'll need to see how we can expand this to our different applications. And then we know we have things which are Python 3, of course, but we are sync, which are kind of nice and they have become standard since FAST was created. Sinks like unit test, FAST did not have any. It's built from the start to be able to be deployed and run in OpenShift. It uses Flask as a framework as a side effect. This has also been a multi-month and multi-team effort. Definitely multi-month, it started in January and we are looking to deploy this in full. Currently, being working on deploying this in staging and then production in full. Multi-team, because CPE has been collaborating a lot with the IPA folks and we are very appreciative of everything they have been able to up with. There's more things they have done to free IPA itself for an extension to it for our needs. So that was very nice of them to help us and basically to be able to work on that. The nice thing about that project is it's also a multi-communities project because it affects Fedora, but it also affects CentOS and as a matter of fact, CentOS is going to be using the same Nogging instance as Fedora, which means you as Fedora community member are going to have a CentOS account in the same way that CentOS community member will be getting a Fedora account. So if you want to contribute to anything on CentOS you'll be using the same account system if CentOS folks want to contribute anything to Fedora they'll be using that exact same account system. But it's not the only communities that are involved. The OpenSUSE community is actually looking at deploying Nogging and by the look of it they may actually be the first one deploying Nogging, not us. So that's very nice to see that something we are building is going to be reused outside of the family, although it remains the RPM family but it is outside of the direct family. Many thanks to Neil Gompa and the OpenSUSE euros. I think that's how they call those three summons for everything that they are doing and they have already done the packaging work so that's also something which is nice and they have been reporting bugs and issues so it's a nice collaborative they thought there. So you can see we moved from ride getting which was very Fedora centric to Nogging which is already involving a bit more communities but we also read that basically read that still is our paycheck. So one of the third project that I'd like to put forward and Lee, that's your cue now. Oh, it's not following. You can hear the keyboard, he's typing his email. Ah, there we go. So one of the bigger one is CentroStream. So the way I would defray CentroStream is basically read at Sefort to open its built pipeline, to open up the way it builds real and so it basically the public view to the next row. Dot y release. So Fedora remains the dot x release. The next version of well is based on Fedora. The next minor version of well will be based on CentroStream. That is definitely multi-months of course. It's starting in fall 2019. It's going to end up probably running into 2021 somewhere. Definitely multi-teams because well, it's involved CPE for a number of things since we also look after Centros, but it involves a large number of people also internally. There are no, you know, I can't comment to the number of teams involved in there, but I actually, you know, the people involved in there have a full agenda and their ends are full with meetings and trying to gather the inputs for everyone and moving that forward. So it's a really big project that is impacting communities as well as the business side of our team. But that's not, we haven't worked on only these three projects. We actually worked on some more. So we have worked on DNF counting, which Matthew Miller gave a, you know, spoke about in the state of Fedora. Better statistics based on the count me value that DNF sends about once a week when you treat the repository metadata. It's something you can configure, you can opt in, opt out. It's, you can opt out, basically, you're obtained by default. You have no personal information sent other than the edge of your system. And that's basically Cent only about once a week if you haven't touched the way your default configuration or default DNFs works. We've, we're still working on the Fedora messaging. So we do have a message bus called Fed message as was used on the zero MQ technology. We're moving to a Fedora messaging one that's based on rabbit MQ MQP technology with a rabbit MQ server. There's still some work to be done. A number of applications have been ported. So actually most of our applications today are running Fedora messaging, but we did the minimal amount of work to do that, which is porting directly. Now we need to add message schemas. And this is a requirement to be able to replace something like FMN or to upgrade our data gripper applications. Both of them are actually need to be able to translate the JSON blob that is being sent on the message bus to something which humans can understand. And that's where the message schemas come into play. We've also introduced something called toddlers that you may have heard about yesterday if you were at the pub quiz. It's just a bunch of small programs running around. So it's basically it's meant to be a plugin based Fedora messaging consumer where we are able to consolidate more of the code base to we've been able to retire so far at least two projects. And there is a third one that will be retired just because of we are leveraging the toddler framework and running this easily. Something it does currently basically if you're in the Fedora bugs group in fast, you have certain privilege in bugzilla. That is toddler that sinks. Everybody in the Fedora bugs group in fast should have these privileges in bugzilla. When you do a successful build in Koji we flag the corresponding commits on this gate. This is a toddler doing that. If you run, if you open a progress on this gate, CI runs, CI sends a message toddler gets this message and flag the progress with the result of that CI run. The next one that we are going to be working on is going to be sinking the assignee and CC list from this get to bugzilla. It's sorry we have already written that piece of software but it's in a dedicated project in a dedicated code base. We've dedicated open shift. We just want to move that into a toddler so that we have one place where we can scale all of these. And that's basically a step toward retiring a Fed message. So far the two project that we have retired with toddlers were Fed message based. And that's some of the project we worked on then we also worked on EPL-8 sending up the initial build environments, sending up the build route and the initial work to get module support. We worked MB box, which is module built in a box. That's a way of deploying code G MBS in an unopened shift environment easily. And that was something that the centers folks are using for centers eight. And that's something that they are interesting into for center stream, but it needed to be updated. It needed to be moved to an easier way to deploy as well as an easier way to maintain it. A newer version of code G MBS for examples. So that's also something that we worked on. You may remember that we spoke about RPM Autospec. We're speaking about there is a talk about it tomorrow again, which the goal there was to get rid of changelog and release field in spec files, not in RPM only in spec files and auto generate them in the code G when you build the source RPM. We're going back to discuss that tomorrow. It's still a proof of concept. It's still available. Well, it should be available in staging once we're staging is back. But yeah, we're eager to hear your input on that and if we should continue to work on this or not. We worked on something called monitor getting and was basically it's an end to end testing of the entire package or workflow with the idea of being able to say how well is the package or workflow working? Is it working at one point and being able to easily find out where it's broken? You have to realize that, well, you have to, the package or workflow actually involves more than 10 systems, more than 10 application from Fed package to this gig code G body. I have not list them all here, but we have counted them. They are more than 10, which are involved. We only testing the API pass. We're not testing every single combination, but it does give us an idea on how well these 10 applications are working together, as well as the, instead of having a single point where not just can tell us an application is running for 95% of the time. Well, monitor getting is giving us an idea of how is the entire workflow behaving? So on the top of that, we've also done something very, very small. And I let if I go on this next slide. Yeah, so as you can see, we have gotten through a bit in the last year. I've only been privy to the last nine months of it, but even still there has been a lot. Probably the biggest of which is our data center move. In case you didn't realize Fedora moved data centers this year. I know it was really like under the radar. Nobody really noticed, but we have a couple of fun figures for you on the small project. And I'm using small with a heavy dose of sarcasm. The team worked, moved 107 servers. 76 servers were retired. There was 31 affected services and I didn't include staging in that one because that just happened it in me to count how many things are affected by that. 23 weeks of overall moving, 20 transit days between all of the hardware, 12 months of planning and executing, eight immediate team members from Red Hat IT and CPE. And I'd like to thank you. You can sing this. I would like to just pause on this because we actually had a great help from Red Hat IT. There was a number of people closely involved with us throughout the whole process. They gave us priority. They were on hand. We had seven meetings, several touch points and they are, and I'm going to get their names all wrong, but Tom Wilson, Chris Adair, Seth Shavak, a couple more, Matthew Gaugci and apologies to anybody on this, but thank you hugely for that. We also had three data centers in the finish. We moved from Phoenix and we put stuff in IED2 and RDU. So it wasn't enough to just move from one to the next one. We had three managers managing, two SysAdmin's holding down the entire operation, one product owner praying and all we were short of the partridge in a pear tree. So that was that. Thank you for everybody as well for your patience and your considerations and your understanding through that mammoth task. And you know, on the top of that with a number of projects and on the top of that, there is, you know, business as usual, you know, requests from the community, hardware failures, drive network issues. So what you see on the slides here is the one year basically, this contract is from yesterday, the Fedora issues, the ticket tracker on packet.io. And the graph at the top there as in blue bars, the number of tickets that were opened that week and in pink, the number of tickets that were closed that week. And you can see some weeks we had more tickets closed and open and some weeks we had more tickets open than closed. But the bottom graph is quite nice to see. It's the number of tickets that have been opened in that were open in the issue tracker over the during that week. So you can see we started the year with about 140 tickets opened in the backlog and about spring this year, we actually managed to go down to 90. I think I've seen even lower than 90. It climbed up a little bit more during the color move. Go figure, people are busy and sinks break. So we actually report them and we used to get to track if they are sold or not. And since the color move, we actually back on track to go under the hundred number of tickets and try to go down from there. So congratulations to Smudge and Kevin Fensi there who are definitely the two running points on the running that side of the team. They are doing a great job at this. And if you send them, you know, Nierik++ and Smudge++ they'll love the cookies. That's it for me. So I guess I'm off. How to work with us. So Lee had spoke in his slides about how we're changing the way we work, how we're trying to be a bit more visible with a bit more structure. So I put together a couple of slides, not going to spend too long on them because I want to save time for our question and answer session but primarily the why, who and how. So why did we make these changes? When I came into the team, Lee had done some for running with the team previously with the other managers that people were reaching a point where they just needed to have a more structured day-to-day. And that's hard given that everybody is in a community setting and other people have day jobs and this is our day job but it's not a 24-hour day job. So it was important for the team to feel that we have structure and we can pre-plan the work so that they know what's coming. It was also important to set expectations for our team stakeholders. Believe it or not, we have people that we have to answer to and you could folk here are one of our stakeholders. So it's important for us to have your expectations so you know what we're doing, why we're doing it and what to expect from us. Following on from that is to establish clear goals that our project teams can deliver on. So the work that we're giving ourselves to do in X amount of time, the teams will feel comfortable that they can achieve on that and they can do that. And if they can't, they're empowered and they're confident to say, oh, hang on a minute. What are you doing? But thankfully our team Rockstar are able to do anything that is thrown their way. So the who that talk and discuss and scope and weigh up the options for the projects that we work on and I'm just speaking on the projects, the big scale projects, not the bug fixes, not anything that the sustaining team work do. This is projects that require multiple people a couple of weeks, maybe even months and they need to be pre-planned. Who decides what happens when? We've come up with a review team of sorts and they're broken into the CPE review team. So that would be the management, Lee Griffin and Carl and Stephanie Nadiate and also Paul Freels as our senior manager. Our team leads. So she went for the Fedora side and Brian Stinson heads up the center west side and then myself, I gotta say too. But we form one voice. Then it's important that we understand what the community needs are. So we have a Fedora rep, Mackie Miller is primary on that one and Marie Norden and Ben Cotton also help them too. But I know that they also then speak for the wider Fedora council and the community. We also have Rep 2, which is Rich Bone as their community architect and Brian Exelbeer I believe is also going to be wearing that hat too. We also do have to adhere to the business because we are unpaid for this at the end of the day. So it's important to know that we're on the right track there too and it's a balancing actually there's a fine line to be in and that business rep is Brian Exelbeer. So while we're walking a fine line, I do have to take my hat off to Bex wearing two hats of center west and the business side too. So they'd make up our decision makers seats at the table for our quarterly planning exercises. The how you can submit your ideas and I'll speak a little bit more to that in the following slides but just so you know how you can let me know that you would like something like the team to work on something is you can always email me and lonely at redhat.com. We do have a project template but it is in a Google doc so feel free to just write your ideas into a regular email send it off to me and we can discuss it a bit more. You can open a ticket. If you're not sure if what you want the team to work on is a bug fix or a small enhancement or could it be a little bigger just open a ticket as normal. The sustaining team are getting very, very good for triaging. They're going to be looking out for running this enough that they know that if things are a little bit bigger could need a bit more attention they'll ping it on to me and triage it. So if you don't know, let us figure it out. You can always message us on hashtag redhatCB or you can DM me directly at email lonely on the free note or you can always join my weekly office hours. They are on a Thursday at 1300 UTC at Fedora meeting one. A couple of people have been joining which is great but I'm always hanging around there for an hour if there's anything you want to ask or talk about. Next slide please Sharon. So, Smer Gualgomal I apologize I'm getting this surname wrong intern that was working with Marie in the Fedora project has put together a fantastic infographic for us on our processes. Thank you. Massive karma to Smer. It's beautiful. My next slide is going to look like a bag of shit compared to it. Sorry I couldn't help it. So this is our formal initiative process. It is going to be published on the CPE you did but I just wanted to highlight that we have been working with Fedora we have had fantastic health and like look that so cool but it does encompass the business side and the community side and I'm aware this is a community stop distracting me I'm aware this is a community conference so I'm going to speak a bit more on the community side. Next slide Sharon. So community edit of that as you can see that's my bag of shit compared to the other one it is nowhere near as nice but I hope it kind of does it justice and I know Marie and Mo and Smer are proud that they like this at home and I'm sorry but anyway so as I said in the previous slide if you are a if you are a community member and you have an issue like the CPE team to work on you can just file it in the normal channels so thank you so file it normally it flows into our sustaining team they review it during their twice daily stand-ups if they think it's something that they can action they will just move it straight down into their tracker or if they think it's something a little bit bigger they will triage it to me so I'll usually get tagged or I'll get an email or I'll get a Google chat going hey Maloney check this out but no I will review it and then it goes into the scoping phase so more than likely you will get an email or something from me if you are the requester just asking for some more information if we feel we need it we'll start walking through the process I will then move it into a Google Doc to capture it more formally and we will walk through the requirements walk through the estimated timeline what kind of skill sets maybe needed to complete this task and then we will move it to the review with review with the CPE management team the CPE review team initially if the CPE management team feel that it is something that we should be considering if the wider review team are saying yes this is absolutely something that we should be addressing for communities we'll move it then on to a more formal project brief and we add it to our backup for QP voting if when we're reviewing it by the review team and it's deemed as something that is either just not meeting our mission statement it's not really anything that we could complete or it shouldn't happen we'll let you know the ticket will be closed it will be updated and the requester or the filer will know so that is it in a nutshell if you have something big file a ticket it will flow through the rest of that back in the start of July you may remember that I had sent out an email to announce announced list and this is some of the feedback that I got so I had sent that gorgeous infographic that Samira created for us along with a lot of text around why we're doing this, what we consider big what we consider folk fixes and we wanted your feedback on it it is very important for me who is kind of the gatekeeper of projects for all the worlds to understand what the community wants, what we're doing, that could be better, what we're doing that you like and what we're doing that you think is useless just leave it out altogether so these are some of the points that I got continue to communicate clearly and regularly on project updates less acronyms and abbreviation and comms do you see what I did there publish the team member time zones and doctor Fiora and help to help to find our working hours that was actually a really good point and it was quite an obvious one and I thank you I think Martin was the one who had messaged me that privately so that is something that we can absolutely action also publish the workflow diagram to DOCSIS CPE and then have something filtered that relates to what area you're coming from if you're coming from a community contributor space this is what really you need to look at if you're coming from a red hat business unit space or you should be filtering the request too people seem to enjoy the office hours and also they're requesting a public tracker for bugs as well which I know Pingo has been working on and maybe speaking about already so it's good well no not too bad next slide this is my last slide you'll be pleased to know couple key dates for your diary so following on from my email about the engagement are the members from CPE team led by women managers and has put together and actually Vipple has stitched it all together thank you Vipple has stitched together a survey so our survey says please complete it by August 30th it's been sent to the devil and in for lists if you haven't seen it please check there and filters and complete it's closing by then September 9th is the new initiative submission deadline for CPE so if you do have something that you'd like us to consider working on please email me file a ticket but do it before September 9th which is to no I was going to say it's tomorrow but I'm a month ahead of myself so you have a little over five weeks for that September 18th thank you and September 18th is when the CPE team are going to host our quarterly planning session so by submitting ideas suggestions by September 9th it will allow our team to have two weeks to review them and see how much work is involved in them and we can then confidently add them to our backlog and put them for consideration to the wider review team then for voting September 30th is our quarter three end so we have a number of projects starting from Q2 and they are going to be running to completion or certainly what we want our goals to have been we want them to be hit by then and then October 4th is when CPE will recognise when our quarter four ends and the projects that we will have reviewed on September 18th and voted on as yes the most important we must do will start and kick off then and I think that may be finished yeah thanks Ifa and thanks Pierre for the overview of the projects as well I think what's work calling out just to briefly go back to the slide for a second is these are key dates that will help us line the teams up get the right level of analysis done so that we can actually ship something and deliver it so 12 months ago at Flock I gave a presentation about the process we were trying to put in open source agile I think was the official title of it and since then and just to call her out because she's on the call on the screen we had Sarah join the team so Sarah Finn is an agile practitioner within the team and it's helped the team just get into that mode of delivery because it was becoming frustrating for the team and I'm sure for the community to work on something and get it so far and then get your attention pulled to something else so we kind of set a mantra out and Vipple probably has several memes about this now about stop starting and start finishing so to try and get stuff shipped and over the line usable for our community and that's why we have dates like this this is where this kind of process came from and we are going into our third full quarter of it which ran at the moment for Q3 we're on a calendar year quarter and in Q2 we hit 100% delivery so 100% of what we wanted to deliver in that three month window was completed and that gives a lot of confidence and hopefully to the community and certainly to our team that if you ask for something and if we say we're going to work on it in quarter whatever that you're going to get it at the end of that quarter and I think that's a really important note to call out and they're all the positives and I just want to kind of end on just some of the future challenges and this is not meant to be negative for anything like this at all but this is where the team are constantly moving towards. We're constantly resetting the horizon goal moving forward to try and tackle new challenges and add more value to our communities and something that we want to start tackling and we've already begun this process and you've heard the word sustaining team mentioned by both Ifa and Pierre which probably should have given a couple more slides on that or maybe a dedicated talk but the sustaining team is really capturing the lights on or the business as usual work that we're actually doing in both Fedora and the West. It's actually heavily tilted towards Fedora at the moment because that was where a bigger scope of responsibility with 130 plus applications were and that was trying to get a handle on just what do we do every day where is the source of our pain where is our contention coming from it's to avoid gut feelings to say that we're working in this area because we think there's problems and it's helping us gather stats and gather some information about what's going on and to help inform where we need to put some effort so the package or workflow analysis that Pierre mentioned is something that has come from the sustaining initiative from listening to the community from observing various bulk reports and now we're investing time to see is this actually a problem and what is the actual output that has to come from it so right now out of our 24 actual active developers documentation folks is really what's available to us it's over 50% of that team is literally keeping the lights on primarily for Fedora that is a huge number and it's questionable what value that's actually bringing when you look at the rising number of tickets and it's great to see the trend going downwards in bugs every 10 bugs we fix we're getting 8 to 10 bugs opening up again and that's pointing towards instability that's pointing towards technical debt that's pointing towards just the age of a lot of our services that we inherited and not owning the code base so by having greater than 50% of our team working on these things it's giving us the time to innovate and build new things which I know is what the community want and as software engineers that's exactly what we want to do as well we want to do that as well so technical debt is going to become a focus for us in the coming year, retiring Fed message, retiring older applications modernizing the stack the recent COLO move for all the fun of the 12 days of Christmas song that we had for it it nearly killed our ability to service the project we hit so many problems going through it you would have seen the various mail shots, the daily and weekly updates that were going out towards the end and the incredible stress and strain on both Smooch and Kevin to try and get that over the line so we're trying to get more people skilled up to be able to service and work on these areas and obviously address some of the root causes that cause this instability when we try to bring all of our service stack back online that means we're going to have to engage and have our conversations and make our decisions to possibly retire applications, amalgamate applications or see can we bridge the gap somehow and get more community involvement one goal and target that we've been saying for a long time but it's certainly on our agenda with this technical debt reduction is to make this more community available to service and to try and lessen the need for CPE to be a roadblock on some of these requests so getting more help on the workload always appreciated you see the initiatives, you see that we have them tracked or publicly available and it's been fantastic particularly on the Noggin project where we had a couple of community contributions very early in the cycle and then as the work matured we got a hell of a lot more work which was called out already when open Susie and Neil and the guys working around that area that's fantastic and we love that and please keep staying involved but if we can get more help on the lights on workload that will give us more opportunities to work on fun stuff and really move those initiatives to another level better communications I think that's something that we always strive towards and over the last year we've introduced the weekly mail shop for IFA and we now participate on the Fedora accounts I think IFA's presented twice at this stage just about what CPE are doing and we try to hold office hours we try to get as involved as we can on the mail in this we're always on IRC and this is something we want to keep continuing we did learn a lot of lessons from the FF we want to try and do these things in the right way there were mistakes made and we put our hands up and we're happy to say we made those mistakes and we want to try and address them and two points out of that from my perspective is way more higher touch points on the comms are needed here and we did have a very tight communication cycle with some of the internal stakeholders particularly appointed council reps but we definitely needed more direct feedback with the community FESCO and others during that process and that's something we definitely address going forward when we run the next ODF because I think it's a really good framework and it's something that brings a lot of benefits to communicating in that style and we just sent a mail to the devil tread in the last two or three days about a survey that we want people to take on and let me just paste the link in here I'll get it in a second if Pierre or if you could maybe grab it and paste it in and we want you to complete the survey and there's 59 people on the call if we even got half of those people responding and it'll only take you 30 seconds to go through it and give us our feedback because that shapes our communication cycle and that gives us that ability to evolve and become a better team and we do want to involve you more in our comms and our planning that is exactly the way we had EFON to call otherwise it would be just Pierre and myself talking about projects and challenges and so on but you see how public we're making our workflow you see the opportunities for community members to effectively put a proposal together that can get back by the rest of the community and take it on by CPE and deliver something of value that means something to you and the community so let us know what is working let us know what we can improve and let us know what you want their team to focus on in the coming quarters and we will do our level best to engage with you and give you the time that you deserve to have that opinion heard and if we can help we can help and if we can't we'll certainly help you evolve that conversation and try to get that proposal into a better shape that we can bring it forward and that I think is it from our side so we're going to hit the open Q&A section and our email addresses are there absolutely reach out connect to us on social media we're around the virtual conference for the next two days as well and I do want to take the time again to thank everyone for turning up to this for the support you've had for the CPE team and working with us through some very challenging issues in the last couple of months and we're here doing our best and we hope we're being good servants to both the Fedora project and the community so on behalf of the team thank you very much and we'll start taking the questions momentarily from Ben. You've very cleverly used up all of the time slot with the presentation we'll throw a couple questions in here real quick first was from Nick. Is there still a GDPR concern if things are moved to CommuniShift or Fedora's AWS account and thus are still hosted by Red Hat but are primarily maintained by community members? Yes, so GDPR views a data controller as someone a data processor and a data controller and I can't remember which one is which but effectively if we host it we fall into one of those categories and we are on the hook and we would have to work with the application maintainer to set up proposals and mechanisms but ultimately if that application maintainer steps away that falls on us as a team and it's something that is rather complex and legal and you would be frightened to know what GDPR considers to be private information. The badge that we're assuming getting for turning up the nest is classified as a GDPR footprint because it self-identifies something about me at a specific period in time so things like that make this hugely complex just to lift and shift an application to come your shift and wash our hands of it from the CP team. So the next question is there an update on GitLab or the timeline or scope of that? So the update is there is an open ticket with a number of technical challenges that are related to the Fesco points that were well made and well received by the team and we're working through that with GitLab and there is an open AMA with the GitLab team early September if it will possibly drop the data into the chat here it's an open conversation and an ongoing conversation and I'm hoping towards the autumn slash winter time that we start getting some timelines and clarity around that. The session with GitLab I can actually speak a little bit more to it now. So both myself and Camal Verna have been speaking and Liaison with Maurici Sanchez and your community manager from GitLab and ask me anything panel session on Thursday the 10th of September through my office hour slot. We're going to change it to a half an hour difference so instead of when do I start 1300 UTC it's going to be 1330 UTC for an hour. But two weeks and ahead of the session we're going to have a document or a survey looking after that side of things that people can fill out. I'll send it on behalf of our team and people can submit their most asked questions. They are hoping that it's going to be a very engaging hour but they are going to have some answers prepped based on what the community feel are most important to that too. So that should have been a date for your diary too but mark it down now Thursday 10th of September. Ask me anything with GitLab. Nick said I thought Ospo was going to take over some of the apps that CPE is dropping question mark could you repeat that again Ben sorry I got a glitch. Nick said I thought Ospo was going to take over some of the apps that CPE is dropping. Correct so Ospo is stepping in and taking over I think it's between 8 and 10 applications and we're working in a transition period with them that obviously doesn't solve any GDPR problems because it's still part of the Red Hat ecosystem but they're going to take on some of that ownership and I think the first couple of apps might have been transferred already and just a point on the GDPR and what type of the chat we are heavily engaged in the legal on it and I know Marie is involved in those conversations and Rich Bone as well for both the centre-western fedora site to see can we get a path forward from a community's perspective. Ospo is a great bridge for this but I know we got some fantastically passionate people in the community that are well capable of minding these applications once we can untangle the GDPR aspects. So setting aside the data centre move how has the change of the team's scope and processes in the last year affected team morale and burnout and productivity? So as the manager I don't think I can answer this because it's like blink if you're under geo rescues. I'll pass this to Pierre as the kind of tech lead and I think you're probably best place to take that Pierre you're muted Pierre. We are to have a t-shirt that says that I think you might it. I think this is a journey I think this is a journey that we have not completed yet and as every journey and kind of changes it gets worse before it gets better. The end result is to have to give the possibility of to everyone in the team to step out on the weekend to take time off actual time off not you know I'm officially taking my PTO because my manager told me that I have to spend my PTO but I'm actually working just as a volunteer on you know as much time as I normally do on the being paid. So we actually want to have the freedom and we want people in the team to not feel guilty or feel like they can't move away because they are needed so we want to increase the collaboration we want to spread the knowledge so that you the expert do not become the point of failure because you're the only expert so we want to be more experts in the team rather than having people specialize there. So this is the end goal I think this is the path we want to go and I think everyone would like being able to step away from the keyboard, lock the screen, you should always lock the screen and just go to a barbecue and you know if the phone rings because it's on well you know it's going to be handled by someone else and you trust that they will have the knowledge that doesn't mean that we won't have to call people in when something dooms happens and we actually need someone who has more expertise to that but we don't want the expert to be called in at every single hours of the day and night for first level or second level issues basically. So that's the end goal but this is a chance, it's a chance on how people and especially the people who have been in the team for a long time are used to work, we used to have a lot of independence, we used to have a lot of independence on how we organize our work on selecting what we work on and suddenly we need to take COSR into account, we need to we work as a team which is great I mean loving to work as a team having people to bounce idea as being able to see work progress because you know that as soon as you have a progress upon you review someone else and work will flow so it's much smoother and it does give a good experience to work in a team in this dynamic but it is it is a transformation, it is a change, it is something that we need to accept, it is also something that we need to adjust everything that Lee wants to see is not necessarily something which would be applicable to the team so it's going to be an ongoing communication between where we have an end goal the discussion is going to be how we get there and it won't be the direct pass that some people would like us to do it will be something that goes around and around and you know there is a pretty flowers over there that we want to go and check before we reach that endpoint there so yeah there is I think more it's going to depend on a lot of things the color move has been a drain on the energy on a lot of people some of these large projects they basically rely on two people you can't have a project this side rely on only two people without burning them down so when I say send them cookies you know send them cookies Kevin Smoosh they deserve all the cookies you can get them they have been leading that project so they are there is fatigue there is burn down there is aspect there but the hope is that you know we have two more systems in the team David and Mark and they should be able to be there and end all the day to day work and then let Kevin and Smoosh be able to do the more in-depth work we need it and then Mark and David will onboard and learn how to do the more in-depth work as well I hope I didn't say anything too crazy there I'm looking at you friends and Lee they seem to be smiling so I guess I didn't miss out too much I will add for people who are red hat employees you could also send Smoosh and Kevin reward zone points I'm sure they would also appreciate that so next question is is there a timeline for retiring Fed message and will there be help for people who have Fed message consumers to convert to Fedora messaging I can take that one the timeline yes as soon as possible it's not going to happen tomorrow the tube the tube major roadblock which I see for retiring Fed message are going to be FMN and data gripper data number they both have the same requirement we need to have Fedora messaging schemas on every single application that published to the bus this is an ongoing effort data gripper needs there is two approach we can take to data gripper is either we upgrade the current stack but that means that we are maintaining something that we have built for ourselves or we try to replace it by something of the shelf that can provide the same functionalities and potentially it doesn't put us on the hook to maintain it so data gripper has pretty big requirements Fed message schemas to be done before it can get moved on potentially it's something which would be fine to do easy to do the FMN on this hand is going to be a fairly large project it's going it needs to have the UI it needs to be re-synced from a technology perspective from a UI perspective from a UX perspective it's pretty demanding from a technology side being able to do notification on ERC being able to do email notifications at the level of the entire project as well as being able to let people manage which notification they want to have so it's a pretty complex stack so that one will require some work the bold estimate that we have for that is at least 6 to 9 months for FMN so that gives us if we start working on this in the next quarter that gives us a Fed message retirement by potentially the end of 2021 but I wouldn't put my hand I wouldn't bet my hand on that basically but I think that's the timeline we can hope for as to whether there will be help to migrate to consumers it's pretty straightforward to do I'm happy to help anyone who is in that situation there is the Fedora messaging documentation and read the docs it's actually fairly complete so it's easy Kevin Fengy is just specifying a small data point to give an idea about data number the database behind has about 600 gigabytes in one table just a little bit of data to process Fedora messaging has a lot of documentation on how to port including from FedMessage and I'm happy to help anyone who is in that situation the sooner you can get started the sooner it is for you and the last question on the list is from Paul who wants to know if there is an Irish equivalent for the phrase soup to nuts what does that mean so I think I said the full whack and then Paul outdid me with the whole kit and caboodle is definitely one