 My name is Rashid Khan. I work out of the desperate office. Welcome to the networking track I'm supposed to open the track But talking about the track on seriousness there if you click the networking icon It comes out pretty cool. So there's a whole bunch. I'm just starting like tap dance in the beginning and then There are a whole bunch of people who are going to just be in this room Until like three and then we have at three we have the workshop and Wi-Fi and parallel sessions, but if you are interested in networking and Want to just stick around in a nice warm room next to coffee and bathrooms. This is a good choice Okay, with that said So not fake news development and QE collaboration is possible How many people who are not from the United States get this not fake news reference? Yeah, so as I said, my name is Rats Khan I this slide deck is in collaboration with many people It would have been ironic if I did not collaborate with QE on the slide deck that I was Anywho so It's not on the slide deck. It was too late. So if you have questions feel free to ask me afterwards So what I mean by collaboration is like people take it for granted, but I wanted to just start by Showing what usually happens This is a game called fireball tennis. No, it's not very common in the United States But I just found this video and it served the purpose. So this guy just picks up a flaming tennis ball and lobs it Across so imagine the first guy is development He finds this hot new feature and then over the net He just lobs it and the other cut let's imagine the other side is QE and like I don't know I can't test this back and then no no you have to test this And this is part of the feature. So this is a problem that sometimes happens Not that it happens in red hat at all, but I'm just saying I've heard this happens sometimes Or the other scenario is this that development just lost the feature over the wall boom It hits QE on the head For the people who are not red-haters we call our quality assurance QE quality excellence Wanted to make sure it's too early in the morning. The other thing is oh, I'm a developer. I don't do testing I swear to God. There was a person who worked for me in another company. He did quote that to me I'm a developer. I don't do testing Okay It's QE's job. I Was very shocked The company still exists where I left them. I Don't do QE. It's QE's job. So that believe it or not It does still happen and water for models and all kinds of crap still happens so these are I just wanted to start with the level set and I'll be honest with you before I started this effort a little bit of this was going on This was happening even in here, but we took a lot of your we'll talk about that So what does it result this result in obviously results in failure? If you just lob it over the wall QE has no idea or throw them a ball of hot feature or what you think is hot It results in absolute failure. So enough of the videos now. Let's get to the serious stuff. Oh What Just when I said serious stuff. So the front guy obviously is with a beard and fat That's me and now what we have achieved is we have achieved Without QE working after a long time that we do Have the boat running in sequence and we are achieving great results. So with that said, yes, this is the last one Okay So, how did we do that if you are here to learn something earth shattering? Sorry to disappoint you. I don't have anything So because somebody pointed out Russia that you have a type on the slide so How did we do that just level set? I don't have any earth shattering stuff But we have a whole bunch of small stuff that we did and it has a journey it took a long time to get here I would say about five ish years. So it's a long duration and a lot of effort and energy Well forish we have been reaping the benefits for about two years now, but three four years So the fundamental thing is openness and trust. I Know it's easier to say then I'll show you what I mean by that But the two fundamental things that I would summarize this with collaboration with QA is openness and trust We'll come back to it in a second So let's talk about some practical things that we did. I just want to share with you some practical things We did these are like the back of We all know it in our team in my team at least but I just wanted to share because there might not be obvious So we took that lead in doing our own unit testing and we created this framework called tier zero testing We call it LNST, but other kinds of automated testing that we developed ourselves and we developed with QE So what does that mean? So earlier before we started this effort? Somebody merged a patch in the vice team or our main line main branch It stayed in bedded for eight months And then towards the end of the cycle when QE started testing it it was a complete disaster that the performance was gone to I don't want to say shit, but performance was going to gone down the drain and But the problem so I Refound the problem. This was this patch a month ago It was merged in and I said can we revert it and the answer was no why? because what eight months other patch it had Cascaded on top of it and now we could not revert it is like eight months of development on top of it So it was like a slap in the face for me and say look this has to stop So now we developed a system in which whatever we merge in Even before it goes to our main line or vice team as we call it It's tested it's tested very thoroughly and it's tested for regression detection is tested for any kind of It doesn't cover everything But at least it covers the basic connections and the basic stuff that supposed to be working And then if there's a problem it causes a problem even let's say 5% degradation Then we just hold the horses hold on a second. There's best worst case two weeks worth of patches We quickly isolate which patch and we stop the presses and say okay We have to either revert this out take this out and continue or do this so Bugs are squished in days and this reduced the bugs going into QE by a tremendous amount So now when QE starts they're testing They they don't have day one Stop the presses kind of bugs because we already tested them. So that's pretty cool So there are some quotes that I've sprinkled in the presentation I'm not gonna say them But if you see something in blue on the side or something that's an exact quote that I copy pasted from some QE members Who look at my slides? So see I is really revealing bugs even before the code is merged This is a merit back for me this quote is what I would like to Make a laminated thing and hang around my neck all the time because this is exactly what I was trying to achieve What our team was a time trying to achieve when we started this effort and unsolicited I got this quote from one of the QE person that I really Respect and when I showed him my slide. Hey, this is what I'm gonna show and reply He sent me this quote That see I is really showing bugs before even the code is much awesome Okay, what else did we do? We created a qualification process for hardware certification So again, it sounds like obviously you didn't have that. Yes. Sorry. We didn't have that but what we did was we one of our developers in our team developed some test for his own testing that he's not causing any regressions and It was so thorough and so good and so fast and so automated that we work with QE QE created an automated framework around it And now we have a whole bunch of tests that we give out to our partners even so let's say a partner gets on the door saying Hey, we have this hot new car and a hot new driver. We want it to be part of the Red Hat software stack Earlier we used to bring them in merge patch code is done Send it to QE QE funds issues and then problem starts now what we do is Hold on guys. We absolutely want your car to come in. We love you. We are awesome But here's a whole bunch of automated tests that you don't need a traffic generator for you Don't need any external equipment for you don't need any expenses piece of equipment All you need to do is one or two one senior colleague QE person and then you need 3% can run these tests on your car On your driver on your own and output is a result Which is a spreadsheet which has green and red lights like pass fail pass fail and When you have the whole spreadsheet green, please bring it in and we will integrate the hardware in inside and Believe me or trust me. It has been just phenomenal. I thought that the hardware vendors will be like, oh my god What do you mean? This is extra gate. This is so difficult for us, you know But they also said, okay, sure these tests the other thing was all these tests are unfair We're not going to do it. This doesn't bring the best quality. So how much nothing like that nothing absolutely nothing like that This at show will run them. How much time we have we said about three weeks or whatever time you need You will be part of next release cycle. They said fine. We started a dialogue with them Trust me. They found bugs that they were surprised with So it instead of having everybody come in and then Merged in and then now is the problem. We said, please run this on your own In your own environment will help you set up the environment if needed and then we do it Wow time flies. I talk a lot So the other thing we did was so this was development taking the lead The other side was QE take the lead sometimes and QE develop this network manager testing, which is super auto magic it runs all the time it runs in the background and It covers 90% of the code the LNST that I talked about also covers a large amount of code It doesn't cover everything, but it covers a lot also the other thing another thing we do is So We bring it in we do our own testing, but then we ask you a can you please Run your On this code that we are about to bring in Before it's part of the wise and that works out very well also so it gives QE the preview and there is no flaming tennis balls we look at their test plans and It works out quite well. We start a dialogue They tell us the deltas between release to release and we do a thorough thorough Investigation slash feedback loop on the test plans and we say oh this maybe you need to do this a little bit more You need to do this a little bit less and it works out very well again a direct quote I would really like to stress out that the well collaboration and willing is to help Any time and I didn't even change the font. This is the exact code with that font in there Which is awesome. This is a merit batch for me or our team Developers not surprised QE with features So we have a major release coming out Well, it's some of the features will develop upstream some of the features We'll develop many of the features will develop by non red hackers But what we did was we said we did an audit of the home release what we are bringing in from upstream and we created an MVP list Or most important to test or etc. And we told QE. Hey QE, please watch out This is the new stuff coming in a lot of it is not touched by us But it will be part of the release. Please make sure your test plans and everything is prepared for that So again not like loving it over the wall and go figure it out QE all the man pages are upstream No, we don't do that. We say okay. Hold on. It's we are in this together. Let's work it out together Take preview Sometimes we don't know all the answers So we use the Part of our release, but we are not going to fully support it. We have customers testing it Partners testing it and then once so there are two aspects with putting in tech previews pretty cool And everybody agrees QE understands that but what we did was we collaborate with QE to say Hey, are you we are going to not take this out of They have to now test it much much thoroughly because it's a supported feature So we have a dialogue we discuss it together and figure it out. What if I started at 9 30? Then I have some time Okay, okay, okay Your reminder threw me off a little bit, but I have some time Okay so now So this was all more of our technical stuff that we did Bottom line is mutual respect and trust so the at the lower right of my next few slides You will see this whole team of mutual respect So we we provide career paths for QE team members We have about from my count four or five Previous QE team members who are part of our development team now Sometimes they reach their limit at QE and they're doing a really good job But there's not much that they are enjoying in QE anymore There's not a challenge so the QE manager reaches out to me saying hey, I have this person he's a little bit bored now Would you mind we talk to them? They like us. We like them. They move to our team We give them a wreck. They hire another QE person Awesome works out every single time every single person who has moved to QE from development has been a superstar Which is awesome. Awesome. Awesome. That number three is wrong now that I do a mental audit. It's much more than three I think it's five Hardware sharing so we we put in the cap x and off x and whatever we need to We create our own lab, but it's not like my lab It's our lab. We've done our testing. We want it. We want to use it We will use it all the time and if the QE manager knocks on my door and say here I should We don't have the same equipment or we don't have the test generator or whatever. Do you mind sharing? We say absolutely. Please do. I'll give you a practical example We bought a cheap traffic generator and there was an overhead cost of our chassis Chassis We told QE saying, okay How about we absorb the cost of the chassis and the first card and we will share with you? You integrated into your test plan and then from there on you can just punch in your cards in that chassis And you don't have to worry about that the cost of the overhead of the chassis or this Signaling and etc. Again sounds like the G the why didn't you do it before but a lot of time it doesn't happen That's why I'm emphasizing it. So we share hardware as much as we can And sometimes they qualify something and say Russia. We found this awesome deal in Beijing. It's really cool We're gonna put it in Beijing. The latency is not that bad. The packets can come all the way to Westford now that if the car is no longer part of it I can say the packets are coming from Beijing to our lab in Westford. Otherwise, you would have closed it Right from his phone. So that's pretty cool And then we interview each other's candidates Because again mutual trust and respect we talk to each other say so my counterpart calls me I call her and say hey, do you mind we talk to each other? We even offered money to hire people don't tell my boss Yeah, but I be offered they never took it but again mutual trust and respect, right? So okay, you need to hire someone you are short by whatever 10,000 CZK sure I'll borrow you can borrow from me you can replenish etc. So we work jointly This is not common gave advice to each other for people management team management project management all the time in the end hugs and kisses Mutual respect and trust So I know I rushed a little bit We have five minutes left. So that's good and that should be plenty to answer questions This was high level I did not want to start with like a Java script in the first day It's a first session. It's no e-day. So it was high level just to get the juices flowing for you Questions other other things which were curious that I didn't explain well Yes So five years ago is my example of that eight months ago the patch went in that's when I realized we have to do it Oh, sorry the question from Dimitri was I Said in my presentation that we started this five years ago And it took many many steps and we asked can I highlight some of those steps? So five years ago was the point when I realized that no we have to test sooner So what we did was we dedicated first of all we Dedicated people to start our own voice So we had some people who are in I don't remember I think we hired people or we got some people from QE into our team Who started working full-time to develop this basic regression detection tester voice and We collaborated with many different parts of QE and other teams to develop the new structure that Does run a test compares the results stores the results stores the configuration of everything makes it repeatable Finds the test which failed Creates a script just for the test that's failed and gives it to the developer So we start now this is the end result But we started that it slowly slowly now is so robust that now developers can request An LST test even before they merge We test them before things going to ice cream But the key things were we invested in people and a small team And we invested in hardware That is dedicated and we turn it into a gate So those were once it was ready and mature and we had trust coming we had understanding of the whole development team that yeah It's working we created turned it into a gate So now to use redhead turns you created a sub maintainer branch for networking Jerry is the maintainer for that. He collects all the patches before going to buy We collect all the patches to use a bill puts it in this testing if it's all green Emerges into my stream if it's not green you stop suppressors call rush it's and rush is your patch Do you want to revert or do you want to fix if you want to fix you have a couple of days? So that was step one that we dedicated people for this and they're still dedicated And they keep on enhancing and keep on expanding the coverage of that test And now they're doing like advanced stuff like OBS DP DK SETP it's really it's really robust now So then I think the whole cycle now takes one day to complete on four machines And we are thinking of paralyzing it or my heart was even if it takes a weekend Like we started on a Friday and Monday morning. We have the report still it's fine So I just found out it takes a day still okay But we are dedicated people But about three years ago it was already showing its fruits first fruits and now it's just the orchard is growing Yes, what about the debate is question? What about buying and mind shift? Um, I'll be honest with you. I am quite lucky that on the QE side and on our development side I in red hat did not have a team member who said I'm a developer. I don't test There was no dialogues like that first of all. Yes, there were some people who said, oh We live on upstream testing and upstream robustness, etc. I had to slowly convince them that okay If you have a test script and you have a test cycle excellent Please give it to us and you will make it part of this testing suite So that it works upstream the same was the problem a lot of times to my utter surprise It what we thought was being tested upstream was not as robust as we thought Somebody had sprinkled few tests here and there, but it was not as robust as we need it And then I kept telling people about this my disastrous example that hey guys If you go with that model where we test at the end eight months, we could not revert remember that time So we had to use a crisis In some circles So Stuart is asking the QE and developer is In the organization structure if you look at the odd chart or whatever, it's all there to see levels week before they emerge And he asked me if I can list the advantages or disadvantages I'm not going to list any advantages because I cannot think of it I'll be honest with you. I think that they what we did was we said look I call my Partner partners and look this is not working out. Okay. Do you mind if you collaborate? I saw an open arms on the other side We didn't care who's reporting to home. I Be honest with you I did talk to person boss like I got this boss boss boss is boss And we developed a relationship, which is not like you suck. No, you suck We did not start with that. We did not start with the idea. You're not starting to see the point. We started it. Hey Let's make this better. Doesn't matter who's reporting to me. What do you need? How can we help? Honestly, that's how I started it and And I'll be honest with you. It was phenomenal Once we start a conversation with hey, there's a problem. How can we help work this together? On the other side was yeah, there's a problem Out of time. I'm here