 Hello, okay. Hello everyone. Welcome. Thanks for coming to the lightning talks As you all know the booth crawl is like gonna be right after this. So we'll try to Get you there on time. We're gonna we have a lot of great content. That's why you're all here So let me welcome up real fast Elizabeth Joseph. It's gonna be talking about status.openstack.org Thank you All right So what I'm doing here is just going to give you a tour of this page status.openstack.org because I often find out What is that? Not now. Should I upgrade? All right Oftentimes I run into people who don't actually know what all these links at the top here are for so I'm just gonna do a quick Tour of what this all is So the first one a lot of people are probably familiar with it's the Zool status page This will load up things that are in your in the check queue. So when you upload a patch You can see your patch here. You just want to search down you can search the page for your patch number You'll see how it's doing and Whether it's I'm sorry Yeah, there's a filter up top. Oh, yes, you can filter using the filter as well Look you can sort by projects or reviews, okay And then once your patch has been approved it'll go in the gate queue and you can find it in this list and then there are other The post queue and post release post and other things You can see on this page pretty much gives you an idea where your patch is While it's being tested This is probably the most well-known page on on status at a mistake You then have the elastic recheck page When the elastic recheck bot comes into your patch and says I think you hit this bug It means someone has already categorized the specific bug you ran into and it's usually either a known problem It is definitely a known problem. Either it's an infrastructure or it's in the test has a known bug or other things that have been happening That the developers already know about and it's not your patch. It just happens to be something you ran into So the recheck page is a really nice page to go to if you're looking for The the things that are popping up, especially the uncharacterized tab here Which will show you failures that have happened but have not yet been Put into bugs and are not really being tracked In the system We then have the reviews tab So this has a bunch of early projects for open stack because no one has used this page in a very long time But the idea of this project was that people wanted to see reviews sorted by importance So taking information from Launchpad based based on various criteria This takes all the reviews in a project and sorts them presumably by some sort of review send thing that makes sense So the top review this system thinks is the most Important review to go through So you can go scroll down here a lot Where's the page down? Here we go. So yeah, you can see glance is on here a bunch of different projects All open source so you can you can add Your project to this list if it's not already there. I Don't have time for question Yes, I'm getting to that So a lot of people don't use this and a lot of people have started creating dashboards for Garrett So one thing we added I think a couple weeks ago was a link to this neutron Dashboard that was created by the neutron team I can't actually bring it up because you need to be logged into Garrett to see the results And I don't want to bring up Paul's Garrett even if you are logged in But if you click on the Garrett dashboard link for neutron it'll bring up prioritized reviews inside of Garrett itself So rather than using a review day Probably will eventually Morph into using a series of dashboard links like neutron has already done. So that's pretty cool So next one we have is bug day and this one is it used even less than review day but the idea here was that the bugs are listed pulled from from a Launch pad for a bunch of different projects and it shows the status of the bug and then it tracks them on this graph So the idea was that teams were supposed to have bug days and when they have a bug day They can look in the next day and see how this graph dropped in the bug numbers And that's really inspiring and exciting if people did bug days So some projects do I don't know if they actually look at this But if they did it would be pretty cool to see infrared did bug days, but I don't think we're even on here Yeah, we moved off the launch pad so we wouldn't we I don't yeah We might have used to been on there, so it was fun to see how the stats went down So we're almost done, but we're gonna go to one of the bigger ones that's pretty new Matthew Trinage did a whole talk this morning on OpenStack health this morning. So there's there's a whole talk worth of material here But essentially OpenStack health was trying to give you developers of you into the health of the gate and the CI system So you'll see here a graph of the total jobs run So the blue is how many how many jobs passed and the red is the failures So overall you can see how many jobs are passing each day over time and the failures You can then dig into the jobs so you can search For through a certain project and see if a certain project has more failures If you scroll down Lost my focus, okay You can see things like you can go into all these different specific tests Specific runs and see what what actually failed It's going down a bit more you can see by project what projects have the most failures happening And then this is also you can also search it and then drill into these things And have fun by yourself. That is all the time I have Thanks Elizabeth All right, Paul your laptop is already up here says helps Hello, so I'll try and keep it quick and fast here So we added Grafana dot OpenStack org I think about six months ago if people don't know Grafana is basically an open source dashboard rendering project The reason why I wanted to add it because this is something we have already which is called graphite and it's actually a very terrible interface on how to Graph stuff and I'm not going to go into it But we would we use this today for the stat of the Zool page on status at opensack.org to look at the very pretty graphs on the bottom So one of the problems obviously was when we wanted to add Grafana in we wanted to give users the ability to create their own Dashboards so for example, we have the node pool dashboard and these are live stats that show what's going on in the queue But we wanted contributors to be able to control this and we didn't want to have to set up LDAP and do it like an open ID Back end or something or so on and so forth and we wanted needed the ability to code reviews on this And it's hard to do code reviews on just native Grafana so doing the Infraway we wrote our own little tool to take YAML and basically convert it directly into JSON and then post it to the local instance of Grafana Generating the dashboard and it's heavily driven by Jenkins job builder or JJB, which is the tooling that we use for our Jenkins jobs So ultimately Yeah, so Where are we here? So we have in project config project config is the user side of the infrastructure configuration a Grafana dashboard or a Grafana folder and in the folder you can see all of our YAML files Basically, there's one special file called data source which controls the back end in Grafana to where to get its stats from which is graphite.opensack.org and then every other little file that you see is basically the YAML file for a specific dashboard. So I was just looking at the node pool dashboard I'm kind of pecking here because I'm holding the mic so this is basically the YAML file for it and Like I said, it's it's almost a direct mapping from YAML to JSON and JSON to YAML Except we've added in some basic validation So that when you push your YAML file up into the code review graph YAML runs and actually validates that you're not putting in You know Whatever crafted, you know stuff to hack the server or something like that We just say, you know, is it integer? Is it positive? Is it negative stuff like that? It's pretty dry like we of what it does But these targets are actually the stats be namespaces for pulling in all all the information and and like I said, it's really a tool to give Developers and Opensack the ability to create craft and create their own Grafana dashboards kind of like the in for a way and Yeah, I mean, that's really all that I have on it and I know I'm kind of short on there Haven't used all my time, but I'm not gonna sit up here and ramble. So that's that's basically it Okay. Thank you. Thanks Paul. All right. We got June Yes, all right Going to do the switcheroo. I guess I should entertain you in the meantime How quickly can we Set up Mirion and oh yay desktop of the year right there. It's all you. Thank you sorry for violating the cardinal rule of Lightning talks and using my own laptop, but I promise I have a good reason I'm going to I'm gonna violate the other cardinal rule, which is don't do live demos. So This is a this is a program that I wrote called Gertie It's a console based interface to the Garrett code review system that we all know and love Except some of us maybe don't love its web interface quite as much as we're supposed to I Love text interfaces. I do I write my code in a text editor not a web browser and I run my code in a shell not a web browser Some people do that I understand but I don't so for me like doing all of the code review in a console Application makes perfect sense. So anyway, I wrote this I wrote this thing called Gertie It interface interfaces with Garrett over the rest API, but it does something a little bit more than that It actually syncs all of the data that it needs to work with down locally So as a local copy of quite a lot of the database that's in Garrett so that it can do Be more responsive. You don't have to be on the internet You don't have to be on a fast internet you could be on a plane or you could be in New Zealand and it'll work just as well and Similarly for all the metadata that is in a database all of the actual code is in your git repositories like the same ones that you're That you're working with anyway So you can easily do things like check code out into your working repo straight from your code review system So anyway, when you start Gertie, this is what it looks like. It's a list of all the projects that you're subscribed to So you can scroll through that. I recently added this thing called project topics So you can like group projects together if you wanted to get you know We have a lot of puppet projects in in front, right? And so maybe you want to focus on them or maybe you want to focus on not them So you can you can group them that way and find them. It'll show you how many open changes are in each project How many things have been you've reviewed? When you open up a project here's the list of unreviewed changes This this looks kind of like a change list in in Garrett except that It also kind of looks like a mail reader or news reader and it starts threading changes together if they're dependent on each other That's that's not on accident You can you can open up these changes and there is a Screen here that looks again, not entirely by accident. Let me Anyway, you can see all the information that's that's on the the Garrett change screen It's got dependencies on other changes the files that are changed. You can see comments on there You can open up the diff and it'll show you a side-by-side diff of the change if you prefer unified diff There's a config option. So you can do that you can leave inline comments Just as you would expect on the old and new sides there, right? You can go back and you can review changes You know leave I I don't even know what I'm looking at here. So probably shouldn't I should probably shouldn't approve this but Yeah, so there's oh, there's What else is neat you can you can do? Yeah, yeah, so it like I'm actually offline right now if you see at the top it says offline sync eight. That's Gertie Gertie knows that it's not connected to the internet right now And so it's not even going to try to sync things if it were to get online That sync counter would start counting down on things it knows that it needs to do to catch up the local or remote states So anyway, you can do what I've been doing you can leave these comments you can leave reviews and When you land you can Gertie will start pushing those up for you asynchronously if it If it detects any discrepancies like you you gave a plus two after somebody else gave up minus one While you and we're in the air it'll actually hold that for you and alert you to the To that situation so that you don't look like an idiot Yeah, so Supports searching for things in line like unless right so you can Anyway interactive search all kinds of things so Anyway, it's it's a fun little thing. We're adding things to it all the time. Of course. It is in open stacks Garrett up here. So open stack slash Gertie is the get repo name you can check it out from there you can Check out this branch and you'll be running the thing that I'm actually showing you right now and Anyway, it's fun. You should try it out and Confirmed some changes if you feel like it. Thanks Thank you Jim now, we're going to learn how to add in the big tent a project Joyce thanks sorry for having slides and no live demo So you can all read in the info manual what I'm telling you now So if you take one thing away from my lightning talk use or in from manual use the creators guide and read through it I'll try to give a very short five minute overview of that and only of the technical side of it because You all know if you are contributing to open stack and part of the community that there's much more than a repository to it There's the open stack way of doing development of working in the open the for opens especially and all of that is documented in the project Team guide together with the processors we in info only support you with those processes And I want to show some of that how we support that so open source You need a repository where you want to work on you can also work with all the repository so all of that in Infra handles all its configuration changes like Paul already said we do it the same way like like like developers do It's a git repository that you propose changes to that gets gets tested gets that gets reviewed Gets merged and then it's active. So you check out the project config repository Can make changes to graph yaml or make changes to get its Projects dot yaml at your project there and you can also import an existing repository, but that's also your and the upstream work Word here is important. That doesn't mean that's the upstream version. It's just import. It's misnamed So keep that in mind. It's also your chance to clean up the repository Remove any branches any text that you don't want or if you spit a repository use git filter to only use those History of the file and the files that you want to have a clean repository that you can work work on and start with We want to do open development That means in OpenStack having a core team that can abandon a page that can do a proof of touch page It sets the workflows and so the usual defaults that we set so you need to create that ACL file for it On you also want to run tests on OpenStack I hope you do you can merge without tests, but but you should if it's a Python project The minimal test that you do is the Python jobs, which is a job group Which is something syntactic sugar that we use here and defines jobs in Jenkins so keep in mind Jenkins defines what to run and Then you tell it in the in Zool in the Zoolay out file when to run it when to trigger it So only changing one file will not help you because either you define jobs that are not run That are not triggered at all while you trigger some jobs that are not nobody finds You need both parts of that. Let's think that's the most important part for you to take away Also, if you're changing the opposite Project config I want to enhance your project going forward What's also important is that those two Jenkins and Zool are different projects different namespaces they share the namespace of the Job name but the syntactic sugar we use here the job templates on the one side the Python The Python jobs the pipey jobs is something totally different completely unknown on the Zool side Here's a Python jobs as a job template that will define that's a PEP 8 The Python 27 and so on jobs are run in the check and on the gate queue And we have published to pee pee which publishes your repository when you take it What else else do you want to do you want to communicate you want to communicate on our seed channel? So set up get it but channel sort yaml and Define conditions when your projects should should get notifications of mergers of new pages at give give open stack the open stack bots and the up and success to it in The access port file and if you want to log about it and have the status command so system-wide messages You need to do that in the system config repository. He is also important So that you can use for example also the success command and tell the community about success and then Mike will show Explain that in the next week newsletter You also need to set up in other systems Not this so once you have done all that you can commit your change and get it up for review You also need to set up bug area and launch pad and pee pee for publishing If you are a big 10 project you also want to make a change work for governance and Add perhaps Publishing for dogs of mistake org or set up the translation workflow And all of that together is explained in the infomanual in the creators guide as I said before so trust in time Amazing as they explain on adding a project to open second less than five minutes. Yeah Yeah All right So now we're gonna hear about one of my favorite subjects see I but enabling a better see I Experience so here we go Yes, it's more like enable better see I when you have a hardware dependency Okay, I'm Moshe Levy from melanox software engineer so Today in open stack as you know, there are several features that are heavily dependent on the hardware Features like PCI past through where you need to pass through The hardware to the guest Yes, so you need a special PCI device SROV for networking you need the SROV card that support You need a network adapter that support SROV If you're using ISO for example, you need Which is ice because the overall DMA you need a network adapter that Support our DMA so as you know, as you know in a open stack the you need to have a good stable see I to test all the features and And These features are a bit a little bit more difficult to test because they are the hardware dependent So why why it's so hard to test these features with see I so first of all you need Physical servers and you need lots of physical servers because it's depend of the amount of the Jenkins job and the load balancing of the all the jobs and you don't want that the Jenkins will report like after three days. You want like a quick report like after a few hours Also, it requires you the hardware device. So in our case, we are managing using network card so you need the network out in each physical server and that's saying a network out in each physical server, so that's mean lots of network cards and You may want tested with the also different configurations of the hardware of the audio device this means a lot again lots physical servers and lots of the other device and the cleaning part which is Difficult part for us so you after the job the ends you want to bring the physical device to its initial state and Then again, you have two possibilities either do re-imaging this and that's bring complexity to your CI environment And also takes time or you can try to do what we are doing today Is like just removing DevStack and some Python packages and hopefully you you did the cleanup of the physical server But that's not all always works So what we want to achieve in a perfect while we want to reduce the numbers of physical servers We would like to somehow to eliminate the hardware device dependency and we want to do it tested on a variety of hardware devices and Maybe on a variety of other device configuration so We thought about it in melanox how to solve this problem and we said why not simulated the real hardware and We The idea is just a second So the idea is like adding a module of a hardware simulator to QMU to QMU so So so it will be like removing the other dependency So how this CI environment will look like so basically you got the virtual machine that running the Jenkins and Zool Which is like most CIs but on the right you can see that I've got now a QMU Process with the hardware simulator module and this can be like a different out of simulator per guest And in the guest itself you see the dev stack invite this tech environment VM with all the hardware configuration that you need so this basically Reducing the number of physical servers because we are going to a virtualizing environment and also we don't need actually our nix so We are all in each case the physical device. So it's basically Our CI environment is becoming less complicated So just a bit dive info into so you have the QMU process Which containing the guest with which has the dev stack environment or whatever where you want to install the open stack and in it you have the device driver that actually That actually Is the vendor? Device hardware right but instead of we're talking to the actual hardware itself You're talking so to some something that's in QMU. That's a similar idea of the device So it's removing entirely the the hardware dependency This is like a summary of That's it Okay, no, that's fine. It's just summary of how we do it and What the benefit of it that thank you? Can we switch the HDMI or do I have to do something special cool. All right, so I got Thomas and So how to drive downstream baggers mad. Thank you. All right, so This is my friend Hickelgemar Who's doing packages in Fedora for the last 10 years and he does all things RPM for open stacks in hard you and Here's Thomas Guaron works at Marantis and is in the packaging open stack and devian since the cactus release So let's start we love you, but spreading you bad news in a nice way is hard, but So we've seen the same mistakes repeating over over and over again So we have the choice to either say to like like here and then the most nice nice way possible Or to just complain about someone. Oh, you did the bad thing. We obviously prefer the former so and Open stack is huge. We have 300 Source packages and we can't pet every single package after all there is only one of us Yes, because he's only one doing audio. I'm nearly the only one doing that. Yeah, and so so the types of problems we see Are listed here and we are going to through all of them So network access at build time not a free Ness funny dependencies assuming you are running Dev stack You know the funny errands of depth Dev stack t-shirt And no proper releases. So let's go to the first one When you build the packages in distribution, we have no network access Don't ask us why because we disable it because we want to have reproducible built that is a big thing in distribution we have privacy concerns and Also, if you are in desert Iceland, you want to build open-stack anyway, and you get no networks except if you use pigeons so to continue on that so one of the things that Are indirectly consuming network access is assuming you got git so you will for a reason use git in your build process or How about doing a carry to the 3w.google.com? We've seen that a few times or downloading whatever from the internet just because or even worse like doing a curl By bash because you just want to execute something. Yes. It's so fun, right? And how about you change the content of that that thing and the address Another thing that we see over and over again is interest things. So because we have no network access You can actually use into things is if you want the only thing that you got to know is that we're going to patch it out So your doc is going to look ugly with broken links And also we have all this fairness dependency your own thing stem Do you think it is he does this really matters? No own test runner. Why and also all various requirements depending on the ugly ones also using the wrong versions or don't caring about the version avatar and distributions and Also, never do that changing changing dependency before GA Because we are not do it. We are not doing this at this time. We have other things to do. Please don't do that So dependency versions, it's been a recurring topic over and over again as well so There is open stack within distributions, but open stack is not alone. There's all this stuff to care about So if you have a to restrictive a book up Then probably some other package maintainer will decide to upgrade I don't know a scale alchemy or Django something like that And then it's going to break everything. There is no way we can reverse that because we are not the only citizen in the distro the same way You can have a to restrictive lower bound which can be problematic for packages that have a Lot of reverse dependencies like jQuery or things like that that we can't just upgrade because it's going to break everything else and Also, you see this. This is what Thomas sneaked in in Oslo precept Description hoping that it would end up in pi pi and you see it did end up in pi pi So, please pay attention to what you're doing We're also having automated tools generating some packages and it get that in description and this is bad So, please trim your description and we love you if you do that That's it Yeah, and by the way, you guys are awesome Just want everybody to know if you've been involved in the dev million list that During this presentation Thomas was smiling. So I Have a picture to prove it. You were smiling during this presentation So this is all I Yeah, we're using VGA. Sorry Okay, yeah fixed it Do you see it now? That's that's net fire I'm here for you I've been here, of course. It has to be a lightning. Yeah. Can you just mirror? Yeah All right, so yeah after you here is gonna tell you about why long live bash. All right, so Man, this is gonna be quick. All right, so I'm gonna give you a quick to your bash eight It was supposed to be using bash eight, but it's pretty simple. I think you guys will figure it out So quick who am I I've been working behind the scenes kind of an IBM for four years I've been on open stack deploying it developing it more deploying fixing And I'm here today because I took a bash eight spot py script from dev stack and turn it into a project And we'll be talking about that Mainly I work on you know building a lot dev ops CI CD tooling That's because you know most devs can write code not everybody can deliver the code effectively. So that's what I find anyway So what does bash eight or formerly bash eight? If you're curious about the rename we can ask I can talk about that later So bash is best described as a style and convention checker for bash grips So it's basically like pep eight if you've used that but for bash the output looks very similar Yeah, you'll you just run it the same except doesn't do auto discovery because we had problems It originally came out of dev stack. It was written by shawng did shawnd egg originally And it's gone through several evolutions from that I extracted it basically from dev stack because Well, I had a lot of bash grips internally that I wanted to kind of clean up You may have this problem as well. I find a lot of people do So it has a very simple test suite I added And then it does obviously style checking and it also does some error checking which is kind of important for dev stack mainly Why should you use it? Well in open stack We really like like code quality tooling and like static code analysis like pep eight flight pipe flakes like a hacking Bash eight Coding conventions are good. They give you a little bit of familiarity and otherwise unfamiliar code You know, and I got a couple examples We always have these comp and log things all over the place and opens that code, right? And then readability counts, right? Yeah, and then of course bash eight this kind of one applies to only bash eight It can help you catch a lot of common errors. This is basically what it was designed for You see a lot of common errors in dev stack, right? In fact, most of the patches that go in are from Ian from finding some weird error pattern that like Just some strange like behavior in bash that you wouldn't think would be You know bad, but it turns out like it'll return something different you wrap it in parentheses It's just the you know, very strange things like that that we want to catch and make sure it doesn't happen How do you use it? It's pretty easy. You can just you know, pip install it That was kind of the main thing about turning into a project. It's pretty easy to pull down now You know, and then you just run it on your files, right? Running it manually is pretty simple, but you want to put it in talks, right? If you go look at the dev stack dot talks, I and I that's a pretty good example, although it's kind of heavy weight, too But you'll get a good example of how to use it and you know run it in your CI or you know OpenStack CI for you because I just read your talks on Yeah, and all my checks are documented basically in the read me and I think we have a man page to Ian put that out There which was really nice More about the project so after creation and then adding a test suite because that's kind of important if you're gonna extract a legacy project We had a large like kind of burst of contributions in the community for I mean There were tons of people that wanted to do different things pluggable checks like You know, we had many adopters the info guys were running it No cheese running it dev stack obviously is running it And then there's also many internal projects, you know, I can attest to that It tends to be slower moving now Ian's kind of the main maintainer. I kind of just hang around and reviews patches when he emails me And that's mainly because we broke dev stack eight one time so Break it once kind of learn don't do it again We're still a major version zero and you know, we try to be open about feedback and patches We're not interested in pushing opinions into other projects. So please don't try to do that And skip that one So here's some links and things if you're interested I would say the main takeaway from this is you know, this was basically just a small script in bash Or I mean in dev stack that I extracted and it turned out to be you know, a lot of people used it Just as soon as creating a project out of it, right? It's it's not like a big project. There's nothing really sexy about it It's just a thing that people use because it provides value to a lot of people because lots people use bash so And that's all I got so thanks All right, so that concludes your lightning talks everybody Thanks for coming and you are now more prepared for upstream development with all these Presentations by wonderful people zero for them again one more time