 So do I get started you want me to start? Yeah, well So And hello everyone and the goal of this presentation is to inform you on what we were doing in OSA But also pinpoint on how and where you could help So my name is John Philip My ILC nickname is EVR tab because it's gonna be impossible to pronounce I'm Mohammed. I'm Nasser on IRC. I'm the current PTO and this is our previous PTO So in this presentation will answer the following question. What is OSA? Who is it meant for the community? What did we do recently but also? What are the plans for the future? so our name is not really fancy so In OpenStack, you have like a guess game that you usually do is like what does this project do and And well, we don't have the chance to do that. So I made you this So on the left side, this is our mascot. It's OpenStack Ansible it installs OpenStack with Ansible and Specialty is we install it on containers or on bare metal So, who is it meant for? We mainly have three classes of users The cloud operators they use OSA to build a cloud for the users They telomated for their customers or their internal IT needs We have people from ISPs hosting from retail companies My met for example is an example with with Vex host We have another category which is the bringing our own cloud product thing The companies are creating product to deploy OpenStack I see people in here that create their product around OpenStack Ansible So if you're looking for companies that bundle OpenStack Ansible, you can come tell us afterwards after the presentation They most likely have a product for you use case The last category is upstream developers whether you were building a new storage solution working on upstream OpenStack code Well OpenStack Ansible can help we've built environment to help upstream developers like for Keystone for opnv developers and The list can go on. It just depends on you The important thing to keep in mind is That you're not alone in a deployment of OpenStack And there may be more categories that we didn't think of or we didn't think of and You could tell us about it because it's important for us to know It's not the fight club. You can actually tell us about it So don't hesitate to help others by sharing your use case That's how our community grows and to talk about community I would like to show you a few stats of For example the commits per cycle As you can see you've no stats for Juno and Kilo But and Newton was particularly big in terms of comets OMP was a cation pike was all shorter cycle Queens had Refactoring and gating so it had less patches but rocky started again with features and the projected Number of commits for Stein should be around 2000 which is a 2000 by 100 right now. So it means that Anyone counts in in that matter. So you shouldn't hesitate to propose patches In terms of diversity I Would say that for this graph just to it's about Quenching of a thought right I I'm showing well a lot of colors and things like that. It's very hard to read But the thing you have to remember is for this exercise I took the first eight companies in Stakely tics and I put them in the table and I And show these things per cycle in terms of commit and so what you can see is The colors and the size of the ball are changing which means that companies come and go and There are like more than 20 companies that made into the top eight top contributors in OSI in history so it's up to you to Come with your use case for your company and be part of this top eight and and you see that it It was a project that was mainly driven by a rack space in the past But you can see for example vexos that is taking more room You can see independence that are that are very important. You see 99 cloud it is that is ramping up red at with new partnerships and Well Yeah, anyway, um, I'm working for us So For review diversity As you can see that the trend is even more I would say at the last line for for Stein is The current state so it means that it's not a projected state But you can see that when you check this the size of the first three bars It's it's quite big and but we are reaching like a 50 60 percent of Reviews which means that these three big groups are handling 60 percent of the review so we are reaching a diversity there and We are not a single vendor thing anymore and I think it's very important for the future to continue in that direction So sorry for more stats, but I think it's important So Stein is not a projection again the amount of committers For rocky has been Amazing thanks for everyone involved with broken record of having 118 of having previously 115 committers unique committers to move to 118 and I would love to see that trend going forward and And increasing that's that's amazing the difference in terms of cause Is zero which means no change and that's sad for me I consider that we should have always more cause and We should we should continue and and getting people on board it and and And have new contributors So I will skim to that very briefly. It's the amount of contributors on average. So we have about 15 people that are coming regularly every week there are about 25 35 people that are active monthly and The trend is kind of Going a little down I would like to just make sure that we reverse that and being having more people that are Active on the longer run and if they are more active, I would say weekly Well, there is high chance that it will become core and then just continues, right? It's a it's a virtuous circle and Again more stats So the question for bugs is are we pining them under our bed and Sadly, I would say that Recently we've been doing so because in rocky we've got less bugs reported and and the amount of pending bugs is increasing and I Would like to revert that trend by Reinstating what what I used to do in our cat I'm pike which was running and Newton which was running bug smashes so during these bug smashes we will request help for the community it will be public on the mailing list and It will be an opportunity for people to step in Know how these things can get this problem can be solved learn more about the project, but also Have a way to know what would be the easiest way to fix something so will mark bugs like low-hanging fruits and This will help you Help yourself basically So for Rocky We've been Adding a new feature a new series of features you want to yeah, sure so I can talk a bit about what has happened in rocky I'll try to get even though I'm already really loud But so over rocky we got a lot of help from some of the newer contributors such as BBC Which has been very very helpful. So the team with Jonathan has helped add open to bionic support So with their help you actually can deploy open-stack rocky on myonic You know if we have anyone from canonical here, we'd love to talk to you about how you're doing releases You would make our lives a bit easier if we coordinated a bit more but Unfortunately, we can only start the integration of rocky once the release of rocky is out inside canonical inside from the Ubuntu side of things So but we actually managed to get that in for the initial release And I can vouch for some deployments that I've seen in production that is running it and they're doing great We also had a support for installation from distribution packages So open-stack Ansible's architecture has always been built from source and deploy these virtual environments However, there has been request from several different users that they wanted to deploy distribution packages And so that's like deploying from RDO RPMs or deploying from the Ubuntu devs the Ubuntu cloud archive or in Suze's I guess they use RPMs as well. Yeah, so the idea is now you can actually use the packaging that your distribution provides to deploy open-stack Ansible as well and Another thing that we did was we did a significant reduction of the open stack of the amount of variables So the variables that you can override If there is a lot of them the memory really starts kind of blown up and Ansible really starts slowing down This has helped actually Increase the speed of deployments as well as some very tricky things that we found like cruft from like five years ago That was slowing down our gate by like 25% and making it unstable So that was amazing to find and everything has been very smooth since we've upgraded to Ansible 2.5 So we're trying to keep up with the latest. I think Ansible is a 2.7 now I think so we we'd like to kind of move up and and try to get to it But I don't think that we'll be bumping major Ansible versions and Rocky at least at this point It's probably best to leave it as is And we added system D and spawn which is actually really interesting. So we mode This is a I'm stealing I'm stealing Kevin Carter cloud knows joke here And is like if you're running Linux and you have system D you have a container management software already and you didn't know about it Which is system D and spawn so that's actually installed in all of the existing operating systems that we support So whether you're using Suze's distribution whether you're using Went to or sent to us all of those actually include system D and spawn So our idea was to rather than use LXC as our container or Container platform we wanted to switch to and spawn because moving forwards that means that we don't have to pull in weird Dependencies from other places. We don't have to have issues trying to figure out some Person that is back ported patches to make LXC work on the CentOS kernels and So that will actually help unify and make everything easier But it'll also make it much faster to deploy much easier to manage and it'll really tie in a lot better with the operating system and That that should be that is actually something that we're aiming to do. So we're aiming at working at making a transition In an upgrade path. So I'm moving forwards. You're only going to be deploying on top of N spawn and there's More to that as well. So we actually added a few more roles within that release so Panko, Masakari, Congress and Blazar So these are all roles that have been added so you can actually deploy these projects using OpenStack Ansible And you know, there was a lot of work done by new contributors in order to make this work Especially I know the the folks behind the Blazar role were really working a lot to make it happen So awesome for them to do that Inventory simplification and bring your own inventory So if you've tinkled enough with the super deep things of OpenStack Ansible You've realized that we have a dynamic inventory script that all kind of generates all these things automatically and Some people feel like it's very easy to use and some might feel it's a bit more difficult and complicated And I guess we've we've made some changes to allow you to really maybe not have to use that Inventory file and bring your own one and maybe you want to run a container with three services all running at the same Container or or whatever use case you want to do and now that's kind of possible We did a lot of role cleanup and simplification So we actually had a lot of redundant repeated code across roles like as part of all of our deploys We have to drop in a system D unit file and doing that like 70 times across seven year roles And every time you like need to fix a bug you have to do like 70 commits for all of them So we've actually replaced that with common role and so we added two of these roles So config template is now something that we include as a role and actually he's been consumed by Seth Ansible as well So it's cool to see our roles being consumed in other Projects and the Ansible role system D. We have system D mount system D Service and a bunch of other stuff and those allowed us to just do an include role inside of all of our roles So if we want to fix something in the system D stuff, it's one patch only and it's our life is not hell And then we also did kind of a more distinct separation So when we added distro installations, we wanted to make sure that both paths are very Separate and you kind of can easily toggle between source and distro So we don't support switching from one to another but by setting that one variable you get that and we still install From source by default First time there's a lot going on And so first thing is updating Ansible. So we want to get on the latest release of Ansible So we try to keep up with the latest one So when the time for us to release we're not like releasing with like a three-year-old version of of Ansible and we're actually John Rosser from the BBC has worked with the project maintainer of midogen So that's a project that does a lot of like really fast accelerated Deployments, it's kind of an engine that somehow uses SSH in a different way I'm really paraphrasing this but the results are like we've tested some stuff in our CI and like the cento is deployment That took like an hour and 20 minutes somehow took like 15 minutes instead So it will really speed things up and will also help us remove some of the Technical stuff that we have to maintain such as our own connection plugins and more complex things and kind of maybe work with an external library to do that we added Intern internationalization support thanks to some work by The internationalization internationalization team. I should just say I 18m. That's why it's there for so that we've actually had contributions to add German translation to the upstack as well documentation and so if you want to help translate any of this It's actually we're actually kind of the like the project that people are trying to see how they can translate documentation So, you know, you can go and talk to the team on how you can help translate that if you want to do that Jesse who I can see is like all the way out of the room because it's been so busy He's been working a lot on helping refactor the code that we do to build over virtual environments So he's kind of moving it more towards a just-in-time building of virtual environments rather than like a whole build process That happens way after this is going to speed up deployments make it much more stable and make it much more sane to deal with So it's kind of like really really simplify things a lot more which really ties into the repo build refactor We've increased ability for cento as so this was a long effort But it involved things like working with upstream and adding mirrors And so now cento is probably one of the more stable ways and we we worked with the RDO team to actually use their promoted packages So we are actually testing with the latest packages of open stack like master Rpms from RDO in order to test our distro installs, which is really really exciting And that's cool, and we've worked a lot with other teams So the tempo the triple O team is actually doing their own has their own kind of ansible role or Puppet role that it's kind of helping to do their CI and we've realized that as the Triple O team was trying to get an ansible roll up to do tempest and we were like hey We're doing the same thing already So we've actually started working efforts together and so we've done a tremendous amount of progress There's a red hat. Well a triple O tool called tempest conf which does some automatic detection of things to Get all the settings for tempest in place, and that's all already include like it's merged already So hopefully we're looking for more ways to cooperate as Triple O as far as I know is doing more of a transition to using ansible to lay down configs and things like that So I think there's a lot of potential in Gathering more contributors and working together with some of the triple O teams To work on nuts Duplicating our efforts and we've talked about crossgating. So we had a really good talk at the Summit so what do we do beyond stein? Well, hopefully I'll be around for that. I don't know But we want to improve our role testing so the way that our role testing is done We kind of have a way of doing role tests and another way of testing our integrated repo We're actually trying to fix that during this cycle and make sure that we don't any more break our integrated Deployment by breaking something in one of the roles and so this will increase our integration and make it Far more stable product which means that at any time if you deploy over stack ansible, it's gonna work Which is great. We've talked about doing things like promotions of roles So if we make a change in a role, it has to be tested inside open stack ansible And then if that's valid then we promote that role to kind of be the one that you're being used Which means, you know, your stuff is not gonna break magically all the time better offline installs So that's one of the efforts that have been led a lot by Some kind of the racks team and they've been awesome at doing that. So It's important for some of the environments where you know You don't have access to the internet very easily and you might have to build all this stuff externally And then bring it over an internal server point to that server and do that deployment So we want to do a lot of that and be able to support it and then the community goal of Python 3 everywhere I think that's gonna be a bit of a challenge Given that I do a lot of the maintaining for the CentOS stuff and CentOS like hasn't hinted anything about what we're gonna do for Python 3 So Tough luck. We'll see what happens Thankfully, the other distros already have Python 3 so but we're hoping to be able to solve that and have Python 3 everywhere All across are we gonna get on time? Good how to give feedback I want you to join us So I'm hosting a forum session about deployment tools and feedback for deployment tools So whether you're using open site console triple low using the chef you're using juju Whatever you're using I thought it would be really good to all kind of be there and hear the operators and say What is the common issues that you're seeing with deployment tools? So we want to hear them What is the struggles? This is gonna be really useful for us to take that feedback in so when we take architectural decisions We can say hey, let's not piss off the entire world because we think this is the right thing to do Is everyone doesn't like that decision? We're also all around here in the next few days I mean you see us wearing our badges. So feel free to drag us down anytime. It's we're very all very Happy to talk to you about open stack ansible anyone the core team or just the contributors We are on IRC our channel is open stack dash ansible the great thing about this channel It's not a dev only channel if you have questions If it is 3 a.m. And your open stack ansible deployment has crashed Maybe someone another time zone can help you but it's definitely welcome to any support questions So if you need to make any changes you need to help with some of the variables Or you're not sure how to do something feel free to come on and then on the mailing lists Send whatever you need the tag that we use this open stack dash ansible So just send an email with the subject open stack dash ansible and hopefully one of us will be able to respond to you How to contribute? So hopefully you're all really excited about this and everyone now wants to push a patch So we have a project on boarding that is happening. I believe after lunch So it is at level three room one. That's what it says here. I'm trusting that you put the right place And then that's where we're gonna be there We're gonna be helping everyone kind of get started if you have some questions on how to contribute if you have questions Maybe as even as an operator and like hey, how do I get started with this or maybe you want to help with some of the Documentation whatever you need will most of the core team will be there a lot of our contributors and our operators will be there So it'd be great to kind of share and heads up. That's not gonna be a session where I'm gonna be here talking the whole time It's gonna really focus on all talking together as a Conversation all together and sharing what you need and some of the questions that you have so we'd love to kind of see everyone here there and Contributing doesn't have to be code by the way. Sometimes it's reviews. Sometimes it's buck triage. Sometimes it's helping write documentation So, you know, don't feel like, you know, it's technical thing That's above me operators are actually one of the very most helpful people in in this type of thing Because sometimes we need someone to ask question to and say hey as an operator is this good or is this bad? And that is really really helpful So that's it. I spoke a lot as usual and We wanted to have some questions if anyone had some questions about OPSAC ansible or what's going on in a few cycles or the future project plans Someone has a question We just need one person to start and everyone will start asking questions Yay So actually we have been merging and backporting a lot of the code to make it and like make and spawn working in Rocky I would say that when we add it right now We are more in a stage where we call it experimental as in like we probably don't feel comfortable Stamping the production sign on it, but we also don't think it's like stuff that will break it Actually runs at a fair amount of scale I saw Kevin earlier and I believe he runs like some testing cloud that's running 50 nodes or 100 nodes using and spawn So, you know, it's definitely somewhat battle tested But we'd like to be more comfortable with more see eye on it more eyes on it to tell you okay now to go I think the goal that we discussed in the PTG and please correct me if I remember this But in rocky and spawn is going to be added as experimental In Stein, I think we're gonna be really saying okay. I don't know if we say we're gonna do Yeah, we see I was gonna be there and I believe we said that we're gonna actually have try to aim to make a default one You can still deploy either or but both will work and then for LXC We'll probably keep it around for a few releases just so that you know It gives you time to switch out But part of that is actually working on our upgrade jobs to make sure that the transition is as smooth as possible And then we don't break your stuff as it upgrades, but we're really excited about it About and spawn So so actually as James James Denton is he it he's here Yeah, so we have James Denton who actually wrote a book about open-stack networking and he actually has contributed a lot of drivers for these for these SDNs, so I believe we're actually we have open daylight support we have No, like actually inner inner CI gates like it will test it Yeah, and and a lot of that goes to the work that's done in open NFE So open NFE uses open-stack ansible to deploy its CI test environments So a lot of that is implemented so open contrail, which I think now is tungsten fabric That is also tested in CI and and deployed So and I think there's a few and Calico is also supported and I might be missing one if someone remembers I think there's four. I can't remember the fourth one open flow. I don't think I've seen open flow But but the way that the roles are actually built Open-stack ansible deploys all these for you But if you want to deploy that SDN and all you need to know is override a few variables inside the neutron config to point to The SDN you could do that very easily So you can do whatever tool you need to deploy open flow and then all you need to do is just change the neutron config Overrides to say point towards this SDN and open-stack ansible will do it with the project that we just named directly ones where Open-stack ansible will actually help deploy the control plane for that as well. So that's kind of a different I'm sorry These documentation are very specific so they are in the role itself But we intend to change that to move towards using user stories to explain From end to end. How would you do with running SDN X or Y? I think that there are Some of the those back-end drivers that would say for neutron have specific tasks that needs to run and those are Basically pluggable and you can have You can basically do a patch set if you want to improve and add your own solution That's a really good question that we spent a lot of time on it The of last BTG so what we actually had decided to do is we've actually backported a lot of the patches to Be able to get you onto rocky Inside 1604 now It's not the super cleanest of rocky But it's enough so that when you're on 1604 plus rocky and this is for source deploys right and so you can be running Rocky on 1604 and then at that stage you would just you may be kill off one of your controllers Rerun it using 1804 like redeploy 1804 and then just rerun the playbooks to get that up and running also, I believe that there has been some effort from some of the Rackspace side of things is this is a big problem that they have to solve and they're actually going to be working with upstream in Or to document a lot of these processes in how they're going to go about doing these upgrades and having some sort of Upstream process that people can follow because that is a problem that we recognize and You're not the only one that is that so So our networking doesn't actually touch that part we assume that the networking is already wired up So thankfully we don't have to Not a problem Yeah, well, it's it's a process that we've done in the past So if you were if you were running older than then Queens and for example, you're running Juno Well, yeah, so you can do Juno in 14. So Juno Kilo Liberty Mitaqa Newton Newton you switched to to 16 and then Newton Ocala Pike Queens Rocky then you move to 18 and so all of that is part of the code so you can upgrade from Juno to Rocky if you want Infineo has support contracts with canonical. Please tell them to just like have a release behind that be nice I saw another hand. Yeah. Yes Yeah, so So basically what we what will have in the latest branches is to Have an easy way to use area directly Inside the virtual amp that is the virtual amp that we use for for the open stack Ansible deployment You can see it already working for example in the gates of open stack Ansible you can check Can you see the woman stack infra as already an area run and and We can you can drill down into a different folder and you will see the area run of open stack Ansible So it's already working with different area if you want yeah, you can so just Briefly show everyone because I think I think this is really beneficial to make everyone's life much easier So just to kind of show off what We're hearing from here. So this is a rule. Oh look look at that a patch to move area at the end of bootstrap See where we're someone already heard you and pushed up the patch. So, you know, we can we can look at Like this deployment here. That's the back port and so you can see like here We have an AR report and that's the whole Deployment that process that goes through and all the plays and tests are there So if you're stuck with anything like that, please feel free to snow the author of our David Lives in the same city that I live in so we keep in touch often and he'd be more than happy to also help some of the users and and making this Fit nicely, but this is one of our very useful debugging tools that we used to debug the gate Any other questions? What until when does our talk finish? So what time does it finish I Mean we have eight minutes. We have eight more minutes. Awesome. We have eight minutes of questions I know everyone's hungry and wants to get lunch, but I don't think they're serving lunch yet Yeah Just he was around the perfect answer So the DID right now is you could have the repo server Which has where you basically pip install things or it will bill and build the things Currently it requires internet, but we intend to have this part to be movable around So it could definitely be something under deployment node or So we just threw you under the bus we had a question about offline installs and artifacts basically Hi Offline installs it's been a journey It's still on the way Open-stack Ansible from its very inception was never built to be an offline installable thing It does a lot of online fetching of keys packages get repos all that stuff and What I've been doing of the last few cycles is trying to just slim that down to something where you have a much smaller set of Things you can easily just replicate So you can install build from source, but you would be able to do it in an offline manner It is possible to do it, but it's a lot of work To get all your mirrors in place Yeah, in fact Do one of our core contributors John Rasa from the BBC labs He does everything in a mostly offline manner. He's got a very strange security setup So he needs to do it that way But it's it's complicated and I've been doing work of the last few cycles to try and reduce that complexity to make it simpler, of course another option is using the distribution installs those are That distribution base installs those are kind of mostly offline as well But it'll be probably at the moment. I'm working on just simplifying the way the pattern builds work That's my main target for the cycle Once that's done, we'll be able to follow that on on actually targeting offline installs as being a thing That we can properly support So for the moment, I'm not focusing on offline installs, but it's an on the horizon and it's something that's being Thought about as part of the set of requirements So hopefully soon we'll have CDs like the AOL CDs for free and like install install opens that cancels on a CD If that's a particular interest to you you can ping me online Or you can catch me afterwards I'd love more hands. So and if you want to focus on that particular era I'm happy to work together with someone to make that happen At the moment, I'm a little short-handed Instead of the stable branch So the idea is that Well, it was for operational purposes before that we were used to use Shars in the documentation and we got these questions about people doing things and we're forgetting to do to get Weird char thing. So we we got bug reports on on on that and it was not really conveniently readable to say well Get check out blah, which is Compared to get check out with a with a typed version and so we regularly tag open stack Ansible Main repositories for all the branches So we do it around every two weeks Those those stable branches Have rolls that are not tagged anymore, but we are using roll shots So basically everything is set in stone so that when you are running With a certain tag you are expected to have a series of things that is well tested Get tested periodically tested upgrade tested and that is that is well known for everyone And so if you're starting to if you want to redeploy that one year from now, it's going to be exactly the same code It's not going to be a moving target So if you want to use a stable Yeah, it's it's so we recommend always to use the tag for a stable branch because we back Even if we backport things a backport could at some point be wrong between two tags, right? It could it could happen even if it went through all of those extensive testing Yeah, so So that's actually one of the things that we're kind of trying to improve So in the stable branches we fetch all of the stable other roles and testing the next Yeah Proposed release and so one of the things that kind of also happens is you know We don't have a deliverable that is just our service only we have a lot of external dependencies We have to deploy rabid MQ after deployment Galera and sometimes we've seen things like you know Just a few weeks ago rabid MQ decided to change the URL where they store all their Signing keys and that broke everything So the idea is by having these tags is like we can kind of say okay You might have checked out stable rocky at a certain point in time And then that's stable right at a certain point time Rabid MQ broke something and we made a new release and it makes it really hard to identify like okay Are you broken? Are you not broken if you give us a shot? We're like okay is this include the fix or not include the fix did you actually pull in all the roles? So it helps having like one kind of a unit of a release But we're actually trying to do part of the stuff that we talked about promoting shaws Which could help us actually make it so that the stable branch is always Valid no matter what you're never gonna get pull a broken one But you know there's work doing there's work needed to do that and and we're trying our best to To help improve that but for now tags are the way to go but for the long term. We're looking to make I would always recommend use a tag first And the tag may not work if there's a known issue Which happens to be fixed and only if the tag doesn't work without the insert use the idea the idea of Of the promoting is instead of waiting for two weeks to get something Latest I would say freeze version of everything would do that on more regular basis even a daily basis Because well, we've got these kind of CI tested anyway But it's work in progress So we're really trying to make up a second syllable like continuously deliverable And right now there's these kind of few things that we need to iron out And yeah from roles to even the actual project themselves So like our roles will pull in the latest stable rocky or whatever So if Nova broke something then you're gonna you're gonna be like oh, it's not working But the reality Nova merged something that was broken. So by doing that we're really it's gonna be continuously deliverable and Maybe someone will actually do continuously deliverable open stack and syllable deploys. I know some people someone might be working on it. I don't know In fact, we do have someone doing that. Yeah, I know I'm looking in the back. I think they're So I think we're probably running out of time we can probably take one very quick question unless everyone's very hungry I Think yesterday they ran out of food bit early. So if you want to get some like hot lunches And find us at the project at the project on boarding if you want to come for more questions