 Hello everyone, welcome to you have hundreds of clouds deployed now what genius To start us off introduce you to the lovely talented and fierce Kayla from me And this is Lee revere so AIC is a massively distributed large open-stack cloud It is a platform that hosts or has many tools to have automation from end to end There's a lot of effort that goes into deploying one cloud site times that by 80 or a hundred or hundreds And you have a lot of automation that needs to be put in place And it's not about automating each individual piece, but from end to end When you've got 80 plus sites With two regions per site, you know 160 plus is owned so 160 individual clouds more or less and Managing that when you have tenants who aren't in but to Possibly all of them. How do you keep track of where they are? How do you put their loads in the right place make sure their tenancy set up their images are there? All of that becomes a bigger challenge So we had to develop a number of tools and our whole goal for everything obviously is to automate all of it There are a lot of manual processes that we're taking days hands off with different teams And we wanted to make that streamlined so that one person could do what several groups were doing and the only way We could do that obviously was That was how All about automation So we created a number of tools to meet our specific needs the first one's actually under development this is how capacity planners take their plan of execution plan of demands and Put them into a tool that they can model things and and change on a regular basis I mean these things change weekly So we need to be able to reserve things like You have the tenant that's coming in and you need to reserve their capacity But is that capacity that they're bringing on new hardware to support? So you don't want to reserve the existing infrastructure and take up for other tenants capacity for expansion So you need to be able to reserve things that aren't even deployed yet as well as things that you needed more immediately After that we have a IC formation This one's had a significant impact we can reduce the workload that These are things we've been doing for 20 plus years spreadsheets, Vizio diagrams all of that It would take two and a half or more weeks for the groups that were involved in putting those together We can do it in 10 minutes or less Up simple is actually our automation for all of the deployment So these tools all talk to each other So there's an API available in formation that when all of the design work is done There's automation to other systems for IP allocation for DNS It makes all of that information available in an API that our deployment software reads to pull all the information It needs to do the actual deployment From up simple you have the self-service portal now that up simple deployed the site and has the cloud running You're going to start provisioning tenants on it. So this is just a Nice web interface that allows tenants to come in and create their you know Their tendency in this number of sites that they're supposed to be in and it gives it that front-end for ease of use That then hands off to like up open-stack resource man resource manager and ballet to go actually do the deployments to put The images in to create the tendencies to put the flavors everything that those tenants need in the places that they exist It's fancy. I just like watching it Formations the one I'm the most proud of This is one that I've been a network engineer for more years and I want to admit and We used to have to do these manually. So we'd create a rack elevation and Vizio or a spreadsheet Populate a bunch of information about all the different devices Then you do a cable map spreadsheet, you know, you got thousands of cables Great opportunity for human error E-mailing them back and forth from stage to stage where it opens up for mistakes and Things getting out of sync which then turns into a deployment error because you have formation feeding your deployment automation We'd lose days going back and forth on spreadsheets So then ops simple will take the information from formation Create the necessary YAML files used for the deployment and go through and do the deployment start out with the seed node You know metal as a service type stuff And then feeds into fuel and deploys the entire open stack environment as well as our VNF's that are our non open stack infrastructure components So after your site deployed and we have the cell service portal that Lee mentioned and it serves as a centralized portal for anyone to go in and order their Resources and for a tenant to be created After your tenants created or while your tenants being created SSP feeds information to ORM open stack a resource manager and ORM creates the flavors Keeps the images consistent so that we don't have different types of images across our massive platform It also allows flavors to be created in the similar fashion And again just enjoy the animation or the While you're creating a tenant And we have clouds around the world specific VNF's and tenants need to be placed in certain locations and also have other things to be included while to be considered while that VM is being spun up so like affinity groups and maybe you want to be two VMs to be close to each other. Maybe you don't know valet acts as a Plug-in to the Nova scheduler to do those things It has a little more enhanced features to it so that you know you don't want to put two high capacity workloads You know on the same node So it takes into consideration what those loads are and separates them out further than just saying I want these nodes My two redundant nodes in different racks It looks at how much you're putting in this one node and make sure you don't overload it with the similar workloads So after you have your clouds deployed your tenants created Now what? Continue the automation I thought you were gonna start talking about automation 2020 So like we talked earlier Automation of a platform is not just about automating the individual pieces But bringing those things together using APIs and microservices and building an ecosystem of applications for Platform with many different cloud with many clouds So if you think about all the things that we just discussed You're introducing this new hardware that you had a tool designed for you It's created the cable rack spreadsheet. It's created the elevations and all but somebody has to go back in and put all then into inventory So why just why do that more than once if you've got a tool with the API as your inventory systems will pull it You give a notification that you have an update or a new site and they can pull just the pieces or the entire site As well as feeding all the additional tools passing that same information down instead of just Telling somebody there's another another site giving them a spreadsheet. They take it modify it do whatever they do with it Just keep it all in the same system and and passing it down the chain before we go into any questions the LQ is a working group that AT&T participates in and leads to work with other large telecoms and other large companies for contributing operator needs to the open stack so Any questions? We have a lot of customized code For our requirements. Yes, but it is still coming from trunk We have a few different versions running right now. Yeah, we have Juno and Kilo running and Next year we've started the open stack home project to do Kubernetes based deployments with open stack And so we're looking to jump to as close as possible to the open stack current open stack release next year of open stack Well, some of the tools we just mentioned and like ballet that's being pushed back to the the community That's already been posted. I think ORM is we're in the process of doing so we're contributing back these automation tools but to your point We do do use open-con trail and even the things that we've done in fuel all of the work that we've needed that has been customized Morantis has pushed that back into the community as well So for the most part all of the things that we're having people customize that we need is getting pushed back up into the The core I was supposed to repeat your question. Thank you So there are like 100 data centers. I guess are different the open stack so as those different open stacks they have they They have some kind of the upper verse one more layer above that is doing orchestration doing some kind of federation or Some kind of unique unified management for all those different Overstack So you might be talking about own app as far as stitching together the WAN or doing some of the resource orchestration for tenant space Or you're talking about ORM That goes out and does all of the placement of the tenants and the various locations that they're supposed to be as long as their Images and and the other unique things for that tendency Yep, so that was the answer ORM does a lot of it for pushing the tenant stuff around so that we're sewing all these clothes together You know so you don't have to go and log into each horizon dashboard and do these things You know push it once through ORM and it starts with a self-service portal So that you know the users have a web interface to do it and then ORM knows to go and create these things in all these Different locations, but then own app to Paul's point would be the network portion to stitch it all together and Even substantiate or instantiate the tenancy VMs It's all been given to the community Does that answer your question If you want to have some follow-ups on it, we'll be happy to talk to you about perfect. I think we're out of time. Thanks guys. Thank you