 Hello everyone, my name is Andrew Lisik. This is a continuation of kind of something we started in Austin and Barcelona, just kind of our continued evolution of what AT&T challenges of dealing with CICD for our scale deployments and I'm Andrew. This is Larry Rinsing and this is Malin that has joined me. So next slide. Biggest challenge since post-Barcelona is what we've been banging into people's heads is really a cultural transformation. That's the people problem has been one of our biggest challenges of taking a credibly skilled DevOps engineers and Taking them back to thinking of a zero-touch cloud provisioning mindset through the labs and through deployments production deployments maintaining consistency back up I'll bring the mic up. Okay LCM fuel life cycle management. These are some of the things we've introduced For people that have been kind of following us. We are a fuel nine dot x shop With the Moss distribution for the AIC 2.x reference architecture, which is a virtualized reference architecture We host two different kinds or really three It's kind of two different larges for specific VNF workloads and then a medium type of site deployment for a workload Which face it which introduces a quite a few challenges from a delivery standpoint But what we've done is we've tried to drive consistency through our pipelines and automate everything Our big challenges are greenfield brownfield upgrades. That's a deployed site and upgrade it with things like kernel upgrades or Open stack component upgrades version upgrades things like that. They're the heavier weight things contrail upgrades were contrail 3.x shop, too And then we've done moved into a release candidate model to allow us to do sort of rapid patching within those sites using puppet and fuel nine dot x and Then our rapid patching which is you know, even a little bit lighter weight We can turn those things around in like an hour Out into production and control that through a canary which we call sills our user acceptance sites before our production application To get assurance out there then rack expanses or something else We needed to mechanize because we need to grow compute nodes quite a bit Some of the other things we've touched on that have been really valuable to us have been able to certain components in our infrastructure that we deploy We automate the upstream sinking So it's just mechanized through Jenkins that we pull down as kind of a few think of a maintenance release So we can kind of consistently get in and choose what things we want to cherry pick in our deployments more rapidly than waiting for the Entire product to go say GA or something if we need a patch Then the AVT emulated scale lab one of the challenges we faced In our deployments when we were using the various products is once we got to three to five hundred nodes We would see some of the some of the infrastructure show stress and would flex and so We just to be able to do something consistently and rapidly we introduced in this emulated Scale lab we can talk more about if some of us have dropped by afterwards. It's literally just made of three Dell 730s and With two of them having 150 emulated VMs, and then one acting as a control node It allows us to do some of those bulleted items there to kind of do some sanity Consistently kind of over our deployments and we can kind of observe times and you know alert when we start seeing things get out of whack if somebody's introduced some code that Introduces a scale problem. We'll have some historical data to flag. So jump to the next slide Some of the stuff we were talking about was we look for consistency. It's really really important to us So YAML describes our deployments We use a Jenkins mechanized node that we Actually, it's just a slave node and we've containerized it and we use that exact same node From our avi from our basic the farthest left CICD labs, which we have dev labs We have RC labs system test labs and up to SILs. We use the exact same tools exact same processes To make sure that we can sort of what we do in these same things is that we have a repeatable consistent site We tried in 2015 to deploy many many sites Where we had a lot of dev ops ninjas having to account for things and we had to eliminate that as much as we could because we deployed it, you know 80 sites and Across the world and you know, we're consistently deploying more of those this year. I think we have another 60 on the books plus 60 total more this year, so Maintaining that as a challenge and consistency and automation is the only way to achieve that. So That's what we've introduced this concept of full CD And full CD is literally like I said just using Jenkins to Mechanize the entire deployment procedure and all four of those items on before that were listed before those different levels of Jenkins automation and the pipelines and Introducing our quality gates within those pipelines. They're almost all voting gates for us So we do regular coding gates as well as deployment gates. So when we're actually deploying We can stop a deployment early in our labs before it goes anywhere if it fails And we have multiple dimensions of those tests that we're kind of always evolving and I think that's it for what we've achieved since then Thanks, Andrew And you talked about gates and this side will this slide will give you an idea of what gates we have in our CSED and deployment workflow Right from the point when someone merges a piece of code to the point where it hits That's at step one and to the point where it hits production. That's in step eight. So Two to seven are all the gates that piece of code has to go through before even it gets deployed in production Right. So we have some of the basic ones like in two you can see the usual syntax checks unit tests Packaging we don't also run some package checks to just to be proactive to make sure it will be the piece of code is going to be deployed successfully we have our nightly sonar cube Scans which are the test coverage scans. We have 45 scans that we have fully automated Which which which are basically security scanning to find vulnerabilities in the code Step four we have the nightly avd avd is a we call it a IC verification test. It's It's integration test framework based on fuel DevOps and fuel QA Once it's passed and yes, we use we use OSTF which is open stack test framework and of course Tempest for our automated verification Step five full CD Andrew mentioned full CD which is Which is also one of the gates Beyond that these are there is a there is something called at least candidate integration lab where developers can Test their piece of code in as it would be deployed in production System test in step seven does their own automated testing and That's I mean that's that's all it takes to hit production So it's it's it's a bunch of stages that you have to pass until until you can get there Here are some of the matrix that will give you an idea. I mean I do mention we are in more than 80 plus production sites right all over the world Let's give you an idea of the scale that we are operating at right and these are again. These are old numbers highlighted some of one sums with some important ones in the red Like we with full CD. We are able to deploy in eight hours, I Think I'll let you see if I think we're running out of time. So I'm gonna pass it on to Larry Yeah, so it's it's no secret at this point the benefits to containers and things like Kubernetes provide Internally at AIC we have an effort to containerize our local control plane leveraging the open stack helm project and Using helm to deploy our open stack control plane The benefits this is pretty much common knowledge Kubernetes provides that abstraction layer that sort of separates the hardware from the actual software running if you don't really have this Snowflake syndrome going on So this is an example of our workflow the top one is if a developer was going to make a modification to a project like Keystone if we push a bug fix Assuming it passes like the pep 8 unit tests. We're gonna perform a docker build Stand that container up do some basic health checks to make sure it doesn't fall over on top of itself when you bring it up and then tear that down Pass that docker image ID to another project called armada, which is going to deploy that in a helm chart Once we have our full open stack cloud up we're gonna run tempest and Rally against that to get our integration test and benchmarking results If that all goes well, we're gonna tag that as a release candidate and then push that out and then the second pipeline Beneath is if we were to make a modification to a helm chart, so we would Push that up bring up a single node Kubernetes instance using something like kubadmin package up our helm chart do a dry run to make sure there are no syntax errors and Pretty much the same from there deploy it again If you want to contribute to open stack helm or armada of the tools we talked about The github links are there if you want to hear more about open stack helm We have another session on Thursday that will be presented by Alan Meadows and Brandon Joseph the information is there So we'll be outside if you have any Q&A