 Without further ado wanted to introduce our first speakers. We have Jasmine James and Chris Bolton from Delta You probably heard Jasmine give a keynote and she teased this talk and I think everyone's really excited for it So let's have a round of applause for Jasmine and Chris. Thank you so very much John. Well, welcome It's after lunch. I'm hoping we can get some energy going here. I know everyone's nice and full So the title of this talk today is how we're leveraging cloud native to be to make Information technology a differentiator at Delta So I think we're gonna start it off with some introductions talk about our team a little bit And then we'll dive deep into our journey with Paz at Delta. So I'll start with the intros So my name is Jasmine James. I'm a manager of the DevOps Center of excellence at Delta So what is the DevOps Center of excellence? It we essentially provide guidance and best practices as well as tooling to enable DevOps at Delta And Chris go ahead. I'm Chris Bolton I am the lead engineer for the app dev portion of the Paz team I focus primarily on the onboarding process the developer experience and the supported technologies for the platform Yeah, so I think we need to talk a little bit about what the Paz team is and really how our team has evolved over the year So initially in 2017 Chris and I started about two months apart at Delta And we solely focused on creating a tool chain I talked a little bit earlier this morning about our legacy tools that were not cloud native friendly at all So we were tasked with bringing in all of the new tools making sure they were ready for developers to start adopting those So since then we've kind of like evolved into like three different arms I think now so we have our core dev tools team which managing manages and administers the tools We also have a speed-to-market dojo that enables Development community through education and immersive learning and we also have the Paz team which I'll let Chris talk a little bit about And this is also our first slide. Oh, there's that okay So yeah, the Delta Paz team is an agile or cross-functionals team that acts as a lean Startup so if you notice the cross-functional aspect to the team That really means if you work in a large organization, we're broken down into different orgs So we have a security org we have an infrastructure org and we have an application services org So really we came together to form one team and we call ourselves the Delta Paz team And so the Delta Paz team really focuses on four different sub-products So developer experience onboarding process supported technologies and platform lifecycle So the developer experience is really around the automation of what makes the developers life easier So we listen to our customers developers at Delta and just try to automate processes for them to make their lives easier The onboarding process is basically day one to production So I just look at okay if a developer joins Delta today What do they need day one and then what do they need to get to production? So then the supported technologies is the middleware basically so It's something we've kind of term supported technologies at Delta It's really what containers you can run in production on OpenShift So we even have a way that if you want to bring your own container if you're an app team And you want to use something that's not a part of our supported technology I don't know kits or whatever You can bring your own container as well and it has to go through something called our architecture review board But then and then finally the platform lifecycle that is basically our cluster lifecycle so lab all the way up to our production clusters and I primarily work with a guy named Aaron Morris and he's the leader over the infrastructure side And we work together to really make one holistic successful platform as a whole to like the app teams at Delta Yeah, so one thing to add about the past team So we talked a lot about cloud native and being multi cloud This was our first stop on our cloud native journey So we implemented OpenShift within our on-prem data center and we've since enabled a lot of development teams to deploy there So the next step in our of our journey is to enable teams to kind of stretch into the different cloud providers that we're going to work with In the future so all the products we plan on just carrying them forward within each individual multi public cloud Alright, so moving on to our tooling. So we talked about the past I talked about tools this morning kind of but I'm just going to kind of talk through exactly what our Implementation was maybe some of the challenges we had in implementing some of the tools I'll cover the CI and Chris will cover the CD. So kicking off this kind of Picture here. We start with the plan phase. So we use something called version one anybody familiar with version one. Oh I someone looks scared. She's like, oh, man horrible. Oh, man No, okay, so a lot of folks use JIRA for this But because we're working on kind of scaling agile at Delta it it more readily supports our Team of team. So we're using version one for the software development life cycle And as you all know get lab as well. So when we first started our journey with get lab It was solely create, right? We were leveraging for for version control distributed version control So upon implementing that we thought okay There's a lot more features here that we weren't really counting on leveraging So we kind of put those other features on the back burner and got with and got what the developers were asking for Which was sona type nexus and Jenkins for CI? So in implementing sona type nexus we you know, of course created our internal repository that Proxied our maven central MPM JS all of the the large Module providers out there and that way our developers could come to one internal place for all the dependencies that they were seeking We were hoping in the future to provide visibility into what open source libraries They were using so that we can eventually hopefully govern that and provide some visibility into any just risk that was out there So after implementing that we got sonar cube, which of course we used to manage code quality and to ensure that our development Community is not building in technical debt to all the new development that is happening at Delta We're also very very very focused on implementing test-driven development As you all probably know that is a huge Mindshift for a developer to make especially if you're not using you used to writing any test at all So writing tests first it seems like whoa, why would I ever want to do that? So we're working very closely our dev practice lead is actually right there on core. You want to wave your hand? He's the driver of that and he has a very hard job. So thank you anchor for all that you do All right, so moving into this CD portion. I'll throw it over Chris Yeah, so I worked with Jasmine a lot on the CI portion in 2017 We really both joined Delta and then worked on bringing in get lab nexus and our cube, etc So some of these other ones We actually didn't help bring into the company other teams did But we use service now for our change management process incidents and things like that We use open shift as our past solution So whether that's on-prem or public cloud and then we use Dynatrace and Sumo logic for logging and monitoring So we didn't really have a big I Guess insight into bringing those into the organization But they do play a poll, you know part of the whole picture. So But one thing I do want to note on this slide I guess is if you're looking at just kind of the whole picture Across the board if you look at this slide it took six teams to onboard all of these tools into Delta And so and then today we have four different teams That represent all of these different tools and there's even more tools out there that are not on this picture But if you are an app team at Delta one thing that we're looking at is okay I'm an app team and I'm starting to become a dev ops team and I'm having to onboard all these different tools So basically now I have to Engage four or six different teams to kind of onboard to their product, right? So I have to go to the Dynatrace team and I have to you know Get that set up in the Sumo team and get that set up and then open shift Etc. Etc. And so what we've found is we've done something called a metrics-based process mapping exercise And if you look at kind of all the tools and everything they have to go through It's quite a lot. So one thing we're starting to look at going forward with some of our tooling is okay What can we look at from a holistic viewpoint of okay? When a developer an app team comes forward and they need to use all these tools How can we make that more seamless to the app teams? How can we get them up and going faster so that it's less taxing to the app teams and just kind of a waste of Their time and a lot of admin work. So that's one thing We're going to be looking at as far as kind of like pipeline goes Yeah I think one thing to add to that is that a lot of the teams that we just talked about the Dynatrace team Sumo logic They're not familiar with the infrastructure as code concepts So we had to have a you know really like an education session with them to get them familiar with these concepts So that they can leverage the automation to enable our development community further So that was another step that we had all right good All right, so this is like the meat of our talk and it's the timeline of our cloud journey at Delta I hope that you all can kind of see that but our first stop was Q1 of 2017 so I start started this morning I talked a little bit about Before a whole joining the company and focusing on availability and reliability So in 2017 we kind of kicked off the process of what type of waste are we introducing at Delta? so we did some research and figuring out what what areas were we wasting the most time and Out of that conversation, you know infrastructure one of the number one on the list infrastructure it took I think four to six weeks to get a server provision and all of that time that developers were waiting was is Was waste really so We started having conversations with past providers to just really enable developers to quickly provision Infrastructure and have it just ready to go when they were ready to deploy to it So that was kind of what kicked off our journey and Chris actually had a hand in selecting our our paths that we went with Which was red hat so I'm gonna throw it over to you to talk through what that selection process was like Yeah, so first on the infrastructure side. I think it was more like four to six months. Okay It was a long time so that's it took a long time for You know developers to get up and going with infrastructure and different things like that So we were definitely looking at different paths providers and one of those was open shift and so we looked at open shift and we did a six week POC with red hat and We basically just worked with an app team and and just looked at kind of the product and worked with them And we just we signed a contract with red hat at the end of 2017 And we made that the the paths for Delta So that kind of kicked off initiative of just the past first mentality that you know We're everything's past first going forward. So, you know developers they think they know we have a past solution They don't really have to worry about the infrastructure as much. They just know, okay We have production clusters out there and they can start deploying to this production clusters So it got them up and going a lot faster. But before we got there we went to we first started with Three pilot teams in Q1 and so we actually started not as the Delta pass team We actually started as the cap team. It's the container adoption program And so this is really just kind of a project spike Where we said, okay, we're gonna do this thing for six months and we're still kind of in that stage We haven't really gone from project to product But cap team was kind of a project spike where we took three applications and we said, okay We're gonna deploy these three applications to OpenShift We're gonna, you know build the infrastructure and the clusters as we go up to production So we actually picked three applications. They were checking API airport inquiry and another one I'm drawing a blank right now. So checking API is actually really cool checking API was It basically is seamless check-in So if you fly on Delta and you fly on another partner airline when you go check in on Delta It checks you went to both flights at the same time So if you've ever flown like say Delta and American Airlines You've had to check in on Delta and then have to go check in on another airline like American or something like that So that one is now in production and running and it's supposed to be a seamless check-in experience for our customers They don't have to do multiple Check-ins. So that was really cool To see that go live and take traffic and then that's when we started to form the the past team So we said, okay We're not, you know, we get the whole project thing. This is gonna another project spike whatever We're gonna try to act as a product team. That's where we want to be we want to form a backlog of items We want to see what our customers are needing and we don't want to just have to go off and try to leave this here and go You know eventually have another project spike come along so we formed the past team and we kind of just stuck together in a room and You know, we're busting out of that room now, but basically, yeah We just said, okay, we're gonna do this and we're gonna have features and a backlog and that kind of thing So and then along came at mod. Yeah, so Out of the past team and the cap team implementing those three pilot apps There were some patterns that were established So we were able to figure out what it took from start to finish to get an application from ideation All the way to your production. So that was a good thing that we established So out of that pattern came application modernization efforts so at Delta we have a ton of applications that can do many many different things so in Q3 we began identifying the applications that should be retained retired or we platformed into more modern infrastructure So 60 applications were targeted for replatforming into more modern infrastructure and that became our application modernization effort So we brought in a large team of individuals that went on to implement these 60 application Use using middleware such as jboss eap open JDK. I think there was some node in there No JS no JS in there as well And this was all new territory for us because we had used standard middleware before but now we were trying to figure out Okay, how do we get 60 applications through? I think it was in like a six-month time frame or a nine-month time frame is about six months six months So it was very very intense But the great thing out of that came out of at mod was an established pattern for now middleware So with middleware we had standard legacy middleware at Delta. We had I think blade logic We had some web sphere in there things like that But when utilizing these new metalware technologies, especially in conjunction with the past We had to not only get the app teams up and running but get the middleware teams up and running So we had a chance to partner with them and bring them into the past team so they can contribute those Images and things like that. So that was really beautiful So after at mod we really had a community going So we had all of the app teams that were now in production from at mod We had the three pilot apps. So we had a little community thing happening So we did what anyone would do we started a slack channel so that we could get the community to support the community that For everyone that was using our openshift platform as a service and another important important piece of this The community aspect was passport. So it's this is a very clever name passport So all of our documentation now lived in one place. So those patterns. I talked about that came out of the at mod effort We had all of the documentation there so that everyone could consume that and your delta organization Fast forward to Q1 of this year We're really focused on developer experience as I talked about so making it easy for developers to get onto the platform and Creating a low barrier of entry. I think one of the key things that we put out this year with certificate management So not having to think about oh, how do I get this certificate applying it to? In open shift and actually creating automation around Annotations within your version control to say I need to start give it to me So I think that was really cool. You had a hand in that you want to talk more about it Yeah, just to add a little bit to the past community as well We have about 400 people in the past channel now They can get a little noisy at times But it's a it's really great because everyone basically at Delta knows about it and they go to it and it's growing and A lot of times, you know, it's just a simple link like oh, here's a document, you know That we've created for this question or whatever so And then we've even had people out in the community You know answer questions and help us out as well as all of our Anything with open shift is all in get lab And so we basically kind of done the inner source thing and said, you know Hey, if you want to add documentation or if you want to add some automation or something like that Hey, you know go here and you can do it You know, I'll put in our backlog and we'll you know get to it and whatever couple sprints or whatever, right? So that's been really really big and really fruitful for our organization So and then on the developer experience side. Yeah, the certificate management. So We started out with a certificate management. I know that Jetstack has created the cert manager and at the time we use Venify as our store for certificates At the time cert manager was a little earlier on and they didn't have Venify as an issuer so we actually Worked with red hat to create a product called cert operator And so we did a lot of work around that to automate the routing layer and open shift so that we could just automate that Certificate level So that was really great It just provided an easy way basically an app team would say I need a cert And then it would just go to Venify create a certificate and apply it to their route It was very easy for them. It's fully automated They did everything through a pipeline, you know everything was in their git lab repo. It was it was really great So then that was awesome. And then after that we basically kind of rolled into q2 of 2019 where all of our infrastructure was on just temporary hardware so then we started moving to the permanent hardware we call it and So basically we we had our clusters and we built clusters on the side and we said, okay You know, these are the permanent hardware for applications now, you know, you have to migrate over here We did side-by-side clusters instead of in place upgrades since anyway, but Basically what that enabled is our mission critical applications That was kind of the whole initiative of this permanent hardware aspect and moving away from some of the old So some of the applications that are onboarding now are something like snap so snap covers You know baggage and things like that. So when you go check your bag in, you know, it's on your phone They even have integrations into clear if you're a clear customer They have a lot of you know, just places throughout the airport you go You're in there. You're talking to snap in the background that enables other other applications to onboard as well So that was kind of the effort of q2 So and then going into q3 is Basically securing images. So we talked about some of our supportive technologies and that kind of thing It was really about, you know, we have one central repository for all the containers Based middleware containers that can run on our platform and we've kind of taken the approach of you know Here's all the containers we can run and if we create a cluster We basically just run our pipeline against that cluster to Make all the images available to the app teams So it was basically about securing those making sure they have like the Delta cert to them and that kind of thing But you know kind of the Delta labels or whatnot on there so that's been you know also Really awesome because we've had app teams contribute like agents and we're looking at like get lab runners and things like that For the community and you know pretty exciting stuff. So do you want to talk about the q4? Yeah, so into encryption and fully automating our process So we talked about onboarding in the beginning and how disjointed that process sort of is right now So our goal for q4 of this year is to automate automate automate How can we make it easier for application teams to onboard and how can we make that process just really? You know predictable So what the first thing that we're going to do is automate the provisioning of sandboxes for us That's the first step for developers to get a sandbox within the open-shift environment Pick and choose which middleware technologies they want to leverage and just get their feet wet essentially because if you think about if we enable Them in that way we can have more people getting educated just really based off of that based off of automated sandbox provisioning and then we're going to take it a step forward further into higher level environments like DVL and and hopefully Eventually production and that that I think will help us kind of onboard a little bit faster in the end going into 2020 So that is q4 Thinking forward to 2020 now that we've signed deals with public cloud providers We're also looking at installing an open-shift within the public Cloud providers that we've selected so that we can kind of prove out What are some of the models that we're going to have when are we going to choose to leverage the public cloud? And how are we going to facilitate that from a development standpoint? So I think that's definitely one thing that we'll have to look at in 2020 any 2020 thoughts for you Well, has anyone worked with operators yet because that's that's what we're basically using to automate all of this You know the automated sandbox provisioning. It's basically just an operator sitting there Someone just goes to the cluster and automatically provisions a namespace for them Some things are get driven So we're looking at a lot of the automation and just really how do we if you go back to the tooling slide How do we on board someone and just a holistic manner so that it's it's more seamless It's less taxing to the app teams and that kind of thing so continuously improve that process and Make it faster and like she said we're we're going to AWS as well So we're going to be running open shift out there and right now We're kind of in the sandbox land for AWS But we do have a few applications that are going to be going there as well So that's super exciting because we'll have some running on-prem and also in the public cloud And so the communication back and forth and those kinds of conversations are happening But so it's pretty exciting stuff. Yeah new challenges very exciting So hopefully this time next year will be back to share more about how we've grown in the public cloud space But I hope that our cloud journey has been meaningful and educational for you Looks like we have about six whole minutes for questions. So does anyone have any questions and Someone's gonna go on the mic One question about data data leaks Do you process any kind of data in the in open shift? Are you connecting? To I don't know. I mean, do you have a spark? How do things like that running open shift is see separated or have you switched your your Work load something something like that Yeah, so right now like the four Things we can run an open shift or open JDK J boss EAP JDG and no JS So there's no persistence on open shift yet right now our backing services cluster FS and so We can't run any databases right now. Basically, that's We've spent a lot of money off platform for database Hadoop etc. And so but eventually yeah, we are looking to potentially run some pieces on there In the near term probably object or object store type stuff And so yeah, we're looking to replace cluster FS with like something like pure storage or something like that. So Any other questions? Yeah, hi guys, thanks for the presentation a question I had you in the beginning of your presentation or whether when I came in you mentioned that you are Or are transitioning or did transition to test-driven development? My question is when did that journey begin and who initiated it in your organization and who supported it I'm trying to look at the culture side of the test-driven development. Yeah, I would say that it started Right around when on core joined our team in early 2018. I would say Little bit before that so on core came from a large channels organization where they were already doing these things But our challenge from a COE standpoint was propagating and disseminating that knowledge across the organization So that we could provide the best practice and practices in direction First and foremost, but then from a dojo standpoint also teach teams how to do this So that was a large piece not only just mandating it but also teaching them from the ground up What is test-driven development? What are the benefits and our CTO? KK he supports supported us in this educational I think journey I would call it Through funding the dojo and allowing us to kind of build out curriculum and provide that to teams And he's fully supported by our CEO CEO Rahul Samad as well So we have lots of support for it and the value is there We just have to get people to just do it I think and a lot of that is the enablement that is happening in our speed to market dojo right now