 So a little bit about myself, I'm a DevOps engineer at GoGo. I've been there for about five years, so I've seen the organization go through a lot of change, a lot of shift, going from a very non-DevOps culture to now a DevOps culture. So this kind of talk is to kind of describe the changes an organization kind of goes through to kind of see that happen. So why am I here? Why am I in front of you all here today, besides being in front of one of the best conferences in the city, DevOps Days? Because I want to talk about the five-viz of DevOps. So when you're making a shift to a DevOps culture and note that I'm saying a DevOps culture, it's daunting, I mean, it's scary, right? I mean, it's, you're changing processes that have probably been in there since day one of the company, right? You're trying to maybe improve some processes, maybe take some changes to them, and you're also trying to, when you're really making these changes, oftentimes it's seen as grief, right? So the titles, the five-viz of DevOps is kind of a play on the five-viz of grief. And it's a little dark, but yeah. Well, the organization, and also it's important though, that when you're doing these changes, you don't try and force the culture onto your company and your organization, right? You got to find something that's natural and just works in a holistic manner for your company and your users. So about that being said, let's dive into phase one, denial. This is fine, you know, there's a fire in the corner, it's okay, it's been there since day one, it's okay, it's okay, let's just leave it. So in this phase, you kind of know if you're there or not, by, you'll hear things like, everything's fine, despite everyone knowing that things are not, right? Or worse, people know there's an issue and they just don't want to change it. There's an issue and it's how it's always been, you know, I'm afraid to change, I don't know what will happen if I don't do that process. You might see an attempt and establish an automation culture, and by an automation culture, I mean, you start writing these scripts around processes instead of asking the fundamental question, is that process really needed? And if there's a problem, hey, just hire 10 more engineers, toss it at them. We're great to go, right? Not necessarily, sometimes this causes more problems than not, right? If you know you're also in the space, if this is how the deployment looks, I know, because this was me a year ago. I had a much bigger desk actually. So, you know, I had to include a mandatory, what does DevOps mean, right? We all have our own definition of DevOps, and I think that's important. I think the diversity and having your own definition of DevOps within your organization and also for your team is very important. You know, every company kind of established, again, establishing a natural way. I mean, your first instinct is that DevOps is automation. Simple, right? You write some scripts, you get to go, you deploy into prod, you're moving fast, right? Not quite, right? DevOps is asking you the hard questions. You know, it's really trying to figure out what processes can be eliminated. Can you empower your developers? And ultimately, are you busting down silos to increase speed and agility? So, DevOps is not just a team, it's a complete culture shift. It's something that your whole entire organization needs to embrace. If just your DevOps team or your automation team or whatever you call it is doing this, you're going to have a really bad time. So again, like I said, you got to bust down those silos. Empower your developers. That's something very important that we at Google really are trying to push. And automated support, others do the same. So what do I mean by busting down silos? Oftentimes in the legacy organization, kind of like in the non DevOps organization, you kind of see this I pass the ball to QA, QA just chucks the ball, and then the ops guy just gets hit in the face because another work. It's just a bad time, right? So really what it should be is go to this. Yes. Yes. Clickable button. And I mean, we laugh about the first picture, but it's kind of true. Because it doesn't, it's not just funny, but it's also causes some hostile behavior within the organization, right? So your developers away coding ways coding a Java, you know, it makes an awesome program shifts it off to QA, pretty much they have moved on to the next project. Then you call your QA is now testing code. They might not have accurate requirements, they might not be able to have a direct channel to the QA organization. And then the QA says, Hey, you know what? It passed my smoke test on to ship it off to the ops team. And the ops team and like one of the morning or whatever I have to deploy, of course it doesn't work. And then everything has to be rolled back nightmare. It's not a good scenario. I mean, in that culture, you also hear things like hostile comments, right? You hear things like QA doesn't know how to test code, or the developers can't code their way out of a paper bag. Or ops team do they even know what Linux is? Do they know how to deploy this code? And my question is why? Why would you kind of draw this hostile environment? Why not make a change to where you empower everyone to be on the same level platform and make deployments a click of a button? Yeah, shaving and eliminating it. Organizers earlier this morning talked about it. But I always love to use Malcolm in the middle gift because great, it's like, to fix the light bulb, we're going to get a screwdriver and get the WD 40 to fix the drawer, then I got to go fix the car. It's about limiting your gas in your organization. And you know what yet having the X in your organization is a normal thing of growth, right? You have a process and you just keep building on it. But it's important that you look back on them and you're saying, do these processes serve the same purpose they were initially implemented for? Have a well defined mission statement and use this statement to drive your choices and draw your tough decisions. So for us, it was easy. It was developers, developers, developers, developers, you know, my friend Steve got it back in 2000. So our, you know, it's everyone has to come up with their own answer. But for us, the great place to start was Empower your developers. They're a great tool to help you drive automation throughout your entire culture. So we always asked ourselves whenever we had a tough question, a tough thing to implement, we're like, you know, will this make the life easier? Will they make them? Will this empower them to drive change? Will this empower them to make their applications better? So I always like to tie it back to, you know, an example of how, you know, a real world example of how we kind of started. So Q3, we started this DevOps initiative. And if you were like, huh, oh my God, like, what DevOps, what do we got to do here? So we started the basics. Get your kid, get your kid instance, get hub, get lab, whatever you want. You know, build on that, have your Jenkins server, you may already have a pieced piece of these infrastructure as well, leverage packer, Ansible, and then start doing infrastructure imaging immutable. And if you can consider an immutable infrastructure, please do so. And the great thing about immutable infrastructure is that you can if you build it once, it's going to work. It's just you don't you don't build every time and you ship the same the same piece of code, same piece of hardware, ship it through. I'm talking about a cloud based microservices infrastructure, but the same can be applied for data center as well. So things like auto scaling, Jenkins slaves, get hooks to check sanity, Jenkins Java DSL, if you don't know what that is, look it up. It's awesome. It's like a lifesaver. So that was phase one. Phase two, fear and doubt. So this is what everyone thinks automation is, you know, that steel beams are gonna come and just knock over all that iron and just fire. And I knew it. It was not it was not good from day one. So some common fears, right? Some common fears. This is what you're kind of here in this phase. You'll hear that it's always been done that way. Why are you being so difficult? Why are you trying to change that? Right? And I think we all heard it before. What if you make things worse? What if you don't succeed? What if you cut over a process and you just completely fail? Right? This is the way it's been done for the past five years. It's important that an organization takes these kind of risks and tries to make their process better. Right? This is the whole point of DevOps. That's the whole point of technology making things better. And really the last one is the fear of the unknown. What does DevOps mean for me? Right? Sometimes it can be seen as a hostile thing in an organization. Like am I gonna get replaced? My take on that is if you're going in for DevOps to eliminate roles, you're doing it wrong. Right? DevOps is about collaborating and driving you to share platforms so you can focus on the more important things. You know, deployments, they're menial. Find some of them maybe more complex than others. But at the end of the day, it's a deployment. That's not the fundamental issue of your business. Your business is driving and improving your services for your customers. So, DevOps. So, within your team, as you're kind of growing and you're building your pipeline, in this phase, you're kind of saying you're making key decisions on tooling. Right? You're setting up your continuous integration. Right? You got your infrastructure, code, all that Amazon, open stack, whatever have you. You're setting it all up. It's great. And during this phase, your DevOps team, you'll probably have questions like, hey, are we doing this right? It's like everything to the T. Is it perfect? I don't want to do this again. Is it going to be perfect? And are you 100% sure it's going to work? And you know what? The short answer? No. It's not perfect. I'm sorry. It never will be. And that's okay. Just make sure that you have the way and ability to improve on it in the future. Start small and set standards. So when you're getting started, you're going to be very tempted to just take on all the projects and possibly your legacy data center or in the cloud and revamp them. Right? Don't set yourself up for failure. Start small, pick up that one project, succeed well on it and start improving on it later on. Right? So this is going to be things like take one project that maybe your core business or not maybe necessarily core business like an edge project, figure out the king's in your pipeline. This is going to be something that you've never had in your organization. It's brand new. Right? You're going to have problems early on, especially. So with that being said, pick a small homogenous base. So start off with your base operating system, you know, get up and get one, just pick one. If you're a window shop, you got windows. If you're a Linux shop, pick, you know, Fedora, Ubuntu, whatever, you know, what's your favorite? Start small. Don't compromise on the standards you set. Early on, take a good, hard look at your business. If 75% of your business is Java, does it really make sense to target 25% of your node developers? Right? Sure, you want to support them. But at the early on, you want to make sure that you have a platform that meets the needs of your core business. And some examples like that we ran into are early on, we wanted to support all the base operating systems. We want to have nightly builds for them. We wanted to have all these great implementations for them. But really, we had to take a step back and, you know, we asked our developers, so what do you want? You know, some of them, yeah, they had the preference because they wanted some benefits of the operating system, but those are really edge cases. Majority of the people said does it have Java? And does it have Tomcat? All right, I'm good. I can deploy my warfile on it. And again, I touched on it earlier, Java versus Java and JavaScript. Succeed well on one, and then go back and add your tooling later. Again, improve on the tooling of the future. Remember, these initial customers are going to be very skeptical of your process. They've never seen DevOps before. This might be their first interaction in DevOps culture, especially for us, five years in green doing a same manual deployment, it's hard to break down those walls. You're also building a trust bond with these developers. It's important that they kind of feel empowered that, hey, things are going to work, things are going to be generally be better and they're not going to suck like before. So the best way to determine issues in your tooling is honestly to deploy fast and deploy often. Making deploying the fastest part of your pipeline. So you can do that by making a click of the button. And this is empowering your developers, your quality engineering organization, your operations, or whatever person in the team to click that button, get reliable deployments and more importantly, get reliable rollbacks. So remember that whatever tooling you do implement needs to be easier, needs to be faster, and needs to be more available and more reliable. And just really again, consider an immutable architecture, infrastructure, deployments or rollbacks. If you make your server no longer the secret piece of infrastructure, you're reliably deploying that from dev to stage to prod. You're moving the same piece of code. So if it works once, it works everywhere. So some tooling that we used to kind of do this was, I think I mentioned earlier, if you haven't heard of it, there's a tool Spinnaker. It's a great open source tool. It's written by, primarily by Netflix early on, and then Google kind of got involved, Microsoft, I believe was Pivotal, and some other companies are getting really ingrained and giving back on it. And the best part is it's open source. So if you find an issue, submit it back. With this, you've got to get a robust EPI to create a pipeline for your infrastructure. You get manual judgment for all the people who need to do SOX auditing and all that stuff, and PCI auditing. You've got an index of your applications. The thing is invaluable. What's actually deployed in my infrastructure? I challenge a lot of people to take a look and say, these apps are actually running, these are in dev, these are in stage. This gives you an accurate view to see where your stuff is at. And, yeah, and custom pipelines for each application. Some of the things to consider is leverage the tools that are great out there. There's a lot of DevOps people. I'm sure a lot of you have heard a lot of these tools, Terraform, Vault, Console, Vagrant. We love them. It's great. It's because they just work. And in the fact that they're open source, it just makes them that much better. So to kind of tie this all back, you see a part here where I say Asgard. Asgard was what came before Spinnaker. Don't use Asgard. Sample application, that's going to be your, you know, you bring it on your application, you're setting up your production, you're migrating your first step over, you're setting up your tooling, all that stuff, you have your, your cloud trail, all that migration goal rush is what we call our mass upcoming of applications. But really, by the end of December for us, we had half of a dozen apps. May not seem like a lot, but when we see the point, we mean DNS records, security groups, I am tagging, slide notifications, bunch of other stuff. Face three, compromise. Shut up and brace DevOps. So in this phase, you're kind of asking, why does this specific process exists? Right? You're asking the question like, does the process still serve the same purpose? Maybe the reason it was implemented the app is no longer in your infrastructure. Do you really need to keep doing that process? Is the process really required? Or is there something better? Like is there a better tool? Do you really need it? For us was, does this process empower or impair my developers? If it impairs my developer, honestly, right now, it was for us, it was alright, this is already negative on our part. But then also considering that does this process also speed, increase my speed or delay my speed to time to market? And does this help break down the silos that I talked about earlier? Does or does this just keep perpetuating them? In this phase, you're also going to probably be eventually in the concept of DevOps. So by evangelizing, I mean, you're going to be explaining DevOps a lot. I mean, in DevOps conferences, we explain DevOps a lot. So you can only imagine to explain DevOps to non technical people. What does DevOps mean? You know, a lot of people say buzzword, but not necessarily. It's really about explaining the key for success and things that like you used to, you know, you're early on drivers for success to use metrics from your early applications, you onboarded. So say like, hey, this app used to take two weeks to deploy. Now the pipeline we have built, it takes two hours, right? So again, use those metrics in your in your favor, explain the benefits of change and improve support, right? DevOps doesn't necessarily mean ripping out process. It means improving the support for them. Understanding technical debt. So in this phase, you probably already counter technical debt as a team. You know, but what I'll say to kind of start off that phase is, you're not special. You're not a beautiful unique snowflake. And that there's never tomorrow, but there's always a today. And by that, I mean that, you know, in this phase, you have customers, what be it internal customers that are relying on tooling to do their day job, right? Early on, you probably could take the time to bring your Jenkins server down. Early on, you could shut down your GitLab server. No one made a cry. You take down your GitLab server, you just fundamentally stopped all your developers from deploying or from committing their code and then ultimately deploying their code, right? So you got to make sure that you do this responsibly and know when you need to write tooling, when you need to implement tooling, when you need to do changes and have a kind of shared agreement with everyone, right? Don't try and make everything perfect. Don't let perfect be the enemy of good, right? You see this circle here, we're trying to like align this puzzle to be this circle and perfect circle, but it's never going to just click. If there's already a tool on the market that does 80% of what you need, is it really worthwhile to write your own? You know, because if that tool breaks, who's going to fix it? It ain't going to be me because I didn't write it. It's going to be you or someone else on your team supporting that. It's going to be a technical debt that's going to keep eating away at you. And if you're having this many pounds early on, it's only going to get worse than your scale. And then ultimately, the last thing is don't try and force a tool to do something it wasn't built to do. Just don't do it. If it wasn't built to do it, just don't, please don't do it. So we took an initiative to say goodbye QA and hello quality engineering. So this means that QA is still right through tests to do all of great, but QE is now empowered to write code and test as close as possible. So I'm going to stop here and say developers in the crowd, this does not mean they're going to write your unit tests. You're still on the hook for that. And you know, so it's important to have testing integrated in your pipeline and have faster reliable feedback. So when you deploy it to dev, if an issue occurs in their code, they should be able to know line 76. I see the issue patch it up, move on, deploy to dev, looks great, deploy to stage as to be fast, has to be reliable and give that feedback to developers, again, empowered developers. Metrics and monitoring are now our first class citizen. So this means that your metrics and monitoring team are now considered DevOps. What does that mean? So consider the case that you're now using your logs in the sense of to kind of look at for errors, you're using actual application level metrics. So using things like graphite to send out application level alerts, like services sending higher or lower latency, using things like history circuit breakers, if you're not concept if another concept of circuit breakers, it's great, awesome, has some default, default way to respond if a downstream service is having an issue. Considering these things early on. And remember that with the shift to this developers are now the ones on call. So if there is an issue, you're the one getting called. But for that to happen, they need fast and reliable metrics. It's a shared responsibility. The OS needs to provide accurate metrics. What's the JVM? What's the heap size? You know, what's, what's how, how much memory is it using? How much CPU is using? On the flip side, developers also responsible for application level metrics. Is this function working as expected? Is it too slow? Is it faster or slower than the last, the last deployment? And then service level metrics, right? Can app be talk to A to C? So you overheard me say the name hysterics, but this is kind of how the tool looks. It's a cool little tool. So the little circle, the circles you see there, they're actually green, the slide's kind of a little funky. The green, good, red, bad. This is a couple of things. The bigger circle, the more traffic. You see arrow rates there. You see RPS, you see your circuit breaker status. It's a great way just to look and say, yep, my application is looking great. And strive for developers to be proud of these metrics, right? Have them readily available so they can see them real time when they do a deployment. Is it working better? Is it working worse? Graphite and Grafana. Great stuff. I mean, just this is a tool we used to hold a company called hosted graphite, just kind of put our metrics in there, getting things like latency, saying, hey, that deployment after we did it started getting a little slow. All right, maybe let's roll it back. Let's take a peek. Let's do some more stress testing back in our local stage environments. So how does this kind of look? Again, we did a lot of QE. We implemented Spinnaker. So we migrated from Asgard to Spinnaker. And then we had our security, developer metrics and all this stuff. So phase four, acceptance. Yeah, I have to include a cat. I'm not a devos person if I don't have one. So people like us. They really like us. So at this phase, you're going to start hearing like educated conversations. If I ever take an educated conversations, I mean like, how do I make my code better? How do I make this implementation work for just more people than myself? You know, do I have circuit breakers when things go wrong? And do I have enough metrics? You know, can the organization see the status of the application and get fast, reliable information as to how the application is performing? At this phase, your early adopters are going to start seeing a net positive. They're going to start seeing that, hey, things are working, things are better. And keep in mind that these are going to be your champions moving forward, right? You can go to these guys and say, hey, how's the tooling? They're going to be the ones to provide you the accurate information. And you know, I should listen to them when you're onboarding new applications. And at this phase, you're probably going to have a lot of customers, a lot of internal customers, why do I say customers? I mean developers wanting to get on your tooling. But you know what, if you did it right, a lot of it's going to be cookie cutter. Java, Tomcat, move on. Java, Tomcat, move on. Now that being said, there are going to be edge cases and for that, it usually comes with the fun acronyms of security, PCI, socks. A lot of this fun stuff that we spend our day jobs kind of trying to meet compliance for. So how does this kind of look in practice? Again, to tie it back, I got a metric that we did 395 manual deployments. This is like some login server, a CP of the war file, restarting Jboss at the time while flying out. So when we started to look at it within a single, almost not a quarter, but quarters worth of time, we saw that we had 18 apps deployed to prod in the new pipeline. Right? Okay. But the most impressive stat was that in this almost a single quarter, we almost did as many deployments as we did to prod in a whole year. And then keep in mind, this is our first iteration of it. So the hype is real. I guess that's what you can kind of say, take away from that. So for us, it was within a quarter, we're already seeing benefits of this stuff. Some statistics artifacts, a lot of it, a lot of logs, a lot of GitLab projects. Want to bore you guys too much with details. So again, we focused on stocks, cost management, multi region, all that stuff. Asgard deprecated, we want to get out of our infrastructure. So phase five, wait for repeat. Right? Good stuff. We're here. Things are stable. That app engineered along these use iterm 2 of 12 concurrent terminal sessions to restart all nodes at once. And that app engineers on the one up here talking to you guys today. Yeah, I had to kind of include that because my SVP said you better own it. So toss it out there. Deployments are faster than ever. Things are working great. Developers are empowered to make their own changes and they have fast reliable feedback. Keep in mind at this phase is when developers say it's taking too long. It's taking 10 minutes. Hey, I saw you last year. It took two weeks to get your app to prod. All right. Take a chill pill. All right. So that's it, right? I met my desk right now. This is probably me wearing my shades, feet up, drinking them, drinking my beverage of choice. Then your SVP, I don't think he's in the crowd. Okay. So my SVP kind of comes around and says, Hey, do you want Docker? Do we have Docker support? Cost tooling? Do we have GCP? Do we have Google Cloud? Do we have Azure? Do we have open source? How about a hardware, hardware malicious software to find networks? Hey, what are you going to do about that data center that we have? And to that I say, this is fine. It's always a repeating cycle, right? And that's the best part of DevOps is that you always improve and get better. So you heard me mention, I promised some of the guys in my team that I pitched this. If you're already using Spinnaker, we're working on an open sourcing or tooling. It's essentially a pipeline DSL language. So if you Spinnaker, you know, to create your pipelines, you got to kind of go in the UI. We're essentially trying to create a language that you can kind of define your infrastructure in a file, toss it in there, into your, into you within your Git repo and have it in there and it'll by the fall, Git will scan it and then it'll create your pipeline for you. It's a pretty cool tool. It's called Formas. We're working to open sourcing now. It's still not there. It's just finishing the final touches. And you're probably asking why in the Git repo? We want everything to be one source of truth, right? We want someone to say, this app in Dev needs Amazon term T2 medium. It's four gigs of RAM. Let's say it over run. And in prod it needs 16 gigs of RAM. But you know, finding requirements. But you should be able to know those requirements early on and know what it takes to build that code, know what it takes to run that code, and more probably know what it takes to deploy that code throughout your infrastructure. So what I'll say is if it sounds interesting, come talk to us. I think some of us are in the crowd and we'd love to talk with you guys about it. Even if you just use Spinnaker, I think it's always cool when you meet someone else that uses Spinnaker. It's kind of cool. So yeah, with that being said, thank you for coming to my talk. This was my first talk. So hopefully I didn't, I haven't passed out yet. So