 Welcome to a DevOps discussion with Gary Grover. I'm Suri, a member of GitLab's content team, and I'm thrilled you're joining us today. If you have any questions for Gary, please use the Q&A function at the bottom of your screen. If you have any technical difficulties, please use the chat function. Today, Gary Grover, a DevOps consultant and author of Starting and Scaling DevOps in the Enterprise, and John Jeremiah, a Senior Product Marketing Manager at GitLab, will discuss the details of a DevOps transformation. Gary and John would love to answer your questions throughout the webcast, so I encourage you to add your thoughts to the Q&A box throughout the discussion. John will lead the conversation as we learn from Gary. Hello, John. Hi, Suri. Thanks so much for setting this up, and I'm really excited to be here. Gary, welcome. I'm really excited to be able to spend some time with you and have a quick conversation and talk about DevOps. Good morning. Great to be here. Nice to be getting questions from everybody. I'm really looking forward to this discussion because I'm so passionate about trying to figure out how to help other people realize the benefits I've seen by applying some of these practices and principles. Well, Gary, you and I have done a lot of these sessions together where we've presented to audiences around the world and we've talked about DevOps. And I've always found that when you're answering questions from the audience or from people that are there, we learn more together. And so I really do want to encourage the people that are in the audience to post questions either in chat or Q&A. I'm waiting for those. But I've got some questions that we've talked about in the past and I want to sort of get started and walk through those. And hopefully that will encourage people to add their questions and we'll see where we go. So let's start with a question that I think a lot of times comes up and you've seen this in your experience writing books and working with customers and clients around the world. Why are people just getting started? I mean, isn't DevOps already here? Why are they just getting started with DevOps now? What do you see? I think a lot of times people thought it applied not to them. It applied to these unicorns and applied to these people who are doing things. And the majority of software exists in these large, tightly coupled organizations, enterprises. A lot of the developers are there. A lot of the work's going on there. And I think people just didn't feel like it applied to them and they couldn't see the benefits. And this whole idea of deploying thousands of times a day was just so far away from where they were at that they didn't see how it was possible or how it applied or how it made sense for their business. The reason I like DevOps is as you force the frequency of even if it's just a test-deploy cycle, what you end up doing is you force out inefficiencies that have been in your organization for years that it'll low cadence your organization that was just able to brute force and toy through it. And I think DevOps is gaining more popularity, more momentum. People are seeing that it can and should be being done in the enterprises. And I think people are starting to try to learn how and why does this apply to me and why should this matter? And as they're seeing more and more people try it and see the benefits of it, they're starting to go down that path. And as I go around the world talking about this, I always start with the two to three X improvements in business productivity we got at HB because I think that truly represents the opportunities and the more organizations out there that realize that, the more organizations will start embracing some of these ideas and practices. Because at the end of the day, Gary, what you're reinforcing and what I've heard, I've talked to, like you have talked to hundreds of people that are doing DevOps to the enterprise. It's not about, it's about the business. It's about business productivity and it's about helping the business to be more successful. I think that's the reality that people are coming to understand that DevOps for them means improving their business and being more competitive. And I always start out when I work with organizations is, why do you want to change your software development practices? What are your business objectives? Let's start there because we are leaders that run business and we own delivery of results to the business. And if we start there and we understand what about your current software development processes aren't meeting those needs, then you start to go to your toolbox, whether it's Agilene, Combon, Six Sigma, DevOps, any of those things, and you start to pull the tools out of your toolbox that will help you improve your processes to deliver to your business objectives. Because if you can't start there in what matters to the business, you're going to have a hard time getting the sustainable momentum you need to get a transformation working and you won't be able to find the investment, the dollars to make the changes and improvements that you need, and you'll lose that momentum. So I always start with what are your business objectives and then start pulling out the toolbox. So interesting you say that. So when you think about this and you talk to organizations, what do you see as the importance or how critical is it to have the right people involved in helping the transformation? And then on top of that, who are the right people to be involved in making a transformation happen? I think it's critical that you get the leaders involved. A lot of times you'll see DevOps starting an enterprise at a build team or a dev team or a lease team and they'll do a little bit of work, but after a while, they'll have a hard time influencing peers that are coupled together with them in the system or leaders to give them the capacity or the tools to invest and make these changes. And when I go into organizations, I always start with the business objectives and then I talk and I work with the organization and I try to get the leaders. And the leaders is a nebulous term, right? It doesn't have to be the CIO, but I tend to think in terms of deployment pipelines. And this is code that because it's coupled together has to be developed, qualified and released as a system. And if you go through an integrated system test environment, those applications need to be considered one system because if a lot of applications in that system stay releasable and apply all these principles, but two or three don't, you're never going to be able to release on a more frequent basis or have a stable environment. So I tend to start with the leaders over a specific deployment pipeline. I start with their business objectives and then I try to draw out where the biggest sources of inefficiencies are and I try to get their fingerprints on the plan. And this will be the managers and the leaders of that, but it's also your technical leaders who can't get your technical leaders to embrace these new ways of doing it and champion it and doing things. You're not going to get the momentum and the movement that you see. And it's going to be a journey and if you really start with what matters to the leaders of that organization and what they're going to drive for results, then you can really start to find, they will find the capacity, they will find the momentum, they will make sure that they hold people accountable and they'll drive the organizational changes that are required to make these successful. So I spend a lot of time doing executive workshops. I do execution workshops where I get the leaders, but my goal as I go on to organizations is that I'm asking the right questions to draw the plans out of the leaders because if it's their plan, they'll make it successful. And this is a big thing about organizational change management. Yes, there's technology in their stuff that applies to the process changes you make, but if you don't have somebody that can coordinate the improvement across the deployment pipeline, you're not going to be seeing the type of results that your organization could be getting and deserves to get. Got it. So what you're really saying is I hear it and what I think is true in many cases is you have to have the top-down leadership, but it has to be bottom-up too. The stakeholders, the people that are going to have to go through the change have to own that. So it really takes a team of everyone who's responsible for the business to make that change happen. Those are the right people. It's a group of people, right? Yeah, and a big part of what I do when I get going with organizations down the continuous improvement journey where I start with, let's agree on what we're going to do first and then once a month we review what got done, what didn't get done when we learned what we're going to do next and then the other thing that I think that I think is important is that the stakeholders can in the beginning analyze where the opportunities are and get a common view of that across the organization, but the problems that you see and the things that come up are always evolving and executives on high just saying go do DevOps aren't really going to understand what's getting in the way of people. So I talk about the role of executive and what you're going to do it becomes out of investigative reporters throughout in the organization trying to figure out why things aren't getting done and it's not, John, why didn't you get this done? It's like, you know, John I'm a leader of, you know, 800 people across the world and I'm in your cube asking why it didn't get done. The first time I walked into John's cube he's like, dude, what are you doing in my cube? Am I going to get fired? Am I in trouble for something? I don't know how it mattered, but it didn't get done. Let me tell you about what got in my way. I ran into all these things and I ran into these issues and the IT organization wasn't supporting me. I couldn't get, you know, whatever that happens to be, then as an executive leader you can start to figure out what you need to prioritize for the next iteration to start from the barriers for the organization. So it can't be, you know, tops down completely comes up and needs to be a collaboration and, you know, people talk about the cultural change of DevOps and I think it starts with that conversation. The first time I walked into John's cube and said, why didn't you get this done? He's like, all worried, but when we started prioritizing the feedback that he gave me in making improvements all of a sudden you get transparency and the next time I go into John's cube he's like, dude, let me tell you what happened this time. I got stuck by this and this and this and these got in the way and we just did these things differently. I think I could be much more productive and much more effective. With that trust comes transparency and with transparency comes the ability to fix things that have been in the way of your organization for years. So that interaction between bottoms up and tops down and really being an investigative reporter and a subservient leader trying to get out there I think is one of the big changes in your organization that you're really going to successfully change. Totally agree. I think it's really insightful Gary and the thing I like about what you're saying is it's about learning. It's a characteristic of going out to discover where are the bottlenecks where are the constraints and so in that spirit where do you see people going when they go to get started? What's the right place to get started with DevOps and Enterprise? I really always want to start where the biggest where you get the biggest bang for your buck in terms of business results. So what I've made available to you guys and probably to everybody on this webinar is an e-book of my starting and scaling DevOps and Enterprise which is how I go into organizations and analyze where they should start their journey where the biggest sources of inefficiencies are. That's the book, Elephant with the Unicorn horn. And I think if you don't have it you can get a hold of it and get lab and they can help you get a copy of it and be pretty straightforward. That tells me where to start and what I see in a lot of organizations is the first place they need to start is the ability to build and deploy more frequently and get into smaller batch sizes in back terms of that in the test cycle and then we start running tests automated tests and if you have a few maybe you have a few and we start to analyze those and we analyze why they are failing and we get a pre-do chart of what's failing in those environments because those failures are happening now, you're just logging them as a defect and people approve force on their way through it even if it's manual and you're fixing whether it's an environment or deployment issue or a test issue or any of those types of things and what I find when you put those automated tests in place frequently it's going to take one to two months to really build the test on a more frequent cycle and then you get 10 or 15 automated tests that you're running into each build and then every organization I work with it's kind of a different direction. One organization had 30,000 automated tests, they ran them once a week and they only ran them once a week because 40% of them failed each time and the guy over in India spent the rest of the week debugging and triaging the ones that failed and it turned out the environment that he had to run the test on just was not size large enough so it was always timing now and so simple things like that that you wouldn't consider or maybe you have an F5 that's intermittent in your test system and you can't get you need to stabilize those things or you find when you move from a dev environment to a QA environment the environments configured differently or maybe you're manually deploying and it's not consistent so you use whether it's an environment issue, deployment issue you know frequently what I find is organizations go in there and they may have a lot of automated tests that they've had those automated tests they've been running them and then they've been manually debugging them if you're really going to use them as a gate they need to be stable they need to be maintainable as you change the application you should be able to change your automated test just as quickly to keep up and they need to be easy to triage most organizations I would say almost every organization I've worked with once they've really started gating code with an automated test have had to go back and re-architect their test framework for automated testing and really improve the design patterns that they use for automated tests one of the things they tell organizations to do is as quick as possible get to the point where you're trying to gate code with automated tests and get the Pareto chart because that's going to tell you where to go next and most people start to try to build deploy more frequently and then run a test and then depending on why those are failing most organizations tend to go in very different directions after that point so that's why I say there's not one can solution there they shouldn't go in the same direction you go and analyze and look at different things you would never imagine that you had 30,000 automated tests 40% of them failed because the environment just wasn't size large enough there's simple answers to a lot of things sometimes and when all those tests fail then it generates this whole backlog of work that people use to try to keep up with it because they know at the end of the next week they're going to run those tests again and it becomes a vicious circle I got a question actually so thank you very much we had a couple of questions come in so thank you for adding questions to this discussion so here's a question what's the relationship as you see it and it's interesting you know Gary you wrote a book on Agile and you've written a book on DevOps so what's the relationship between adopting Agile and Enterprise and adopting DevOps which one comes first you know the idea of DevOps is nothing more than an Agile principle of releasing code on a more frequent basis that got left behind when Agile scaled to the Enterprise just because it was so hard and it was outside the team's control and nobody could figure out how to do it you know at HP we focused a lot on how the teams came together and how we made the code more releasable and we left the teams with a lot of flexibility in terms of how they work with Agile principles and some sort of worked in more traditional ways at the end of it we looked across the teams and we didn't see dramatic differences in productivity between the individual teams but we saw a 2-3x improvement in business results and our conclusion was that in a large tightly coupled system with a lot of people having to work together the first order effect is how teams come together to deliver value and the second order effect is how the team works together and I think that the teams made their ideas work which with a lot of empowering them and giving them flexibility but if you don't have that process for pulling the system together and making that work I don't worry about it so you know I go into large organizations that have COBOL mainframe systems and we couldn't possibly do Agile or anything else that are building and releasing on a more frequent basis and trying to test against you let's talk about how often you can build your test cycle and how often you can do that so it's more of the DevOps principles and I think once you get to that point where you're building testing cycles are very small the Agile part of it starts to become much easier and much more natural so create a system where it's easy for people to do those types of things and then you know talk to them about it some teams want to embrace it, they want to embrace it but if you don't have that system for putting it together and a deployment pipeline for releasing it more frequently you know but I think the business results from Agile become much more constrained or you don't get as much out of it so I would always start with and call it the Agile principle of being able to always code on a more frequent basis to the customer I would start with that piece because what I find is when you have a large number of people that have to work together that deployment pipeline is the forcing function that aligns the organization one team can't check in something that doesn't sort of work with somebody else's and they have to resolve that and figure it out and if you just let the teams go independently and then somebody else is responsible for putting that together I just don't think you get the benefits that Agile was attended to achieve and I don't think you're doing what Agile really is and that's a lot of reasons why you know the first book was the practical approach to large scale Agile but what we realize is where we got our biggest benefits is how do you coordinate to work across teams and that kind of evolved to what DevOps is today so I've kind of turned my focus to that you know if you've got an Agile transformation in place great, want to go forward, want to do but let's start working on the problems that have the first order effect which are how do you integrate the work across teams very cool you know it's interesting it's sort of the water scrum fall model that a lot of people fall into the trap of which is you know we're going to do all these big requirements up front the development team will run in sprints and they'll run as fast as they can and then they hit the blocker of getting to production which is where DevOps solves the problem and it helps to address it so I think you're spot on you know it's interesting that you talked about your experience at HP because this might actually this next question might be relevant for you just and maybe you should talk about your background there because this is a question and let me read the question and you can share your background and why how you can help so here's a person who's actually working for a small to medium company and there's dozens of them and they make some embedded systems and there are lots of DevOps material that teaches and talks about end user applications in the web where you know tools you know can lead you through things like you know GitLab and Kubernetes and web deployment making it easy what are your tips and tricks or your insight about applying DevOps for an embedded system where there's hardware buttons and a GUI I think you might have some background on that yeah so I started by transforming how we did software development for Hew Packard for the laser jet printers which is very much embedded software and embedded firmware that goes into these systems and the firmware had been the bottom line for the organization for two decades and I figured we had to do a different way so we started down this path of continuous improvement I didn't even know I was doing Agile for the first six months I just thought we were doing common sense trying to set short-term milestones and improve and get better and a lot of the things that we ended up doing and if you really want a good case study of that it's a practical approach to large-scale Agile development see if I can grab that one it's this book here and it talks about the journey that we went through at HP and the types of things that we did and a lot of the things we did was you know how do you coordinate the work across teams how do you test how do you automatically do continuous integration how do you gate the code how do you work through those pieces and that's clearly there I work with a lot of organizations around the world that are doing embedded work and the same principles apply you know you don't have the deployment issue that you have in some you know large organizations where you're deploying to a website and environment but Kubernetes can be very helpful for your test cycle right at HP we had I think it was you know it was like six or seven racks of servers running 20,000 hours of automated tests every day and to manage that and do that we'd have a simulator we'd spin it up we'd deploy it into these server farms and we're doing that automated testing and feedback which is critical to being closer to your release on an ongoing basis it's critical to making your triage process more efficiently because you have a smaller group of change sets between the last build and the next build you may not be deploying to your customers as frequently because they don't want to upgrade their firmware on an ongoing basis but where we got a lot of those benefits was really automating our testing going to one trunk based development instead of a lot of branches and going to smaller batch sizes in the change sets between test cycles that enabled us to triage efficiently and that got us two to three X improvements in productivity I'm working with other large organizations that can't release very frequently just because they had a lot of branches and a lot of trunk sizes or a lot of test cycles length and a lot of this applies very directly to embedded systems and you know grab a copy of the practical process large scale agile development that is truly step by step the changes that we made and the things that apply and if you have additional questions just pop me an email and maybe we can get on the phone and talk through it because the benefits are very real with embedded systems even though you may not be deploying quite as frequently Cool. I knew you'd like that question Gary. So let's double click and go talk about some of the common challenges that you're running into and you're seeing and please more questions from the audience love having questions and the insight because that's a we're here for you and we're here to help you learn so that's our goal today when you do workshops and you've done these with lots of different industries and lots of different situations what do you think as you see it what do you do are the top two or three challenges that are most common that people are running into the issues that are them overcoming your workshops I think like I said the first one that people need to be able to do is to get to smaller iterations they need to be able to build and deploy on a more frequent basis and I work with some organizations that have got a pile of scripts that they've had laying around for builds and different things and when they really start trying to build more frequent places and deploy you find that they didn't have a very efficient process it wasn't working so you know somebody would do a build a week or build a day and they would just manually brute force getting it done when you're going to try to increase that frequency it requires really fixing that build deploy process and I think some of the guys some of the things that you guys are doing with your auto DevOps can help a lot of organizations get that process cleaned up much quicker but you know I'll spend nine months with organizations that are really just trying to get that problem solved because it's been so difficult and then really quickly behind what you've been able to build and deploy you start trying to run some automated tests and most organizations and while they've been doing automated tests some a little bit here there a little bit there really don't have a good test framework and don't have good design patterns for tests that enable them to be maintainable triageable and all the things from that so as you go down through that process and start working on it it really starts to highlight the things that you need to do and the reason I love DevOps is as you start to go through those things you will find that people are deploying code into an integrated test environment that's not ready and is unstable and it's causing everybody else in the organization to run into the same defects they log a defect somebody's got to close it out the deployment wasn't done correctly people log a bunch of defects you got to close it out you got to fix it you got to go through that you know you really shouldn't allow anybody to do manual testing on an environment until it's at least passed through some set of automated tests and the first step is really getting those going getting those stable and requiring those to pass before you don't have anybody have access to it so I tend to try to work on finding and fixing that defect as close to the developer as I can and I talk about three different types of ways that you see in organizations one is a developer working on something that won't work with somebody else's code won't work in production meet the business need and you can't optimize that like you'd optimize a manufacturing process where you change the process and you measure that but because you're changing both the product and the process at the same time and improve those efficiencies of that what you really need to do is reduce the feedback cycle time and you need to improve the quality of feedback every developer wants to do a good job thought they did a good job until you provide them that they expect to be getting better so a big part of what we're trying to do is really improve the quality and frequency of that feedback the next thing that organizations spend a lot of time and waste doing is just triaging stuff this is you know how can I figure out who caused the problem where did it come from those different types of things and if you want your test environments become very unstable you can tend to create a lot of waste entering defects closing defects doing all those sorts really you saw that by going to much smaller batch sizes and you do it by gating code before it gets into these integrated tests environment to make it more efficient and then the third thing that you see is repetitive task and software this is a build this is a deploy this is set up an environment this is a test those things can and should be automated right you don't want people repetitively doing the same thing over and over again that a computer can do reminds me of jazz Hummels favorite joke do you know what computers do when they find out that you're manually doing something that's easy to automate you know they answer that one John go ahead tell me what do they do Gary well they sit up at night and drink beer and laugh at you I mean doing something is not repetitive task being manually done is not very enriching for the employee it's not very valuable you tend to not do it the same way that's fine and the other part of it is when you automated it makes it affordable to go to these smaller batch sizes that are easy to triage so it's working through those processes and get them worked got it so that's great thank you I got a couple other questions came in so thank you again for sharing questions with us I'm going to try to rephrase or at least get this question so one of the asked what would be a good architecture when you have multiple teams to build easy and quickly how then the question is how good are common solutions like GitLab Jenkins and others at scaling that would be your opinion and then the question goes on is I was looking at self-service tools and portals to make it easier for more teams to help themselves but there's not much out there you found something like AWX not sure what AWX is but what's your take on architecture and building and scaling well you know ideally you want an architecture that enables small teams to independently develop, qualify and deploy code and that's what all the unicorns are doing and you know I've gone through this debate with Jez and Jean about you know architecture matters when you go to do DevOps and it's the types of things you do for large systems are different than the types of things you do for small systems and the last puppet report or two they've gone in and tested for that came back to you know you're right most of the people that are doing DevOps really well are the ones with loosely coupled architectures so if possible you want a loosely coupled architecture that enables people to independently develop, qualify and deploy and what you'll see is it's much easier to coordinate the work across six people than it is to coordinate the work across six hundred textural changes can enable that to happen and wherever possible you want to go down that path the reality is that changing an architecture in a large tightly coupled system is a long journey and it's a very difficult process and I having done that at HP where we completely rewrote 10 million lines of code I tend to avoid that at all cost if possible if you can do it great but you know I think the opportunities for improving the productivity of organizations is much larger in these tightly coupled systems where you're coordinating lots of people not that they'll ever move quicker not that they'll ever go better but just because the inefficiencies of coordinating six hundred people are so much larger than coordinating six and so I start with that if you look at the starting and scaling DevOps and Enterprise book that's there or even go back and look at the webinar that was done before this with GitLab you can kind of go through how I approach putting together that system and the process and where to start for large organization and as opposed to just enabling teams to do continuous integration on their own and having a self-service portal for that if you need bunches of teams to come together and integrate a test environment somebody needs to design that deployment pipeline somebody needs to optimize and somebody needs to put it together there are lots of tools out there people are doing this with large tightly coupled systems with the things you mentioned and others and you know the key is getting the people that are going to be using it and making it to own the tool selection because every tool has great breakthroughs and every tool has works and if you let the team pick the tools that they're going to use they'll look past the works and as Darren Bender who worked for me used to always say we will bend any tool to our will if it's a tool that we choose to make work and so it's that process you know there are some really nice things that GitLab's doing with AutodevOps that really helps you if you've got a small or if you've got a something that you can put in a container you can stand up that deployment pipeline pretty quickly and get that piece working so there's tools out there I work with a lot of vendors so I tend not to be very tool specific but there's a lot of things that can really help you move forward out there Thank you Gary and I agree I think AutodevOps is pretty cool but I'll let people check that out on their own and that's online you can Google it Mateo asked a question I think he's got a common challenge which is how do you convince your boss to start from zero a DevOps business transformation in a scenario with lots of old code so what do you like we talked about it's top down bottom up it takes a team to make this happen it's not it can't happen grassroots up only and it can't happen top down only right so what's your sales pitch how do you convince leaders that they need to think about DevOps I have opinions but I'd like to hear yours I'd almost not start with DevOps I'd start about know about your current software development processes are not meeting the needs of your business what is he trying to accomplish that he can't get done he's got pressure from his bosses to do more with less to release more frequently from the business side or whatever it is what about that is not meeting his needs and then you can talk through talk through now here's some inefficiencies that I've identified in our organization and that starting scaling DevOps can help you collect metrics can help you show how your process works and then talk to them about you know here's some big inefficiencies I see in our organization and if we went down a continuous improvement journey of addressing these things I think you would see more capacity freed up to do the types of things that you want to do to win in your business so I would not go talk to an executive about doing DevOps I'd go pull out of them what they went to accomplish and why and then and only then would I start talking about pulling tools out of my toolbox to show the types of things to do and you know it's it's really hard to measure productivity in software development and what I've realized more and more as I started going into these companies what I've gotten much better at measuring is waste and inefficiencies in our organization and if you can put that in front of people and they can see it and and they can go on a journey with you that that's helpful you know I write my books fairly short they're usually about a hundred pages they're designed to be if you've got a boss that's going on a two to three hour plane ride they should be able to get through it from start to finish and figure out where they are I do a lot of these different webinars you can Google that have that out there and and have a watch a presentation where I've gone through that piece and kind of gotten it out there but talk to them in terms of what matters to them not in terms of just doing DevOps for the sake of doing DevOps now I think that's great advice Gary because at the end of the day executives aren't interested in DevOps as a fad or a trend they need to understand the business value they need to understand the risk and they need to understand I think the reality that more and more businesses are facing competition from places that they don't expect and so being able to ship software faster becomes a competitive advantage for them not not just a competitive advantage but it's also a way to avoid and to deal with the threats that they don't see coming think of Uber as a classic example so I think you're spot on and I think that's really what the conversation has to be about is how do you go in that direction assuming we're talking to an executive I have another question about grassroots movements so what do you find is the best way of tying together grassroots DevOps movements within a company at the CXO level often innovation starts low in the company where people start organic kind of DevOps issues and initiatives the one team starts doing and I'm going to paraphrase you know one team starts doing it one way another team starts doing another way it becomes fragmented before there is a higher level oversight or management of it so that it potentially becomes either an IT ops problem or a security risk you know I really if the if the teams have to work together on a deployment pipeline that has to be integrated and tested together I think they should be on a common deployment pipeline a common set of tools and a common piece if I've got another team that doesn't ever have to test against it doesn't have to deploy against it sometimes I think the efforts to make those two teams common are more trouble than they're worth so in a lot of instances you can maybe you pick some common tools and you get some leverage across that in your organization but absolutely if the code has to come together and tested together it should be a common process and somebody should be looking over that piece and in that case I would go to the leaders of that and get them to design that deployment pipeline figure out how to release that code on a more frequent basis what I frequently see in organizations is I think they're doing DevOps because they've got a team doing continuous integration at their level and they're doing that and they're doing their agile sprints and then they hand it off the release team later to get it out the door and my mind that's not really that's kind of what you can do from grassroots it's kind of what agile ended up doing when it scaled to the enterprise yes we do stand-ups and we do demos in our dedicated environment we do all those other things but we don't release on a more frequent basis because that's the release team's job and we hand it off so in those cases where you have a you know integrated system I think it makes sense to put it together you know would I slow down two teams that are making progress and deploying their code on a more frequent basis that can work independently to get them on a common tool and common process I don't know that I would I think if I was a security person I would be trying to look at both of those deployment pipelines and trying to figure out how do I put my security testing in that deployment pipeline in the appropriate place and figuring out how to put it in but even if people are using a little different tools a little different processes you should still be able to get that in there so I don't think you're going to have security risk if they do it different is necessarily valid but you do need to get your security testing as part of that deployment pipeline because you know DevOps is all about how do you release on a more frequent basis enabling all aspects of quality and whether it's performance, security, functionality any of those types of things and if you're not pulling that security feedback as close to the developers you possible can so as we talked about increasing the quality and frequency of feedback that needs to be with security as much as it does everything else and I go to so many organizations that sort of say well I'm the only one that cares about security I'm like really so when do you run your testing when they say they're done I run my testing I find all these holes and nobody will fix them because they say they don't want to delay the release it's like well duh but you know sure if you make it a last minute you're either going to move the release or do security but if you give it to them feedback exactly the day that they're writing the code I don't know of any developer in the world that doesn't want to write secure code so you know it's it's more of a process and approach and when you give the feedback than it is specifically everybody's got to use the exact same tools in the exact same way good feedback Gary and I you know one of the things I've become really thrilled about when I build things and ship things here at GitLab as I watch our security tests run right away whenever every make and commit it happens right away in the pipeline we run the security scans and the security tests and so we know right away whether whether I've added anything whether anything new has been introduced and so that's feedback that's incredibly valuable but I'm with you another anonymous attendee asked the question and this is a I don't know if you're going to be able to answer this question if I want to do too much but I'm going to read it and then we'll decide how we want to handle it my organization of roughly 100 engineers is currently using Bitbucket and Bamboo today for their source code and CI and I'm proposing my senior leaders that we move to GitLab because of the source control because of the source control pipeline definitions and this person's finding it difficult what do you tell people when they they ask you know how do I justify one tool versus the other and and to whoever asked the question paying me I'd be glad to have a conversation offline with you is to understand better where you're at and what your challenges are and figure out how how to help you make that argument because I'd like to know more information before I go too far down that path but Gary what do you tell people when when they ask questions like that about tools and how do they just pick the tool this tool that tool you know I I'm going to turn that one back to you John because it's I think it's a perfect opportunity for you to describe why you think GitLab provides something better than a lot of the other options out there and and how you would recommend people go to their executive leadership and help them understand you know how to well so I mean the way I the way I would talk about it would be from the perspective of and I and I want this to be your webinar I want this to be about you but so just quick sidebar you know GitLab offers a very unique capability of being a single application that covers the entire DevOps lifecycle from tracking and managing ideas and planning and planning to work and planning how what you're going to build to managing the code and managing the reviews and managing that process to then linking it to continuous integration if you look at the entire tool chain of all the different tools and all of the different integrations that you have to touch from beginning to end it can be incredibly challenging to manage it to manage it efficiently to manage it effectively it can be difficult and it requires people to actually become full-time plumbers if you will of building and maintaining your pipeline I don't know about everyone else but as a developer I want to focus on solving business problems not managing IT plumbing and and so the question really is how can you get a pipeline and deliver and ship business value faster without spending all your time on the plumbing and and I think GitLab is an incredible answer to that problem because we provide the entire flow from idea to production we've used that in the past but it's about how then how do you work and how do you enable people to come together and work faster more efficiently so again I will do more webinars with GitLab demos and talk to the GitLab value proposition love to do that but I really don't want to take away from Gary your expertise of helping people make these transformations so thank you for the platform for a moment but I really do want you to tell people about what you've learned and your insight which is cool Gary when you what do you do about tightly coupled systems I mean I came from insurance and financial services working there doing testing and helping to drive SDLC transformations our enterprise architecture diagram consisted of of incredibly old legacy technology and it looked like a spaghetti diagram if you will how things flowed from beginning to end and changing it was incredibly hard so we have this large legacy tightly coupled set of systems how do you do DevOps in that kind of world well you know that's a lot of what I cover in my starting scaling books but it's really how do you take it and maybe you can go to the slide that has the tightly coupled system John and that would be a good way to get into it but you know you start with this big complex system where you have a lot of on down a little bit down one more yeah you start with this large tightly coupled system with all these things that have to be in an integrated test environment together and you know ideally you just do continuous integration on everybody everybody checking code you do a build you do a deploy and then you do a test cycle and you figure out which of the three people that made a change broke the test the reality is when you have this large of a system with hundreds of people working together that because it takes a long time to deploy it because it takes a long time to test it you're in and out doing batch sizes that may have a hundred commits and so it's really hard to debug and tree off that thing and when you get into that large tightly coupled system what you tend to do is you click down on to this slide one what you tend to do is you break this into subsystems using service virtualization and label you to test and then click one more slide and what you do is you end up with each of these three different subsystems where you're able to independently test and qualify those and then if you go into the next slide what you're really trying to do is take each subsystem one you're trying to do continuous integration on both component a if you click forward one and it shows that coming in and then component b and what you're trying to do with this is you're trying to keep defects from component a getting into subsystem one and causing instability in that test environment and you do that by running automated tests that you gate it with and then you're trying to increase the quality and frequency of feedback to developers in component a and you do that by gating their code so if they get something in through there the test automatically kicks it out then they own resolve in it and you've got a lot fewer people that need to look at debugging and triaging that then if you do if that code made it into subsystem one so it's process of building up that deployment pipeline and then you just do the same thing for each of the subsystems and then you put the subsystems in together and so it's continuous integration of continuous integration and it's gating of things and trying to keep that integrated test environment stable it's that process that I think of and so if all of these different parts of the architecture have to work together that's what I call one deployment pipeline and that's what I call one system those people need to be on the same toolset on the same process and there needs to be somebody in a leadership role that's over that entire deployment pipeline that's prioritizing the improvements that we make the changes that we make and really leading the continuous improvement process and interacting with all the different components and taking waste and inefficiencies out of that system cool thanks Gary that's really helpful and I think it's wise advice as to how to break those systems down and to think about that we've got a few minutes I want to try to I don't want to run much longer we've got a watching our time because we've had a really good conversation let's just close on the question to you is about what's the advice when you talk to people what advice do you give to your leaders who are trying to figure out what to do with this DevOps thing so if you talk to executives you talk to leaders all the time what's your advice what do you tell them we're going to start with the business objectives and then we're going to try to get everybody that's part of the deployment pipeline to have a common view of the elephant in my first book because most organizations I go into a lot of people are excited about doing DevOps they all have a different definition what DevOps are and they all have different feelings for where they would start and if you're going to look across that entire deployment pipeline what you really need to get is everybody in the team to have a common understanding of what the issues are and what needs to be addressed and that's the first step and that's what I do with my execution workshops is one I go and I look at where the biggest sources of waste and inefficiencies are and then two as I go into these organizations I really spend time trying to make sure that everybody on the team in the workshop has a common view of not only how does DevOps not just how does DevOps work for small independent teams but how does DevOps work in a large system give them a common understanding of that a good map of the biggest inefficiencies in the organizations and then get them started on that continuous improvement journey and it's a journey as I engage with organizations on the execution workshops I don't do it unless we're signed up for at least six monthly checkpoints to review the progress and get that because you're going to go in and think you've got the biggest problem understood and as you start digging into it you're going to peel the onion and you're going to find it's a different layer so I really try to engage with them on that journey and I think it's important for the executives to engage with their organization on that journey so that they can learn and evolve with them as they start digging through the layers of the onion very very cool that's awesome thank you so much we've had a great session Gary I thank you for sharing all of your insight and engaging and I think you gave great questions from the audience as well Suri I'll let you if we want to see if there's any other questions we have a few minutes I think left if there are more questions although we've answered a lot already if there are more questions we'll take them now it doesn't look like we have anything no it doesn't so Suri I'll turn the floor back to you thank you and thank you all so much for joining us we hope that you enjoyed hearing from Gary and from John if you'd like to learn more please visit about.gitlab.com slash devos thank you again for joining us bye