 Welcome to Starting and Scaling DevOps in the Enterprise. I'm Suri, a member of GitLab's content team, and I'm thrilled you're joining us today. If you have any questions, please use the Q&A function at the bottom of your screen. If you have any technical difficulties, please use the chat function. On this next slide, you can see that we have Gary Gruver, the renowned DevOps consultant and author of Starting and Scaling DevOps in the Enterprise, and John Jeremiah, a senior product marketing manager at GitLab. They'll discuss how to apply DevOps and agile principles at scale. We have several interesting topics to cover today, and I'm excited to hear from John, who will first explain how to align a DevOps transformation to business objectives. Welcome, John. Thank you, Suri. And I'm so excited to be here with you and Gary. I've had the opportunity to work with Gary in the past to join him in his webinars and workshops, and so I'm very excited to hear his talk. I wanna start, you know, really to talk a little bit about how we align and how we think about DevOps from a transformation and how it drives and should align to our business objectives. So if you think about it, Mark Adrysson said this a few years ago, that software is eating the world, and it really gets to the point that we have to move faster than ever. We can go ahead and move to the next slide there, Gary. And as we try to move faster and faster because software is ever more critical to what our business objectives are. A lot of organizations talk about a digital transformation and they talk about, and we see this everywhere, we've seen the disruption where digital has changed industries left and right. Look at retail, for example. How often do you not even consider going into a physical store but simply go online and order typically from Amazon? How often do we rethink, have we started to rethink transportation all because of a mobile app on our phone? Digital is disrupting transportation, it's disrupting retail, it's disrupting financial services. Every industry is facing a disruption and some of them know what that disruption is today, others don't. We have to be able to move faster and ship software faster because more and more the relationship and the experience we have as consumers and we have with our customers and our partners is defined in the digital realm. And so in order to be competitive, the faster we deliver, the more competitive we can be. But how do you align? Let's go to the next slide, Gary. How do you align? Do you need to be able to ship software every five minutes? Is that how fast you have to go? In order to answer that question, you really have to take a step back and talk to your business partners and look at your industry and your market. If you're in an industry and a market where the pace of change and the pace of innovation is at a pace which is six months of change and the cycle is slow, then perhaps you don't have to be moving at the same speed as Google or Amazon or Facebook. But if you're competing in an industry where the speed is faster and it's accelerating or if your industry is one where you're facing this disruption, then you have to learn how to move faster without sacrificing quality. It's not about just fast, it's about speed with both security and with quality. And the only way you can determine the right speed, the right velocity is by working intimately and closely with your business to understand the demands in your specific industry and your specific market. It's, this is not an IT problem. This is not a problem that is solved simply by figuring out how do we go faster to ship faster? It has to be done at the pace the business needs. If you ship an application faster than the business can consume and you're pushing changes to the business at a speed that they can't consume, then you can be more disruptive than helpful. So the key to understanding how fast is fast enough is understanding your business and your users and the needs that they have. This is really about aligning with your stakeholders and shipping as fast as they can, as they can consume it and as fast as they need it. Now, as a part of the GitLab team, I wanna just stop for one second before Gary does his content because I want Gary to get into talking about how do you define and scale a DevOps transformation. But for those of you who aren't familiar with GitLab, I wanna share with you a little bit about GitLab and who we are and what GitLab is all about. GitLab is an innovative solution. We are a single application that's been designed from the beginning in order to address all of the challenges within the DevOps lifecycle. Whether it's how you create and manage your source code because our foundation is based on Git and distributed version control, we can go from there to look at how do we do continuous integration and build, automate the delivery across the entire lifecycle. And the power and the advantages that we were able to help companies and help teams by being on a single platform are listed here, but it really is an opportunity to bring teams together so they can collaborate and have visibility with each other. And it's in the act of collaboration that you can really start to achieve efficiencies and velocity. And so we'll have more on GitLab and I don't wanna spend a lot of time on GitLab. I wanna just anchor you on understanding what GitLab is, because we're so much more than just Git. We really are the end-to-end lifecycle. And now I wanna turn you over. I've started this off with a little bit about alignment and a little bit about GitLab, but I think the real exciting conversation and the interesting conversation is about to follow because what Gary is gonna share is really the insight that he's had from his years of experience, leading DevOps transformations in large organizations. And I always love to hear his stories. So Gary, the floor is yours, sir. Thanks, John. If you could improve the productivity of your software development processes by two to three X, would it matter? What if you decided all this was too hard? All this was too risky, didn't make sense in your business, but your competitors went all in. Can you see how that'd make a difference to your company, to your coworkers, and eventually to yourself? I start every presentation I give in this space with those questions, because one, I think it truly represents the opportunity with embracing these new ways of working and also highlights the risk associated with ignoring these trends in the industry. And as I've spent a lot of my time over the last decade going around the world and being fortunate enough to tell the story about the transformation that a great group of people at HP led, where we easily got a two to three X improvement productivity where we dropped our R&D costs almost in half, went to where the software was no longer a bottleneck and we're supporting 140% more products and increased our innovation by eight X. It really was a two to three X improvement as conservative. And I think that's the opportunity that this represents. And as I've been fortunate enough to go have these conversations around the world with people, what I'm starting to realize is that more and more people are realizing what we did, which is a lot of these benefits are more associated with how do you work across teams, which is a lot of the DevOps stuff that's being coordinated. And we did these things because they had the biggest impact on our productivity, not necessarily because we wanted to do DevOps because DevOps wasn't a term that, but it's exciting as I go out in the organizations that I'm getting to see more and more people really excited about doing DevOps. The challenge I see is when I go into these organizations it's like five blind men describing an elephant. I'll go talk to a developer and I said, are you excited about doing DevOps? Yeah, yeah, yeah, I'm way excited. This is gonna be great. Really, so you're gonna make sure when you write your code it's all covered with an automated test. You're gonna make sure that job one is making sure when you bring your code in it doesn't break any existing functionality and all those automatic tests are still passing. You're gonna make sure that any change you make to the infrastructure is done in code that's in a SCM tool like Git that enables you to tell exactly who changed what when and that all your deployment mechanisms are automated so you have consistency in this process. So I said, no, no, no, wait, wait, wait, that sounds like a lot of work. I couldn't possibly do that. No, I wanna do DevOps because I heard at Netflix where they're doing DevOps, the first day on the job you get to push code into production and you and I have no idea how hard it is to push code into production in my organization. First I have to go to the change resistance board and get it reproofle and then I've gotta go through the whole process of making sure I get on a launch call and I gotta get my code over to a branch for a release branch and then I gotta go through and debug and triage that. No, no, it's such a headache that this is, I just wanna be able to push my code into production and between you and me I'm not sure this is gonna work till we get better operations people because right now I knew I patched the code and I knew I needed this patch in my operating system before this could go and the production people didn't really get that and so when operations wasn't there it was halfway through the launch call before we figured it out and they never even did open the firewall that they needed to interface with this third party and they should have known this. No, this DevOps stuff isn't gonna work until we either get better operations people or I don't know, operations people that can read my mind and then I go talk to the release team and they're way excited about doing DevOps too. I was like, really, you're gonna make sure that you do multiple releases a week, maybe multiple releases a day so you have very small batch sizes of changes so it's easy to triage and figure out the exact impact on the business. It's like, whoa, no way, that sounds too hard, right? I only have my change resistance board on Tuesdays and if I had to have that every day I couldn't possibly do it. No, no, I wanna do DevOps because I wanna make sure that everything's in version control. Right now, I'm the babysitter of the organization. I have developers off changing things, off patching their systems, off doing these different things and they don't tell anybody what's needed and how it's gonna be done. What I wanna make sure is that all of that is infrastructure as code. It's in an SCM system. I can tell exactly who changed what, that's consistent. I can make sure that that goes from a dev environment to a QA environment to a pre-prod environment to a production environment in exactly the same way and that's used. I'm gonna have automated tests to where when I have a showstop or defect come in I don't have to guess at which tests I can run and how many I can afford to run because it's really easy to kick off all my automated tests and know exactly where we are in our release. No, no, I can't wait to have DevOps because I'm tired of being the babysitter of the organization, trying to go around and figure out who changed what and where and why and I'm gonna have it all under version control. It's gonna be easy to replicate and we're gonna do these deployments so many times in a QA environment using the exact same processes and tools that when we go into production it's gonna be boring. And then I go talk to operations people and they're way excited about doing DevOps too. It's like really, you're excited about doing DevOps? Yeah, yeah. So you're gonna make sure that you never go in and manually touch a server again. You're gonna make sure that all the definitions of that infrastructure is done in code. It's in the SCM tool. You're gonna make sure all your deployments are automated and repeatable. And you're gonna make sure that anytime you make a change or a patch of those production environments, you're gonna start all the way back at Dev and put it at the front end of the deployment pipeline and make sure that it goes through all the different stages of the deployment pipeline in the production and you've got that working together and you're really collaborating with your QA people and your Dev people to make sure that all this is under version control. To which the answer is typically no, no, no, wait, wait. You don't understand. You have any idea how busy I am? This business would not run if I wasn't spending my time full time deep in the details of figuring out exactly how to make each server run. I've set up each of these servers by hand manually myself. I know how each one works. I've even started to name them and treat them a little more like pets because they're there. And if I wasn't doing that, keeping this place running, if I was going to script that or if I was spending time with the development people, this place would fall apart. No, no, I wanna do DevOps because when you do DevOps, the developers have feature flags and when some millennial developers stuff some code in on a Friday afternoon that breaks everything and I get a call at two o'clock in the morning, I wanna just be able to go in and turn the feature toggle off, go back to bed and let them deal with it on Monday morning when they get in and clean up the mess that they made. So everybody's excited to do DevOps but they've got a little different definition of what DevOps is. What could possibly go wrong other than you create, you've got all the right pieces, you've got all the right parts for functioning DevOps implementation. It just doesn't look like a DevOps elephant. It doesn't act like a DevOps elephant. In fact, it's more of a face plant than delivering the business results that you're helping for. I spend a little bit of time on this one because I think I can throw a little bit of humor in it but two is those really start to identify the cultural changes that you need people to embrace in the different ways of working and we can do a lot with technology, we can do a lot with these different pieces but if you can't get people on board to embrace new ways and different ways of working, I think you're gonna struggle with your implementation and a lot of people have different views of it and a lot of what I do when I go into organizations is try to get everybody on the same page, try to get a common view of the elephant so they can start going down this path of improving and the first place I like to start is everybody's got a different definition of DevOps and a lot of it is based on practices they've heard at different conferences or different places. So I like to back up to Gene Kim's definition because he spent a lot of time really cultivating this DevOps community and forming conferences and spending time around that and for him and I think this resonates for me too is it's all about the outcomes it's not the practices, it's what you're trying to achieve and it's the ability to release code on a more frequent basis while enabling all aspects of quality and that's pretty straightforward and easy and the good news is you'd be doing that today if it was easy and straightforward. What I find when I go into organizations is there's lots of reasons why you're not doing this and it has to do with waste and inefficiencies in your system and from my perspective having grown up as a manufacturing process engineer you can't quite apply the same processes to improving your software developments because it's different you're doing something different each time and when I really go in and look at it there's three different types of work that I see going on in organizations the first work is just the new code you're writing, the new feature, the new application where you're really going in and trying to write something new and optimizing that and because you're changing both the product and the process at the same time you can't really do designed experiments to figure that out and improving that productivity is really based on providing higher quality and more frequent feedback to the developers every developer wanted to do a good job thought they did a good job until you give them that feedback they don't know that they haven't done a good job and so the key to taking waste and inefficiencies out of that process is reducing the time that developers end up on working on stuff that won't work with other people's code that won't work in a production environment and won't need to be need to the business so we're really trying to reduce that cycle time and take it out the next thing that large organizations do is they spend a lot of time debugging and triaging the source of problems and if we're going in and we're looking for a problem that has been written by somebody that's been committing code over the last month or so it takes a long time to debug and triage that and look for cues and I find a lot of organizations spend almost as much or more time debugging, triaging the environment, the code and getting it to release quality than they do actually writing in the features and the capability so DevOps we go to try to improve that by going to smaller batch sizes more frequent feedback, smaller change sets so we understand those changes and the last thing that happens in software development is the repetitive tasks these are give me an environment do a build, do a deployment all those different types of things and we go to improve that through automation and we automate for a few reasons one is if it's automated it's more affordable to run more frequently in smaller batch sizes which makes the triage easier the other one that really comes in there that helps a lot is when you automate it you get it done consistently the same time every time, whether it's a different server it doesn't matter, you don't have to name them you don't have to treat them as paths you can start to treat them more like cattle because they're treated the same and those sorts of things so you go after those things and the challenge or the framework that I use to think about how do you do this? How do you take that waste and inefficiency out and one of the reasons I'm a big champion at DevOps is what I find when I go into organizations is when you start to increase this frequency it forces you to fix waste and inefficiencies that have been in your organization for years when you're doing it at a low frequency you're not very often your organization can muscle its way through it and use brute force to get the releases out the door when you start to go into smaller batch sizes and more frequently you have to fix those things once and for all and I start with the framework that David Farley and Jez Hummel came up with which is a deployment pipeline which is nothing more than thinking about how do you start with a business idea hand it to a developer get an environment for testing move the code over for testing decide that it's approved to go into production roll it into production and monitor that's making sure it's working pretty straightforward and easy process I don't know why we continue to have these webinars to do something that's so simple but in large organizations something as simple as it starts to become more difficult and you look at the types of things that people do in an organization and we'll start with a deployment pipeline for one developer in a small organization and then we'll talk about how do you scale it to a team and then we'll talk about how do you scale it to a larger organization there are a lot of inefficiencies in this process that happen in most organizations for the business ideas typically most organizations start with way too much inventory in the system in terms of requirements sitting in front of the developers and if lean manufacturing has taught us anything it's that whenever you have excess inventory in the system it's at risk of waste in terms of rework and expediting so if you find out that when you start talking to the developer about the business ideas and you have all these requirements written up they have different ideas and you have interactions and you have to end up reworking that or if the market condition changes you have to rework that or if your competitor comes up with something new and you need to prioritize a whole new set of requirements and you de-prioritize the ones that you've invested a lot in then you end up throwing out waste because you're not using a lot of the effort that you put into it next you need to move the code to the developer to write the code and then the next step is you need an environment for testing I know one large financial organization on the East Coast that started their journey to DevOps by just putting in a ticket and seeing how long it would take to get Hello World up and going in a production environment and they turned the experiment off after I think it was 250 days and the reason they turned it off is not because they had Hello World up and going but they thought they mapped their process and understood where the bottlenecks were and next they tried to figure out how long it would take them to get Hello World up and going in Amazon Web Services and that was three hours so they understood the opportunity they understood the bottlenecks and for them this DevOps journey and how do they release on a more frequent basis was a lot about going after cloud capabilities and those types of things to enable them to do that in their internal environment so that people weren't waiting for the environment piece after you've got that environment and it's consistent working well then you move the code over into the test environment and make sure that you test it what I find is most organizations this is probably one of the most frequent bottlenecks that occurs next is that people are relying on manual testing or they'll have some automated testing or when we really go in and we start to work with them to try to gate code with their automated testing what you'll find is their automated testing isn't maintainable, triageable and stable and as you go through that process and a lot of organizations I find the forcing function is when you start trying to use automated testing the gate code is when you really start finding where are some of your inefficiencies and bottlenecks are in the system after that you need to move these things over into production, make a decision some people talk about the approval and let people directly push it in what I find is that that is a regulatory burden or hurdle and a lot of times this deployment type pipeline is taking three or four months and the approval takes half a day I'm not sure I would take on that challenge of just saying anybody can push into production until that starts to become the bottleneck next you move it into production and this should be a straightforward process because you're using the same exact definition of the environment, you're using the same deployment mechanisms, it's all automated you've been doing that a lot in your environment testing program and as you move it into production what you find is that nothing new should show up but what frequently happens in organizations is between the dev teams and the ops teams they haven't really agreed on the common set of tools they haven't agreed on the common set of processes they haven't put things as infrastructure as code they haven't automated all these things and so each deployment into production is a new and unique event and you're struggling to debug and triage it a lot of what we're trying to do here with dev ops is get that a lot more consistent and then the last thing when you go into monitoring you need to make sure that everything's working well I've had a lot of experience where you do all the testing you go into production and something very unique shows up in monitoring and one large release I was doing for a website where I was at and everything worked perfectly backdoor testing everything we put live traffic on it we found an error that didn't show up and it only showed up for this one function that has gotten added to the release and it only showed up in IE8 localized to Spanish the reality is you're not gonna be able to test for all those things for in dev ops they've moved to things called canary releases or feature toggles which enables you to put a small amount of traffic on there to find those things that you wouldn't find otherwise and it's not really a license to say that you shouldn't test and validate things before you go into production but it's a realization that you're never gonna find absolutely everything and you need a way to release code into production without putting a lot of different things at risk so we've talked about a lot of different things here and we put them into place you can't really go do all of these things at once so I tend to go into organizations and talk to them about metrics around each of these pieces to understand where their biggest bottlenecks are and where their biggest issues we talk about metrics around the planning capacity we talk about metrics around the environments about the testing about the release and about the monitoring and we look across those with everybody on the team and we try to get a good feel for where the biggest bottlenecks are and what improvements will have the biggest impact on business and this is all the inefficiencies and priorities that you can have for a team of one developer but I imagine most of you out there have larger teams than one and you need a more efficient process for those as we start to scale the team what we're really trying to do is take a business idea scale it across a bunch of developers and integrate it into together and make sure that it all works together this is a lot of what the scrum process was designed to do is closer collaboration across the developers and closer collaboration with the business and the developers to make sure that we're getting that feedback on an ongoing basis this really though does need a big cultural changes one developers need to be integrating their code together on a more frequent basis than a CI environment they need to be prioritizing keeping that build green on an ongoing basis and to do that they need to learn to develop in different ways they need to not modify services but version services they need to go to evolutionary databases they need to prioritize keeping the code base running over just shoving the functionality in and they need to figure out how do they bring that code in without changing it and this is one of the biggest cultural changes that you're gonna see in the organization as you go through this process next this is scaling to a small team of developers and in organizations depending on the size of it some of these teams can independently develop qualifying deploy code in other organizations the architecture so tightly coupled that it takes larger groups of team working together first thing I do is try to really segment your applications into those different types of group if it's a non-business critical application that you're not updating not changing very frequently I wouldn't bother to do the investment for the change management to get here and I'd segment those out and not even work on the changes there next you can look at loosely coupled and tightly coupled loosely coupled are these small teams that can work independently and this is a lot of what you hear about in DevOps where you empower the teams you remove the barriers you get out of the way you give the developers a pager let them fill the empathy let them keep things running and then you have large type of coupled systems and for the loosely coupled systems it's a lot about just that CI process being able to move into production and letting these teams work independently you may decide that there's some common guardrails for your organizations you may decide that you wanna use some common tool sets like GitLab so you can look across them but in a lot of cases you are gonna wanna empower those teams to do what they seem to think best and be able to move quickly where I get involved with organizations frequently though is these organizations that have large tightly coupled architectures and ideally you'd wanna just do the same thing where you're doing a very frequent builds continuous integration process but what you find with these large tightly coupled systems is that you've got so many developers in each build it's too hard to debug it because you've got a lot of developers working on it and the time it takes to do a build deploy and test cycle larger and you end up with too many change sets in each iteration to make the debug and triage effort more efficiently. Ideally you'd go to a smaller icon build acceptance test that just really a kind of a smoke test type of thing to make sure that it always passes that stability but even that is not enough frequently and in those cases what you try to do is look for clean areas in your architecture where you can break this down into sub components and work through that and use service virtualization to break those apart and what you're really trying to do is break this down into subsystems of different pieces that you keep stable and then when you have each subsystem what you're really trying to do is create a deployment pipeline for that. So for each component each team in there each CI process they go through a gate before it goes into phase two and phase three of this pipeline or stage two and stage three and you do that for each component as you go through and there's a couple of key things that you're trying to do here when you use that stop light at each component you're doing two things you're increasing the frequency and quality of that feedback you're also making it much easier to debug and triage because it is associated with that one small team and additionally you're keeping these large complex enterprise environments more stable by gating defects from making their way into that process you do this for each subsystem and then each subsystem you take this and you build that up into your enterprise system that you can release in the production each stage and this is what I really think about and work on and as I go into organizations really trying to get clear about the deployment pipeline and what we're trying to do here after we've got the architecture understood the deployment pipelines figured out we really start to take and put on top of the metrics that we looked earlier what are the things associated with this complex pipeline? One of the first things I look at for Waste and Organizations is branches ideally you want all the developers putting their code together to make sure it works with each other and provide that frequent feedback what I see in a lot of organizations is that they have let those integrated environments become so unstable because they're not gating code that it's forced them to create a bunch of different branches so people have stable places to work in this case I tend to spend a lot of time with organizations you know they have the branches for a reason but every time you have a branch it represents duplicate work somebody's got to check code into it somebody's got to do testing on it somebody's got to work through it you got a Debo, you got a triage, you got to clean it up and what I tell people is branches are evil branches are evil, branches are evil every time you see them you should think of that as waste and duplication in your organization and they're there for a reason but we need to work on the reasons that are there and work to eliminate them over time so you can gain those efficiencies the next thing I look at is just what is the time that it takes for code to go from a development environment all the way out into production and in this example you know stage four for your integrated thing is going in four hours for bat tests that's pretty good your regression test is taking two weeks that may be a good opportunity to automate deploying into production is taking 18 hours you're not going to deploy on a very frequent basis if it's taking you 18 hours to deploy into production and subsystem two is taking three days and if you look at that it's a day to deploy in two days to test that may be an opportunity to look at automating it and this is assuming that you're deploying test times with the bottlenecks what we see in a lot of organizations is they don't even move into these integrated phases until they reach a milestone in their development process something some people call it functionality complete so you wouldn't even go into an integrated stage for environment until you reach functionality complete and that can be weeks or months after people started writing the code so the other thing that we look at is once we understand the time that it takes for code to flow through the deployment pipeline we look at where we're finding the defects are we finding 50, 60% of the defects in stage five and the regression testing and that's happening six weeks after somebody was writing code if you go to somebody six weeks after they're writing code and beat them up for a defect they entered in the system they're not even gonna remember what they're doing and it's gonna take you a long time to triage and figure out who the individual was in the first place if you can make that much more frequent and give that feedback and try to find it as close to the developer as you can ideally if you catch it in subsystem one it makes it much easier to triage and the feedback's quicker and you work through that process so one of the biggest things I see in organizations is just the delay in that feedback and these are the metrics and these are the maps that we use to find that out so that we know which process improvements to go after first and finally, a lot of times we'll wanna start gating code assuming that DevOps is all about keeping built green and working on that and the developers need to do that but one of the things I find frequently is we'll go down this journey and we'll start to pick a stage and start to increase the frequency to build test cycle and what we'll go into and realize is that not very many of the things that are occurring are problems with the code if you look in stage four a very small percentage here is code the majority of it is tests I go into very few organizations that really start gating their code with test automation that don't have to go through and rework the test framework and the design patterns for their test automation and this is one of the first things we find out and this one, there's also a lot of environment issues and deployment issues if you ask the developers to try to keep this build green you're asking them to fix things that are outside of their control so we really need to make sure that we get a solid test framework and environment and both different types of things and this points us to you can't do infrastructure as code, test automation all these things at once as we get down this journey and we start to get these metrics and know what's going on we use that to prioritize the things that are causing the most waste and inefficiency in the organization because if you've got almost half your defects or environment and test you need to fix that because those are the types of things that your organization are having to brute force their way through an ongoing basis next, you can't even do each subsystem all at the same time so we tend to look at the subsystem that's causing most of the defects and going through that and as you're going through and trying to set up one of the big complex enterprise environments it's important to understand that those are differently hopefully I frame that out as you have loosely coupled system that's more about empowerment as you have tightly coupled systems it's more important to have leadership to coordinate the work across the system map these out, understand the inefficiencies you have a lot of generalists with the small teams and you want to put operational capabilities on that you're more likely to have specialization as you get into a large tightly coupled system somebody's going to need to set these up and work through it and there's other pieces that are there that are different but I think you will find if you get these maps you identify the inefficiencies what you're really going to do is get a good understanding of a common view of the elephant and people know what you're going to be going after there and if you go down the process and you use these approach I think you're going to be able to deliver these types of improvements to your business I think these are real these represent what's going on in the industry and it is what you can expect it's going to be a journey you're going to have to free up capacity to sharpen the saw and make these process improvements but the breakthroughs that I personally experienced and I've watched my clients experience are just dramatic and are certainly worth the effort So I'm going to pause there we're going to do another one of these next month where John and I are really going to go in and have discussions around what is a typical transformation look like what are the types of things I'm seeing what are the types of things and then get more questions but we have some time here today so I think let's open it up to questions if I'm right John Thank you so much Hi, sorry Hi, thank you so much so we'd like to take some time now to answer any questions you have so please submit your questions using the Q&A box at the bottom of your screen and our first question is from Ben who asks what does a DevOps workflow look like in a small company with only a handful of devs and IT staff where everyone wears many hats I think that's the more loosely coupled thing that I said especially if you haven't created a big internal data warehouse I see organizations move to Amazon web services or Azure or Google or one of those cloud environments fairly quickly and I can see them getting it up and working so it's more of that continuous integration monitoring and feedback and it's empowering the team and I find organizations get there fairly quickly and those will always be the most efficient they'll deploy more they'll deploy more frequently I tend to like the large, tightly coupled, complex ones just because not that the level move faster but just because the level of inefficiencies that you have trying to coordinate hundreds of people is just so much more pronounced than the inefficiencies you have coordinating a few people Thank you so much Gary, can I elaborate on that? What about where you've got a situation where you've got a lot of people wearing multiple hats so you're doing everything from operation you're wearing the hat of operations and security and other things and I think that's a part of this how do you see that as where someone has to do multiple things and how DevOps can help them? You know, I think that's key I think that's where DevOps started really if you look at the Netflix or the Amazons of the world what they're really saying is if you can do this in a small team then you're gonna have people having that broader view and you're gonna need people that become what's referred to as full stack developers they know all the different pieces inside and out the thing that I would do is leverage a lot of those cloud capabilities if you can because that's gonna keep you from reinventing those capabilities Has that worked on? Yeah, I think that helps That's a good question Go ahead, sorry Back to you, sir Thank you both so much So our next question is how would you recommend testing the integration between a new internal app to a large legacy ERP? Typically, I see suggested to abstract the connection to the legacy app and mock the API layer that this still leaves the abstraction relatively untested I tend to do... I think that's correct You know, if you looked at it we tested each subsystem with service virtualization but I would still put the complete thing together once a day and test it in I would try to find as many of the things as I could in the subsystem first but I don't think that that allows you to not do the end-to-end testing because of what you're saying You're gonna find you thought you had the service virtualization right you thought you got it correct but when you put it together you had different interpretations and you don't want to wait till the end to find those things and you know, you can do ERP type of stuff I'm not sure if people I think it was like 2016 Rosalind Radcliffe that does San Francisco gave an excellent of how to do DevOps on a mainframe and it's possible and you should be doing it There's nothing bad about mainframes we just shouldn't be doing development along the same way we did it 30 years ago and she kind of really pulls that apart but you know, you still need to be putting the system together and testing it in Did I get that one, John? I think so I think it was a good answer I mean, you do have to have that end-to-end test you can't just rely on service virtualization for everything I mean, yes, service virtualization gets you going faster and while you answer the next question I'm gonna look up Rosalind's talk it was a really good talk and I agree with you Okay I'll put the link in the chat Okay All right, Gary, our next question What's an ideal duration for BAT and regression? You know, at HP, we started with our regression with manual and took about six weeks when we finished the transformation regression would take 24 hours and it was, you know, farmed it was probably 20,000 hours of testing farmed across a bunch of VMs that we ran every night and made sure we got there We got our BAT testing a build and a BAT testing and this was, you know, 10 million lines of code and 400 plus developers around the world got to the point where it was about three hours for that process to sort of go and get through and, you know, if Jess was here he'd be screaming up and down and yelling and saying it needs to be less than a minute or two minutes and I think that really comes from thinking about how these small teams do it and do it and keep it stable and what I found in large organizations it's almost more important to keep that build stable and get stuff from coming in than it was for speed so we found that balance and that's gonna be unique for your different organizations you know, if you're releasing daily then Jess is probably right or less than daily, Jess is probably right you need it minutes but if you're releasing once a week once a month type of thing and you're trying to keep it more stable and you have, you know, hundreds of developers that you got to triage through it's probably more important to lean a little more towards stability and better coverage than it is towards speed Thank you so much, Gary Thank you all so much for joining us We hope you enjoyed hearing from Gary Groover and that you will join us for our next webcast in which John and Gary engage in a DevOps dialogue If you'd like to learn more please visit about.gitlab.com slash DevOps Thank you again for joining us, bye