 Hello, and welcome everyone. Thanks for joining us today. We're excited to have you on. During today's webcast, we will be hosting a panel discussion on DevOps within the public sector. My name is Agnes, and I work on the marketing programs team here at GitLab. I'm joining you from San Jose, California today. Before we get started, I'm going to cover a couple of housekeeping items. First, feel free to ask questions throughout the presentation. You may use the Q&A function at the bottom of your screen for that. We'll have a dedicated time for questions at the end of the panel discussion, but you may go ahead and send in your questions as you think of them and we'll make sure to get to them at the end. If you are experiencing any technical difficulties, you can use the chat function to get in touch with me, the moderator for help. Now I'm going to turn it over to Tom Sutter, who is the president and founder of ATARC, which is the Advanced Technology Academic Research Center. Tom is a two-time winner of a Federal 100 Award, which is a prestigious honor that recognizes government and industry leaders who played pivotal roles in federal IT. He also co-authors 27 white papers related to DevOps and IT, and he serves on a variety of technology advisory boards. Over to you, Tom. Thank you, Agnes, again. I run the Advanced Technology Academic Research Center, and we have a lot of fun bringing in government industry and academia to solve the big problems that the federal government runs into. We're talking cloud, we deal with mobile, we deal with AI, and of course we deal with the DevOps aspect. We run a conference every year in the federal DevOps summit, and it's been the fastest growing summit that we've ever had. So we've got over 200 government people there. So we see there's a lot of interest in there. So we're really excited to do this today. We've got a great panel. We definitely welcome your questions. We were talking to John, and he's out of Amsterdam, and we were talking about challenging questions. So we really want to challenge him and the rest of the panel with these questions. Feel free, and I'll try to interject you in there. So we're going to start off with everybody making introductions, and let's go ahead and talk to Rob Brown first with USCIS and find out a little bit about what Rob's doing about himself and a little bit about what he's doing over there at DHS. Rob, if you can go ahead. Thanks, Tom. Started at USCIS actually working with Booz Allen Hamilton. As a contractor, primary focus was standing up essentially a center of excellence training various sort of activities focused on DevOps and DevOps adoption. From technical tooling, automation, to just sort of even to understanding what Git really is. And after some successes and actually rolling out some large platforms in capacity as a contractor became a federal employee about a year and a half ago. And I've taken the charge in many phases from being a division chief and senior solutions architect for enterprise infrastructure. And now in the enterprise architect for USCIS and moving towards essentially a domain driven design event driven architecture at USCIS that essentially transcends all of OIT and business portfolios trying to do a lot of alignment. At that same time, trying to help move from a rogue nature of multiple orchestration tools and platforms to a more mature platforms that multiple teams or multi-tenant teams can actually leverage. So one of the missions that we have is essentially e-processing for USCIS, essentially digitizing all collaboration benefits at USCIS. That's sort of the existing director mandate we're working towards. I'm trying to have that completed by 2020. Great. And if I can ask a quick question, you know, Mark Schwartz, we worked with him and he was one of the DevOps leaders. What are some of the challenges of being a leader? You guys really went in the DevOps trail very, very early. What are some of those challenges that you've run into just being first in so many ways? I think first was education and getting everybody to understand the value and benefit. A lot of folks were stuck in non-agile waterfall-like program management practices from a sort of operations perspective, pretty antiquated monitoring, still a lot of segregation of duties. And even if I would even say really antiquated ITIL practices, we're trying to overcome all of that. One of the successes that Mr. Schwartz essentially provided was a very tactical approach to essentially go and run as quickly as possible. Adopt as many new technologies as you can and experiment as often as possible as well. So actually changing a bit of the culture in the mindset really did help us. But those challenges still exist today and trying to move folks to, again, not so much turning screws but delivering code in an organized fashion. I know we'll talk about compliance a little bit later, but that also has been a large challenge for us being in a federal agency and dealing with all the rules, regulations, and just security in general has been a challenge in our dev ops or dev sec ops adoption. Great. Thank you. And now we'll turn it over to Leo, who is in a location that you can guess doesn't allow video, but we were lucky enough to get him by audio today. Welcome, Leo. Hey, how's everyone doing? I'm Leo Garcia. I'm the Chief Technology Officer at the Joint Improvised Threat Defeat Organization, aligned under the Defense Threat Reduction Agency. And I'm on, I would say probably month 18 of my dev ops journey. We've been up and running a pretty mature pipeline now for about a year. It's been an interesting journey to say the least. And I think, you know, kind of where I'm pushing now, and my big focus, right, besides just expanding kind of our ecosystem out to a broader community, i.e. not just the folks that sit in my building, but out as far as to the technical edges I can, is to really look at how we can democratize some of the capabilities we've built, right? Not only from a pipeline to be able to push software analytics out faster, but push that all the way out to the edge so we can have functionals out at the edge really, really doing a lot of that heavy lifting for the analytics that they're looking at. And since we focus so hard on building out that secure dev ops infrastructure and that pipeline and really wrapping it around kind of that DOD flavor, pushing it as far out to the edges we can. And other than that, I think I'll leave that because I want to get to the bigger discussion. Yeah, I just maybe one quick question. I think that tactical edge is kind of underestimated. How do you get these capabilities to somebody in a that's disconnected from the internet, maybe go in a little bit more about, maybe a little bit more about that would be great. Sure. Yeah, definitely. I think, you know, when you look at how commercial space is doing right now, right, very distributed development. I think one of the opportunities that we have once, once you've kind of matured and modernize the infrastructure to really have that type of ecosystem to support dev ops, then there's there's a lot of tooling out there and automation that's already ready to go, where you can actually push it out as far as you want right where you don't have to have a team sit in the same, you know, build in next to the same knock. Just just trying to build capabilities out. So two pieces that right one is live real time so if you have a functional sitting on the edge somewhere who can get to like Jupiter notebook or something like that. Be able to actually write up analytics on the fly against the stack and then use the rest of the pipeline to actually get that out to production to the rest of the force. I think the other piece on the disconnected ops right I think similar right we have a bunch of capabilities that are sitting out there right now that allow you to build some of those analytics out. Disconnected and either once reconnected or even just hand pass being able to bring that back to the enterprise and quickly pushing it through a pipeline to get it on to production. I think that's that's a big deal. I think that's where the future is going to be. And I think we'll start treating a lot of this, you know, right now it's kind of cool and everyone's excited. I think in two three years we're going to be treating the DevOps infrastructure just as another piece of our infrastructure on the day to day. So my focus is pushing that out as far as I can. Fantastic and batting cleanup we have with this John Jeremiah and John is tell us where you are right now in your travels that's just kind of interesting. Absolutely Tommy I'm super excited to be able to join everyone today I happen to be. I live in Florida, just just south of Cape Canaveral, which is kind of a fun place to live because I get to watch rockets go off all the time but today I'm in Amsterdam and I am traveling with one of the thought leaders who's been writing a lot about DevOps and I've been working with Gary Gruber for a number of years, frankly. And Gary and I have been doing road shows going and talking to customers and helping people to learn about where they're at in their DevOps journey and how do they go on that transformation we've we were talking with banks and heavily regulated industries and government agencies we had and then yesterday where we had the Department of Work and Pensions and we had government agencies talking about how they're how they're on their own journey to do the transformation and what what I've experienced in my career as a software engineer and it leader. I haven't always been a vendor I used to actually implement software and was leading transformations in a financial services organization before I joined HP and then started to lead marketing and talk about marketing to help people understand how different tools can help them. And the thing that I find to be most amazing and fascinating about how people are adopting DevOps is how it becomes a combination of cultural change, along with think embracing continuous improvement and technology together are enabling really amazing results. And I'm excited to be part of this panel and part of this discussion. And with a little ways back in my career I was a naval officer and I, I was on the receiving end of a lot of the technology that that this team builds as a communications officer and and really receiving the technology and using it to help accomplish and achieve missions so I understand speed to mission and I'm excited to be part of the part of the panel. And I've got a quick question for you. Where do we, where does the federal government compare to the rest of, you know, the world in this DevOps journey are they, you know, they always say the government's behind how far behind are we are we behind. It'd be good to get some perspectives on where we compare to other sectors. Tom it's a great great question I think it in a typical consulting sort of answer it depends. It depends on the agency it depends on the leadership. You know when when leaders take on organizations and start to drive a transformation mark is a great example, and how Mark helps to drive a transformation, you know, at DHS and how, how they evolved there and adopted and embraced DevOps so there are evangelists who are driving transformations. And I think there are other parts of the, of the federal government that are laggards and I think this is exactly the same as true in in the, in the civilian side as well in the commercial where there are organizations that are the culture is resistant to change they don't feel the need to go faster. It's very similar I think there's a lot of parallels I don't think I don't think in general, the government is that far beyond. Sometimes the government thinks they're further behind than they are in every every kind of aspect and really they probably aren't first, but they're they're definitely not not too far behind and before we get into the Q&A, just from my observations I've seen the entire the entire federal government. I do think that the biggest issue we have. I definitely think it's the culture and I think the development cycles from doing a lot of software development myself is, you know, we're kind of used to waterfall we're kind of used to building out a process and security kind of comes at the end and I think it's doing those corrections at the end that cost about 30 times as much. So I think it's just getting this getting this culture changes obviously I think one of the biggest challenges. I think the good tech is out there. So anyway, I will, I have a few questions but I would love to get some audience participation. But anyway, the first question I have is in maybe Leo you can touch on it. What is the main cultural challenges faced by the public sector when it developing a dev ops practice, how do I even get started on this. And Lee if you could flesh that out a little bit and any of the other panels can chip in. Sure, definitely and I'm glad you said what you said because because I'm with you right I think one of the things that we see in the federal space is that kind of traditional model and that's what we've been used to. And around that is the fact that both both how we acquire capabilities right whether it be software or or an integrator come in and how we contract is definitely very much tied. And, and then it's own little box that that really supports waterfall and that traditional model. So, so how do you get there right so it's interesting right the cultural piece. It's challenging right it's and it's a it's a people thing more than anything else I think what I tell folks when folks kind of asked me hey how did you get from a to be besides having a smart answer very deliberately. I think it really is about doing that transition and some big chunks right and and those big chunks are the ones that that kind of help you get there. And again this is at the enterprise level not on a small project. I think that first thing was the transition right so transition from from a more traditional waterfall. To an agile approach, I think that that was kind of our first step and the first step that we took on our journey and that and that was a little bit further down line right that was about four years ago. And what that did is it drove everything else right it drove the changes to our processes to our tactic 60 procedures to kind of tools that that we were using to just see ourselves and get capability out the door. We reshaped the policy that we had, but we started thinking about what a next generation kind of contract construct would look like or even organizational construct. And that was the first step that next step really is to build out an ecosystem. Right, and what I mean by that is you know, if you go into commercial space right companies build their DevOps pipelines very focused on the type of capabilities that they're building right so if you're a commercial entity that's pushing out web apps right you're going to kind of tool yourself to focus on web apps and put some left and right limits around that. That was really our next big piece is to start building out that ecosystem knowing that we were moving in the direction to kind of make our entire enterprise really follow those base agile principles. And what building out that ecosystem did was it showed us that, hey, we can work the configuration management piece we can work the process piece we can get all our contracts lined up to do agile. We can bring all these tools, but in a 24 month period we went down from like 92 days to get something out the door accredited to about 23 days, and we couldn't move any faster than 23 days. And that's really when about little less than two years ago, we said hey, you know the demand signal is we got to we got to push capabilities out even faster. So how do we get there. And that's what we really started looking at, building out that that automation piece and starting to bring some of these DevOps capabilities in. But I think you got to do it in in subtle chunks to get there and the only reason I say that is. So, if you look at most traditional enterprises right their policies their processes that the skill set of folks they have is for what the enterprise has today. So you do have to do a pretty deliberate transition into the space that being said right 24 months is pretty fast for government agency. But, but some of that really was, you know, having that initial shift agile that build out of the ecosystem. Those pieces parts kind of laid the foundation to really have a scalable enterprise or very quickly we could move to to a much more automated fashion. I think the other piece around that right culturally is is a shift and you hit it right on right so so that initial shift to agile caused us to move our security to the left right of from a process perspective. And as we moved into the DevOps space we very quickly realized that we had to move it all the way to the front right so as you had those daily scrums right you're authorizing official on your network was sitting having a discussion. Right and really thinking through what risks, just based on what was being developed was going to be on the network and and making those decisions up front. And then being able to go and put those thresholds within your pipeline to get capability out I think that that probably was the biggest cultural challenges, shifting where people were in the process. Right. And, and integrating them to be able to get stuff out the door. And that's important right I think you see a you see a shift over over several months to really matrixing your folks in the right place at the right time to get capability out the door I think the cultural challenges are real. I think we and we still see them right we saw some some larger challenges with that acquisition community as we try to to to move stuff through the process that we're still fit I think we're still figuring out. And it's getting shaped by by the delivery process on a on a daily basis does that answer your question. I think that I can't see everybody on the other end but I think everybody was shaking their head and cordons on the cultural issues challenges in the federal government. Robert john you want to add anything Leo responded with. I'm happy to. I think when we started is why skills was the right skill set for my, you know, having that skill set matching with the right culture as we're trying to adopt it was a big challenge so a lot of education training illustrating benefits, going through that. Definitely just adopting automation and letting everybody understand and showing that true value and illustrating that you still do have a job. After you start to automate some of those tasks that you've been doing in a manual fashion. So that I think you know having a story to tell and showing and showcasing those examples for us really did make a difference. And sort of moving that culture moving that needle to to more of an automated manner for these folks and really to help set the station foundation for truly like that DevSecOps. The other the other cultural challenges that we that we faced was just in communication and having teams that are from different companies, you know, we're in an organization where a lot of the folks are contractors and they're from different companies and some of them are competitors, some of them are friendly, some of them are company mates, trying to navigate those waters and sort of set a baseline of like this is for the mission guys this is not just for the profit of the company. And so overcoming that I think is was also a large challenge. And I think one of the ways we overcame that was to be able to focus on the technology and change from more of a static communication of constantly shifting or moving things around an email and starting to leverage the technology that we had or adopting new technology. So we had real time comms that transcended essentially the silos of people and contracts and projects where it was more mission based that really helped a lot. So that real time comms and understanding that there was real technical challenges to solve that again was for the agency and for the mission and sort of capitulating those those challenges in that manner really helped us and that we see that continuing to grow today. I can only imagine you don't want to gain a throne situation with all your contractors. Yeah, and, and this is Leo you know, and it's interesting right so my philosophy has always been when you've got a workforce and you're doing a major transformation is re skill retool or redeploy right those are your choices. And I think we tend to take, especially in the federal space a traditional view right where we'll send them to this class. I think where we saw a lot of list was by actually looking at it a little bit different from the training perspective is really getting folks plugged in and stuff that was happening with Sarah and ed X and some of some of these free gate courses that were out there, and then really pushing getting to meetups right so focusing on the meetups that were happening in the national capital region, and getting folks engaged with the community that was moving, not necessarily totally inside the federal space, right but with the experts and really gaining that that that blue network right and growing it. And that's been a huge help for us that kind of really helps us with the transformation. And on the integrators piece right there's there's some great options out there. I think that one of the things that we found as we moved into the DevOps space was right because you're going to have different contracts that that's just real is really focusing on building really smart service level agreements. With all of our vendors and and build in what I would say would be across task order service level agreements right that's that's a relationship between the government and and the integrators. But I think right still focused on mission with the mission being the primary focus has been the driver, but but there are some tools that tidy up very nicely with with having kind of DevOps capability on your enterprise. John anything you want to add from the commercial perspective and how they've handled these challenges. You know, I think the reality that I see when I talk to customers and in all sorts of different industries is that the cultural change is often one of the largest. They've spent many people spent large parts of their careers and building software in a waterfall way or thinking in a waterfall way or a, or maybe even a water scrum fall way where they're trying to dabble with agile but they really haven't committed. And the reality is that they've learned that the big fat document you know the BFD that they would write was usually the Bible they write the requirements that they write the design they've learned a process of doing that and having them let go of that requires leadership to give to give them different goals and a different objective, if you will, a mission for them about going faster and evolving. And what I think is amazing and you know when the speed at which Leo talked about the change in the transformation is a lot of organizations it takes them years to start to incrementally go faster where they start to apply some of the learnings. They make mistakes they learn they start to go faster and it becomes it's truly a pattern of of crawl walk run in order to achieve speed and that if you tell them about what it looks like to go fast. You know for us at get lab we're releasing on our website 60 times a day, we're deploying, you know, we're just all the time. And for people who aren't anywhere near that they think that's crazy they think that's insane. And it requires a maturity to get there and evolution and I think that's the cultural challenges leaders of how do you take people on that journey. And so this this is true in the public sector, as well as is where you know where I see it across the board and commercial as well around the world. Thanks, John. Next question I have is how have the legal regulatory and policy changes. And let's add procurement to that question affected the world out of DevOps in your agency and maybe Rob you you can lead because I know that you want it's one of the things you've had as a leader you've run into that. Yeah, I'll speak high just a high level of procurement but we're really from legal regulatory really from just compliance in general. You know, we were from the federal perspective of course every other industry has some level of compliance. Finally, it's, you know, we're looking at the missed 853, the sort of the FSMA or FedRAMP compliance standards and then within DHS as well we've got our own layer of additional controls that we all need to meet. So I think, you know, we've, Leo's already brought this up, but understanding what those challenges are, how to move to something like ongoing authorization and building that into your pipeline. So really adopting that, you know, it's a pipeline not people perspective, taking on the the compliance is code perspective has helped us a lot. So we've actually, where does that stem from again it's back to infrastructures code and picking the right tools to manage that level of automation and having the right pipelines set up that sort of are parallel or part of your master pipeline and showing that evidence to your auditors and those regulatory bodies. That I can essentially be again immutable and not impotent at any step of the way that for us is proven to be effective in those building blocks to move forward. We've spent a lot of time energy and effort here to essentially create a lot of these building blocks that can be leveraged across the enterprise and are working with other DHS components to share a lot of this code. Essentially, these repositories so that they can leverage this as well just from hardening operating systems, hardening, sort of pervasive applications that actually have benchmarks that we can harden against something like red hat or windows to specific databases like post address. So actually moving in that into that direction and providing just that that repository or that code to other groups, not only within the organization but now essentially externally for us has been a huge win. And then essentially again recently adopting pipeline as code as well. So not only just having you know infrastructures code but also having our pipeline as code so that entire change management process is essentially documented and is a pipeline unto itself. Then provides those bodies that that complete audit ability so they can essentially see our move from a really a risk based task driven or task list driven sort of an approach to more of an outcome based approach and actually see the results at any point in time and really on demand. So, you know, I could go on and on about this forever, but I, you know, from a, you know, it's getting started I think it's getting those building blocks in place that can be leveraged across either even if it's just a team but again across an agency, promoting that and showing how efficient it is, as has helped us and helped essentially speed up our ability to, you know, achieve ATOs or authority to operate in a much faster timeframe. Now there is a bit of an activation energy up front of course, but I think over the past, you know, a few years, we've been successful in and having pipelines just for those core building blocks that a lot of our other teams and portfolios are taking advantage of. To the point now where we are essentially providing a lot of our development teams and operation teams, essentially a parameterized DML file for them to fill out where their endpoints are so they can move forward and not have that labor intensive, you know, initial energy just to get started with a project, an automated project. So I think we're moving towards a more advanced method of reusability as well as just, you know, continuous improvement of that pipeline and of the tooling that we've got. I would say that, you know, just to hand off as to where we're heading to deal with, you know, the constant changes in this sort of venue from a security perspective regulatory perspective is truly again more on the compliance as code. And so the 18F group, I'm happy to provide these links, the 18F group has done a very good job of illustrating how they've moved from essentially an authority to operate from six months to 30 days, and some key sort of principles and best practices that teams, groups and agencies can adopt. And then working with NIST, NIST has got, I think it's 870 documents that they published in February of this year, which is essentially a checklist that helps or can help guide a lot of groups down this journey, put them on the right path to deal with these, with these types of auditors, these types of regulations and not just within federal space but easily adopted for groups like PCI or SSA-16, et cetera. Sure. Good answer. Anybody, Leo, John, anything to add there? Yeah, I'm with Rob, I think. Go ahead, Leo. Yeah, no, I'm with Rob, right? Like, and that's where I talk about this, like building out that ecosystem, right? I think our big push because it was all about speed was not just that infrastructure code, but the whole process, right? Bring SRE into it. Make sure you can document as much as you can. Make sure you have visibility into how each piece of code, right? From birth to deployment and then to death can be seen on your enterprise and really just automate all that, right? There should be no Excel spreadsheet that tells you where stuff is or what release you're on. That's insane and absolutely makes no sense when there is all this automation that's ready to go and available. And I think that's the key, right? When folks come and ask me, hey, all right, we're going to start here. I'm like, your end state should be continuous authorization, right? When you look at the NIST cybersecurity framework and DOD, the risk management framework, right, at the end of the day, that's what it tells you to do, right? Continuous monitoring, continuous deployment, continuous integration, which really gets to at that point when that means that you're doing continuous authorization. So I think that's one piece where the rules, right, and what's out there as our security framework is actually set up to support DevOps. I think that the challenge becomes how do you translate your policies, your processes, and how you approach the problem to get there? But that should be everybody's goal, which really is continuous authorization, because it's really getting that capability out to the customer. So that's a really big piece to it, and I'm glad other folks are moving in that direction. John? The thing I loved about both Rob's answer and Leo's answer too is I was listening, I was taking notes, and I was noting where were we talking about things that are unique and specific to public sector and to the government versus what commercial organizations are facing. And the thing that I absolutely loved about this discussion was 80% of everything we talked about is exactly what banks and healthcare organizations and commercial organizations are trying to do everywhere about trying to go faster and be able to meet compliance to be able to demonstrate their auditors that they're doing what they said they would do, and that they're able to prove that to them. So what you're doing and what everyone else is doing is so similar and so parallel. It's incredibly powerful. Fantastic, John. We've got a question from the audience. Caleb Cooper. How can you use DevOps to manage your fleet for multiple OS's on a variety of device types, servers, clusters, desktops, etc. What change management process do you use for day to day changes that fit into your DevOps workflow, and maybe we'll start with you, John. Well, I think it's a question of building out an automation. I mean, first off, a lot of what I see people taking advantage of is the ability to containerize and leverage Kubernetes and containers to help manage the deployments of environments and deployments of applications. When you're looking at, I think, managing down to the desktop or down to the actual endpoint, the actual final device, then it's a question really of building in the workflow and automation required in order to actually update those end devices. That becomes a key part of it. I'm not sure whether or not you were using DevOps at managing, you know, the actual applications on the deployed devices or the web applications where a lot of times people have the opportunity to go so much faster leveraging containers and containerized applications. Anybody else? And this is Leo. Automate it all, right? I mean, that's really what it boils down to. I think when I look at it, same way, right? I don't look at this as at the desktop level, though there are some SRE approaches to do that. And there's a lot of capability out there. But at the capability level, I think this really gets to, well, and containers are the biggest help, right? I mean, that really tidies up the enterprise in a way that makes it a lot easier. That's the approach that we've taken, but we've also pushed out some capabilities, not necessarily in containers, and we found that really automating that deployment process is important. And I think that, and it's a great question, right? Because I think my biggest lesson learned day one was, so we pushed our first tool through our pipeline, and yep, it's in production, but, you know, some DNS entries weren't made, some IPs were changed. So the rest of the network wasn't, you know, flexed with that deployment. And at that point, I stood back and said, well, this really isn't DevOps, so what do we need to do to get to that next step? And really, that's when you get into building out those YAML files, right? And doing the configuration stuff and baking that into the pipeline. So it becomes not just, right, it really is the true nature of infrastructure is code, where you're really pushing it out and exercising, not just the software that you're developing, but exercising the entire network to get capability out. So my stance is, automate everything. Everything that can be automated should be automated. It gives you better pedigree too, right? I think the other, and here's the piece that I think a lot of folks miss in this. My first aha moment after deploying a couple of things on the network was sitting back and looking at a screen and thinking, wow, this is the first time ever that I really feel comfortable, that I have full situational awareness and understanding of what's been done, how it's been deployed, what the risks are, right, and how that fits into my programmatic portfolio. That's really powerful. It really does bring that next level of governance to your enterprise, because you can see that data in real time and where things are at all times. All right, that might be tip of the day, automate everything. I like that. I wrote that down. So what is the hardest challenge you had to overcome and how did you navigate it in your DevOps journey? And each of you can take that on and maybe we'll start off with you, Leo. People, right? I think the people challenge was hard. Some of this stuff is actually really scary for a lot of folks, right? I think Rob said it up front, the concern that people have that, hey, this box is taking over my job or this automation is going to take over. I think what we saw was quite the opposite, though. I think we saw people being remissioned and rebranded, leveraging their skill sets in different areas. But I think that was probably one of our biggest challenges was kind of looking at not just the automation but the process changes and being able to document that, right, bring it into our day to day operations in a smart way. And have folks really be on board with where we were going. And it's hard, right? It is fundamental organizational transformation. So it takes some time. And I would say too, I think something that we miss a lot is some of our naysayers were actually now are some of our biggest advocates because they'd learned, right? They made mistakes in the past and they brought up good things that made us kind of redirect some of our process and policies and the way we did business in a much safer manner than I think if you just got a bunch of technologists in the room. So I think the change, right? It is a drastic change. And working through that change is probably the biggest thing and that's a people piece. And again, right? We focused on going out there and having a little bit of a different flavor to bring folks up to speed and using some what I would consider non-traditional means from a government perspective. And I think the other thing too is getting people out and talking to folks in the commercial space who are doing it, right? And really getting them engaged and building out that thick blue network of folks who are in the game. Because what we found there is building those relationships really did help us a lot and kind of soften in the transformation. Leo, that last thing you said, I think we've had the repercussions of the GSA conference scandal where I think they're going to see the vendors and the training and some of those things. Some organizations still are like, oh, I don't send people outside of my office to learn because it's waste of time. And I think we definitely, we've tried to do that at ATARC. It's really turned that around. You need to collaborate with other government colleagues, but you also need to collaborate with the industry colleagues. Great. And Rob, do you have anything to add to that thread? To echo what Leo stated, people is always, anything technology is relatively easy. It's always the people. I can share an anecdote, you know, and a sort of a tool and approach that we took here to help with that. We actually started to help with this adoption and help with, you know, the cultural bit, the people part was we, this was Mark Schwartz, we created first essentially, essentially it was a management instruction. So it wasn't per se a policy, but it was a, you know, a CIO set of instructions policy like to help people get spurred along. We started with just a sort of an agile policy or management instruction. And then I was lucky enough to participate in the next management instruction, which essentially provided a carrot for development teams, as well as operators. If they followed these sort of best practices, essentially where we've got a release readiness review, which is a governance gate before you can deploy to production. So if teams portfolio started to adhere to essentially continuous integration, automated testing, certain number of deployments per, per every two weeks, they would actually have this release readiness review gate completely removed, essentially giving them the ability to deploy at will. So we use some of those types of tools per se, to help spur along these teams and get them, you know, not only moving and ready to move to get rid of these crappy governance gates, which are just waterfall in nature, but to empower those teams to take on the challenge of meeting these essentially DevOps 101 criteria from at least a continuous integration perspective. That to us was awesome. And we have since moved on to another sort of management instruction, essentially promoting from an outcome perspective, DevSecOps, and again providing a bit of a carrot for all the teams here, both on the dev and operation side to follow some of these best practices and have some of these governance gates removed so they are empowered to deliver. Great. John, you have anything to add. What are you doing. I'm curious as you talk about it because I love I love the conversation I agree with people are the hardest part of the whole thing and the idea about making the right path, the right way to do it the easy way and make it that the easy path people will choose that over the hard way. What are you doing as far as there are you doing anything in regards to doing DevOps training and DevOps dojo. I know target implemented a dojo model a few years ago. When I was an HPE we had a dojo where we would bring teams in and teach them the principles of lean and is that something that you see happening in the federal sector as well. Totally. The first contract that was on here was actually the end part of standing up a center of excellence which included a variety of just general training that could be a couple of hours of just sort of listening to full hands on workshops that would cover things like agile and lean to really DevOps principles like understanding how to build a pipeline, understanding what a Jenkins file is and how to actually move that into your into your pipeline as opposed to being sort of drag and drop and click and move around as well as just some fundamental infrastructures code some various tools that we use here and how to get started with that. So we definitely have a pretty, pretty a large wealth of training that's available to federal and contracting staff alike to get them up to speed and where it may seem they get to choose again that's a huge spot. I think some of the most recent additions we actually have is some serverless training as it relates to how you can use serverless and some of your to essentially, you know, bait the need to have a standard pipeline for apps that just don't need to be apps. We've adopted and have adopted kubernetes and production here for the year and a half. So getting giving people a good understanding, not just how containerization works but what orchestration looks like in a k8 world. How to construct proper YAML files, how they would leverage or how we stood up kubernetes here and how to take advantage of the wildcard certificates and essentially onboarding those tenants and just focus on deploying and working on quality code. They don't have to worry about anything else but still giving them that that comfort level or that confidence level that that this is how things work. Yeah, I guess that's that's key because you can't just compile, you know, cultural change or people change it's not a kid compile and people just operate differently. You've got to continue to reiterate and teach and train so that that's but that's happening in the civilian sector it's happening across the board and I love the I love what you're talking about here. It's common. It's a common problem. It's a common solution. Good thread you pulled there John and that'll go into the next question I have is how do you get started. We probably should have asked that first but how do you, how do you get started you're you've been parachuted in a new organization they are their waterfall they've been doing things a certain way how do we get started I know Rob you mentioned the Center for excellence how maybe you can lead us off and then we can go to Leo for a bit. Sure. Yeah, we from and I can just speak to what we've done here at us as was again providing teams that ability to fail first of all, culturally and like failure is actually a success. So starting there I think was a big help. And we started off pretty, you know, every team essentially had their own Jenkins server they had their own tooling it was really Wild West. And I think, you know, depending upon your maturity level, where you are is going to probably help dictate where you need to be and what you need to start to adopt. And there's plenty of, you know, you can do a self assessment there's plenty of professional organizations out there like Dora that can help you understand your maturity level. But if you're, you know, ground floor depending upon the size of the group, I would, you know, budget some time to promote experimentation and power those folks to sort of Wild West a little bit and see where they fit in. What, what are the right tools? What is the right communication structure? How are they going to work properly with the goal of, you know, speed and quality at the, at the forefront. I think it's first and foremost, and then as, as we've matured, you know, I came in here and there was probably 100 to 200 Jenkins servers that were rampant, starting to do economies of scale and picking the right platforms. To help manage that we actually first started with aggregating essentially all of our code into one platform and promoting that platform, which is a good platform. So that this became essentially a hub for all activity, not just for developers but also for operations to include even something as simple as standing up a website on that platform. For folks to be educated and understand what was happening. So good, proper solid version control that can be centralized and aggregated that you can work from I think is first and foremost. And then you can start to move down the sort of that whole CI CD pipeline perspective. If you've got a bunch of different teams using a million different artifact repositories. Good luck. So start to aggregate on a good common set of dependency management tools and essentially a good supply chain for your developers and your operation teams to include a good registry where it's transparent for all and not just those individual teams. I think, you know, that's a probably a really good start. And then you can start to move into whatever flavor you want from a deployment perspective, depending upon where you're actually deploying to. Again, we focused on, you know, really doing a push towards microservices and event driven architecture. And so Kubernetes has been our choice of orchestration. And again, working within any sort of cloud service provider also provides you that flexibility and the ability to sort of work at that scale that everybody needs. Great, and Leo. Yeah, I'm with Rob. I mean, I think we took a very, a very similar approach. I think the only thing I would add to that is right scale matters, and there's not an easy template for this. I think the bigger thing from a getting started is a really focusing on what you want to deliver. Right. And what I mean by that is doing this at an enterprise level where as for any capability you push out is actually. It's really hard. I think starting off with really focused on, hey, I want to, I want to, you know, kind of build out a Delos pipeline to focus on delivering analytics to, to a big data stack or to push out to, you know, this web app or that web app. Right starts building out that ecosystem that slowly kind of gets you moving down that path. I think that's an important thing to think about. I think leadership buying to right and we've said it a couple of times part of this is, is really having a push from leadership from a mission perspective, ie, I need to get capability out faster I need to be more secure I need to be able to scale development. I think those things right there and having that that mission focus. I think that that's a luxury that we've had in the joint provider feed organization is that that focus on on on speed of delivery. Kind of naturally started pulling us in this direction. I think when you look at other enterprises that becomes a little bit more challenging when it when it may not necessarily be a speed thing. We're getting those lessons learned if you're in the federal space right talking to folks. There's a couple of us out there, we have pretty mature capability now, and being able to show that hey yeah this is possible we can do this in a safe and secure manner. Just really helps spread, you know, get that word out and get folks to start buying in. Yeah that that leadership things pretty important it's hard to do anything without that top cover from leadership that is on board I agree with that. We got a couple quick questions that maybe we can fit in before I can't believe it's almost one o'clock Eastern standard time already it's gone by quick. How do you manage the development of the architecture platform within the within the with the DevOps approaches. I think I understand what they're trying to ask. Anybody want to take a shot at that. If not, we'll move to the next question. Yeah, I know I know and I'll take a shot at this is Leo and that it that's actually a great question and it's not the first time I've been asked this question. Right that's the easy answer so so I think something to think about is right so the tools are always changing. New requirements that you may have are for the most part always changing. So you really have to be flexible my approach that has been and this this will be a little bit of a scrum fall for lack of a better term from a requirements perspective is to kind of work that out in cycles right so I try to work really in six month cycles where it's six months of hard and fast modernization the six months of hard and fast innovation, and then you go back to modernization right, where kind of your main effort or with those, you know, the big rocks that you're trying to tackle I think the reality is, and just because capabilities are changing so fast is you, you have to have some kind of organizational enterprise plan to get at that problem. Or if not why we'll run into kind of the problems are trying to solve with DevOps on on you know buying down tech debt and all that what you don't want to do is you don't want to buy a lot of tech debt into your pipeline either. So there is a certain amount of programmatic surround that to really focus on the longer term efforts which are not DevOps I think but making sure that you're moving that tooling, and, and do it in a smart way to keep up with that modernization cycle that that would be my answer. I mean, I'm just reading it the way I'm reading it thinking of, you know, some of the platforms we've already talked about, and this is a get lab webcast. And actually, I think the approach we've taken is that all of these types of tools and platforms live within an automated pipeline themselves. Whereas when we're doing an appointment of some sort of platform. It also is delivered in again in a completely automated fashion where applicable or where, if possible, some platforms don't allow for complete 100% automation. And at the point where we're actually doing for a lot of our Kubernetes work, everything is essentially a blue green deployment, where we have zero downtime for any of our tenants on that platform. So I think taking from it again, looking at platform and architecture, as things upgrade and as you do move from versions and or moving from different orchestration engine. Again, this is just part of, you know, development and engineering work that fits very nicely into any sort of pipeline that could exist. Great. And one more question from Ben Taylor. What tools and standards do you use to maintain and verify security compliance for containers. So mist mist actually has a very good security document 800-190. If you need to look it up. It's public. All these this documents are public. So from the standards. CIS also has some very good hardening security guidelines that you can follow as well. And there's plenty of tools out there that you can embed. Not only from a, you know, looking at what vulnerabilities you have and your image that may sit in a registry, but also as you spin that container up at runtime. And again, most of these tools can be embedded through API API calls into your pipeline. Great, great. Well, we've got about three minutes. Can we get the kind of the last word? I would always parse it is, if we do this webinar in five years, what are we going to be talking about? And maybe we can have you back clean up, John, and get our government colleagues first. Where would you like to be? Sorry, I had to throw a stumper in there at the end. Yeah, I think just like any sort of data that we that we would like to gather on the business side to provide insight into how we operate and move forward. I think we need to be thinking the same as we operationalize and move forward with our pipelines, the tools that we've got aggregate that data and provide meaningful, useful information to the developers, to the operators, to the security folks, the platform engineers, the SREs. And ultimately, I would hope to start to apply levels of AI and ML to that information that could be done in real time adjustments to how we do business. This is Leo. I hope that in five years, the conversation that we're having is how machine learning and some AI capabilities are allowing us to actually physically have the right less code and be able to go through our whole corpus of code that we're pushing out to really validate the level of security. I think that's the, that's the longer term piece that's missing. And it's probably the most, even now still time consuming piece from it from a human perspective is at the end of the day we have to triage stuff. Right. If it's a third party library and it's bad, a human has to go in there and do a lot of work to determine whether we want to put that out. I think that's, I hope that's what we're talking about. In fact, I hope we're not even talking about DevOps as I think it's just infrastructure at that point, right? It's just like buying a pizza box in your enterprise. And really, we are talking about stuff on top of that, right, whether it be on the analytics level, or really leveraging some new and emerging kind of techniques to look at code to make, get our quality and our security to go up. Thank you, Leo. And John, I'll give you the last word. That's a tough one. These are tough act to follow, Tom. So I'm going to do my best. I think really in five years from now, I would second the comments about using machine learning both to increase that our velocity and our DevOps pipelines and our and also to help us improve how we code. But I really hope that in five years that as we start to work together in a collaborative way that we're really unlocking the human potential that we're enabling our operators, our users, the people that use our systems to come together. And so we can innovate faster and more collaboratively. And that's something I think that is right now it's a challenge because there's so many different silos of the way we work and people working different systems in different places. And I hope the future and I hope that the future is really one in which all of us are able to come together and collaborate as a team and to work and to leverage the technology in order to be more effective and to solve more problems faster. So I think it's really about solving business problems. It's less about just technology for technology's sake, but it's about us solving the problems of the day and five years from now we'll have a whole new set of problems to solve and these will be pleasant history, I hope. Thank you, John. We're going to end it at that. Thank you, Rob, Leo, John, and I really want to thank the GitLab team. They're very, very dedicated to this educational process and it really matches the mission of ATARC and everybody have a good day. And I'd like to thank the audience who are very good questions. Appreciate the interaction and everybody have a good rest of the day. Thank you.