 All right. Good morning, everyone. Before we get started, I am obligated to read that the opinions, interpretations, conclusions, and recommendations are those of the authors. That's us. Do not necessarily reflect the official position or policy of the Air Force Department of Defense or the U.S. government. Now that that's out of the way, I'm Captain Brian Kroger, active duty Air Force. I spent seven years as an intelligence officer doing a lot of targeting, spent a lot of time in and around the Air Operations Centers, which we're going to talk about today. But now I find myself as an acquisition officer at Air Force Life Cycle Management Center. I'm on Project Kessel Run. I'm the chief operating officer. Basically means that I'm always covering down on the next most valuable thing that nobody else is working on. And I'm Victoria Galvin. I am a civil servant for the Air Force and have been in the acquisitions group for about six years now. And what I used to do before this job was price out the cost of major weapon systems. And what I do for Kessel Run now is run the chief, I am the chief of business. So I run the whole acquisitions arm of basically figuring out from requirements to funding and contracts. How can we make all three of those facilitate the delivery of software in an agile manner? And so today we're going to be talking about cost of delay. We're going to be talking about it at a strategic level. So not using it to prioritize features within the backlog, but rather to inform and guide, in our case, the Air Force, on how we should approach continuous delivery, why it's important, and what the return on investment is. And so first I'll talk a little bit about Kessel Run. It's an alliance. It's not just one program office. So we have the Air Operations Center program office, another one called the Targeting and Geo-It program office, both at Air Force Life Cycle Management Center. Also brought in expertise from Defense Innovation Unit Experimental and the Defense Digital Service. And ultimately we've come together and we're trying to find a new way. It's not a new way for a commercial sector but within the Air Force to approach continuous delivery of valuable software. So first I've got to start about talking the problem. And it's going to take me a minute to cage out the problem because we're talking about government. This is how we go about acquiring software in the Department of Defense. And shown a similar spaghetti slide, separate but related, General McChrystal said, when we understand it we'll have won the war. But if he was looking at this slide he'd be wrong because the most important thing is the fine print in the upper right corner which basically says, it's so complex you couldn't ever possibly hope to understand it in one slide. And this is how we acquire software. And there's a good reason for every little diagram, every little box, every little arrow on that chart. But you know the path to somewhere is paved with good intentions. What does it actually get us? That's the question we need to be asking. What's the outcome? And a lot of my peers when I spent time in intelligence, they thought that maybe my job looked like my family always wondering what do you do? I thought it was like a set out of CSI Miami, or maybe an episode of 24 with Jack Bauer. I got this ops room and I'm throwing things off my iPad and they're showing up on the walls. I've got holograms everywhere and it's really high tech. But actually the tools of choice for modern warfare, I don't even really need to show you because you're all very familiar with Microsoft Office. And it's kind of a jest but not really. Actually when we went out to the kayak, I can't tell the exact number, but we did wildcard search for PowerPoint and Excel documents and it's on the order of millions. And so we're using these, you know, PowerPoint is a visualization tool, right? We're doing a lot of our analytics, which is basically basic reporting in Excel. If you get really fancy, you might find somebody that knows access database, but then they leave and then everything's gone and you can't find the data. So all that said, we have an acquisitions community that's definitely like working hard, trying to get capabilities out to the field. The problem is by the time they get out, they're irrelevant. And so how did we get there? It's the project timeline. So when we talk about this study was done by the Institute of Defense Analysis back in 2016 and it looked across the broad spectrum over 25 years to field any major weapon system. So we're talking airplanes, submarines, tanks, and even software. The median cycle time from when the contract kicked off so like we awarded and someone begins work to when it was first fielded to the warfighter is eight years. It does not matter. There is no correlation between the type of system being developed. That is just where we stand as you can see in the scatter plot. That's not even the whole time frame. So before I can even let a contract out, you have to have a requirement. I have to be able to go submit to the president's budget, which takes an act of Congress. And then finally, I have to write the documentation. If I'm being nice, that's about a three to five year process before I even let out the contract. So from that median cycle time just went up from 11 to 13 years. So it takes us 11 to 13 years to even start that feedback. This isn't just a problem in the DOD. This is a study done by the United Kingdom. And when you're in stable times, you tend to build these really big complex system. What also causes that to happen is it takes me a really long time to field. So if I'm going to take a long time to field, I'm going to tell you everything I want because in 13 years that system better be the best thing that ever fielded. So it's super complex, which very much looks like any Star Wars fans in here. We start building Death Stars. And then if you move through the trilogy, it keeps getting bigger and bigger. And what we're trying to change with our Castle Run effort is focusing on products, not these big large projects. And why this matters, right? It should matter to all of us because it's our tax dollars that are informing these government projects. And the Empire taxed an entire galaxy into oblivion to build this thing. And we don't want to start seeing that in our government, right? So the change is really important. But another thing that really frustrates me is when you look at that scatterplot that Tori showed you, there are a number of projects, programs in that scatterplot that are calling themselves Agile. I don't really think that word means what they think it means. In fact, this is a safe diagram from one of the major weapon systems in the Air Force. And as I like to say, we don't always do Agile, but when we do, we like to do it like waterfall. In fact, it's water scrum fall, right? So scrum is very much in vogue in the government right now. And so what we've done is we've taken scrum and inserted it in the middle of our waterfall process. And so on the fuzzy front end, you've got studies, approvals, design, then you've also got literally an act of Congress to get your budget, and then ultimately letting out that contract. Three to five years, that could be being very generous, pretty long process. Then scrum, right? So you have this development process, typically you'd see on the order of two years for most medium complexity software. So there's already the shortest portion of the cycle where we inserted Agile practices. And then on the last mile, the testing processes, getting authority to operate from a cybersecurity perspective, and then actually fielding things, right? We're sending out fielding teams to eight different locations around the world to physically install hardware and software, a lot of which doesn't even have automated installers. So very manual, intensive process. Another five-year process for the Air Operations Center on the low end, sometimes up to seven years. And so you have about a 12-year process here, and what we've done is we've gone to the warfighter, in our case, and said, we're going to do Agile. We're going to do this new thing, it's called scrum, and even if I take at face value that was scrum, you could get twice the amount of work done in half the amount of time. Really, instead of a 12-year process, I'm going to the warfighter and saying, I'll get it to you in 10 and a half years, very Agile. That's inherently not Agile, right? That still looks very much like a Death Star. And so what keeps happening, we need to stop building those, they keep getting blown up. And so what we want to keep focusing on are outcomes, not outputs, so turning that on its head. And what we like to say is the most important outcome for us is the continuous delivery of valuable software. And to harp on that, if we say if it's not continuous, then Agile hasn't manifested here. And so this question always really resonated with us. It's from a talk that Jez Humble gave where he asked, what if you could get valuable software released faster but with higher quality and reduced risk? And in Kessel Run, we've been proving that over and over again that we can go fast and still have quality, still have reduced risk. There's no trade-off anymore between speed and security, speed and risk, but it's not just on the software development side of the house. We're living with an entire acquisition system. Yeah, in the acquisition system, I've been in plenty of classes where they teach you how to do this and they talk about the iron triangle. And the iron triangle is cost schedule and performance and the famous saying is picked to. And that's what you get in your system. You can't have all three. So we've tried to use this question but rephrase it within the acquisition mindset as well. So we're not only challenging things from a software perspective, but we want to challenge the whole process so that we can increase those timelines. And ultimately, this is what continuous delivery is all about, right? It's not just about going fast. It's being able to be stable to have high quality and reduced risk. And when you start talking about continuous delivery, Jez Humble also said the first thing you get is that sounds great but it won't work here. And there's a lot of stated reasons, right? And we hear the same things in government. We're too regulated. I mean, we're as regulated as they come, right? We're not always building websites. So even in the Air Operations Center, we have interfaces with everything from radios to the platform itself that's flying across the world. Too much legacy. We've got a lot of that. And our people are too stupid, right? That we don't have the skill sets built up in our system. But that's actually not true. We'll talk about the people a little bit later. But the actual reasons revolve around our culture and our architecture. Those are the things we need to focus on, culture being chief among them. And it's funny in the intro, this was actually the exact thing, bringing the people, the technology and the processes together. But a lot of times we stop there, right? We try to go acquire that. That's just a snapshot in time, right? The most important thing that people always forget is that it's about learning the fastest. I bring those things together to learn. Learning is the outcome that we're going for. And if you're not learning, you're always going to be irrelevant. By the time you field something, the way technology moves today, it doesn't even matter if you can field on the order of weeks, you're still going to be behind if you're not always learning. And so that's what our team's been all about. That's what's made Kessel Run successful. If I could say one reason why, it's learning, learning, learning. Instead of going out and acquiring buzzwords, right? Let's hire a DevOps. Let's acquire some cloud. We'll sprinkle some agile on top. Maybe some other practices, right? Continuous integration. That's not what it's about. It's about learning. And so we focused on acquiring an ability to learn. And that's all about avoiding disruption. We need to transform into a software company that delivers air power. That's a controversial statement right now. It's not always well received, I'll be honest. But our position here is that we're living in the age of disruption, right? It's been a while since it was penned that software is eating the world. But if it's eating the world, it's certainly eating the war. And the last place you want to be disrupted is on the battlefield. And if we're going to change that, we have to start thinking of software as a strategic asset. And we have to be a software company that's capable of managing that strategic asset. That involves a lot of change, especially in the government. But our leadership gets it. If you don't like change, you're going to like relevance even less. And for us, relevance is all about a mindset. So this is really important in government, especially if you're trying to do a transformation. Your people get scared because they don't feel they maybe have the skill sets necessary. The thing that we hire for, the first characteristic we look for is a growth mindset. And these are some of the things that we look for. Bias for action. Strong opinions, loosely held. A lot of soft skills, right? Trusting the process, trusting the airmen. Organizing for success. We've had to completely reorganize the way we do business. Flat org structures. You know, when you go to our labs, there's no uniforms in our labs. So you might have a senior airman, which is very low ranking for those of you who aren't familiar, talking to a major on a balanced team. Typically, there's not a lot of balance between a senior airman and a major when you're in uniform. So changing some of those things is a key to our success. And then last, but certainly not least, is not feeding the bureaucracy, which is something that's really easy to do, especially in government. I'll just send this one email and get them off our back. But the more you feed something, the bigger it gets. And that one innocuous little feeding of the bureaucracy turns into a monster that you can't fight anymore. So we always say, don't feed the bureaucracy if you're going to do anything, feed the innovation. And our airmen have come together in a way that is just incredible to me to continuously deliver war-winning software that our airmen out in the field love. And it's been the most impressive thing. I just mentioned it. One of the things is, oh, our people are too stupid. Let me tell you, there is no talent shortage. If we could just all agree walking out of this room, there is no talent shortage. Not in the tech industry, not in the government. What there's a shortage of is environments where people with a growth mindset can learn and thrive. And if you can create that environment and then get out of the way, I've seen our airmen do incredible things that I never would have thought possible just eight months ago when we started this project. But a couple of things wrapped up here. Government's always, acquisitions always trying to develop war-winning capabilities. That's not new. But figuring out what is the war-winning capability and how you get to that involves a lot of lean practices that we're just not used to. And so we've incorporated a lot of lean product management, but also continuous delivery. Although continuous delivery might seem foreign in the government, actually our regulations actually speak to that. Yeah, so the Federal Acquisition Regulation, which is a riveting 3,000-page document if anybody wants to read it. But if you read it up front, the vision is to deliver in a timely manner. So just like we challenge our teams to always continuously improve, we're taking that one step further and saying we're going to continuously deliver in our mission. And then the last piece is something that's also somewhat new to the government, and that's building capabilities that our airmen love. So typically we're focusing on, if we're looking at the pyramid, we may get to the usability stage. It's usable software. We're trying to push well past that, making useful software, but then making joyful software, which is kind of a funny thing to say, not just in the government, but especially in the military. I walk into some rooms and say, I want to create joyful software, and people kind of laugh at me, but that's okay, right? It's really important to our airmen out in the field. I would say that our airmen out in the field have hope again. It's really great. It's really refreshing to see. We've seen quotes like, my eyes no longer have to bleed when I'm writing a misrep. And that's important, right? It's not just making software that works. If people are actually going to use it, instead of reverting back to Microsoft Office, it has to be joyful, right? We have to get rid of all of those pains around the software and figure out a way to do that quickly. And so this is involved bringing in design practices, and that's somewhat foreign in software development in the DoD right now. It's up and coming though. So we're employing user-centered design as well as Lean UX. And ultimately, we want to revelationize the way that the Air Force builds and delivers software, which software delivery, unfortunately, is not all that common. So how did we get here? So our strategy was, like we said earlier, prototyping a new methodology for the Department of Defense. And we did that by partnering with Pivotal Labs on two fronts to fix our broken architecture, which we're relying on the platform as a service Pivotal Cloud Foundry, and then fix our broken culture in which we have all our airmen pairing in the Pivotal Labs experience. And we've proven this. We've delivered capability to the Air Operations Center. If you go to the next, our successes to date. Like I said earlier, we awarded this contract in 60 days. I had said it takes about one to two years. So we crushed those timelines just with this contract delivery. And then since award, we've been able to deploy an unclassed dev environment in 60 days to classified production environments in 130. And then those and what was really key from an application standpoint, the transition from the unclassed to the classified environment is seamless because of the Pivotal Cloud Foundry running underneath. It gives us the ability to be the first organization in the DoD to continuously deliver to SIPR, which is our classified network. And with the prototype, we've taken five applications from concept to operations, averaging about 124 days. And the most important thing, this is calendar days, not even working days. So we, again, talking about a third of a year to deliver capability with our best time being 88 days. And to do that, the platform is essential. Raven, I'll highlight the 88 days one, actually. So we do discovery and framing. That included the upfront user research. There's actually 33 coding days to get that capability live on SIPR and use in operations. So some things that are key to us on the platform from a government perspective is abstracting away all the things that we don't have to worry about, right? I just now have to worry about my applications and my data. I say me, I'm not a coder. Our airmen, coders, are primarily focused on their apps and their data. A couple of things on this. So use commercial cloud infrastructure. Right now, the battle for who's going to be the cloud providers on SIPRnet is still shaking out. Maybe one provider, maybe multiple, but at the end of the day, we don't care. The economies of scale aren't really at play in the classified networks. And so whether it's public, private, hybrid, doesn't really matter. So we took a way to hedge our bets against that was to have a platform. And we wanted to choose a leading commercial platform. A couple of things on that is in housing the platform layer. A couple studies. And if you want to get offline with me, I can provide those to you that show that in housing the platform layer is actually more expensive. And we think, we believe, that it will lead to an inferior product. And at the end of the day, we just don't have the development maturity to in-house a platform layer compared to Pivotal has 500 people managing Pivotal Cloud Foundry. We need to have portability across infrastructure choices so that when that cloud infrastructure becomes available on SIPRnet and on Jwix, that we can seamlessly port across those. And then the last thing is having this pipeline of processes from the PM all the way out to deploy gives us an unprecedented level of visibility, our transparency, where we can provide very detailed dollar estimates of what Congress and what our stakeholders are getting for their money. And so that leads to a level of accountability that you just don't see in the traditional acquisition cycle. Ultimately though, don't build it when you can buy it. Don't buy it when you can rent it. Commercial industry has solved a lot of these problems for us. Government doesn't need to be in the business of building platforms. We should focus on the things that are unique to us, which we believe is not a platform, but the applications and the data, and even some cases, not even those. I do want to talk about one thing the platforms provided us, and that's around automating GRC. So Enterprise sees this, certainly we see it in the government. I'm just going to talk about one and that's the risk management framework, which ultimately allows you to get your authority to operate and put your capability on the network. Typically takes on the order of 10 months, once dev is complete. So this is a stage gate to production and it's about 10 months in our legacy program. And so the risk management framework has 800 and some odd controls, some of which are really important to security. Others involve things like making sure there's a water shutoff valve within like 15 feet of your server. Any dev that can answer that question, I will like anybody, multiple deployments around the world and they're supposed to be able to answer that question. It's not possible. So what we've done is taken the infrastructure and the platform layer and applied the controls that apply at those layers and then anything we build on top of it inherits those controls. What we see is we're able to get about 90% reduction, 80 or so controls, about half of which may not be applicable given the tech stack that the application team is using. And so ultimately what you end up with or what we're seeing on average is about 40 controls we're worried about at the application level and that allows us to get our authority to operate in a week and we're actually working on refining a process right now where we will be able to get it automatically coming out of our pipelines. This is really important, right? Again, dev complete, I used to have to wait 10 months just to get authority to be put on the network. Now is where I can start opening up continuous delivery capabilities. And there's other stage gates to production, developmental testing, operational testing. If you're in the Department of Defense, you're very familiar with these. But also stage gates to production and what we've done here, it's a lot more complex than we've made it look with this policy math. But you can sum it up in this idea that we take the concepts behind these technologies and these practices and show how they can meet the intent of things like developmental testing and operational testing. And in the case of authority to operate, we're actually producing all of the required documentation as well. So things like having infrastructure and platform as a service paired with TDD, continuous integration, continuous delivery. That sums up, I have confidence now that at a developmental testing level that I've met all of the requirements. And when I pair that with user-centered design and do acceptance testing with the user in production, now I've met my operational testing requirements as well. And that's important because getting rid of those stage gates to production, again, three to five years in the Air Operations Center to have that happen, there's a huge gap between when the thing's complete and when it gets out. And that results in things like this, where we're planning tankers. And this is the capability I'm going to talk about for Costa Delay. We have other examples. This is the only one I can talk about today. They are planning on the whiteboard all of the tanker missions for their theater. And they're using dry erase markers, pucks, moving these around and basically they're beautiful-minding this thing. And that's really difficult in and of itself but there's a lot of constraints that are applied on them as well. So we're fighting with a coalition and some coalition partners can't refuel other coalition partners. There's just the asset compatibility between the tanker and the asset they're trying to refuel and a number of other things that go into this. And so when you have all this planned out and then a change happens, you have to erase the whiteboard. There's a lot of cursing involved and start over or you can launch another tanker. So there's a lot of efficiencies to be gained here. These are really easy problems for computers to solve. And so what we did is, again, they had this whiteboard but that wasn't it. They would have to call across the room to the gonker who would enter the information into the gonk which is a spreadsheet that has a lot of scripts and macros, calculate fuel and make sure everything's compatible. And then that person would call it out to another person who would hand jam it into the legacy application that ultimately produces the plan. So we no longer have to do that. That's really slow. I can't give you numbers but trust me, it's really slow. And we're a lot faster now because we just have the planner and the tanker planner application and they're able to do the plan. This is what it ultimately looks like. They're using a touchscreen with it now. It looks still a lot like the whiteboard, as you'll notice. That was actually one of the things that they liked. So we were able to maintain a similar user experience for them but calculate all of the math and the constraints in the background. And that led to this. So what does that get you in terms of cost of delay and the whole point of this discussion? So we spent $1.5 million when it was first used in operations. By the time at that point, you start saving with all the efficiencies we gained with the application, $214,000 per day in fuel alone. So in a couple weeks, I paid for the application but let's then look at it from a bigger picture of if we follow the process and we'll be nice and say it took us five years and it feel just as good of a capability which we pretty much know it probably wouldn't have but let's make that assumption. That's $391 million of savings basically for this method versus if we had still followed by the waterfall. This likes to harp on the fact that we are showing we don't have to make that trade off that they so famously teach us of cost schedule and performance. We're getting a better performance at a quicker schedule timeline and then for cheaper cost which we can easily depict out of here. And that helps us as far as driving the conversation like this is a better way of doing business. There's a big part of us that has to still convince that within the government but also there's another side of it that we can't just look at it from a cost perspective. Absolutely. And so the methodology again, the key here is this makes the case for continuous delivery and not batching up work. I do want to highlight, I don't want to oversell it. Tanker Planner Tool is nowhere near feature complete. Never will be, right? Software is never done but it was very much an MVP when we fielded it and we fielded it as an MVP with not a ton of features, just the ones that were most important to do the tanker planning and that's what allowed these cost savings. And if we had waited and waited and waited and built the perfect capability, that's where these costs come in. The cost of delay is high but a lot of times we're not just talking about dollars and this is where it gets heavy, right? So I was a targetier. I've seen this firsthand. Sometimes the cost of delay is measured in lives. We deal with life and death situations every day. Whether we're talking about our own troops and force protection, we might be talking about our coalition partners in the case of this CNN moment or, you know, innocent civilians, you don't want to have mistakes happen and so if we can get capabilities out to the field that are high value and reducing the amount of errors, reducing manual entry, those types of things are very important. Let's get those out to the field first. We'll finish up the rest of the features later but let's continuously deliver, iterate on those capabilities and keep making them better. We don't need to wait until they're perfect. The warfighter can't wait for us to get it perfect. And so at the end of the day, the big game changer here is continuous delivery and the way that we've been able to sell continuous delivery which, believe it or not, is not always a popular concept. People push it back against it is by using cost of delay to show this can't happen. We cannot afford waterfall anymore. So that's it for our talk. I do want to say, shame was plug, we're hiring. So kesselrun.io slash join. You can also go to the menu and order really cool t-shirts. Agile Air Force stands for Air Force. Any questions? I'm not the perfect person to talk about that but I can say that that has been one of our biggest pains as an organization. If I'm being frank, I mentioned, you know, our development maturity is an issue. What we've seen is we hire for a growth mindset and we've been able to build these skill sets. That skill set is obviously a lot harder to build but our team has really risen to the occasion on that. They're continuously improving their practices all the time. We recently just, we can now, we have a pipeline for our unclassed PCF so we can, we have a pipeline for our platform now which was a big feat for us, right? But yes, it's been a struggle. We are working that every day and continuously improving. But at the end of the day, I mean, our platform out forward is stable so we have much higher uptimes even despite all of those issues that we're having as an organization, learning how to do platform operations, we have higher availability than our legacy systems out in the field. I think it can be applied across the services from an operations center perspective where you have like central command and control nodes. I think there are definitely challenges that we haven't solved yet that the Navy would face, for instance, when you're putting a boat out at sea. I think those things can be solved but we haven't tackled those yet. But largely, yes, any of the services, I believe can do this. I mean, RMF is not unique to the Air Force. A lot of the other GRC that we've dealt with not unique to the Air Force. They're DOD-level policies that we've tackled and now everybody else can take advantage of. So we're trying to do that right now. I will say it gets hard because we aren't an organization that's typically concerned with profit and loss which you would have a really good baseline of metrics to start with to estimate out features like what this feature would give me an ROI. We, as a department, we don't have a lot of those baseline metrics. So what we're doing right now is we've made a commitment to start collecting the metrics on the new capabilities we've built and then new features within the new capabilities, these new cloud native capabilities, being able to measure that. That's something relatively new. I don't want to oversell. That's something we're still learning. On the legacy side, though, it's a really difficult problem. So from an OPSEC perspective, I have to be really careful. I can't go into too much detail. What I will say is that outside of the security practices that we've employed within our pipeline, we're also starting to do things with pen testing to give the ASICs or the comms directorates the assurance that what we have built is secure. But yeah, absolutely. From a security perspective, we can show that what we're doing today is more secure than what's in the field. And then the other thing that we always harp on when we're talking to cybersecurity professionals is not only that, but we're more secure than we were yesterday, like Kessel Run, and tomorrow will be more secure than we are today. The most important thing that we have from a security capability is speed and responsiveness to change. Speed is the new security. So there's a cost of delay associated with security as well. Whether you're talking about a legacy system that you have out in the field that has unpatchable vulnerabilities, there's a cost of delay associated with not replacing that quickly. There's also the capabilities you're deploying every day. The more often you can deploy patches, specifically patching security vulnerabilities, the better off you're going to be. And so that message is really resonated. I'll say all the way up to DOD, CIO levels. I don't know, but I'm not aware of the DHS activities, but I do know that, you know, I've seen a lot of press around them moving to the cloud as well. And how are we doing on time before I... Good. I think we just hit. One more. Yeah, I mean, I'm looking for that too. Because I know that like cloud.gov has learned a lot of lessons that we probably could have leveraged, and like we frankly didn't know enough about them to go leverage them before we got started. So that's always a problem in the government, you know, the right hand not talking to the left. But at the end of the day, I think there's also this aspect we always say we're hesitant to have people start going and adopting the things that we've done because we're still learning and we're learning a lot of lessons every day. I don't want to oversell where we're at today. Like there's still a lot of pain. There's lessons, hard lessons that we're learning. And so, but at the end of the day, yeah, absolutely, we need to do a better job. I know that when we're onboarding vendors, we're doing a lot better job of that now of informing them where we're at and where they need to be to work with us and get through our pipeline. Good. We'll talk later. All right. I think that's it for our time. So if you have any other questions, Tori and I will be available afterwards.