 Well, hi everybody, thank you again for joining opengov con this panel. I'm really excited about it's near and dear to me So what we're doing today is we've got a collection of folks that are gonna talk about the do these efforts to do platform engineering Which we started calling the software factory And so this is a grand adventure where the department has started to adopt DevSecOps and they wanted to find a way to not have to recreate this DevSecOps team custom platform unique skill set in every single program inside of the DoD which is hundreds and hundreds and hundreds of programs and so today, you know, I'm excited just to kind of gain the perspectives from the team and We look forward to opening up for questions after just a couple of quick ones for me. So to kick us off again I'm Kyle Fox. I was in Air Force for about 14 years I was with the Sentinel weapons system, which is the ICBM replacement program $100 billion weapons system in the first cloud native DevSecOps enabled nuclear weapons system and Worked a lot with everybody up here What I'll actually do is we're just gonna do questions and then everybody will do their intro as part of the The questions that I hand out. So let's start off with Over the past three years the DoD software factory ecosystem has exploded from only a handful of teams to nearly two dozen Why has the ecosystem proven so popular and what's next for this community? Anybody that wants it go for it? I'll take the first first shot at this So hi, my name is Mark Alpin I work at a company called tidelift which is about Building relationships hopefully between governments and open-source maintainers But before that I've been doing other companies in the DevSecOps space and before that I Was one of ten lead systems engineers on what was then the Army's largest software program a nice ACAT-1 program and all that lovely jazz and I've been thinking about this and this question Kyle since you did give us this question ahead of time And I was thinking about the fact that almost exactly nine years ago the chief architect of our government program got a service-level award for being the first ACAT program to implement agile practices Under the under the far under some new changes in the far that have been made about 12 years ago. I think it was to start trying to allow agile practices quote-unquote And we got this award because we had managed to push a release to the field in 18 months So lightning fast lightning And I think I think that really just answers your question right there that you know on the one hand that that was a really amazing accomplishment by the standards of an army software program and on the other hand I Left to go to industry about six months after that and We pushed something to the field a customer complained about a bug At 3 p.m. And we had it in their hands at 9 a.m With a fix and it's like oh, this is the difference and this is why We need to do better and when I started started interacting with With platform one Back as it was just getting going it's like wow this is this is finally Enabling this great capability Yeah, 18 months to sub 18 hours. That's that's awesome Okay, any other thoughts I'll take a swing at that so kind of it in the similar vein So I think For those of you that haven't worked with the federal government or haven't worked with the DoD specifically the DoD tends to be a little bit hierarchical We literally have you know chains of command drawn all the way up to the the president in order They're on the wall of every building so What happens is you know that when we talk about DevSecOps? Well, we've got a dev community. That's the acquisitions community. We've got the security community. We don't really know where they are We've got multiple ops communities because you've got like the IT ops side of the house and then you've got the actual end users and Everyone kind of forgets about them and so I Think what's actually powerful about the software factories It is not even the technology and that pains me because I'm the CTO at platform one But it's the fact that the users now kind of had a voice in the entire system and and so Normally to get those requirements to go back into the acquisitions community to buy software because because even if you could deliver You know every 18 hours instead of every 18 months and call it agile You'd probably still have a requirements document. That's 10 years old And I wouldn't capture what you need and you know when I was at AFRL and I would go talk to people in the field They're like what I actually need to be able to do is right-click and copy that text to the clipboard so that I can paste it Into this other window. That's all I need. I don't need AI. I don't need ML I need to be able to right-click and copy And it takes like 20 years to deliver that so I think what's made the software factory ecosystem really explode is The users having a voice in the software development That's great. Yeah, I was actually gonna say something similar there With so I'm Dan Fedek. I actually work at Hashicorp. I am subbing for Armand who's our CTO, which is kind of crazy But I'm glad to be here But I think the biggest thing that the software factories provide as far as I'm concerned While working with other organizations in the Air Force and the Space Force is that the software factories are actually creating Known artifacts that everybody else in the DOD can use Actually in the DOD is the right answer there because in the Air Force everybody's using it But also if I talk to anybody in the Army Space Force, you know, the Navy, they're still using Iron Bank They're still using Big Bang as some of these artifacts that are available to the rest of the DOD and that and like Camden said, you know The users are actually able to make minor changes to those and then actually give that feedback back to The original program and the artifacts that are there. So the nice if you think about, you know Software design and design patterns. So what the software factories actually giving is those design patterns that are useful And then can be codified and then can be iterated on over time so if you think about the there's a the concept of the Spotify model if you've heard of an application called Backstage so the idea is that I have this CICD workflow Or I have a specific way that I do something and I post that that workflow into a common repo And then anybody can choose to use that workflow and over time the best ones end up moving to the top of the stack Right, so the workflow that uses GitLab and you know has GitLab actions And you know whatever workflows are being used that are the best ones or the ones are going to be used the most And that's kind of what the software factories are offering for the Air Force in the Space Force Yeah, awesome. Can I Yeah, just to compliment what major Katie brought forth. I Really think that the explosion of essentially these teams across the space Represents, you know pain essentially felt across the space, right? And so if we just kind of like invert the notion of pain and treat it as an opportunity people rally around Problems and opportunities to solve problems and usually What is the most important problem or opportunity to seize or problem to solve is the one that is the most Proximal to you and so if we just think about the concept of user pain and user experience and just like Functionality of being able to achieve, you know your daily grind easing the pain of being able to Deliver your daily grind is something that I think everybody is interested in and so, you know teams across the space across the country Being remotely constructed etc. Are a representation of that, you know collective Collective mindset to solve Problems and achieve, you know the opportunity space Yeah, I'm bunny by the way Yeah, so I Support platform one in enterprise design specifically for the engineering side of the house mostly Great to be here awesome Okay, next So the dody has platforms approaching a hundred years old like the B-52 for example The B-52 pulling on a thread has computers that are roughly 40 years old So how has the dody successfully approached? Bringing in modern technologies that may be four months old to operate in an ecosystem What's such diverse timelines? They could mark could be a great starter So Fun story my first boss Back when I started as a contractor at white sands missile ranch I'm not gonna say how long ago, but he had The decade previous taken out the last punch card machine from the dody In the late 90s So I was Thinking about Robert talking about the the eight-inch floppies and thinking to myself Yeah, and I was using three and a half inch floppies In production in the army in 2014 so When most people thought floppies were completely dead It was our best cross-domain solution because they had that that little switch that you could set it from Read only to read right good UX Isn't that still really awful? I mean, yeah, I mean, so I guess part of the answer Is I actually think this is something dody is in some ways good at is having Such a range of technologies, right? Like I got to see in an a cat one program one of the things you get to see every once in a while is the logistics spreadsheet for basically Every person with an intel MOS in the entire United States Army, which is a lot a lot of people and all of the equipment they needed and you get to see that You just have to you know, some people get better equipment than others and that sucks But that was like one of the key decisions that you end up making right is that okay If you want to keep integrating new technology, you just have to kind of accept that it's going to take time But that if you're willing to bite the bullet It will lead to a better experience for the user 98% of the time Hey Kyle. Yeah, can I ask you a question back? Of course, okay? So basically my name is Cameron Benosky. I'm with she bash so Has policy like from your perspective like where where's innovative policy? driving the ability for The You know discardment of 40 year old computers Yeah, so I'll start with you know, what is the mission require right and we'll start with DoD But it applies to anything. I mean there's pressure to provide services and for the DoD the purpose is to keep the United States safe And so with what we have I agree with Mark I mean the DoD is incredibly scrappy and It has made the best with what it's had and it's been incredibly creative with those solutions now as we start to see pressure To do better right there's more of a discussion on how do we look at one? Truly making the use the most optimal use of what's out there today And then to getting serious about how we do platform engineering work things like that to consolidate That also is true of hardware devices and starting to retire platforms my last soapbox thing on this I think this is an area where technology really helps us We used to deliver monolithic applications that had a very tight coupling with hardware And you have to change everything to change anything now We've seen it abstract from hardware and we have services like Kubernetes that even abstracts it from the OS So we're able to see quite a bit of change even though we're not having to influence the underlying hardware Cool, okay. I have one quick comment there So when I came actually I went in the military in 96 I was part of the DCGS program and we were working on Basically you to imagery which almost every imagery analyst at the time was actually Training on so everybody knew how to look at a grainy piece of black and white Image and actually know it was behind us. This is a surface-to-air missile or whatever And we went from that to the predator which was this you know Video feed of people loading things into you know trucks or whatever it was like amazing the difference But they were still using that integration point of hey We still need to know that this is here and we still have this so how do we? How do we use the old technology and the new technology in the same platform and we did that through integration points? It's almost like software Software development right so I need to be able to interact. I need to have those integration points And I think that's you know one way that We just continue to build integration points between older legacy hardware and new hardware. That's awesome Okay, so I've got one for you good sir Okay innovation in highly regulated highly regulated environments such as the DoD is clearly a huge challenge You've heard about it the stories up here If you could equip a federal software factory with a superpower What would it be? Yeah, so So the superpower that I would give you know the the software factory overlord if you will Would be the ability to look out into the world and just see this little halo around just little glow Everyone out there in the world that the DoD or the federal government has empowered to tell that that software factory overlord. No And the reason is we Inside the federal government for good reason and particularly within DoD. We've empowered so many people to To stand up and say nope. You can't do that. There's there's a lot of policy There's there's an opinion that says you can't do that and where things fail is when you discover that late in the process And what I would love to do is if you could just look out and see everyone in the crowd that can possibly tell you know You bring them to the table in the beginning and you say, okay, what is it? You actually need here and I like well I need I need 36 months to analyze your piece of software and decide if you can To play I'm like, okay, but what do you really need here? Like you don't need 36 months there. There's something you're looking for during those 36 months What if I can give you that in 36 seconds or 36 minutes? I think where we've gone off the rails sometimes is When we we get like 90% of the way through the process and all of a sudden those last 10 people they get to tell us No that we've never met before they walk into the room. They're like, okay. I'm here. You're done Stop I Love it a decision clarity person. That's a great. That's a great superhero any other powers. Yeah I mean having just Either it is zero cares or no for who is who? Really has been helpful Arguably is what's gotten me to be able to talk to you today so Having kind of a fearlessness and justified Technical rationale for what it is that you're bringing To say whether that be popular or not and not caring who's on your email Yeah, I echo that completely I mean I've heard many people say that innovation in the Air Force is run by young captains with scissors, right? And it's this idea that now our software factories, you know, they're very young professionals that are on these things And it's amazing because they're interacting with You know three-star equivalents regularly and that's something that's very different from a command and control military Model so it takes bravery to do that That's super power. I'd like to give them is just a look into the future when they're successful because I think The biggest thing for me, I see you know kind of getting to Camden's comment about all the people that can say no is all the people that say you just can't make this work and I talked about being an a-cat one program. We talked about innovation and modernization You know us lead systems engineers would get together and talk about how can we do it and so many times we were stopped by people who said You know this will never work or it's gonna cost way too much And there were so many of those things that I can say in hindsight with certainty having been an industry and Seeing them in practice that if we had stopped development for a year and done about half of them It would have saved the army money Right, right, you know like yeah, and That's I think really what I want to give The future super proud love to give them is you know if you pull through this software factory model It is going to be such a competitive edge for the United States military in the future That we're all going to you know wonder why it took so long to get there Fantastic. It's worth it person. I like it. I Think being able to jack into the matrix and and upload all the communities code would be a pretty cool one red dress great So the superpower I would bestow upon Whomever would be the power to bestow the ability to ask the right question So in my experience People carry around a lot of assumptions that's you know really founded on a baseline working knowledge usually informed by either a functional role or their perspective where they sit in the world and What I find you know being a part of so many different types of engineering conversations is that You know people sort of assume the meaning of a particular term Or we assume the issue on the table or we even assume the outcome that we're after And you know start getting to requirements or something like that But we don't take the time to slow down and ask our questions even if the questions You know are non-technical Or seem out of scope or seem dumb right the ability to ask the right questions to drive down to You know root cause analysis in order to determine outcomes and outcomes really being you know Something achieving something that is valuable right which is The concept of value in and of itself is very very proximal and rooted in reality. So yeah, I love that so clarity person insightful person bravery person and then You know hope it's worth it. That's fantastic So I really want to start dragging in some audience members here Really fantastic panel that we have up here today and we're curious on what questions are on your mind. I can call on folks Okay This is a science car So a quick question like there is actually a softer effect is very overloaded term first of all It depends on you ask it depends on who person does like a building a one pipeline Building only one application. I heard people are coming off their factory. So I don't got a discussion but my main question is Do this more about the consumer of the product comes from the vendor perspective We are not really building a lot of code in reality But there is a good connection with the DevSecOps infrastructure soft their factory means we kind of Accept the code or drops from the vendor With you starting as a benefits of DevSecOps of receiving the vendor drops To address the one of the biggest things about the verification validation which we are receiving the code from the vendors In reality, we have more code is coming from the vendors that versus we are producing in-house Any thoughts on verification validation? I have a lot of thoughts on verification and validation Arguably has spent a good amount of time digging into that question So to answer your question, I don't think that there is a good verification Mechanism in place today that comprehensively could account for source code provided by vendors that Makes its way through some secure pipeline And I think it's sorely needed and we heard Sysa talk about transparency That fundamentally breaks when you look at policy and What's needed is a new PKI for This supply chain that enables a transparent view from a to Z Yeah, like communication piece is key. I'm I'm curious on I'm curious on your thoughts of like, you know in the seat today Platform one CTO, how's it going with vendors and customers, you know, trying to bring them in and participate in this DevSecOps and platform engineering. Is it helping or yet? Yes, I think it So the benefits of Incorporating the vendors and getting those those code drops onto the platform Really fall into two spaces, I think so one is the ability to work directly with the people instead of keeping them at arms length Whether that's the developers or the support personnel or whoever it is that that you're working with from the vendor It really shortens that that feedback loop and then two is the more internal feedback loop. So Yeah, we're probably only writing maybe a single-digit percentage of the actual code Within the software factory and sending it through through our pipelines But the advantage there is more that at every stage of the way You know on every merge. We are testing the integration with all that vendor provided software and I think that's the To me that's the the big change in the big win from the model where I get a delivery every 12 months 18 months 10 years I Get a drop from the vendor And it's this big monolithic thing. I mean it's I've had source code mailed to me like a printout of source code physically mailed to my address as a contract deliverable I've gotten, you know binaries on disk Right, I mean, yeah Yeah, this was not a long time ago. This is And so I think it's that constant integration piece and so I I don't want to find out what normally happens is I I get the software and someone goes, okay It got delivered in my mailbox. So I'll go ahead and sign that seed roll and I'll pay it out And then I hand it off to someone else that then has to go integrate it with the system That the end user is going to use and I find out after I've already deployed it to production when someone can't do their job That it doesn't integrate, right? I want to find that out Like right now is immediately as possible. So I think that's the benefit of even if I'm only writing a very small amount of code on my platform It's it's getting it getting people into CI and doing that integration testing is really where the the win is and what? Like what what are the requirements that you want to see from the vendor? that gives you experts and Like what what are you doing to verify those other than testing them against like logical unit tests I? mean because arguably, you know That that's where the risk was in my opinion Yeah, that's great So go ahead. All right. Hi. I'm on a crayons that contractor support to do DC. I owe and my question is as you heard Rob in the last talk say, you know, we're tracking about 50 to 60 Software factories in the DoD And so, you know we started We have Some and now we want to get you where software factories, you know, or sort of the default So I'd like to hear from you. What is your? recommended steps to growing software factories to the point where that's our default way of doing business I Would flip the model of business over to innovative small companies that are allowed to do flexible scopes of work and Then let the Sustainment part happen with the companies that are doing that anyway, but really taking advantage of I Guess the current model of lazy large integrators so letting integration happen with small innovative companies and And letting them subcontract to large prime Integrators as is today and let them be the butts and seats that enable a sustainment model rather than letting them drive Direction of any type of technical validity Because they're profit driven by nature Yeah, so I think what I would do is I would I would make it a I Would flip the competitive marketplace around within the DoD so that the folks providing Software whether you're a program office or whether you're like afwarks with the cyber program or something like that The users have a choice and and you're competing for the users So we see a lot of really bad incentives on like the internet overall where it's it's ad-driven and things like that and so you're actually incentivized to Like keep eyeballs on a page for for like one second so you get that ad payout or something I Don't want the DoD to look like that But what I do want is like I have a mission to do and my mission might be you know in acquisitions to write reports Or or something like that still a mission if I could compete across or if I had a choice of different software factories to work with or different pieces of software to use to complete my mission and then That got fed back into what programs are supported in the future Instead of just getting something handed to me that I'm required to use I that if we could somehow make it a competitive marketplace within the DoD where I had like a DoD app store and The bad ideas would eventually fade and the good ideas would percolate to the top Kind of the model that Dan was talking about earlier that that's what I would do I think you would find that where software factories deliver compelling value They'd float to the top and where software factories are really just a bunch of people have thought it was a cool idea to have a software factory because they They got someone to draw on the cool logo like if they're not delivering outcomes They just won't be there anymore because people won't be using this stuff I'm actually gonna pull on money. So you had a really good comment on value earlier So let's just pull on that jam on that a little bit. How do you hope a DevSecOps team? articulate their value so In design and design thinking When it comes to understanding user experience user pain and The gap or the conflation of the two Really what There's one tried-and-true method. So and I'll just I'll I'll keep it simple Since you know simplicity is is beauty You don't ask a person a user what they want You don't even ask them what they're after necessarily You ask them what their experience is And you're not supposed to do it by just asking what is your experience, right? You ask You first of all Understand who you're gonna be talking to and then you understand the stakeholder that is responsible for delivering something to that person that you're gonna be talking to and You understand what that stakeholder cares about, right? And so that forms your questions going in and Having a subset of questions that gets after what the user is experiencing today Actually lends insight into, you know, where are the breaking points? Where are the gaps in communication? Where does the process break down? Where is the tool not working? Where essentially is their disruption in the pathway to the user or the users or the teams? Getting what they need to achieve their job essentially, right? And so in understanding multiple perspectives all Feeding into delivering, you know actual outcomes for the user subset You really establish What needs to be accomplished? Right in order to deliver to the user something usable So that they achieve whatever it is that they need to achieve faster better healthier I love that, you know, don't ask but discover that's great. Yeah, right Yeah, please so the three main metrics from a commercial perspective is usually cost The risk and the speed to market so in the military. It's more speed to mission, right? So how fast can we get to mission? How how quickly can we do that? but also have policy and governance as a first-class citizen and what's the best option there and then So cost risk, so that's and then speed so lower cost the lower risk and the fastest to mission So as we start to create competitive examples of CICD workflows and Policy is code and infrastructure is code and you know credential management and identity management all of those things You know we can create artifacts that allow for the rest of you know the platform teams in the US or in the government To get to market fat or get to mission fastest and I think that's going to be the kind of competitive marketplace that we need to Really foster. Yeah, excellent. Hey Kyle. Yeah, I did have an answer for Anna if I get really quickly So I would say three things One thing is content generation So understanding who the different stakeholders are that are required in order to stand up a software factory whether it's program managers Acquisitions folks, etc. They have lots of questions. They don't have anyone to ask right so having Continuous content generation to answer those questions to break it down in understandable ways from those functional roles I think is number one number two Money right being able to Understand and relay to the participants across the space how much things actually cost by pointing to actual use cases Right so that they can prepare their budgets two years in advance, etc As we know the acquisitions fiscal years, etc All requires advanced planning and then the third is much in line with what camden mentioned, which is having a place for users across the space to Voice their their pain their experience their Failure to do something right and collecting those those features or stories or whatever you want to call them so that vendors or Software factory, you know future potential software factory participants can say, aha You know, I do that really well or we do that really well We work in edge or we want to work in edge, you know, we can go after that particular subset of issues, etc three things That was great. So we're we're right at time. So we should probably Maybe take one more really quick question Yep Hey, austin brine. Good to see a lot of familiar faces So it's been a while since like the cato movements of maybe 2016 and a lot of these innovations lately have been around optimizing for that security policy identity like what's the next big challenge because We've kind of and who and who's championing it maybe is a bonus answer How do you offer multi-cloud to the government in a consumable easy way? How do you make cloud one? like A multi-cloud like like live up to its name I want answer Maybe so If I were to to be a little bit of a Revisionist historian here, I would describe the the cato journey. So continuous authority to operate journey as what what came out of inviting Some of the the smarter more progressive Cybersecurity folks to the table and designing a process with them and what came out of that was this continuous security, you know Guard rail system so that we get accredited a process instead of accrediting a particular series of bits So we were going to load into memory and execute Um, I think the next big challenge is to pull in more communities and do continuous everything. So Um, I've been talking with a few people at this conference about like The test and evaluation community. How do we we get them on board faster? But um, how do you do continuous acquisitions? So You know, we talk about milestones. We talk about deliverables. How do we take the human not make those a human subjective decision and Maybe, you know, we could actually bid out. What if we could bid out an algorithm? Or what if we could bid out a ui or, you know, some element and then automatically accept that after the fact so Just shrinking all those those big long cycles we have into shorter and shorter iteration loops I think continuous everything is what I would say the next thing is and and you had next step would probably be the The dtot community getting them on board the way we did the cybersecurity community That's awesome. So we could I said a short question, but you know, thanks for throw it out to you. Um Thank you again. Let's just give it a round of applause for this awesome panel