 Hi everybody so glad that you came back from lunch. Hopefully you're not too tired. I'm a little tired myself That's all right so I'm so excited to be here and What I'm gonna talk about is maybe just a little bit different than open source However, I can tell you where it fits in all the pieces, but before that I'll do a little bit of an introduction My name is Robin Yeemann. I currently work at Carnegie Mellon the Software Engineering Institute As the space domain lead prior to that I spent 28 years working at Lockheed Martin I retired as a senior technical fellow and I had the opportunity to work pretty much I always tell people everything from submarines to satellites So a whole range I began as a software engineer So I spent most of my career as a software engineer But then transitioned into systems because most of the things Lockheed builds are pretty big How many of you guys have heard of Lockheed Martin? Okay, so a few So what I talk about is is a Applying Agile and DevSecOps beyond software right how do I bring it together for the system? And one of the things that I ran into pretty frequently is if we did a great job And we were able to get the software out fast It still didn't mean that I got my systems out fast right my customers still didn't experience anything different So the key is being able to bring these together now over the last couple decades We've seen some pretty good things with with Agile and DevOps. Would you guys agree? Right, we've got higher morale better visibility faster progress all kinds of great things What I want to do or what I'm most interested in is how do I take those great things that happen to the software level and Bring them all the way up to the system level right? How do I how do I do that and Before we go anywhere with that why why do you think we'd want to do that? Do we want to do that because it sounds cool? Or you know, it's a great buzzword That it doesn't even have generative AI in it. It's not a good buzzword right so the reason you want to do that is because Traditional development traditional systems development waterfall operates very well. Let's say in a simple domain, right? Pretty simple. I'm repeating things that I've built before multiple times. However Most things that we're building today are Complicated and complex right and that is where Agile thrives so Things like the f35 right there their lines of code is going through the roof Orion the you know the space vehicle Things like satellites, etc Highly complex and when I start putting a lot of them together. They have what we would call emergent behavior All right, so so there's a lot going on and that is why I would want to move from a traditional kind of Waterfall approach to an agile approach leveraging DevSecOps principles Now how many of you guys have heard of the waterfall? How many of you have had an opportunity to read the paper by Winston Royce that first instantiated the waterfall? That's good now the interesting thing about that paper is he worked at Lockheed 1970 he was in Burbank, California And you don't even have to turn the page before he says I Would never use this approach for anything larger than three months without 100% well understood requirements So fast forward to me beginning at Lockheed in the early 90s And I spent probably three years building requirements for a submarine before we did any code before we did any design For we did any development right so I I think this I Believe that we're all so busy that we had time to look at the picture And we're like yes, but not time to read the whole 20-page paper That's kind of what I think happen. All right motivation to migrate demand for faster development I got to think just like in the States. You guys need products faster We need to be able to manage change and then really handle this increased complexity One of the things I see pretty regularly My children are both graduating from University of Central Florida one's in computer science one's in biochemistry and Their degree looks exactly the same as my degree looked like almost 30 years ago In computer science coming out of Syracuse University. I Believe they're being educated for last generation's problems not necessarily for this generation and I honestly think that In order to build these systems faster and to handle the complexity The answers are going to be in the liminal space right so the subject matter expertise of just software or just systems or just test Is probably not going to to meet the need that we're looking at And we got some early adopters early innovators Those of us, you know, you've seen some of these guys SpaceX heard of them, right? Planet Labs. I'm a huge fan. They actually launch satellites every three or four months, which is really fast Bosch Formula one Relativity which actually has one of the largest 3d printers in the world now both them and SpaceX say they have The largest 3d printer. I don't know who's right. I haven't actually gone to visit, but they're both claiming that and Tesla All right, so these are what we would call those early adopters and they've really taken off so What we want to do is make sure we can do that for the rest of our systems Otherwise, we're all going to be working for the same 10 companies in in 10 years, right because they're they're just killing it Now recently a Colleague of mine and myself we published a book It's called industrial DevOps hence the name of the topic and it may not be the best title It was the one that came up at the time and so we've kept it but it really is about how you apply agile and DevSecOps to Safety critical cyber physical systems mostly large-scale so things like fighter jets submarines space vehicles etc All right, and based upon the the book we actually found Maybe eight or nine principles nine that we have here That we found pretty successful Repeatedly on programs. She's from Northrop Grumman. I'm from or I was from Lockheed Martin It's hard to stop saying that after 28 years when you're with a place like you just always forever kind of associate yourself with them so I'm gonna walk you through these principles and my goal is you know for you to ask questions maybe give you Thought Things to think about even if you have areas that you want to push back. Nope that that won't work Any of those things are are on the table All right, make sure I'm on time here All right First one organize for the flow of value organize around value What does that mean? Well Many of the programs that I've worked over the years. We have very large stove pipes within our organization So I'll have software engineers of systems engineers. I will have mechanical engineers hardware electronics etc all all completely separated trying to build business outcomes and Then what happens is we actually have to come together to deliver the system The other problem that we see with these large stove pipes is communication follows the org structure Meaning everybody in the software round the soft They they know the software speak and they know what's going on in the software organization But not necessarily a crossed right the goal of most of my customers is how do I get a satellite into orbit in Less than 60 months right that's which is a long time But that would be the goal of many of my my customers Some of the problems we also have with these organizations is you know, we've got program managers They're speaking in things like lean lean startup great Systems engineers. They're talking about systems thinking maybe design thinking software referring to things like agile dev ops Operations they're talking about things like ITIL or IT infrastructure library. Here's the secret Each one of those things is actually about trying to deliver an optimized system and We're all talking past each other So I had this great experience just before I left Lockheed and they'll they figured it out right there they're smart folks, but we had a brand new missile contract and You know, we had to go fast right you've heard that we were by the time we start. We're already late so we had to go fast and on day one You had your systems engineers going off and getting reusable documents reusable artifacts We had our digital twin folks going off and getting a digital shadow to begin with We had our modeling and simulation engineers getting reusable models and Our software team downloading an existing code base to begin from Now the interesting thing is They all picked a different missile Because we don't talk across organizations and when we have programs with thousands of people this happens pretty regularly Wouldn't it be great if we had all picked the same one now again, like I said, they're gonna they're gonna get there so one of the biggest things we want to do is make sure that we organize around Value and that's not around the functions. That's how I get a system into the hands of a user Here's a good example. I stole this from scaled agile, but you know, I Could have built one just like it up at the top. You're looking at the operational value stream Notice that the operational value stream doesn't say Requirements software design. No, it has all of those capabilities that we need to do And then we've got what we would refer to as the development value streams The development value stream is typically that value stream that's feeding the operational So perhaps you're purchasing a vehicle, right? Your operational value stream is I want a vehicle I procured it. I have a vehicle really short value stream, but that would be it The development value stream would perhaps be the people who are building the transfer case for your truck Or they're building, you know, your infotainment system Those are development value streams that are feeding the system for that overall vehicle Very next one multiple horizons of planning Now many of our programs, especially if you have been big into agile, how many of you guys have done a little bit of agile They're looking at planning for two-week sprints or maybe they're planning for a quarter Fleet ballistic missile is a 60-year program that that Lockheed Martin has right? I can't plan two weeks at a time to build that I need to plan much longer So typically for a large program I might have a five-year plan and I break that down into an annual plan And I'll break that down into a quarterly plan and then I'll break that down into a sprint plan and a daily plan now some of you May say what is the difference between that and an integrated master schedule? Well an integrated master schedule is a predictive plan and basically one of the things that typically happens for an integrated master schedule that I've built out for five years is Your program managers are going to beat people to get back on plan But keep in mind you built this plan when you knew the least amount about the system So we want to move to empirical planning an empirical planning is Okay, so I have this five-year plan But I have regular review points now if I'm looking at my daily plan And I planned and let's say complete ten things today, and I completed seven It's pretty likely that my two-week plan is also going to be about that 70% complete There's usually not a case where we magically catch up So what we want to do is take that data and maybe I wouldn't take it from an individual day But if I did have a couple of sprints where I was 70% complete I would use that data to further inform the next horizon of planning So that I can be predictable now I can either do any of the things that I would typically do in that that iron triangle I can add resources I Can change my schedule? I can add automation any of those things, but I need to do something Because it's likely that we're not going to hit that predictability mark Now here's an example This is Orion right so Orion has a series of Experiments right big things that they're going to do and currently this is the plan to get to Mars by 2030 but Based upon things that we learn along the way. This may be adjusted We're not going to just wait till 2030 and go oops. I didn't make it You've got a series of experiments. You're going to use that empirical data to feed back into your plan so that we can actually get to a predictable system the next thing I want to do is really look at Implementing data-driven decisions. Now data-driven decisions is not decisions that I make based upon looking at paper It's decisions that I make based upon looking at an Executable system now. Maybe I don't have the hardware for the rocket yet Maybe I don't have all of the components complete and I have to mock out areas but I want to demonstrate My work not just paper every single Time box every single whether it's a sprint a a larger quarter and an annual I want people to actually look at a real system and Use that data to determine what we need to do what what progress was All right, typically This isn't happening now The cool thing is we have even better tools that are getting better and better all the time So we've got simulators. We've got emulators digital shadows digital twins digital threads 3d printers all kinds of things that allow us to get to fast feedback things that we might not have been able to Do maybe ten years ago We can now right? I heard this morning Which I won't take back to anybody, but I heard this morning He said oh the difference between hardware and software is hardware slower. Well It is a physical product, right? So we're not gonna we're not gonna you know make fun of our hardware brethren But we do want to tell them that there's digital mechanisms now that can actually validate and get us information So the old way of building doesn't have to stay the same Here's an example the VC 25v program from Boeing the hind schedule right and They've demonstrated and seen variants multiple times and there's multiple reasons But the fact is that they're looking at it, right? They're not looking at just the paper I actually have a couple friends that work at Boeing and They have some pretty good data about it This is not new you guys know this this we need to do this in software But we have to do it in hardware too, right architect for change in speed. We need modularity and standardized interfaces It looked like and I got a chance to sit in the um software defined vehicle portion of the linux Conference yesterday and and it looks like they're making huge strides in exactly this Typically a lot of our larger systems the satellites stuff like that that that I built they're all one-offs, right? They're they're constantly it's not product line engineering. It's not standardized, but it could be right and If we did that then we can commoditize the things we know And innovate on those things that we don't but much faster All right, so let's make sure that we're we're using this modular architecture both within the software and the hardware Here's an example This is smart sat from lockheed martin. So just like you got software defined vehicles software defined satellite It came out I want to say about four years ago And it's used on a multiple Multiple sets of satellites, but the cool thing about this Just like a vehicle is I can change the mission from weather to intelligence Over the air anytime, right? It's ultimate ultimate flexibility In being able to you know change what it's doing All right, so This would be a good example of a modular architecture Software defined that gives us maximum flexibility to make changes when we need it Iterate and manage cues. This is not new The theory of constraints has been around a long time. However What I tend to see especially when we're building systems is most of my teams are working eight projects and Just little piece parts and we have a lot of work in progress All right When I was a kid I got a chance to work on the assembly line at chrysler building transfer cases Third shift, which is the middle of the night. I absolutely hated it, but You know One night I came in and we didn't actually have all of the pieces For for the transfer case But the line manager, he's not measured on whether we're building transfer cases correctly He's measured on whether we're getting 532 pieces out a night So we built the transfer case minus one part, which is problem Um, the next night I got called in to come in and work for double time I learned economics at a very young age So double time double pay to come in and tear down 532 transfer cases. Awesome And then the following day was actually a sunday So we got called in again and we got paid triple time to rebuild those transfer cases Which potentially could be why chrysler cars can be expensive at times Now you're thinking to yourself That would never happen, you know, that's that's manufacturing. That would never happen in our world. Here's here's the problem It does You know, we constantly are having a lot of rework and we can't visualize our work One of the things you can see on the assembly line when we have work in progress, you can see those physical products We have knowledge work. You can't actually see all of those But the damage is equally if not worse Right, so having multiple things in progress not managing to the theory of constraints and creating rework Is causing us huge delays In our system delivery as well as quality This is planet labs again. I'm a huge fan. They have an entire approach that they talk about for agile manufacturing But they release new designs And they're able to launch said every three months. So they've completed 14 iterations on their satellite design Since 2012 Now I had a great time and a good experience at Lockheed, but I can tell you we didn't have anything even close to that fast Ever All right So these guys do a really good job at iterating on designs minimizing rework and managing that work in progress cadence and synchronization again, this comes out of How many of you guys have had a chance to read uh, don reynerson's principles of product development flow Never mind, but it's a really good book highly recommend it Um, and he's got a lot of mathematics that actually show why certain things work And one of the things we want to do for large scale teams that are building cyber physical systems Especially when they're they're located in multiple places as put them on a common cadence a heartbeat And synchronize them right because we don't want individual portions of the system Building up we want to build up the system Now for manufacturing the number one priority is to reduce variability remove all variability. You guys agree with that So for product development, it's a little different I want to exploit good variability And I want to remove bad variability And that's much harder So one of the things that doing this does is it removes noise from the system It allows me to actually be able to see those variations and identify whether it's good or bad Is it something that I can innovate on or is it something that's bad variability? We may need to remove from the system. So, you know cadence and synchronization removes that noise All right, and here's an example right so building up a car and you can see here I've got my enclosure team my pcb team my image sensors fpgas. They're all on a common cadence with key integration points to build up a system This is another good book It's dentaire austere walls the lean machine and it talks about how he really improved the delivery of Harley Right, so the delivery of motorcycles Integrating early and often Uh, again, I don't think I'm telling you anything new But how many of you guys think this happens? As a normal part of business Like we we often say that we want to integrate frequently But I find even even with just my software teams, right? Everybody's working on their own machine works on my machine Not necessarily doing continuous integration. Even though we talk about CICD everywhere Um, and it's even worse once we get to Things that have hardware software and firmware right It's very easy to stay in our lane in our stove pipe and build up product It's much harder to get everybody together and to integrate it So we really want to integrate as often as we can Now I say this and I know it Uh, but when uh, Suzette and I were actually writing the book we're both on travel a lot And lo and behold, we both had downloaded a copy off of you know, google docs of the book to work on on planes, etc Literally two weeks. We've both done lots of work in in chapters and guess what? We had to merge. I was like nobody's even gonna believe this So even though I know that you need to integrate early and often We had to go through the pain of merging and let me tell you it's not that easy with google docs, right? It's just doesn't work that well So even though we know Vegetables are good for us and dessert is not we don't always follow it Uh, here we're looking at things like building up an mvp minimum viable product And then the next viable product right and you can see here how I'm building up my my rocket over time It doesn't mean that every team's integrating every time. So I'm integrate slower than others But you only go as fast as your slowest integration point Here's Alton labs um, and again, they show both their software team their, um Mechanical engineering team and their EE teams their regular Increments and how they're integrating all of the time So not separately not I'm building my software here. I've got my firmware here. We're gonna we're gonna plan a big integration point They're on a cadence. They're integrating all the time Allowing them to get much faster and learn much quicker shift left Probably many of you guys have heard this term before but when we say shift left don't just shift left test Shift left security shift left safety shift left compliance Begin with your constraint um So for many of the programs that I've worked We have a tendency to build up again a lot of product And then validate it later now it could be that we're getting our um our test systems too late Or we think that we can build compliance in afterwards But we want to and it's much faster to build it in from the start All right One of the things that I had the opportunity to do is we had an additive manufacturing team and they'd built this old school Excel spreadsheet that basically looked at risk looked at requirements from a risk perspective I've worked a lot of proposals and built, you know, a lot of responses to those proposals From from various requirements and one of the things that it allowed me to do Is look at a whole bunch of requirements for the system That looked completely reasonable They look they look good. They didn't look bad at all, but I thought hey, I'll just play with this tool try it out um When I got done I found that actually 32 percent of the requirements I had were not testable They seemed like they were good, but once I had to actually determine how I was going to test them It I was able to ask questions now typically we would have found this out much later Again, we'll get the system to work, but we've got rework that gets us there Here I was able to reduce time in schedule exponentially By understanding how we're going to validate the solutions earlier Another thing I typically do with my teams is when we kick off anything software systems, etc We build out hacker personas Most of the engineers that I have are very good at knowing how to build the system Not always right there with how somebody else is going to use the system in a way that you hadn't intended So building up hacker personas Anything you can do to shift the constraint first is going to allow us to reduce time and increase quality Formula one knows this and so here you can see mclaren talks about digital twins and they shift Everything left now funny story about this. I actually use this slide At the dev ops summit and I had a different car Because I'm not a car person and all formula one cars look the same to me But I've got mclaren at the bottom and somebody comes up after my presentation. They said, you know robin. That's it's a cool slide But that's not a mclaren So I've been able to fix it I think this is a mclaren now. So we're we're good But yeah leave it to me to get up on stage and like Have the wrong car However, I am a huge fan of Formula one their digital twin technology and their their innovation. They it's just amazing I highly recommend you go out to their website and just review some of the the techniques they do to Optimize the speed of the vehicles a growth mindset Now this seems completely logical too, right? But it's harder than you think right, so Typically when I'm working, you know, large systems. I'm always going against the grain I'm trying to get leadership to operate in a different way. I'm trying to get them to iterate all of these things Most of the executives let's say at lockheed martin became executives because they were very good engineers Right. So when they were building systems, they were very good at it And when we come along 10 15 20 years later and say, hey, how about we do things differently? It actually Is is almost personal to them, right? They they don't want to change So, you know, one of the things I try to focus on is you know Not that you were doing anything wrong before but we've got better tools So when I was a kid, I had an eight-party line I'm sure that most of you have no idea what that means But it meant that I could hear everybody's Business in the entire neighborhood. It was an awesome telephone. I could just Stand next to it hear what's going on. It was it was terrific. Now It was good then But I still use my iPhone now, right? So again change happens And so when we're looking at applying a growth mindset, we have to remember That we're continuously learning and even though I know this Just a couple years ago team topologies came out It's a new book and it talks about how to work with Different types of teams whether you're a complicated subsystem team Or a feature team or a platform team, etc Now the first thing I did when I look at this book is I was like, oh What are they even doing? I have been telling everybody we need cross-functional feature teams for the last 10 years And then I realized my mistake. I'm doing the same thing Um So, you know, it's it's not a one time growth mindset. It's a constantly questioning what you think you know because It's changing Now one of those companies that's obviously pretty good at this is SpaceX They talk about their their run. They're, you know, rapidly unscheduled You know decomposition or whatever it is And they congratulate their engineers because They're learning something now. I know not everybody agrees with this, but it is Demonstrating a growth mindset So if you want to use principles like this, here is a mechanism to get started, right? Creating strategic alignment delivering at the speed of relevance and basically we mapped out all of the different principles to this Over the last five years, we've written a number of papers Through it revolution. So gene kim Is the publisher for that he wrote the phoenix project the unicorn project the The dev ops handbook things like that And he has sponsored a lot of these papers most of the work that i've done really Talks about Not just the software piece, but how do I deliver a system piece, right? So cyber Systems engineering model based systems engineering digital transformation um, one of the things that I I constantly see is you know, every company has an Agile initiative they have a cyber initiative. They have a cloud initiative. They've got a digital transformation initiative They've got a model based systems engineering initiative. Now. We've got an artificial intelligence initiative um Really, I think we should have a build better systems faster initiative and use all the tools in our toolbox to do those and with that Here is a qr code if you want to get the first chapter of the book and and read it for free and I would love Any feedback so if you want to reach out to me on linkedin Or anything like that Say, hey, it's good. It's bad. You missed huge gaps here. Um, you forgot all of this stuff I would absolutely welcome that I am a huge fan of feedback and a lot of passion for how we can deliver Systems that are safe and secure at what I refer to as the speed of relevance Right, so not continuous all the time speed of relevance speed of need So depending on what we're building it may depend or Determine, you know, how frequently we need to actually have that. All right, and with that I hope you have a couple questions. I know you're tired, but even if you just ask one, it'll make me feel better So given software is always changing and getting bugs updated in the cyber security space How can it most effectively be integrated into these systems going forward? Open source Yes, um So I don't actually have the answer for that. I did listen this morning where a gentleman said, hey Make sure you always have the latest kernel. Otherwise, you're unsafe I I agree with that. Make sure that you're you're constantly keeping your system current Make sure that we're validating But there's not one answer like We even had, you know, folks who are like, well the sbom will do it It's a lot of material. It's a lot of data, but are we really using it? So TBD like let's keep working on it Uh, I guess I have two questions that are pretty closely related The first question is some of the changes. I mean very reasonable like when you talked about going from kind of Test an engine whatever to organizing around the value But I mean doing that especially at a company like Glockheed I imagine that would require buy-in at like an extremely high level And there would be a lot of people like super angry and upset about you like destroying their testing fiefdom and whatnot So i'm kind of curious how you I mean over 28 years managed that and like I mean Were you just empowered by like the ceo to go piss people off and then the The second question is, you know, given that you were successful. It seems like as part of this change Has this movement in the industry changed how contracts are negotiated or anything like going from the kind of like we're just going to assume it's Here's the five-year plan and we're just going to magically do it to learning about things, etc So good questions. No, I definitely wasn't empowered to piss people off. However, I did it every day I'm definitely a A disruptor And you know, I at times felt like I was just banging my head against the wall because you're absolutely right If you've worked your entire career to finally get to the vp Of tests, the last thing you want to do is to give up that power that, you know, etc However, we've really found that you get much better systems that way now What are the things that I typically did? I volunteer a lot at the national defense industrial association In cozy, etc And I worked a lot with the government who really wants to get systems faster and in some cases Help them kind of write policy That actually forced our leaders because no, I mean sometimes they listen lots of times You know, I I got the heisman So I I get that Have they come around by now? Like is that change thoroughly? It's it's it's made a huge Yes, so it's not as fast as I like if you were to look at you know, because again I had my first agile project in 2002 because the intelligence agency started requiring it. It's not because we're bleeding edge Um And we've made a lot of progress since 2002. Do I still run into problems? Absolutely talking death 35 leadership. They were like nope. We're doing this. We're doing this Um, honestly, they the jpo and the air force don't get along in themselves. It's almost, you know, impossible Um, I think you just have to keep trying the other thing that's helped a lot is I know that you may not know what this is, but it's called the government accountability office or the gao And the reports from the gao have been consistent on we have to go faster And we have to get cheaper and the way to do that their belief is iterative and incremental Uh, the other question was has this influence like has this change in the industry change how contracts are structured and how kind of deals are made so Yes, um 100 no What I will tell you is within the us we have what's called the far cone of contracting which has like a bajillion different contracts We have opened up the aperture for people to try other contracts multiple different types Now I would say that I could use almost any type of contract And I know not everybody agrees with that and still leverage a lot of the agile practices But obviously the most flexible ones are kind of what we would refer to as an idi q these short term contracts where I I do something I validate it and then I get the next contract Um, so we are making progress and I was recently helping with what's called do d 5,085 Do d 5,085 is primarily for weapon systems. And that is the the um updates or feedback that we gave So with that I'm done, but thank you so much. I really appreciate you guys coming to hear my talk