 All right, greetings programs. My name is Michael Broadhead. I work for Stark and Wayne, and I'm here to get some input from you. I'm going to keep my prepared remarks as short as I can, and then I'm going to ask you a lot of questions. At Stark and Wayne, we often find ourselves helping our clients with their Cloud Foundry deployments and with the supporting services around those Cloud Foundry deployments. So in practice, this means we wind up working a lot with Bosch and a lot with Concourse. Now, Bosch and Concourse, of course, both configured with big YAML files. And once you get beyond the very basics, if you want to do anything interesting, inevitably, you need secrets in those files. So you wind up with database passwords, SSH keys, AWS creds, et cetera. Many organizations succumb to temptation and just check those configs right into source control, secrets and all. And I will confess here publicly for the first time, I understand that temptation. I felt the pull many times myself. But it's not a great way to do things. For starters, these days when we say source control, most of the time we mean GitHub. So now you've got the keys to the kingdom hosted on a public service online. So if GitHub has a security issue or you don't manage your team memberships properly or your permissions properly or your GitHub SSH keys properly, well, now the keys to the kingdom have leaked out. Moreover, once those keys are in source control, now you have a copy of all your secrets on the laptop of every engineer who uses that repo. Unfortunately for us, there's an easy way to address at least that first issue if not the second one. With both Bosch and Concourse, we can pull our creds out of the main configuration file and put them in a separate file that we don't check into source control. So with Concourse, we can pull in from that extra file right with the fly command as we're setting up our pipeline. And with Bosch, hopefully you're already using a tool like Spruce or Spiff to assemble your manifest from multiple files. Now this is great, and it gets rid of a good chunk of our risk, but it still leaves some issues behind because those secrets are still on the local disk of all your engineers. So if an engineer loses a laptop or a laptop gets stolen, now you've got to treat those creds as compromise. You can reduce the risk of an actual compromise by enforcing a good password policy, by making sure that people have disk encryption turned on. But as a practical matter, once you lose physical control of that asset and it's in unknown hands, you have to treat all the data on it as compromise. Also, engineers will sometimes we will accidentally check files into source control that don't belong there. And we've all done it. Everybody in this room probably has done it. And sometimes those are the credential files. And so if you accidentally check your credentials into source control, well now we're back to square one. Now if you accidentally check the credentials into source control and it's a public repo, then things get very unpleasant indeed. I actually found that totally by mistake, but it had to go in, right? So both of these things amount to human error. And I've been in tech a really long time and I've worked with a whole lot of engineers. And so far, every engineer I've met is a human being. And as human beings, we make mistakes. That's just how it is. Ideally, the way we want to deal with mistakes is not by gritting our teeth and just hoping really fervently that mistakes don't happen, but to seek out tools and processes that keep human error in mind. So they either make it harder to make mistakes or easier to catch mistakes or they limit the damage that mistakes can do. Now in addition to these existing problems we haven't solved yet, we took on a new problem, distribution. Previously, source control would make sure that all your engineers had an up-to-date copy of all the creds. You've taken that away from them. So now the engineers have to figure out how to distribute that data out of band and we just hope that the approaches they use are reasonable. In practice, and many of us have lived this, the moment you discover your creds are out of date is when you're in the middle of doing something else. And you suddenly realize, I've just broken the pipeline, I've just broken the deployment. So you go on Slack, you hope there's somebody around who has an up-to-date copy of the file, they have to go dig it up, then the two of you work out how they're gonna get it to you and eventually you're back in the saddle but the whole time that's going on you're not advancing on whatever story you were working on. Now, having heard all these limitations of the approach of pulling the secrets out of source control but still having them on the local disk, you might think that it's a bad idea and it's not actually all that bad. It makes sense in a lot of situations and the reason comes down to risk management. In security, we don't get to eliminate all of our risks. We would love to do that but the real world is always gonna be imperfect. So what we have to do is look for acceptable trade-offs and we shave risk where we can. In this case, the cost of implementation is really low, it's really easy to do. The added downside is manageable and it gets rid of a good chunk of your risk. So if you have secrets checked into source control for your Bosch deployments or your concourse pipelines today, I urge you to take a look at that approach. It's probably gonna be a net win. We can take it a step farther. So Genesis was created by some of my colleagues at Stark and Wayne and in addition to it being cool and I wanting to plug it here, it's actually applicable. So Genesis is both a tool and a methodology for managing Bosch deployments. If you have an automated deployment for any app, then chances are you need to deploy that app into multiple environments. At a minimum, you've got your production and your pre-production ends. And if you have a big team or complex setup or really large user base, then you may well have several different environments you have to deploy into. And then across all these environments, you've got some settings that you want to be the same everywhere. You've got other settings that you want shared across subsets of those deployments. And then you've got settings that you wanna set uniquely for every single environment. Genesis gives you the tools to manage all of that without getting into copy and paste craziness. And the reason I bring up Genesis today is because Genesis talks to Spruce and Spruce talks to Vault. Vault is an open source data store that is specifically designed for storing secrets. And my experience with Vault has been, and that of everybody I've talked to who's worked with Vault, is that Vault works as advertised. It's easy to set up, it's easy to use, and it's pretty well thought out. So with Spruce, talking to Vault, when you write your manifest, instead of putting in a secret, you can put in a placeholder that identifies that secret. Now when you invoke Genesis, and Genesis under the hood is invoking Spruce, Spruce looks for all those identifiers, looks all your secrets up in Vault, fills them in, and then the master manifest that it produces has all your secrets in it. Now if you're not making manual changes to that manifest after generating it, then if you want, once your deployment is done, you can delete it. It's a derived artifact and it's easy to recreate, so why not? So now with Genesis, we have addressed the distribution problem for secrets and we've at least made a dent in the problem of secrets on the local disk. And that brings us to air-martial. And it means it's time for me to take a drink because my throat is getting very scratchy. So the original idea for air-martial came from our fearless leader, Dr. Nick Williams, who founded Stark and Wayne. And a lot of the work on air-martial has been done by my colleague Norma Bromovitz. I wanna make clear what air-martial is not. Air-martial seeks to address the problems of secrets on the local disk, secrets in source control, and the distribution of secrets. It does not attempt to solve the problem of a malicious engineer abusing secrets. And the reason for that is pretty simple. If you look at statistics for data breaches for recent years, more than 80% of successful breaches are due to an external attack. And so while the problem of the insider threat is worth worrying about, we have to take our best crack at the big risks before we start to take on the medium-sized ones. So air-martial, in essence, is a proxy that sits between your command line and the back-end service you're dealing with. So either the Bosch director or the concourse ATC. So when you configure your CLI, you lie to it and you tell it that air-martial is its back-end. You create manifests much like if you're using Genesis and Spruce with placeholders instead of the secrets. Now, when you go to kick off your deployment or configure a pipeline, your CLI takes that configuration, sends it to air-martial. Air-martial identifies all the placeholders, looks up the secrets, fills them in, and then sends that on to the back-end. Now, of course, when you're setting this up, you probably wanna configure the networking so that the back-end is not directly reachable. Otherwise, some of those deployments and pipelines just aren't gonna be using it. So therein lies the basic idea of air-martial. And as I said, we've built a couple of prototypes internally and we're really excited about it. And we wanna hear from you about how to take that further. And I thought the stage was gonna be brighter and I had these questions written down. For starters, can I get a show of hands? How many of you are at an organization that is using Bosch today? All right, so large proportion, as we expected. Now, keep your hand up if you personally are one of the people working with Bosch at your org. Okay, how many people are at an org that is using concourse today? Yes, gentlemen, I'm aware. And keep your hand up if you personally are working with concourse. All right, so pretty decent representation. I was really wondering what the concourse uptake would be. So that's pretty cool. So those of you using Bosch, wondering how many deployments you've got, put your hand up if you have more than five apps being deployed with Bosch. More than 10, more than 20, more than 30. All right, I'm not even gonna take it further than that. That's good enough, that gives me the idea. So concourse pipelines, how many people using concourse have more than five pipelines? All right, so a big percentage of the people using concourse. More than 10, more than 20, more than 30. All right, and that's far enough there. Okay, any of you with your deployments or your pipelines, do you have actual deployments of pipelines that you're using for real production purposes that don't have secrets somewhere in the config? That's usually how it goes. Are any of you at organizations that are subject to compliance requirements for something like SOC2 or HIPAA or PCI, any of the above? All right, so yeah, a decent number of people. So what about other compliance regimes other than the ones I mentioned? Anybody? FedRAMP, good times. Hard to read and lots of stuff. Okay, let's see. How many have a requirement where your engineer is doing the deployments, you need to partition access so that the people doing the deployments are able to deal with some of the deployment secrets but not others? So do you need granular authorization? Nobody, oh, just one person, okay. All right, yeah, I would love if you came and talked to me afterwards to hear about what you're doing. And what about key rotation? Who has issues with doing the rotation of those creds in production and those files All right, if I can editorialize, if you don't have an issue with key rotation, you should. Okay, is there anybody that would, and I know you guys, but is there anybody that would prefer to use a different secret store than Vault? No, that's funny. I saw a hand back there, what would you prefer to use? Okay, cyber-arg, EPV? Okay, all right, so thanks. That's a lot of good information and do you guys have questions for me? If you want, you can either ask them here or you can come talk to me afterwards or feel free to grab me in the hall at any time. Yes, sir? It can, and one of our prototypes is doing that. We're not yet fully confident enough because we basically, we have to be parsing the output that's coming back. So, and that's why we say at least for now, the intention is to address the mistakes and the outsider threat and not worry about the problem of a malicious insider. But there's definitely something that we're looking at and keen to do and I would love to be able to get up here and say, oh yeah, I can do that too. Any other questions? Sir? Uh-huh. No, not anything other than big pie in the sky. It's certainly, I mean, conceptually, could do that with anything. I mean, the nice thing about those two is you've got a local CLI that's making HTTP calls out. So you have like a well-defined interface and they both work in very similar ways. But yeah, you're right, that concept could be extended. Sure, we haven't played with that but it could. Yes, sir? We have not. That is a good idea. I having, I've done enough work, compliance work with auditors that I'm pretty sure I know what the answer will be, but that is probably a good idea to check just in case. I mean, a lot of this stuff, the actual requirements of something like PCI or SOC2 are fairly broad because they know that they have to apply to a lot of different places and so how you choose to define that and apply that to your environment winds up being a negotiation between your staff and the auditors. So, I mean, it really depends on the relationship you have with them but that's a good idea. I'm gonna run that by some folks. Well, so there are two different pieces. If we're using the Genesis spruce approach, then you do wind up with a local file that has those creds. If you're using the air-martial approach, then those local creds don't go on your local disk. So what you create just has placeholders. You send it to air-martial with placeholders and then placeholder, then air-martial does the look up and the fill in. That certainly is a good idea and it means you have fewer moving parts. The advantage is there's a lot of things in common between the problem for Concourse and the problem for Bosch and so we only have to touch one code base rather than multiples and yeah, it's at least easier at this juncture to deal with the thing in isolation but yeah, that might make some sense is to have that be right in Bosch. Yes. We haven't looked at how we do it with UAA authenticated Bosch. I'm sure we could make that happen but I haven't thought it through. Let me write that down because I, let me forget that. So yeah, that's a good idea. And that's an area that we've been kind of glossing over so far with the prototypes is different compliance approaches. At this point, we're just, I'm sorry, the different authentication approaches. At this point, we're essentially passing authentication through so if you authenticate to the back end then you're authenticated to air-martial and it's probably not really robust enough for the real world but at least it lets us prove out what's happening. Thanks. You're welcome. Any other questions? Yes, excellent. I was wondering if you would be here, thank you. It was, Concourse has an open issue related to this and so the suggestion is that I go take a look because it's closely aligned so. Yeah. Anybody else? Yes? No, but there's no reason we couldn't point it at KMS. I mean, once you point it at one back end and you have an interface, you could plug it at different things and we've done custom work for one of our big clients that wasn't part of this but was related where we're talking to another big enterprise key store. So but yeah, there's no reason why air-martial couldn't be configurable to multiples. Yeah, so KMS, Amazon KMS might be a good choice. Or for that matter, it could be something that you add. Yes? That really depends on interest. I'm pleased by how many questions I'm getting that gives us the justification to put more time into that. What are the timelines? And then I gave a hand wavy answer. Anybody else? All right, folks, thanks very much for coming and again, if you wanna come up and talk or grab me in the hallway, I would love to hear from you.