 Hello, Boston! Woo! All right. Wow, this is really low on the ground. I'm going to see if I can do this without hurting myself. So I'm Brett Mogolevsky. I am the product lead for cloud.gov in 18F at the US General Services Administration. Oh, that's so nice of you. Thank you, Peter. And this presentation is about preventing tainted apps with staging gates and electric fences. And I'll talk a little bit about what cloud.gov is and does first and the problem we're trying to solve here. And I will warn you, this is an end of the day presentation. You've all heard about compliance plenty. You've all heard about government plenty. This is more a little bit more tech and dirty down on the actual process, apart with Cloud Foundry. So security and compliance, as most people probably know, is really pretty hard. In government, it looks like this. There's this mountain of risks. And you have to figure out a plan for dealing with all of them in order to get over production. We call it Compliance Mountain. How do you get over Compliance Mountain to the other side? You'll notice it's very dry and arid on Compliance Mountain. You might die. You should be well stocked with water. So the question is, what does cloud.gov do? And why does it exist? And the goal is to take risk mountain and shove as much of it down to the icy, cool water and leave you with a nice, cool, plenty of water thing at the top there that sticks above. So you can basically delegate most of your risk management down below the surface. And what remains is something that's actually tractable. We've made them out into, well, not a molehill, but the tip of an iceberg. Now, that seems really awesome. And who wouldn't want to concentrate all their effort into the thing that actually delivers the value, which is a small part of your mountain? Well, what we find is that a lot of CISOs don't see it that way. They look at the value and they see risk. And the reason they still see risk is because it's their risk. And we're putting them on the spot. Cloud Founders is a very self-service platform. You can get things production very, very easily. And so now it's like, OK, well, now we have this rapid ability to do something remarkably technical, but there's this responsibility left on us. And they get very concerned about that. And so you might ask, well, why? Why are they so concerned? So what happens is now we've enabled them to not require a whole bunch of teams with a whole bunch of different expertise and a bunch of handoffs, but just even a really small team to actually deliver a project end to end. And so they're concerned because, well, OK, so now our scope is down to just that. But those dev teams are now dev ops teams. They're supposed to be handling the whole thing. And they have to do things that normally there are very tight controls on. They don't usually have that easy access to production. So we want to see that those dev teams will prevent CVs from shipping in an app. They want to detect CVs that are already out there or new ones that are discovered. And they want to make sure that if an app is compromised, that it doesn't hurt the customers of that app or the organization as a whole, reputationally or operationally, that's a big concern. And government compliance is complex. And the idea that any one person in any one dev team who's just eager to make the impact by getting something to production, they're not necessarily going to have all these concerns in their head. And so they're worried about delegating that responsibility to them. So you see that. Everybody says, well, can we just shift the responsibility to the left? And that's what Cloud Foundry does. It lets us shift all the responsibility to the left. And now you have devsec ops. And you have devsec compliance ops. And it's like you're iterating really quite rapidly inside the team. Why don't you just shift it to the left of the pipeline, where this is just the end of the pipeline. That's in the left. But a lot of organizations are not in shape to do that. And they might be looking for a platform earlier. And the platform is an enabler for them to do the stuff on the left. But you have to be ready for the idea that they are coming to the platform before they've managed to get their actual development culture under control. And it's not that they can't have control. It's not that they're prevented, but they're struggling to figure it out. And this is a common pattern in transformation. You end up with a bunch of contributing factors that there's not enough security and compliance expertise to go around. All of a sudden, you've made really effective dev teams that need security and compliance expertise all the time, because they're shipping all the time, and those people aren't available. We haven't scaled up. There's not enough of them. You mint them quickly enough. You might have either unavailable CI, CD capability, because you've always relied on doing it by hand, or you've always delegated it to vendors or humans. Or you might have a process that is just not there. And you haven't got it as a culture. Your maturity for actually doing CI and CD is really low. You might have a low trust environment where people have not traditionally been given this much responsibility. And maybe you have a relatively transient workforce, and you have maybe vendors that have things are going to end, and you're worried about what happens after they leave, and that we have a problem, or you're worried about people who are on a term appointment, and they have to rotate out of where you are. And so you're maybe having a lot more concern about contingency planning for losing people, not technology. And in general, this is the case across the government, very low tolerance for risk. We always used to joke when we started HCFs, like, there's no such thing as FISMA jail. Like, nobody's ever gone to jail for a FISMA violation. But the reality is FISMA is there to squeeze risk out. And the government is really, really, I won't say allergic to risk, but it's very concerned about it. We talk about front page to the Washington Post, but think more about people's tax is going to the wrong place. Their tax return is going to the wrong place. Or think about people's locations being just all kinds of different things you can do a sense to say, all kinds of things. So the government does not have a lot harder to risk. So as a result, when you have this institutional need for this, it gets cemented into laws and regulations and bureaucracy in your organization, and so on. So cloud.gov is actually a shared service offered to multiple agencies. And for that reason, we can take care of the bottom of the zipper, the platform, or the right end of that pipeline, and zip it left. But we can't help every agency go through that transformation journey on their own. There's nothing we can do at our level. It has to come from the top of their organization down. They really have to tackle this as a first-class problem inside their organization. The technology or the availability of the service is not going to do it. It's just an aid. So we have to kind of deal with that. And so we get a lot of people who are reluctant to use cloud.gov for that reason. But their question comes back. They basically say, OK, assume my process is leaky and something's gone wrong. I want you to basically, the platform is like the gate of last resort. I want you to prevent those things from happening. So can the platform do these things so the dev teams don't have to? Can you push even more stuff in that iceberg below the waterline? You make it even a smaller hill. So hence the name of the talk, we have your staging gates. You must be this tall to ride. And if you're not tall enough, you can't get over it. And if you're in the wrong place and you touch the electric fence, you're out of there. I have no idea where that came from, because this is Jurassic Park. So we took a look at it and we said, OK, well, this is a common need. Everybody's got this needing cloud founder. This must not be that hard, right? And it's a little bit of a surprise. So I'm going to tackle that third one first, which is preventing the use of compromised apps as a foothold. So in this case, what we're talking about is the idea that whatever your best efforts, somebody got into an app. And hey, it's all container abstraction. They can't hurt their neighbors. They can't break out of the container. That's cool, right? It's like, yeah, but they're in your app. And your app has customers, and especially if it's in the clouds in the outside world, they might be serving malware on the IRS website when everybody's checking for their refund. They might be mining coins on the taxpayer's dollar. There's all kinds of different things they could be doing. They could be embarrassing the very tiny operations team that somehow keeps this amazing shared service going for the rest of government and threatening their existence. Who knows? So the goal here is that we want to say, OK, given everything else, somehow this container is running in production, let's detect if there's something wrong with it, and then let's do something about it. So we want to look for unusual behavior, and we want to at least warn or error in the logs so that the operators of that app, the team that deployed it that pushed it, can see that. And if it's suspicious enough activity, we want to just go ahead and kill the app. And the reason we can do that, it's a really simple thing we can do, because we know we're on Cloud Foundry. It's going to get restarted. And it'll get restarted from a clean baseline. So go ahead and kill it. There's nothing wrong with that. It dies if it runs out of memory. It dies if we replace the stem cell. We're going to kill it anyway. There's sort of a contract that every single app has to expect that the Cloud Foundry platform is going to restart it. So if you have a problem with it, shoot it in the head and let the platform restart it. So question is, can we do that? And the answer is, yeah. This turns out to be a fairly well-explored space with a few different solutions. But we always prefer open source if we can, if it already exists. So we went and looked around. And it turns out the best sort of way to look for rootkits is at the kernel level. So we checked around. And there's this thing in the kernel level called the Kernel TracePoint API. And it's used by Dtrace and Sistrace. And it's also used by Sistig, which is super cool. But we're not concerned about what's happening at the host level. We're concerned about what's happening inside the container. So it turns out there's a Sistig open source project called Sistig Falco. And Falco is basically taking that same notion of, hey, I can see exactly every syscall made by every process. And it makes a container aware, namespace aware. And so what you can do is it can attribute specific syscalls to particular containers. It can then also use rules to say, hey, I'm looking for patterns of a certain kind. And if I see that pattern, then that's a trigger for me. And then if it sees that pattern, it can take actions. And the actions could be run this program or log this here or kill the process or whatever. So what we've got here is an open source namespace aware snitch and enforcer. It's like, yes, this is exactly what we need. And it's open source. It was ready for us to try out day one. So we're not sticking somebody's unknown binary blob in our kernel. It's one that we compile ourselves. And so you can write these rules. So here's a Falka rule, which is to look for unexpected shells. So if somebody starts a shell in the container, and you can see that the format of the rule, the rule is called shell in container, it says notice shell activity within a container. If a container is not a host, if it's an actual container, and the process name is bash, then say in the log, shell in a container, and then give it these attributes to kind of give you a sense of what happened, including all the way down to the command line. So if somebody just popped a shell into that container, how do they do it? You might actually see the command in there. And then the priority is warning. Okay, maybe this is allowed. Maybe you've got CFSSH turned on because you forgot to turn that off for production. And somebody's CFSSH did. Oh, okay, well let's go turn off CFSSH. But at least you notice in your logs. But maybe it's not. Maybe it's a bad guy. This one is, if there's a binary directory that has the exec bid on, we wanna know if somebody creates a directory under there that's a hidden directory where they're gonna stash their rootkit binaries. So again, it looks for the condition, trying to make a directory, and it's that directory, and it's not the package management process. Well then, there's a directory been created, blah, blah, priority error, you can even tag it by file system. So you can say a different kinds of rules. So Falco comes with a whole bunch of these rules. They're not necessarily tuned for Cloud Foundry. As far as we can tell, it had never been used with Cloud Foundry. It was sort of designed for Kubernetes and Docker, OCI registry OCI containers. So like, okay, can we make this work with Cloud Foundry? And the answer was yes. You run Falco on each of your Diego VMs. We made a Bosch release that does this. And instead of having Falco directly kill the container, we just have it alert to this process, which is called Gardner, because it brings your garden, right? And that then for that alert, it actually goes and talks, it walks the process tree to find the guardian parent. The guardian parent is the part that's run by Diego. And then if it's found, then it looks up the app, go ahead and from the BBS system and says, okay, what app am I talking about here? Which app will claim this container? And then it emits that custom Falco log that was generated via Logregator. So it just emits the log, right into the log for that application. And then if it exceeds the severity threshold, then we just kill the instance with CAPI. And that's it. And then the rest of it is just normal Cloud Foundry stuff. Oh, that thing went away and it came back. So here we're gonna watch it in action. And I think I have to, let's see if I can get this to work. Okay, so on the left, we have CFCLI. And on the right, we have the logs for an app. So this is gonna be really hard because I have to look over my shoulder and do this. Let me see if I can turn this around here without ripping the cable out. Okay, so I'm gonna pause it a couple times through too. So what have we got? Let me first make this full screen. Hope that doesn't break it, there we go. Okay, so like I said, there's your CFSSH into instance one of application foo. And so these are the logs for foo. So what we see is, let me see, I guess I have to be down here. By the way. Okay, so Falco just emitted something into the law, or sorry, the gardeners submitted something into logs that Falco noticed. And it says, hey, there's a debug shell spawned by this process, all the law. Here's the binary, blah, blah, here's the debugger, all these different things. It's got exactly what the command line was, et cetera. That's pretty good. That's not bad. And that would work even if the application that you had deployed spawned a shell. You would still get that information, right? So it's pretty cool. So this is a clear indicator that somebody has not turned off CFSSH or somebody's popped a shell into your container. Now on the other side, once he got in, he did an echo and he made a little program that all it does is try to do a set yoid to root called evil.c. And then he compiled it, and then he ran it. So the other thing that's happening, if this happens kind of quick, the other thing that's happening is that we see, hey, guess what? So there's another thing. Unexpected set yoid happens here, non-sudo, non-root program. Here it is. So somebody attempted this, and normally it would just fail quietly and you'd never know about it. And somebody would be mucking around in your container trying to get root. Well in this case, Falco noticed it, they made the rule to Gardner. Gardner figured out which app ID it was, stuck it in the logs. And now what's gonna happen? Well, they're still in the shell there, but watch what happens. Oops, failed. Error, whoops, didn't want to continue. Remote command exited without exit status or exit signal. That was the electric fence. They touched the electric fence, they got desorged out of their entire session because we killed the container. The process is not even there anymore. So what's gonna happen? It says stopping the instance. And now we can see it continue. Or look at the instances. Okay, destroyed the container. So now we can see that one of them is dead, one of them hasn't restarted. So Cloud Founder's gonna go ahead and restart it. So we're waiting for that to happen. There it is, just restarted that instance. So there's a couple that came out just as it started. This is container became healthy. All right, cool. So let's look at the app. Instances are running, cool. Okay, let's get inside them. All right, so now we're back in. There's that SSH meshes again. So now I'm over here in this container. Notice it's got a different, if you're watching it's got a different container ID because it's an all new container. And let's check out that evil program. Oh, it's gone. So we've denied the foothold. The minute they did something that crossed our threshold, we killed the process. And they can keep trying, they can keep trying, they can keep trying, they can eat. They might have an automated process, but the minute they hit that rule, they're out. So that's pretty cool. And so we were pretty excited that this worked. There's a couple hacks in it. I'll give you some of the caveats in a bit, but we're pretty pleased. So what do you think? Is that cool? All right. I didn't do any of that work, by the way. I did zero. I did nothing except playing credit for the idea. And maybe make some slides. It is not production. And the reason it's not in production is that it is considered by FedRAMP to be a significant change request. A significant change request has to go through the FedRAMP JAB technical reviewers. Then it has to be checked out by a FedRAMP third party auditing organization to make sure that we have not introduced new vulnerabilities into the system. And then we are allowed to put in production. And by the way, if you don't have a 3PO on tap, you have to procure one. And that, if you do that unexpectedly, then oops, now you're trapped on the other side of Procurement Mountain, which is like Compliance Mountain. It's right next door, Procurement Mountain, which is why we're waiting to get that in production. But there's other things we still have to do. One thing is that the FALCO rule set we have right now is just the default FALCO rule set. And it's fairly comprehensive, but it's not really tuned for Cloud Foundry, because nobody who's using it has been using Cloud Foundry. The other thing is that we wanna add some rules that are very specific to Cloud Foundry. If you look in the Cloud Foundry documentation, it talks about hardening containers and there's all this extra stuff that happens. So it's like, this is mounted no exact, this is mounted read only, et cetera, et cetera. For that reason, we know more about the operating of that application. If somebody was working on that in development, they would have seen those warnings and presumably said, oh, I should stop doing that. So if you see it in production, that's really suspicious, because they didn't even know they were in Cloud Foundry. So we should be able to make additional rules that basically say if you try anything that would have normally failed in a Cloud Foundry container, we should be able to kill it and say that's dangerous. So we can go further to kind of get the normal case in there. The plumbing between Falco and Garden is not very robust the way it's done. We are looking at the Guardian Container command line and that is how we're figuring out what app it is. It's really not, it's sort of an undocumented API, effectively dependency, so if they change that, it'll make it break. We don't like that. And the goal eventually is that we should be able to give this to all of our customers. Our customers should be able to say, here's the Falco rules for my app. And then Falco actually, you can have custom ones. So even beyond the base ones the platform has, if you know exactly what your app should be doing or there's particularly sensitive data in it that should not be touched in a certain way, then you can actually add your own rules. So that's another thing we'd like to add, but for the development. And then the far future of Cowboys and Rainbows and Unicorns, Cowboys, Rainbows and Unicorns, I don't know why Cowboys are in my future, but it's to generate some rules based on machine learning. And we'd have to have it in production and watch for a while and see what normal behavior looks like for apps to decide, okay, let's make a rule for this, generate a rule for this thing that we saw that was abnormal. So we don't know until we actually have an introduction and get more time to observe it. It's open source, like everything else that we make, it's there. We put it in the community a while back. That's called Gardner, Fakwa Bosch Release Gardner. Super cool. All right. All right, I'm gonna go real fast. Detecting CVEs, I used five minutes at the beginning though. So really I have 10, I'm gonna waste all these people time. They're here for the end of the day. They have nowhere to go. That was just data. Why wouldn't they just, all right. Okay, so detecting CVEs and already shipped apps. This one, we did a lot of thinking and just recently kind of came to a conclusion that it's not a big problem. To some extent it's not a big thing. It's not that hard. First of all, the question is where do CVEs come from and we identify there's basically these five places where they come from. They come in the stem cell for the actual hosts. We have it in the RudaFest, which is in the container. We have it introduced in the build pack in the language stack. Or it's in the application dependencies that it pulls in. Or it's in the application code itself. And when we looked at this, it really turns out that three of those are under platform control. And it's only the last two that are gonna give us headaches. So these aren't really that much trouble. Talk about them real briefly. I'm gonna start going faster now. You thought I talked fast, but I'm gonna talk faster now. If there's a new CVE in a stem cell, we update the stem cell, there's no customer impact. Boom, this is Bosch, Cloud Foundry, this is how this works. It's awesome. As long as your deployment of Cloud Foundry is itself, something you can easily push the button and push out the stem cell, you're in great shape here. The Cloud Foundry community is really good on security. There's a lot of scanning upstream that people don't talk about. There's a lot of commercial vendors, obviously, very investant, have Tiger teams working on it. And in general, if you have a question about something that came out, you can go into the hash security Slack channel and ask a question. They're usually very forthcoming and they're very helpful. And we've already told customers their apps are gonna be restarted. That's part of the contract. So they don't have, then they're supposed to have multiple instances for production. So don't worry about it. Okay. So if there's new CV in the RootFS, we update the RootFS to the new CV in the BuildPack, we update the BuildPacks. We've got Cloud.gov set up to automatically install the BuildPacks as soon as they pass all the tests. And so we're putting up there as soon as they're possibly available. But for a customer to pick up, they have to restage their app. That's what causes the BuildPack to actually run again and repopulate the droplet. We can't do that for customers because we don't know what it's gonna mean to restage their app. It's probably gonna mean taking their app offline. As we saw in the keynote this morning, the Cappy team is working on zero downtime deploys and zero downtime rollbacks, which is fricking amazing and long overdue. When that happens, then we have a much easier time telling customers, hey, we're gonna do this periodically. And because it won't actually cause downtime for their app, or if it does, it's so minuscule. You saw this morning in the demo, there's a rolling update and so nothing's ever fully offline. So we're gonna figure that out. But we think in future that, even that restage won't be needed. Meanwhile, we wanna help customers understand that a restage is needed. For some reason, this isn't part of Cloud Foundry, even though it seems like the most obvious app responsibility anybody has, which is, hey, when there's a new BuildPack that is a new version of the BuildPack you're using, you should restage. And if you don't restage for months until there's a problem, it's likely that the version of the language that you were on is no longer even supporting the BuildPack and you'll have trouble. So you'll be in trouble and have trouble deploying. So you should always be restaging pretty routinely whenever a new BuildPack update comes up. So it's a really simple job, BuildPack Notifier, and all it does is it says, hey, by the way, we typically do this, this is just a template for the repository, you can edit it. We typically do this, it's your responsibility, we don't wanna cause downtime, you should do this your next occasion, here's where you can find out more information. Thanks, security is everybody's responsibility. And this knocked down the number of people running on obsolete BuildPacks to near zero on our platform. So this really worked very, very well. So it turns out people will do the right thing if you just remind them that it's necessary and tell them. They don't read release notes, but they do read their mail. What about CVs appearing in application dependencies or code after they're in production? This turns out to be pretty easy, all you do is scan the droplets in your blob store. So you can do that, we didn't consider this challenging, but we didn't actually do it yet because that's not where the tough part was. Tough part is this other part, the application dependencies, which they're gonna pull in, which might involve a piece of vendor wearer or something that they grabbed off of binary stash somewhere on the internet, or maybe their application code. Now the application code, presumably they're gonna, they're not gonna ship it with the CV because the CV hasn't been named yet. It's their code, right? If the CV appears in their code, it's gonna be after they ship it. So we're less worried about that. But again, the application code itself might contain things based on using, you know, multiple BuildPacks or whatever, or they might curl out and grab something. So that could be an issue. So the question is, can we prevent CVs from shipping apps? So this is just, I know I'm late on time, this is a quick rundown of things we tried. The, I'm gonna blow through really quickly, but the short story is it didn't work. And it was a bad approach. And it took us a long time to realize it was a bad approach, and basically till this week almost. So first of all, are there a bunch, there's a bunch of vendors and open source projects that are already targeting the OCI BuildPack for, or sorry, the OCI Container Format and OCI registries. Can we use those? Generally no. Most of them don't, the momentum around the OCI BuildPack format, and the things done with Docker and Kubernetes and that community momentum is, a lot of people thinking about these issues are not thinking about, and then software supply chain have not been looking at Cloud Foundry. And so a lot of them either don't have native Cloud Foundry support, or if they do, it's a very rudimentary. And I'll talk a bit about what rudimentary looks like in a minute. What about doing a custom BuildPack? I noticed when I was doing some research, the Node.js BuildPack, if it finds sneak variables, environment variables, it will actually go run sneak as it does the Node.js build. It's cool, but it's only the Node.js BuildPack does it, and not any of the other languages, and it doesn't have any kind of hook for any other vendor or any other thing you might want to put in there. It's like it's kind of bad coupling, and it's all kind of mash into one thing. And if you want to do that, you have to, if it's not something that gets in the upstream, you have to maintain and build, and then force developers to use your own BuildPack, which is messy. So it's much more effective to do a couple, I put an issue to say this is the bad place for this, and they seem to agree, but it's not resolved. What about the new multi-build back support? Well, if your scanning BuildPack is the last BuildPack in the chain, that's a problem because the way Cloud Foundry works right now is that the start command is only paid attention to in the last BuildPack. So you've accidentally blown away their start command. That's not big, that's not good. Okay, well what if it's the first in the chain? Well, at that point, the other BuildPacks haven't run, all we have is the code, the dependencies file. So it's good if you're using sneak or using libraries.io or one of these other tools that is gonna look at the dependencies, that's fine, but ultimately you still don't know what's actually gonna be in the ship code. It's not really the final product. So what about the meta-build pack? The meta-build pack is a build pack that runs a bunch of other build packs and then does a bunch of decorators, and it seems really cool, but number one, developers aren't required to use it. It doesn't work with multi-build pack and the new build pack formats and it doesn't work with manifests and it doesn't work with cf push bit dash b. So it's severely compromised in terms of the occasions in which you can use it. But much more importantly, it is incompatible and deprecated even though the repository doesn't say that. So if you're tempted to use the meta-build pack because it looks cool, don't do it. And I put in an issue to say can you please deprecate this repository because this is broken. I will say one vendor at the show that I talked to is relying on the meta-build pack. So this is why I think that this is a leaky abstraction. It's not working very well and it's not well communicated. And that comes to one of my conclusions in a minute. And that's this. This is all voluntary. We're still relying on developers to pick the right build pack or the right build pack to be run. Can't I force it on them? And then it says, well, yeah, you could hack Diego to say, okay, whatever build packs they supplied, add this one on the end and run that as well, but then use the other one to start. Like you could hack it on Diego. But it's like, now you're forking Diego. Do you really want to fork Diego? And for us, especially to be certified, it's like we need to be able to use the correct extension mechanisms that are built into Cloud Foundry. We don't want to do things that are different. So we don't want to do that. Okay, so now we're at the end of the blind alley. Now you know it's a blind alley, here it is. It turns out you can just upload a droplet. You don't have to use the build pack path at all. You do it when you use the Docker support, but it's actually in the API, that link goes to the API where it's like, CF upload my bits for this app. And the build pack chain is completely skipped. So if you're trying to do this in the build pack chain and try and get developers to use it, there's these big gaping holes that and weighs around it. So it's not a central kind of choke point, which is what the person who would want this functionality actually wanted. So, and also Docker containers have the same issue. So here's what you end up with. It might be good for them, security's good for them, you want them to use it, and you think they want it and they all want to do the right thing, but it's like this poor person trying to force me to crick it to this frog. I don't want it, fine, I'll take it, but I won't use it. It's not my thing. So after all this, all this sort of brainstorm me different ways to approach this with build packs. Build packs are the wrong thing. So what do we actually want? Sorry to waste your time on it, but I thought it was worthwhile to think about build pack things. We don't actually care how people build their app, we care what they're shipping. So we have to look at the droplet, we want to look at the droplet after it's uploaded before it's actually scheduled by Diego. So we want a validator hook where we can say, okay, hey, you just told me to drop it was uploaded, I'm gonna go check it out before you run it please, just hang on a sec. And then that ideally would be something we can say per org per space or per the entire deployment of cloud foundry, so our customers can actually decide it. So that's the feature we want, and something we haven't looked too closely at the V3 API. It seems like it might be better for this, there's actually a better place to put it than the Cappy workflow, but we haven't checked. So we're gonna check the Cappy, and if the Cappy team hasn't already done it, we're gonna try and get it in for V3 because it's desperately needed. So another option is this deployed active, which if you saw the keynote, you know that about, that's a weird one where it's like production and staging are completely separate cloud foundries altogether. So they're just scanning it after it's in production, which is the staging cloud foundry, and then after it passes there, then they deploy it in the production cloud foundry. Okay, that's a way to approach it. It was new to me and I hadn't thought of it, so that's something maybe worth checking out. So how do we do, prevent CV from shipping in apps? I mean, we can definitely, the scanning hole is a problem, and the fact that cloud foundry, if you're using it in the right way with the right extensions, that there is not an affordance for doing this easily, as far as I can tell, I mean, I might be wrong, because you know, I'm just a dude. CVs are ready shipped in apps, pretty easy, scan your droplets, keep your stem cells up to date, notify your users when there's a new build pack and root ifs. Preventing use of compromised app as a photo, yes, we can totally do this, and it's not even that intrusive, it's nice. The only thing is that command line inspection thing. So the takeaways from here, we are missing out on community and vendor tooling in that area, we're not able to take advantage of things that people are doing for Kubernetes and OCI containers and registries, that's a problem, because if that momentum and that effort's going around that standardization and we're missing out, then we're missing out, we're re-implementing the wheel. And there is no good expansion point for doing this yourself, so we need to actually fix that. It's not as big a problem as CI, so I just think people have a lot of doubts because of this open source upstream and what's gonna happen, and you just got these open source stem cells and how you know what's going on. If you look at the track record, they're really good. So part of it is just convincing your CISO that no, no, look at the actual track record, how long are vulnerabilities live and what's actually happening here. And then the other thing is that very small changes can improve user behavior. If you just give a user a cue like hey, you have a responsibility, it's time for you to do your thing, they will actually change behavior, so that's good. And the other part is open source is great. We got to this far, like doing all this with open source, we didn't have to talk to a vendor, we didn't, like we got to do all this experimentation, it was a lot of fun and really cool. Great thing about the CloudFinary community, the CloudFinary tech, you know, the fact that CISDIC had FALCO open source is great for us, like it's been, and the fact that we're able to publish this open source is great, so open source for the win. And your pull requests, welcome. That is it, and if you have any questions about this talk, or cloud.gov, because I kept you too long, you can direct them to cloud.gov, inquiries at gsa.gov, or I am bmogalewski on Twitter, pretty easy to find me. And that's it. If you've stayed this long, you're eligible for a valuable prize. So yeah, any questions, any comments, any feedback? Any advice? Do we miss something? I don't know, we got to talk to the Cappy team and see if they're open to it and what's going on. Does anybody know? I don't know anything about V3 API, I haven't looked at it. Yeah, I think we have to figure out, it might be easy to do, or it might be hard, and it depends on how far they are along to locking the V3 API. Any other questions? We haven't, we thought about it. Yeah, it sits out of the mic up. The question was, have we thought about just killing it on any suspicious activity rather than like making a judgment call and then killing it? We've thought about it, but until we have it in production and see what kind of alerts actually come up and how often things actually are being killed and how many of our customers are not actually running multiple instances as they should, we want to make sure that we do it safely first. So we want to kind of run it in production for a while just logging and get everybody used to like this is what you're going to see. And then based on that, we can tune the rules and make sure we're doing the right thing and then if necessary, we'll just bump all the rules severity to high and then that'll be and kill everything. So yeah, you can't do it before they do it. I mean, obviously it's triggered by the sys call. So until they make that sys call that's suspicious, you can't predict they're going to do it, but there might be some interesting things. I mean, we haven't really exported that. So maybe there's some things that come in with headers or things like that that we can look at that might make it as soon as somebody arrives in the container, you've already got them before they even try and run the root kit. So we'll have to figure it out. It's going to take some tuning. We just don't want to immediately start killing everything from everybody without knowing how much of an impact it's going to have. Yeah, well, it's true. Well, ideally, we don't like having separate cloud foundries that have different behavior. We like the staging and production to basically be identical behavior. So hopefully they're going to figure that out in the Dev side before they get to production with it. But yeah, that's an issue too. We might be killing something based on benign behavior. Oh, that was the other thing I didn't put in future work is to be able to say, ignore this rule. So in addition to saying, here's my additional rules, you should be able to say, no, this rule is legit behavior for this thing, let it through. There should be some way to specify that. Some file in the container or something like that. If this file exists, then, so you basically say touch rule X, and when you say touch rule X and then you push your app, then every single rule is gated by, if file X doesn't exist, and this happens. If this happens and file X doesn't exist, then do this thing. So we thought about that, but it's like, first thing was just to get the facility through FedRAMP and get it in production, and then we'll start tuning rules as we go. And it's going to be up in the community too, so herd immunity for all government apps running on it. If you contribute to the rule set, base, everybody's going to benefit. We didn't really do a lot for this particular feature. This one started out as what we call our innovation and planning sprint. We basically have time where it's like, hey, what about this wacky idea? And then it became more important because we had a particular CISO who's like, I demand an answer to this problem because I handle this with this vendor tool X in this other environment, and I don't want to use your environment unless I have something with parity. So in that case, we had a very specific customer case driving it of like, I want something that is essentially cis-cultured behavior analysis. So it was pretty straightforward, like what is it going to be? So yeah, this one did not get what I would call like beta testing or like even a lot of feedback other than that it needs to have a scalability. I don't really care how it does as long as it does it. So yeah, what do you think? Should we wait before we put it out to more or? Yeah, and that's why I said we're going to turn it on in production first and have it just worn. Just log, and that's all it's going to do. And then we can look at those logs ourselves and say, okay, well, who's getting, who's logging the most from this? And then we'll go talk to them and say what's going on. So that's the user accepted testing for us is the fact that we're not going to kill initially, is that we're just going to log in and then we're going to talk to people. All right, thanks everybody. Thanks for staying so long. Bye.