 Hey there, welcome to the webinar. My name is Steve Jageir. I'm a developer advocate for Bridge Crew and in this show We're going to while you just set up on screen What is the cost of a secret identifying secrets with a check off before? public exposure as I said, my name is Steve Jageir. I'm a developer advocate. That's my Twitter and Check out that loose lips might sink ships. I found that that is some I guess you'd call it I don't know if we call it propaganda I guess it is kind of called that from World War two from the United States and it still applies today It's interesting that information leakage can cause some form of damage regardless of the context and that's what we're going to talk about today. How can we mitigate the damage of releasing secrets, but also Find secrets as early as possible in our software supply chain. How can we do that? So Let's go for it. All right. So next Let's look at some horror stories Starting with the fear as always, right? The first one goes way back to 2014 Ryan Hellyer did it did actually a lot of the right things He was trying to give back by pushing his WordPress website into get up He knew that the WP config which is kind of where all the secrets live in WordPress Decide what you want about that from a security perspective, but he knew that was a bad thing to push He put it into his own directory Added it to the get ignore all good things But missed one critical piece and that was that there was a a backup mechanism that created a dot save version of that That was not in the get ignore that got into AW that got into github, sorry and Within well, he went to sleep big mistake Doing things just before you you go to bed six hours later. It wakes up. He has a bill from AWS for six thousand He was running six hundred EC to server instances. He normally has zero So that was a bit of a giveaway there and then cleaning things up was an absolute mess both in terms of AWS and In terms of github so he blocked the whole thing out there for all of us to learn Have we learned not really? Yes and no Fast forward to 2021 you have to be living in a cave to not have heard of solar winds Intern leaks passwords. They always blame the intern don't they? SolarWinds CEO testifying in front of Congress. This is not what you want You do not want to ever find yourself testifying in front of Congress. That probably means that your breach was significant, but in this case a An intern leaked an internal password, which you normally think would be safe, right? So if we move this way, you can see the solar winds password solar winds one two three I Tell you I have worked for a lot of different companies and I can guarantee you even the best intentions Fall sideways when we're creating internal passwords. They're almost placeholders. We still use password one two three solar winds one two three They it matters internal passwords matter as well, but that's probably for another webinar All right, let's move So what is a secret? Let's let's expand our definition of that if we go back to these guys here the hackers cult Slash almost comedy film from the 90s now when they you can see it like on the weird lighting on their faces above me love God secret password had to be in there surely these were the we obviously were not paying brute forcing Let's say brute forcing was pretty easy back in the 90s. You're not that are you a skilled hacker really? It's probably seen pretty easy back then Now though it probably doesn't look much better. This was from a report from Dashlane I believe it was or one password or last pass or a password management company They released there every year. They released their top passwords 2021 number one was one two three four five six Passwords in there courties in there So I'm just kidding solar winds one two three was not in there But you get the idea these are all very simple passwords, and you're probably thinking I don't use past Does anybody use password? Can you even use password anymore because there's so many constraints around what you can do? I'm sure these are legacy, right? That's why they're still there, but we haven't we really haven't human nature has made it so that we never learn, right? That's what we do. That's our that's what we're best at not learning Now what we do instead of writing password is we capitalize the P or Maybe we add special character Or perhaps we add a zero or we add a one or we put an exclamation point on the end and that's good, right? No one's gonna figure that out. That's crazy complicated. Oh, but then the my organization makes me change the password every month So, okay, I can die right got it. So That's kind of what we do now. So why would we expect anything better in the future? Now to make matters more complicated when we're talking about secrets Automation needs passwords now so now we have generated passwords and those can be really messy like if we look at a shot two five six of That word password one it's a big messy complicated thing and When we look at that we're like, whoa, okay Well, what am I gonna do with that? What if you had to use that as you're creating automation? What if you're writing some code and ansible you're creating something in terraform and you need that to? Provision something or to create something or configure something. What do you do with it? Well, your first instinct would be like well, I'll just do I hard card you as soon as you say hard code That's like a swear word in programming, isn't it the H word hard coding So you don't want to do that. You need some kind of strategy for your secrets You need a supply chain for the secrets themselves that keeps them private and That is a whole other problem So let's talk about a little bit more about that. Let's get dive in Secrets are passwords, so we're still in familiar territory But what they represent in the real world is database credentials encryption keys API keys SSH tokens that is not an exhaustive list, right? I'm sure you realize that and How do they get revealed publicly? Well, we saw those two examples You can just forget to add Files to your get ignore by accident or just simply through ignorance You can hard code them into a container image That's there's lots of ways that container images can give away secrets not just that one and we can do a whole webinar on that You can cut development corners by temporarily hard coding your credentials into something Temporarily, you're gonna remove them. It's fine, right? I'm not saying I've done that personally recently Even though I know and I'm doing a webinar on this. I have made that mistake. It's easy. It's too easy to do Finally committing infrastructure as code templates like terraform cloud formation Many are available into your github with the credentials inside and this is very very easy to do and you're probably even thinking How could that even be possible on you might be asking the question? What is infrastructure as code? Okay, it's possible. You're thinking Can you just do a quick Definition of that and I will do that one slide a one slide infrastructure as code So you want to provision things at scale? You don't want to do it manually. That is a massive pain So you there's many formats of code for doing this and there's many types or styles of doing this So there is procedural and that's probably very familiar an example of that would be Ansible and Then there's declarative and a good example of that would be terraform or Kubernetes YAML What does that look like in an abstract way? Well, think of it in terms of like making a cake, right? Preheat the oven to 160 mix flour eggs butter to a fluffy the imagine these are all commands in a script and You want them to be repeatable and do the same thing every single time So you put them into your recipe or in Ansible's terms a playbook You run it it will happen you get the result a cake is made although that is kind of assuming a bit of knowledge that I am a chef and chef interesting, okay Which I might not be what I like better and is becoming very popular is the terraform example as you can see Now I rely on a provider who is to act as the the creative the chef and I simply say I need a resource It's a cake. It's called birthday surprise. This is my spec icing fondant sponge texture diameter layers boom make it And that's all we have to do and I have a cake I really like that because you can even use that to look for a drift to see if the cake changes over time Does it match my code of definition? It is declared state Instead of procedure. So that is a rough way of defining what infrastructures code looks like Now let's go back to how secrets get revealed publicly and talk a little bit about committing the infrastructures code into GitHub How does this happen? Does it happen? How often does it happen? Interestingly, you can see up there. I've got hacker news get a co-pilot which was released just a few months ago in Beta Digitates valid secrets. Does it really? I don't know Get a co-pilot is an AI driven plug-in for VS code where a lot of people create these these infrastructures code templates And what it's doing is it's spitting out Predictive lines of code and even functions to make you code faster in a variety of different language including infrastructure as code templates so In some instances there were some claims potentially conspiracy theory Maybe a little bit of trolling that this co-pilot was spitting out passwords and API keys that were still valid Based on the breadth of its knowledge from the massive github code repository Is this real? Was it not real? Nevertheless people believed it because it it happens that often that it made sense that perhaps this was a possible thing Let me make you make it clear though. I Really tried to do it and I could not do it So I don't know if it's possible or if they fixed it right away, but it doesn't seem like it's doing it now But nevertheless What do you do if credentials are exposed? What if you accidentally did this? Let's let's put a little bit of corrective action in here just to just you get an idea Of course, you should disable the keys that you think were in there revoke them immediately Rotate them look for compromised services like what happened to the individual who Checked in their WordPress credentials and suddenly they had 600 servers running Look for your logs for nefarious activities add probably have to add new monitoring clean your get history That's not easy. I'll show you a little bit that about why that's not easy in a moment It sometimes can be easier to delete the repository entirely if it's new and just create it in your current state If the history doesn't isn't that important to you and then of course monitor your supply chain Look for breakages look for anomalies and understand that you might be part of someone else's supply chain So your mistake might be affecting a lot of other people as is what happened with solar winds So finally and the important thing why we're here today add automation to scan your code for secrets So it doesn't happen again and ideally it doesn't happen ever Some best practices surrounding that create that secret supply chain first Before you need it in automation think about how you're going to do it if you're in dev and you need You just don't have it. You don't have access to it. Whatever always put your secrets into another file Always put them somewhere else and then reference that file and add that file to your get ignore. That's smart Don't put it into the comments like I did and check it in that's dumb because even that even though afterwards when you do The code right because it's not hard coded anymore. You forgot you added it to a comment at one point and now it's like oh If you're in prod, of course use a Production-grade secrets manager like cyberk or hashi-karp free ones are available paid ones are available and almost every cloud provider has one so It's becoming easier and easier to do things right. It's more. I think In dev where things go wrong or when we're rushing or when we're cutting corners that problems can arise Which is why we need to scan Any code be it our application our infrastructure as code anything needs to be scanned before it goes into github Ideally as a pre-commit or on your desktop. So how do we do that? Well, let me introduce you to check off check off is an open-source tool for analyzing infrastructure as code in Analyzes all sorts of different types terraform cloud formation Or my helm Kubernetes yaml serverless framework. There are over 500 different rules that is looking for for common misconfigurations You can see some examples in the image just next to it there However, we have improved check off to add secrets and that is really quite interesting check off It's new feature for scanning secrets is a combination of some of the things we already had in there There were some there was some secret scanning already there combining prowler, which is looking for AWS misconfigurations like CIS benchmarks and secrets with Yelps detect secrets yep Yelps detect secrets was written by an individual named Kevin Hawk and actually full disclosure We asked him to be on this webinar and he kind of didn't want to do that and I'm all right. No problem I never met him. He's very bright guy, but when I looked at his LinkedIn, I saw that all right not the most forthcoming His about says hi no picture no background. Okay got it. No problem You want to just write your code. He speaks through his github account, which you can see right there But combining those two together We are able to create three different types of identifiers and ways of finding secrets and infrastructure as code regular expressions keywords Looking for entropy, which is interesting. We'll talk a little bit more about that in a moment So regular expressions are fantastic if I look at the different types of regular expressions We can see Artifactory AWS JSON Mailchimp Slack very specific ones right Stripe Twilio Regular expressions allow us to characterize a lot of different Credentials a lot of different secret types so that not only do we find the secrets But we can tell you specifics on how to remediate them, which is pretty cool Now also keywords versus entropy. Oh, pardon me If we look at an example of keywords, we're just looking for obvious things like API key password people do that They just say oh password equals and so there are some very obvious common loco low-hanging fruit in terms of finding secrets Because if you call that API key, it's probably an API key So we're looking for wording that fits those those keywords and then finally entropy entropy if you look at a Passphrase and there's a good example there correct horse battery staple for random words in a row that looks weird and You can find that using using an entropy based analysis and Now I could have put an API key up there like we saw earlier that Shaw 256 now that even to the human eye looks Weird so that falls into the same category But it's worth knowing that entropy extends beyond that into examples like this and what we do with checkoff is we even combine all of that so that We're reducing false positives So we're combining all the different checks together But then characterizing the check into an individual type when we're done So we've taken what Kevin Hawk has done. We've taken what prowlers done And we added our own kind of secrets off to that to make it really good All right, you saw that let's take a look. Well, let's let's do it now. All right. Let me just share my screen really quick Screen we are awesome. So I am in a Terraform Directory look there. I have terraform for a whole bunch of different formats. Okay, AWS a Azure It's all part of a bridge crew intentionally vulnerable Environment and I can run check off. So let's do that. I ran it earlier I've still got the command line here check off dash the AWS is going to check my AWS directory And I've got a you think I'm using quiet so that I get I don't get all the things that I successfully did I know I don't need a pat in the back just yet. I'm just gonna look for the misconfigurations So it's that simple. I'm gonna run that and We're gonna see a whole bunch of bright colors go by there we go It's looking for everything. I didn't say just look for secrets. I don't have a configuration file I don't have a baseline any of the other Things that you can do is check off. I'm I'm scanning fresh and we can already see some of the misconfigurations I have in here. So we just look right there like the way the bottom a base 64 high entropy string You might be thinking okay looks like Something bad clearly not base 64 example key there We also could have caught this on secret key and probably combined some of what we're doing to make sure that it was Certainly we caught this one access key AWS access key. We're being pretty specific about that one, aren't we? Private key so we're we've caught this Most likely using a regular expression and we go and we can see a few other ones slack token basic auth credentials Twilio API stripe access key that's getting pretty specific, isn't it? That we found in here and what's kind of even better about this is that We're offering guidance here. So you can see get secrets 17 if I go over here We can see the kind of information that we've been given We have very clean clear instructions revoke the exposed secrets with administrator permissions You can access the stripe api and navigating to the developer section It tells you what to do in the context of stripe itself And we can also see if we look down here Same thing goes for if it was a twilio api key, which we did have we get specific twilio instructions On how to do that. So we're providing guidance that is specific to the type of key because we characterized the regular expression specifically For that type, which is awesome And there's upwards of I think there's almost 20 different types of regular expressions. We've got going on here Now, of course the last step we say clean the get history. We don't get into that Easy just what does google clean get history and just one of many articles will come up That show you hey, let's get started a word of caution. Oh, no, this is not a short article So if you want to know the pain of cleaning your get history You can find it very easily and this is just about having a tidy history That article wasn't about how to eliminate and ensure that you've eliminated a leaked access key That can be That can be a lot worse. Let's say So that's how easy it was for me to find Misconfigurations in my code, but also Secrets that might be leaked in there and of a variety of different types And how easy it was for me to find instructions On how to remediate them Very specific to the type of key that we had which is great Okay, so what are my key takeaways? let's jump back to my screen share and So key takeaways keeps keep secret secrets. Well, that's secret secret That is the whole point of this this webinar be prepared for leaks in advance Assume that's the larger your organization the higher the likelihood somebody is going to leak a secret that would be bad Create your secret supply chain in advance So make sure you know how you're going to handle automation And secrets before you start creating that automation. That's ideal. It doesn't always happen probably rarely happens But understand what you're doing and if you are in dev and you don't have Some smooth automation with something like vault Then always put those secrets into a separate file And add that to your git ignore. So even just maintaining a best practice from an individual perspective is important Remember that internal secrets can still create risk scan all your code Not just your application, but also your infrastructure's code and not just your infrastructure's code also your application Scan locally if you can using checkoff. It's easy to install on a mac. It's like brew install checkoff super simple Add it to ci checkoff has a github action. That is as simple as running it on your desktop It's a one-liner github action and you can embed this into your ci And you can also use that as a pre-commit hook, which I highly recommend and of course finally revoke and rotate keys often in fact when you're choosing services Make sure those services allow you to revoke and rotate keys That can be an important way that you method you used to decide what services you're going to use in the future Because it's really important Okay that Is the end of this Webinar Once again, my name is steve gegar I hope you enjoyed this and you learned a little bit more about checkoff You learned a little bit more of secrets and maybe you learned a little bit more about infrastructure as code That's the end big thanks to the cncf for hosting this And if you have any more questions about checkoff or your And you'd like to maybe join our community go check us out at codified security all one word dot slack dot com Come join the community That's where people talk about the open source tools that we've got a bridge crew and request new features, etc So we'd love to see you there. My name is steve gegar now Let's do the scooby-doo ending All right anonymous hacker, let's see who's really behind all of these data breaches Danny DeVito Whoa, I didn't see that one coming My name is steve gegar. This is the end of the webinar. Thanks for watching