 All right, so next up we have both Dave and Catherine here speaking on avoiding default passwords and secret breaches using open source So please welcome them welcome them to the Torcon stage All right, thanks. So Dave Dietrich The University of Washington for quite a while programmer system administrator security operations when simple nomad was talking about history and D-DOS I was involved in writing the first five analysis of those tools and In trying to get thousands of people on the campus to figure out how to secure their machines Like came a big proponent of Newton's third law of vulnerability disclosure, which is for every vulnerability that you disclose Identifying disclose there should be an equal and opposite mitigation So that's part of the motivation for the talk here And I'm Catherine Carpenter. I'm a business consultant in information security and privacy with a background in health care and ethics I'm also the founder of Zerzer advisors and a licensed attorney, so I have to warn you. None of this is legal advice But your future lawyers may thank you if you listen to our ideas Dave and I first wrote about the Default admin password problem in 2014 when we discussed the ethics of the Karna botnet and using the data it produced And we've been working in this field since then educating and advocating about ethical information security research and Working to find ways to better protect information data and privacy Some of the work that went into this presentation was funded by a Comcast open source innovation grant Okay, the beginning of our presentation covers topics. You're familiar with That's not oh The slides aren't connected I thought that would go over automatically The beginning of our of our presentation covers topics you're already familiar with and Our goal is to get developers to think differently and and behave differently when managing secrets and handling I'll begin our presentation with a discussion of secrets and a very fundamental question. What is secret switch over? Well, yeah Seems to be is there a switch. I thought it was automatic So we're gonna use Jeff Mitchell's definition of secret and that is something that will elevate your risk if exposed to unauthorized entities This kind of information used with networked computers and applications generally controls access or identifies a user to a service And for that reason an exposed secret is a threat to the user or the system and identifiers are related to secrets But they have different levels of sensitivity and identifier like a username May be exposed without causing harm in some cases an identifier must be shared for technology to work for example DNS text record used by DKIM protocols to prevent email spoofing and in all cases a Secret that is exposed causes harm to the user or the system in open-source software projects You'll generally need to deal with the following types of secrets passwords either as clear text salted and hash Strings stored on a system that does the hashing and compares the hashes bearer tokens long strings of characters Representing large numbers including cookies API tokens and JSON web tokens that represent an authenticated user when presented to the server These are more complex than passwords, and so they're not usually entered manually They're stored in computer memory But just like passwords they require a secured transport mechanism to stay secret and Private keys are the private component of public private keys all used for encryption And for any public private key pair the private key must remain secret So where do secrets come from you can create them yourself if it's a password you make it up You can generate it automatically using a system and sometimes you need to get them from somewhere else like a service You're using if the service generates it and you have to retrieve it from them Like if Amazon generates a service you retrieve it from the AWS panel or you get a key Access to a product key for a licensed product so there's there's a secret distribution Distribution problem whenever you're trying to share secrets with someone if all of you need access to the same service And one problem we address with this presentation is how you might share or distribute secrets to others who need them differently than you may now Looking at complexities about dealing with harms and ensuring information is how you want it to be Information resides in and goes between information systems and sometimes it floats around on hard drives, too There are three main attributes to into information assurance availability integrity and confidentiality and Authentication is important because we're talking about passwords We're also discussing protection and reaction capabilities Mostly we'll talk about that later, but it's important to consider reaction capabilities as a way to mitigate problematic situations for example an intrusion of your system or a breach or leak of data we'll discuss availability integrity and confidentiality in the next few slides Information compromise can cause harm if the information is changed or altered and that change Compromises the integrity of the information because it changes the meaning of the information for example change a buy order to a sell order Change a paycheck from $2,000 to $200 that changes the integrity of the information when information is deleted or encrypted There's harm because the availability of the information is compromised And it is no longer available unless you have the key to decrypt it if deleted while there's the potential for recovery There's at least a temporary harm to the availability of the data If the information that's intended to be private is leaked to the public the confidentiality of the information is compromised Because that information is no longer private So looking at second downstream harms secondary effects leaks of credentials allow Systems allow an unauthorized user access to a system and that compromises the integrity of the system because an unauthorized person Has access to the system and can now modify files or access the system or access to the system Since an unauthorized person is able to view the files their confidentiality is also likely compromised Encrypting a file alters the integrity of the file because it changes the file It's no longer in its original form and ransomware compromises the availability of data or information because it becomes unreadable without the key If ransomware happens to encrypt critical system files the files are no longer available and the system could crash or refuse to boot up Our goal is to minimize harm and speed up the mitigation process So we're going to talk about what goes on in the real world We'll begin by looking at the complexity of the group that will use some secrets So you're just one person there's no sharing or distribution problem, but there still is a default password problem There's a small federated group say a trusted group same community or slack channel then there's a Distribution problem, but in this context you might set up a central protected server to share secrets and the turnover of the group may be limited That is an assumption Take a set of federated organizational units say a commercial group its product developers The an open-source developers within the same organization and the group responsible for identity management all at the same organization Say there's a vault server and only employees have access Authorized access to the internal network in this context. There's no sharing of services with developers outside the company If open-source developers from outside the federated group are aiming to participate in a project secrets must be shared Outside the identity management group and that leads to a confederated set of pre-existing groups or individuals confederated here means there's a common goal, but less trust because the outside Participants are non-employees There's still possibly some degree of personal trust. What is limited? Although although trust is limited Here there are no credentials going from the identity management group to the outside individuals But you need a way to transmit secrets Facilitate sharing new secrets and generate or revoke secrets from people who are no longer trusted Went too fast. The secret distribution problem can be summed up this way You need a method for creating and sharing secrets also for generating and revoking secrets as the size of the group grows the sharing of secrets become more complex and Another thing that impacts the distribution or Revocation of secrets is how frequently the membership of the group changes. So I Want to I want to Get a raspberry pi and This is the first the first of the problems is the default admin password problem If you get a raspberry pi you can log in as the default user pi and the password is raspberry Pi has pseudo access changing the password is best practice But it's only discretionary and that's just raspberry pi any open-source technology potentially has a default password So this problem is actually predated the internet Morris worm and the ARP in it two modules that had 432 common English words or names As well as a module that would take the account who would look through the password file find the account name Try that as the password try no password as the password quite effective And there have been researchers over the years who have found vulnerabilities and things like commodity or consumer wireless access points Routers things like that. There's a big list of all these default common defaults that's available it's used by a whole bunch of tools and that's been exploited despite people like David Fifeld at black hat showing nmap scripting engine being used to go out and scan for in this case his camera at home finding it trying a default password getting in and Simple nomad was mentioning the DDoS attacks that have been happening lately from IOT devices So I look back after the mirai botnet came out. This is a Venn diagram I went through the passwords that were used by these tools to gain access to these systems And you'll see that there's a massive overlap The big gigantic bubble there is a program called br.c. That was floating around in like late 1990s or early 2000s brute force on SSH Embedded list of over 50,000 passwords Some of those passwords Indicated that there were some military sites that had common passwords the reason being The passwords and the account names Indicated the bases where these accounts existed so That's one of the main things we've got to get rid of for those of you that saw the truffle hog talk a moment ago the embedded secrets and source code problem and One of the main things that he was saying start to put the search for these things as an audit mechanism into the DevOps chain That gives you the ability to keep the secrets from being breached in the first place All right, so this might be duplicative of the truffle hog talk also I think he mentioned uber, but two hackers found credentials in a private github repo and used them to access uber data Stored on AWS around October 2016 and it was only publicized about a year later The attackers found personal information for 57 million customers and drivers They also found 600,000 driver's licenses Which that alone triggers a reporting? Requirement and uber claims that no credit card numbers or social security numbers or trip data was covered but they also paid a hundred thousand dollars to keep these intruders quiet and Adobe Visual Studio 2015 had a bug in it that Someone learned about because they published a private repo and it ended up being public on github and it had AWS access keys in it and all of a sudden he had $6,500 in usage fees after doing really nothing So his lesson learned from that experience was put your access keys in a separate config file and use get ignore to exclude them in reality Also, people are searching for credentials in GitLab and using the ones they find to steal intellectual property Even though this mischief is not generally publicized in the media Okay, so if you start trying to solve this problem, you're gonna run into two people Anybody here not seen David five feel or David's Daniel summer fields talk It's a very good one He goes into a quite a bit of depth on pros and cons of various tools So I'm not gonna be repeating what he said but Look for that. There's another person max VT who had put together a Google doc that lists set of tools with Some details about those and I'll also throw in that the software development section of my home page has Significant number of resources as well So most of what I'm gonna be talking about here is gonna come from these guys as well as some of the research That I had done on previous project So if you're gonna start looking at Tactics and procedures We've got to deal with this default password problem like it's existed too long There needs to be a better way for people to do the coding If you look at These are like the really high-level recommendations from summer field You're always gonna have this bootstrapping problem if you decide the solution is encryption of the secrets Then the key has to exist somewhere You're gonna put it into a CI CD chain Then now the key has to be somewhere where Jenkins or whatever is gonna be trying to access them So you can never get rid of this bootstrapping problem. There's always gonna be some starting point so his idea or his suggestion is compartmentalize as much as you can and Use tactical human intervention where necessary try to minimize it try to structure it by policy and Focus on audit so again truffle hog in the CI CD pipeline That's one way of making sure that you're looking at these things before they get to the point of hitting GitHub and Automation of all of the tedious things Especially if you're trying to compose a large system from open source components each one of them needing passwords the ability to simplify Rolling all those passwords Revoking credentials when someone's laptop is compromised those all require automation and Whatever you do, don't try to like let's just go change everything that we have to deal with or every policy that we have about Dealing with secrets try to do it in an incremental way Bring people aboard. There's gonna be some training time. So it takes small steps If you just think about let's keep it out of a repo pushing is the problem As was mentioned in the truffle hog talk every single commit that has ever been made Contains something if the secrets are in there you just keep layering commits on top of it Whoever has cloned or pulled that repo has them So if you're using get and you explicitly add files rather than using wildcards You reduce the chance that some temporary file is going to be included get status After doing get add so that you just have some feedback. These are the things that are now in the staging area Or using get commit specific file names Rather than get commit minus a again try to avoid accidentally doing something that you're not really aware of Get diff dash dash cached a lot of people don't think of that that will show you the changes in the files Look through it methodically to see if there happens to be a comment that was left or if there was a file It shows up as being added If you just do get commit without a message a lot of people start doing you know, I'm gonna give a message I want to be in a hurry That's when things will happen that you don't realize How many people here know about get what changed it's a handy way of seeing okay good at least somebody It's a good way of seeing what files were staged what files were changed in each of the commits The thing is these are all discretionary policies You've got to train your developers and make sure that they understand this is your responsibility Take that responsibility So you're only gonna have problems if there are humans involved in coding. So that's a simple thing to fix No, it's not Excluding files with get ignore as has been recommended the people that Katherine was talking about also mentioned and get Rob It's not perfect You can have wild cards, but can you predict that I'm gonna make a copy temporarily with my initials as the extension and Then forget about it and accidentally commit it That you're not gonna find or you're not gonna prevent by a policy So a number of people are now starting to say just move the secrets out of the code repo Don't commit code that has a hard-coded password in it Someone wanting to change that must then edit that file Now it's in conflict with the original file. It's still in the repo Just make them access secrets indirectly rather than hard code Another way to deal with this is to just get used to minimizing the life cycle get used to burning the creds and Restoring them each time you're going to deploy a new system if you're in development mode So that you don't accidentally have a long-lived secret F5 Has an open-source project that requires product key That key they were trying to protect in the repo with PGP So if somebody who had previously been trusted is revoked They still have their private key. It was encrypted with their public key You can't stop them from reading that file So the lifetime of the secret must be less than the rolling over of the the untrusted people and and That also helps when it becomes an emergency and you must now go out and rekey a lot of things When you're choosing your tools you have to pay really close attention to lock-in Are you now choosing a tool that only works on AWS? Are you choosing a tool that requires that you use chef? So these kinds of things will become a problem if you're planning on growing a small group into a larger project And you want to look for tools that are easy to integrate so things that may access things from the file system or by using Command line so you can do inline command substitution. You can now more easily integrate with with other tools in terms of rolling your secrets Think about the general period of whatever the project is that you're working on so campaigns There is a there is an election that happens as soon as the election is done That's the time to go burn all the creds start over again because you're going to have turnover and staff With academic research projects. The main thing is intellectual property theft So at least your period of performance of your grant But probably more frequently if you have graduate students who are coming and going over time Anybody who is left should be considered not trusted anymore. You can't rely on the employee system Making someone who is no longer staff no longer have access and then the bigger problem is really Consumer devices you've got a supply chain of tools that you're using you may rely on upstream services and Each of those may have their own cycle for how they they deal with With secrets So when I talk about emergencies and break-last procedures This is not a good break-last procedure. This is not the kind that we're talking about How many people here know that I can is rolling the DNS sec key signing key in a little bit? Okay, so at least one person This is the key that signs the keys that sign the keys That allow you to have a device validate DNS sec They've been mentioning this for a while They've done a survey to understand what the impact is the impact will be minimal, but there are devices Probably guaranteed that are out there that use DNS sec that cannot be patched or updated That will no longer be able to validate DNS zone information So there will be some pain from this so let's look a little bit at some of these tools and Again, I'm kind of falling back on Daniel Somerfield's Way of categorizing him his first set was the source code management integrated tools and most of you here if you've heard of any of these you've heard of black box or get crypt and basically their Public private key use GPG and cryptos secrets in the repo whoever's keys were used for signing has access to them So again, they're leaving the secrets in some office gated form in the code repo Which is one of the downsides Then there's the orchestration tools So if you're doing some build automation if you're using chef puppet ansible There are solutions for each of these Haven't really looked at barbecue and from open stack, but Here you see that there's at least one Big lock-in if you're using chef and you want to use Citadel Citadel requires AWS I am So you're you're totally locked in on that one Then there's secrets as a service How many people here are familiar with hashy corp vault or have ever used it? There's a few more It's probably the well, it is the leader. It's the one that's the most popular right now. It scales really well But it is kind of complicated to set up the recommended deployment, which is a clusterized environment for high availability it uses consensus for the Unsealing of the vault so three out of five keys are required, which means if the system goes down You've got to get three people on the phone ASAP to get that thing unsealed. Otherwise your user base is down And I'll also note that Pinterest knocks here is kind of interesting and that it uses the files the file system and files in the user space views To expose the secrets from a remote system in the file system So it solves the distribution problem and simplifies the access of the secrets and then there's kind of this Miscellaneous set here. I noticed that Amazon now has secrets manager Again, it's Amazon only Crypt is kind of interesting here because they're trying to do something similar to vault It's relying on at CD and console as a clusterized key-value store for simplicity and High availability and backup And Shopify uses this encrypted JSON Mechanism so they're using JSON encrypting the data in it then there's this thing called Python secrets Which is one of the main reasons we're here So what is Python secrets? It's available right now in Pi Pi I'll be making another release pretty soon with a few little changes probably switching to calendar versioning instead of the kind of semantic versioning The core features for the purposes of this talk is It allows you to remove the need for default passwords You can have a product open-source product where the first thing you do is keys get generated passwords get generated There is no default or it doesn't need to be it also kind of helps with moving the secrets out of the source code base and then managing them in a way that You can accommodate multiple open-source components. So if you have like rabid mq and Console and a couple other things they all require some secret That mechanism to have them all be the same password is the same on all these things a sort of cheap single sign-on Is facilitated you can make them unique if you need to And it uses a drop-in model if you're familiar with like Our syslog.d and cron.d a directory where you drop things into it and then in a modular way The larger file or update.d is a how many people here have not heard of update.d It really makes your life easier in being able to have your bash rc file or your SSH configuration file be handled by breaking it up into small parts. They're all put into one file It supports multiple simultaneous environments hashi-corp's Terraform Implemented something about a year ago that they they called initially environments. They now call it workspaces Allows you to have development production staging and have the secrets and configuration for each one of them separated and To move between them easily There's a program called Mantle that came out of Cisco Which was an automated deployment of a whole bunch of machine machine learning tools and They have a single program secrets.py That has built into it hard-coded prompts for each of the secrets So if you're now going to train integrate some new tool into that you've got to go edit that program To add the prompts to then create the secrets.yml file the big global file that's used for the configuration so What Python secrets does is make it easier to then add new components without impacting anything and hopefully then If promoted it becomes a lot easier for all open source tools to integrate tools like Terraform will take secrets from environment variables and instantiate variables for the Terraform run So Python secrets will make it easier for you to have all these secrets defined export them run a sub program You're now not even needing to put them anywhere near the code repository. They just come from the environment So how does it work? The proof of concept for using this tool how many people here know that NSA has a github page with a whole bunch of tools on it Very few there are some interesting things on there One of them is this thing called go secure, which is potentially Raspberry Pi client and server for VPN and Their instructions, of course This thing is a Raspberry Pi. It uses Raspbian log into it with Pi password Raspberry So I've demonstrated with Python secrets how to to get around that problem So To There's probably time to go through them all so one of the biggest limitations I would say is Currently this program runs with the Python secrets module Which I probably should rename my program because Python underscore secrets is kind of confusing in comparison with Python secrets the module That came into Python at 3.6, so you have to be using 3.6 or greater I I'd recommend doing it in a virtual environment in this example. It's in the default the installed Python I have three six installed separate from the Mac Python to avoid breaking it Help is available at a high level. I'm using the open stack cliff framework for command line interfaces, which is really handy It allows you to define Yep, let's get past already You can change which environment you can change where the secrets are kept So if you need to move them to a shared file system or something like that it facilitates that and then it uses sub commands Which are pretty easy to implement. They're just Python objects So adding to this tool is once you get used to cliff It's pretty easy Each command can have its own help for the unique options So I'll be showing the templating command in a minute in a minute uses Jinja templating and there's some utilities built in like with Amazon security groups You want to allow access for only the IP address of the system you're using? There's a way to go get your IP add that as a variable make direct access Somebody tried changing my Apple password a couple days ago So the concept of environments This was done as Something to facilitate the ansible dims playbooks I'll mention it a little bit where you're instantiating a system like going out getting let's encrypt certificates Setting up a database that has the database secrets in it and it's used by somebody on a regular basis So there's state that needs to be kept all the backups get kept in the secret directory now You've got one place where all the files are located When you first use it, you don't have a directory by default It's dot secrets in your home directory. It gets created on first use. So it makes it pretty easy to get running We're going to clone the go secure repo the go secure repo has within it a description of the variables that are required to Burn a SD card or flash an SD card. So go into the directory the secrets Look like a standard drop D style directory structure with YML files in it Secrets are simply data structures in YAML name of the variable The prompt that you're going to give to the user the type of the variable Not shown here is there's a export option where you can say I want this specific environment variable to be exported So if you need to change the name of the variable to accommodate That can be easily done When you clone the secrets By default the environment is going to be created with the base name of the directory that you're in So as long as the projects that you're working on are unique. You don't need to manually set that So say clone from give a directory name By default it tells you whenever there are variables that have not been set yet You can then key on that to make sure that you don't try to run something and have passwords not be set Look at the time. I that's not going so Okay, so Each open source tool that you're going to add to an environment has its secrets in its own group you can list the groups see how many secrets are in each one and By default when you show variables you get the Descriptions of the variables the values are redacted. So if you're at a conference and you forget You don't want your AWS secret key to be copied by somebody and get 3700 or $3600 bill So you have to use the dash dash no dash redact option you'll see that there are types password and Token underscore hex and string String is something that a user provides the other ones or types that are generable So you start out by saying I have this set of def define secrets generate me some secrets Now you get your password I'm using the XK CD style password Which if you're not familiar with it get people to use it. It actually is really really easy In this case four words the initial letter of each of those words Spells a four-letter word in this case putt So if I put a golf ball on my screen That will remind me pu you pu TT Better than having a post-it note with the real password You can set the variables manually on the command line with a variable equals value You can Set it from a file put an ad sign in front of it You're going to read from the file to set the variable. So this is a way to import secrets from other places created a tour con SSH key So add that one For very long things cliff requires that you do dash dash fit dash width To make everything nicely formatted on the screen. Otherwise it goes as long as the last column goes as long as whatever the value is Cliff is really cool. If you're a python coder and you write APIs or CLIs learn cliff and Here's the solution for the mantle problem to find the secrets say prompt me for them simplify it Okay, so I'll cut it off here. So how do we solve this problem of Go secure requires passwords and pre-shared keys and things like that to make SD cards for the Raspberry Pi client server We do it by using ginger templating So in a ginger template plaintext password is a reference to a variable the variable is in the environment Switch environments Regenerate you have a different value. It's the output of the templating that has a secret in it so what we're going to do is We have The cloud-config.client.j2 and server.j2. They're slightly different You can get fancy and put them all in one with logic, but not doing that oops and Go secure has this helper make file So it's a little bit easier to remember what the commands are or someone who's doing this for the first time It's simplified. We're going to create the client and the server Templates to prove that the files are created outside the repo create a marker in the temp directory Run the find command looking for files in the dot secrets It's up directory and the current working directory and there is nothing that's newer because we just created the marker Create the cloud config say make cloud-config Use the templates there is a mechanism to get a temp directory created in the environment so that you can store the file with the secrets temporarily Templating puts the secrets in the file so this is the kind of thing that would accidentally get committed to github and Prove it's not there We're good so You say environment Path gives you the path to the environment add the tempter option Creates the directory if it doesn't exist returns a path So Some everything up You can protect with policy you can run truffle hog to detect but you're still going to have some problems So being prepared to respond and having tools to make that easy is super important All these different tools exist Python secrets now exists getting people to use them is really hard So I'm encouraging everybody here to reach out to anyone you know who works in some open-source project and Try to convince them to think about changing the way they're doing things to avoid the default password problem at least and It's going to take some time some documentation will be necessary So I'm just we're encouraging everybody here to get involved The slides will be published in a little bit links to all these resources including I was talking about the Ansible dims playbooks for setting up a small-scale distributed open-source system I'm still in the process of integrating psec into that So get involved help out pull requests encouraged So any questions? All right, I guess everybody's hungry