 Alright, it looks like it's about 11.30 now, so I think we still have some people coming in, but we'll get started. Alright, welcome to Hunt Teaming. This is our first time here at Scale, so we're pretty excited. So what is Hunt Teaming? Hunt Teaming is trying, analyzing your network traffic to detect the advanced attacker on your network. This is really a passion project of our boss, John Strand. He started a pentesting company and saw a huge gap in what was available for network analysis. When you look at the market out there today and the tools that are available, we have protection tools, your firewall, your antivirus, and we have recovery tools, incident response, and forensics. There's a missing bridge between that. Like, how do you identify threats that's already on your network? A command and control channel that's beaconing out your data. So when we first released this product in 2016, our boss's mom died from pancreatic cancer and we named this after her, Rita. Yup, and that stands for Real Intelligence Threat Analytics. Okay, so who are we? This is Lisa. She's a computer scientist. She's a security software developer. She's actually a lead on this project that you're seeing here as well as another project. So she's kind of the brains of all of this that's going on here. She's a data nerd. She likes big data problems, trying to come up with hard solutions for those. My name is Joe. I'm also a computer scientist. I'm currently a security analyst, so I'm doing offensive network security analytics pen testing. My boss, Chris, came to me the other day and told me his official position within the company was senior knowledge expert. Since I'm studying under him, I think that makes me a junior knowledge expert. So let me know. That's a shout out to Chris because he's put in a ton of time with education on this topic alongside John. So we're very thankful for all the work he's put into this as well. Okay, so what's the problem we're talking about here? This is probably hard to read for you guys, but this is an infographic from informationisbeautiful.com. And these are recent data breaches from, you know, large companies sorted by the relative size. So the number of people who were affected in the last couple of years. And I wish you went back to the previous years, 2015-14, because what you'd see is 2018 was huge in compared to years in the past. Right in the middle there, there's the Marriott Hotels breach. That was last November. And 500 million people's records were breached there. And that's a problem. When we're entrusting companies with information on us that could be used in the hands of an attacker, we're expecting that they should be able to hold on to that and protect that. And as we can see by the last couple of years, that really hasn't been the case. So some more numbers to go along with that. Yeah, everybody gets a data hack. 2017, we saw almost 1,600 public data breaches. There's a study that asks 1,200 companies if they've been breached in the last couple of years. 71% said yes. The average cost to recover from one of these breaches is $3.6 million. So no big deal, right? But the one we have highlighted here, on average it took companies 191 days to identify that they had been compromised after the attackers got on their network. And an additional 66 days after that to contain it, going back to the Marriott breach, they went undetected on the Marriott's network for four years pulling data out of there. And that's how they got so much out. So this is the problem we're trying to address here. And Lisa will talk a little bit about why it takes so long to do detection. Yep, so why the long detection time? Well, it kind of dates back to how humans are. We have outdated concepts of risk. We like silver bullet solutions and signature-based detection. And like this pan is hot so I won't touch it. Or someone in my village ate a mushroom and died so nobody's going to eat that mushroom anymore. We like to think that solutions work like that because that's how it works in nature. Things are very predictable. But we're dealing with the human element here and humans aren't predictable. Anything that can be automated can be bypassed. So silver bullet solutions won't work here and it's not enough to have endpoint protection. It's not enough to have recovery. We need something to bridge that gap, something that says our protections have failed and we need to implement recovery now, not six months after the fact. All right. So let's talk a little bit about what it looks like when a company is breached. In our company we actually just train cats so it's very easy. They're very intelligent. But no. So looking back at how Hollywood or even the media portrays hacking, you go back to 95 and we have all-star casts, rollerblading around, hacking through phone lines, blowing up oil tankers. And that's why I got into security. And then I was very quickly disappointed because the idea that you can go, you know, break through somebody's firewall or find a vulnerability in their web application and get into the system that way, it happens but it's very rare. For the most part, the way a compromise starts is through like a phishing campaign where somebody's sending emails with malicious attachments or through social engineering where you're calling somebody up, convincing them to go to a website, click a link. And then that's going to go ahead and download a piece of software onto their computer which is usually a staging piece of software. It'll get passed antivirus pretty easily because it's small. It's usually custom built. And then that'll go ahead and pull down the real malware, load it up in memory without touching the file system, so evading antivirus. And that's kind of how it works. Once that malware is running, you're going to start seeing it calling home, calling back to the attacker for commands to run and it'll be exploiting data at the same time. So this is a very sophisticated network graph to represent this. On the bottom, that'd be the compromise system. So that's inside a company's network. Up above that, there's a firewall. That's the edge of their network. And above that, then you'd have the internet. And that's where the attacker lives and the attacker has command and control server out there that they own. So the way this works, we have a firewall on the edge of the network and that's designed to limit traffic that's going in and out of the network. If the attacker could just connect directly to the infected system, it'd be an easy world for the attacker. But that's why we have this technology and firewalls will limit outbound connections as well, hopefully to sources that we trust. So when we write malware, we have to have this mentality or this technology in mind. And so a common way that the command and control channel will work is we obfuscate the commands that we're sending out to make them look like regular web requests. Just make it look like a user going to social media, getting on YouTube, all the fun things that IT loves to deal with. So we'll make just plain HTTP or HTTPS request to the server that we own, make it look like normal web browsing, and then that request will be the server will respond with commands that we want to run on the host. And so we just have these connections that are initiated inside the network going out and we can trade data back and forth that way. And most of the time it looks like just normal web traffic. However, you can see down here the way that works is you send out a request, do you have anything you want me to do? There's going to be a pause and that's anywhere from a few seconds to a few minutes and then same thing, do you have anything for me to do? So we're making this connection over and over and over again, sometimes on a very consistent interval reaching out for commands from the attacker. This gets a little more detailed. So this is a DNS based malware channel. These are pretty popular right now. If you're not familiar with how DNS works, it's an address book. When I go to my browser and I type in Google.com or whatever website you're trying to get to, your browser is going to go out, hit a DNS service, and the DNS service is going to respond back with an address, which your browser can then use to go access the resources that you're interested in. And so for a long time DNS hasn't really been a big concern in defensive security. It's a very simple protocol. There's not a lot of information going back and forth and a lot of companies will just set up like an internal DNS resolver and kind of just let it run. But there is malware that can leverage this protocol. DNS cat is the example we're going to be using in our slides here and it's all DNS based. And so what that looks like is me as an attacker, I set up a command and control server that's an authoritative server for, let's say, evil.com. And so what that means is anybody with a request for, like, hey, what's mail.evil.com, what's the address for that, I have the answer. So we may send it to another DNS server first, like another big public one, but if that server doesn't have the answer, it's going to come find mine so it can respond. And so what this attack looks like is you see on the bottom left there we have some encoded data.evil.com. We're sending that to the company's internal DNS resolver and because that resolver has never seen that unique string before that's going to reach out to the Internet that will eventually make its way to our server and now we have a chunk of data and we just return some data back. And this command and control channel is, it's not the most fun to work with, it's very slow because you can only send very limited amounts of data but what you'll see is an attacker will get on the network, they'll stand up another type of command and control network and then they'll put one of these on, potentially a system you don't look at very often and this is backup. So if you shut down their main command and control channel they can use this one to get more backup and running. So one of our goals with this is moving companies from the mentality of can we be hacked to, it's inevitable, how can we detect it when it happens. And that's difficult for a lot of people because we're very emotionally attached to how we have things set up like this unique snowflake and everything like that and a lot of customers we have don't want to hear that there's anything wrong. They want to believe in their endpoint remediation solutions. So the point of doing network traffic analysis is to drive any systems on my internal network that are exhibiting beaconing activity or abnormal DNS lookups. So one of the reasons we don't look down to the process level is that to us threat hunting is just as much about reducing false positive as it is about finding the threat. The reality of a threat is a scale. There's a ton of gray area in there and you can only be sure of two things. One that the malware will be persistent and two that it will need to call home. And for that you can just look at the network traffic. You don't need to filter through all of the system logs. They tend to be inconsistent and we've had customers with web cams or camera systems set up that called and beacon to China and had no system log output. Okay, so now we're going to do some math. We put together a little example here of the initial math that we use to kind of start looking at these command and control networks. So T shark is the command lined part of wire shark and you can basically point it at your network card. Just you can do it on your local host or if you have something like a switch with a span port or a network tap you can point this at the device that's listening there to collect traffic from multiple hosts. But for the point of the exercise you can run these commands on your local machine and watch your own network traffic. So what we have here, this isn't filtered out already and the next slide will have the command to do that but on the left there we have a source IP address and a destination IP address. And then we have the protocol being used, the port and then the amount of data being transferred and in this case it's all consistent. This is DNS, it talks on port 53 and you can see that the data size there is consistent for every connection and on the very far right side that number is the duration in seconds between consecutive connections here. So you can see by looking at it it happens about one second apart. These connections happen about one second apart every time. And so what we started doing is well okay so this is the T shark command and maybe you can take pictures and things you don't need to get, we don't really need to go into it but basically we're grabbing specific fields out of the packet capture that we're dumping into a CSV file that we're calling analyze.txt there. So we have the source IP, the destination IP, the protocol, the port being used, the data size and then that duration between consecutive connections and this is just being dumped out into a file. From there we're going to use R to just use some very basic statistics on it. So the command is at the top there and then the bottom five lines are the output and you can see our mean signal timing so that timing between consecutive connections is .99999 and this is over a 24 hour data set. So that's kind of confirming that this is happening once per second. At the very bottom we can see that our variance in those connections is 88 milliseconds so this is very consistent. I mean this is obviously, this is on a timer. This is in human browsing behavior and then at the top our max time delta was two seconds so at some point it probably missed a connection. So this is good evidence that this is scheduled and we can do the same thing with the data size. So at the very top our minimum there was 85 bytes and our mean size was 95 bytes and then the max was 290. So looking at this we can see that the majority of these transactions used a data size very close to the minimum but we did have at least one event that used quite a bit more data transfer and when you think about it if we have a command and control channel that's sitting there all day asking for commands to run for the most part the responses back are going to be no, no, no until the attacker gets on the computer and starts asking for things and that might only happen a few times a day so this kind of supports the behavior that we're trying to find. So one of the obvious problems is what happens when a beacon has jitter in it. So this is an example of a config entry for Cobalt Strike which is a malware framework that our testers use and as you can see you can just tell it to create a beacon with 25% jitter and that obviously makes it a lot more difficult to detect. The second problem is how are you going to run this analysis for every unique source destination pair that you come across in your connection file and that's why we created RETA. RETA takes in network traffic splits up all the unique source destination pairs and runs this analysis on there and if you have beacons with jitter they will score lower than perfect beacons it's still your responsibility to go through do some housekeeping, some white listing and research on that. So we usually analyze on a sample size of about 24 hours you can increase that time if you need to and the process is just keep analyzing, keep researching what those systems are keep white listing expected connections and try to find any command and control channels that are beaconing out. So we built RETA with Golang it takes in bro logs so you'd need a span port on your switch or a tab and you need to point bro at it bro will create logs and RETA will take in those logs and put the analysis into an open source Mongo database. So this is a screenshot when you go grab RETA off of GitHub pull it down, compile it and run it these are the options that you have and the way it's working right now this is actually when does the new release come out? It comes out in about two weeks. So this is actually it says version two on there but this is the menu for version three which is coming out soon and like Lisa said so there's an import command you can point that at the bro logs directory and I don't know if we mentioned coming out soon two will be supporting NetFlow and IPv6 so you'll get that in addition to bro logs so you import the bro logs RETA is going to run analysis if anybody has played with it before this next version will be much faster too thanks Lisa and so after that it does all the analysis it stores everything in Mongo you can obviously go into Mongo and look at results there but there are a number of options for exporting different analysis modules there's an HTML reporting feature so you can dump to an HTML file and we have modules for looking for that beacon timing behavior we have we actually have some blacklisted modules so we'll check hosts against known blacklists and we can do long connection analysis so looking for connections that are stood up and persist for 80, 90 hours sometimes those can be interesting to look at and a few other different modules on there that you guys can go play with so here is a screenshot of the beacons output for this data set that we've been playing with and the very top line there 192.168.88.2 that's the DNS cat and control channel and on the left side there what we try to do is put together all these metrics that are actually overflowing off the right side of the screen there's a whole bunch there and we try to combine them into like a threat score so you can see here it goes from 0 to 1 so this one is the perfect one it stands out at the top and this data set had 25 internal hosts and it was 24 hours long and there were a few million connections happening there so it wasn't the small data set but this malware floated right to the top which is cool to see another module is called Exploded DNS and what this does is it just gives you a count of subdomain seen on a domain so for example kind of towards the bottom middle there you have microsoft.com and we saw 67 unique subdomains for that so that's mail.microsoft.com insert product name here.microsoft.com and you'd expect for a company that big to have a fair number that you see in your network up at the top there a couple down you have Akamai that's an ad tracking network again there's 154 you expect to see that there's a lot of data that Akamai is moving and that's normal but at the very top we have r1x.com with 62,000 unique subdomains in a 24 hour period that's not normal that's showing how that DNS command and control traffic works each request we make is unique and just by looking at the number of unique connections there that stands out pretty easily and this is just a screenshot of the html report so in this case it's the user agent module which sorts user agents by time it's pretty pretty pretty easy but you'd be surprised things like curl or wget tools that you can use on the command line to retrieve web resources those will usually float to the top and we can see which systems might have scripted requests out to the internet and then things like Microsoft products will actually kind of show up near the top too and those would be potentially like Excel files with macros in them that are reaching out to the web so for a very simple concept just sorting user agents by the number of times we see them you can pull some interesting results out of it so story time things you'll find when network threat hunting number one you will find all of your security products because it turns out so I was actually doing this with a customer a couple weeks ago and all of these signatures for a DNS backdoor were there for three hosts on their network I mean I saw it immediately it was obviously a DNS backdoor and I reached for the phone to call them to say something bad is going on here and then I looked at the domain name that was being used and it's a security product it's a honeypot so you can install these on your network and if somebody starts poking at them they'll report back to their owners that something weird is happening but what this security vendor did is realized that a lot of people who install these on their network aren't going to give them internet access they're not going to allow them to or they're going to control that internet access so it's going to be hard for them to go get updates talk back to their their home manufacturers servers and so they put a DNS backdoor in and called it a feature because there's no setup required so I showed them that whoa I had no idea that was there Fitbits we had a network that we looked at a year ago or so and they had just handed out Fitbits to their employees for Christmas and just by sorting through the network data I think we showed them that 25 to maybe 40% of their network traffic was from Fitbits like maybe we should put those on a different network guys misconfigurations one of our one of the hunts we did so they had a few hosts in particular that were very noisy to the point where they I mean they just drown out the rest of the traffic in the network and so we had them go look at it and like do you guys know what this is we'll see that with things like internal DNS resolvers they make a lot of requests and if we're logging that traffic as well that'll drown stuff out and we ask people to filter it but this was some random hosts on the network just generating thousands of connections per minute and so they went to look at it they got back to us an hour later and they're like well that's the HVAC system like what? so they discovered when they set up the config file for one of their HVAC systems there was a duration you could put in there for how often one of the controllers talks back to the main server to report temperatures or whatever it was reporting and they read it as seconds but it was in fact in milliseconds so it was talking or several of them were talking every 10 milliseconds and like okay old systems trying to use Telnet we were actually at a hospital they found a system using these tools that was talking out to the internet and they're like we didn't even know that host was still up like it's hidden somewhere inside the hospital and it was a piece of medical equipment and they're like we don't know what's going on there so we kind of dug through it and we found out it was trying to use Telnet to connect to something out on the internet which used to be so the host was trying to connect to they used to own but they don't anymore and so they're concerned that they're spilling patient information onto the internet and somebody's picking it up we ended up going through the logs and found out all those connections were getting shut down by a firewall and it wasn't sending anything it was just trying to make a connection but yeah, just interesting things you find ad trackers and spyware one of the best things you can do for your network is block ads because it takes up so much traffic and when people install little browser tabs and things with spyware in them they behave like malware and they're talking out and sending information about the user and you can see they're scheduled they make connections every so often and send data out and then Department of Defense denial of service so I only heard about this one from our boss John but apparently one of the hunts he did the company discovered they were sending a whole bunch of data to an IP address owned by the DOD and of course this was very alarming because they had no idea why it was happening and they discovered this was a tool that they had developed internally and they made a typo with the IP address for an internal host that everything was supposed to talk back to they deployed it on every host in the network and so they were sending data to the government which I mean they're probably getting anyway so no big deal but yeah those are fun things you'll come across and of course hopefully you don't find malware obviously that's the goal of this to discover compromises but you find fun things along the way that can help clean your network up and make it a better place so to recap we want people to move from prevention and I mean we still want you to use your firewall and antivirus but we want you to do more than that we want you to move towards assuming that it'll happen and executing techniques to locate those machines to locate those calls home and not focus on just remediation individual endpoints and a lot of companies do want that we were just at RSA and so many people walked up and asked like do you clean it up for us do you do remediation and we don't you still need to have a human being look through and examine all of that do their research figure out with that old medical device from 1995 is and do some housekeeping whitelist things reduce false positives so that when it does happen you can locate it so with that I think we have 10 ish minutes left we got done a little early so if anybody has any questions feel free you can go check out Rita on github and while you're out on the internet go to black kills information security dot com or active countermeasures dot com their sister companies they developed this and they put out a whole bunch of webcasts on the topic and on other security topics so if you guys want to learn more this is kind of just an intro to what we're trying to do here but they have webcasts that are two hours long that go into much more detail so go check them out and if you guys have any questions we're here and we can try to answer them so thank you so detecting sequel injection would that be like oh sorry so he asked we don't mention sequel injection here what are we doing to detect sequel injection so I guess I mean that kind of falls into the end point security because you'd be logging that on an end point I mean in theory that is data that could be going across your network though but we're not doing any sort of actual packet analysis so if things are going across encrypted we're not trying to decrypt or anything and really most of that traffic would be potentially encrypted and wrapped up so we're kind of just trying to have a higher level view of it and just look at behavior and not so much what data is actually going across if that answers your question yep so the question is 24 hour period how is that defined can you define that as the user yep and you can point Rita to a bro directory that has more than 24 hours in it and it'll import all of that and do analysis on all of that typically people tend to run it once every 24 hours and that's the minimum we suggest for getting a good sample size how long does it take to run the analysis with this version much faster than the last version so it depends on the size of the network if this is if I'm running on my home network on a machine with four cores couple gigs of memory and the family computer's on there it's going to be done in seconds it's very quick if we're talking 10,000 hosts in a very large corporate network um Lisa's saying now it's going to be minutes in previous version it was hours and so we had people running it scheduling it to run at night but yeah it's speeding up very much now so what platforms does it work on yep it's Linux based so before when we had to do a hunt team we would write custom log parsers which was awful because people's logs would switch field names like halfway through and I don't even know how that would thing and it would take a really long time and then we were putting all of that info into parsing it out and everything like that so yeah this is that's why we moved towards throughout the way of getting like standardized logs so what percent of network traffic is ad trackers it is 100% um again it depends on what kind of network you have if you have a whole bunch of user systems on there it's going to be higher than if you just have some servers running services I don't know if I can give a number company that had 40% when we yeah but it kind of depends on their policy because some companies do block those like content delivery services but some don't and then it's a lot so we do the um so we used to do it like that but now with the v3 that's coming out um we do the import and we only store the analysis so we read through the bro log we analyze it on the fly and we build on that analysis because you can choose to now append to a rolling data set um and you can define like the number of chunks you want in a certain time period and everything and um and then yeah so we only store the results of that analysis and the bro logs just stay we don't store each individual connection just the results yep and then so like when you do the command line output it's just querying Mongo again and I mean you can write your own things to pipe that into Elasticse or anything else that you want yep but some people do tell us that they do that so maybe one of these days we should write something for that so she asked with Wireshark you can do the same kind of analysis but on smaller pieces of data compared to looking at a full 24 hours worth of data I guess I don't have a great answer for you because some of these like we showed here you could definitely detect if it's connecting every second you could detect that in a couple minutes worth of data if you have something where maybe it's only talking out once per hour and it's not necessarily a two way thing it's just at this point sending data out that's where you're getting into more difficult things to detect and hopefully with a bigger data set you can get some visibility there so the question was in terms of system resources like how much processor, how much RAM we recommend 32 to 64 gigs of RAM and a solid state drive if possible for those Mongo lookups but I mean running it on my computer with a bunch of data sets and it's working fine so it's kind of up to you for how you want things if you want things fast or not but you should just be able to do a regular system for that amount of hosts and if you look at the documentation I think there's some recommendations there too any other questions well thank you thank you very much thank you yeah absolutely that's coming out in version 3 in a couple of weeks and it has the capability of doing rolling analysis so it'll right now it's for 24 hours so at any point you'll have 24 hours of data and you'll tell it say you want it has to be divisors of 24 so say like you want 12 chunks it's every 2 hours so you'll put 2 hours of bro logs into it and when it gets to the end it'll push the first chunk out add another one the beginning analysis gets re-aggregated and so you'll have a rolling data set and you can write a cron job to automate that import step cool I think we're good oh there's one we'll cluster this yes we have people doing it so if you have like several egress points on the network throw a span port on each of them and you can aggregate all that data together absolutely just make sure you name your data sets so you recognize those systems that's the main thing and yeah absolutely sure what do you mail it to send them to you right now can everybody hear me okay so it looks like we've got this all sorted out we're going to have to run it from a pdf file because the my machine is completely flubar at this point so my name is Ty Shipman I'm going to go through the introduction pretty fast I'm a PCI compliance auditor for a company called Compliance Point based in Georgia I do all kinds of security audits for the most part for customers around the world so where do you start if you're not a security expert these are some of the possible domains I can't take credit for this slide that's incredibly imposing if you're new to the security environment but these are all the sub areas of knowledge you might need to know in order to be a security auditor or an expert in the field all these slides can be found in my LinkedIn so you don't have to take snapshots just take a snapshot of the last page of the slide deck and then you can go and find it all so with that said my so basically what I'm trying to do is I want to answer the why why are you having a control in a specific implementation a specific security requirement document why is that there most people don't think about why it's there and my premise of this talk is if you don't understand why something is being done you're liable to do a lousy job at the implementation so my whole idea is I want to give you a framework to try to answer so there's an old proverb called want of a nail I had a really nifty video but I can't show it but the want of a nail goes like this for want of a nail the shoe was lost for the shoe the horse the messenger the messenger the message the battle the battle the war the kingdom all for the want of a nail the idea being that the seemingly small detail causes huge problems down the line this is essentially a version of the hacker's playbook you can kind of think of it as the blue team has to do everything perfect the red team only has to find one mistake the one mistake and the want of the nail was the farrier forgot to put the nail in the horseshoe and the kingdom the kingdom was lost because of that if you ever want to see a great rendition of want of a nail go watch this youtube video it's about the uncle delivering a very important message to his nephew in that so in my version want of a patch the patch is the nail for the want of a patch the security of the application was lost for the want of a secure application the host was lost for the want of a host the network segment the network the data server for the data server the data for the data the company was lost all for the want of a patch this is essentially the hacker's playbook in a different form find a vulnerability penetrate move across the network to you find something that's monetizable monetize it ruin the company okay there's all kinds of standards out there that your company may want to adopt or you may be implementing right now most of them are descriptive which is they tell you what they want in the end they don't tell you how you have to do it the idea sets is a little bit different from that it actually says you must put in a firewall you must put in the firewall rules that deny access to everything as a default you must do this there are 12 controls and 240 requirements we'll go through some of those later on with this framework the idea being that what are you protecting what class are you protecting you're protecting the application at that point nothing else right so a patch can be a patch to existing application or the OS or homegrown application that was coded incorrectly it's got a security vulnerability in it you're trying to plug the holes for the security vulnerability because that is what the hackers and the red teams go after okay I contend the PCI is a good place to start because it is very descriptive it covers many many areas it was originally developed by the open card brands, Visa, MasterCard American Express and Discover it's now being adopted by other areas HIPAA in specific Judy PR another place where you see a lot of people implementing PCI security controls in order to secure the data for Judy PR as I said there are 12 requirements and 242 controls that are inside the PCI DSS again the PCI DSS is around for securing credit card data but I choose to make a few modifications to that and we'll look at those just in a second here I'm speeding through some of these things oops sorry too far in the PCI DSS you'll see acronyms called CHD or CDE in the credit card environment that's card data environment I choose to call it the critical data environment it is whatever data you're trying to protect from being monetized your customer data your trade secrets formulas the data in your database that you're building your business around whatever your secret source is the critical data environment and then CHD they call it cardholder data it's basically the data that shows up on a credit card that's the critically held data right what are you holding in your environment that can be monetized okay so it doesn't really matter what you're trying to protect as far as the data goes where it's coming from it all needs to be protected okay so some of the requirements are fairly easy to understand install and maintain firewall configurations to protect the critically held data this is protecting the application the network and the segment this is to prevent general intrusion to open ports that you might have on a machine that you don't want open to the public right go to the next one or down below protect stored CHD so you're protecting the data at that point and there are specific requirements around protecting the data basically encryption so even though you've protected it on a machine that's supposed to be secure encrypt the data in case it actually is captured slow the hacker down so they can't monetize it so they can't extort your customers for you there are 12 requirements we're going to go through some examples of the controls which are what this talk is about why are we trying to do this so if we look at the first one of the first controls is keep a current network diagram that identifies all connections between the CDE and other networks including wireless networks we're looking at the application the segment the host and the data at this point with this control what I'm trying to do is if you look at the want of a patch all of the various levels of want of a patch are part of the onion inside of your secure area you have the outer layer which is the firewall the next layer would be the application the host the network itself so you're trying to build an onion of security through the environment if you look at a specific control it's not always evident why that control is there again this is why I think if you understand why the control is in place and what you're trying to protect with this control you're likely to do a better job at the implementation level and see other places where maybe this control should be applied in your company besides the area that you're currently trying to protect wireless environments connected to the CHD you need to change the password you wouldn't believe how often this is found in commercial systems it is the number one vulnerability of just password control overall the large DOS attack that happened two years ago now was caused by IoT devices and that's because the passwords weren't changed on the cameras and things like that and they use them to reflect against the service provider that knocked out most of the east coast internet for a DOS on the DNS system all because the passwords weren't changed there are specific requirements inside the PCI that say you must do this you must have a plan to stand up that device and change the password okay so going forward you're going to get to apply your knowledge so audience participation maintain inventory of system components that are in scope so scope is a very interesting idea scope is the idea that machines that are connected in the path from the outside the internet all the way down to the data that you're trying to protect all the machines that that data flows through are in scope so the firewall the switch the application the host the next firewall that you might have internally the switches connected to to the application server all the way down to the database all those things are in scope if you're maintaining a system inventory what are you doing there you're trying to protect either the host, the application the segment, the network the data or the company why would you have a list of machines that are in scope you gotta have a list to understand what you have to go after right segmentation does limit scope absolutely and it's a really good idea flat network is horrible flat networks are you have one giant IP space for the entire company there's not a router between the finance group and your technical areas right so segmentation absolutely is a great idea and the PCI actually recommends that you have segmentation it doesn't say you must it just says it's a better idea because the implementation of your network is up to you sometimes flat networks are exactly what you want because everything's in scope and you don't really care other times you want segmentation just to create barriers logical barriers I would suggest segmentation across every network in my house I have four segments I have one for my I have a specific for my IOT devices one for my kids one for my work and audio and stuff like that so my work gets primary output when I'm working I don't care about anything else I really want the telephone call and the data and my screen share to go out so you can also apply preferences to network traffic with segmentation control yeah QOS based on segmentation but you want to have a list of the components so you understand what you have to protect this also comes in play when you're looking at vulnerability lists if you don't know the applications you're running you don't know the hardware you're running when a vulnerability is announced you may not know if you are vulnerable or not unless you have a full component list in the PCI environment it goes all the way down to the libraries that you're using in your software for the applications that you're growing or using because we are all building on top of open source software these days and if you don't have a version of the libraries that you're running when there's a vulnerability on that library guess what happens if it's exposed you get penetrated anybody think of the great example of that happening in the last 24 months experience one machine wasn't patched because they didn't have it and the person who was getting the emails about that component was ignoring them in fact there were 7 people who ignored the email that the vulnerability existed on a system that wasn't patched and hence the great escape so keeping cardholder data storage to a minimum by implementing data retention and disposal policies why would you have disposal policies why are they important why would you want to throw away data I'm sorry you want to shred it or degauss a hard drive or something like that but why would you want to throw away data I'm sorry I heard too many at once others sorry liability if you don't need it throw it away the longer whoops what happened it went into complete because it's not plugged in so liability the longer you hold the data if you get sued that data comes in scope if you have a data retention policy and it says after 3 years you can throw it away and you do and you get sued in the fourth it's not a problem if you don't have a data retention disposal policy and you throw it away the day after you get the summons you've now destroyed evidence that's a criminal offense so as Nixon pointed out it's not the crime that gets you it's the cover up so it might apply to our current president of the future too so you want to have data retention and disposal policies there is a story that I will tell you how many remember Ashley Madison the hack for Ashley Madison yeah so this was kind of a soft porn site where you would go and you would try to hook up with men or women it turns out that a lot of guys were going there and they were talking to chatbots so you could sign up for free but if you wanted your data thrown away you had to pay them 25 bucks an interesting business model so this guy paid it 25 bucks and they had thrown away his data they actually didn't a year and a half later the company was penetrated and the hacker said if you don't shut it down we're going to leak all the data to the world of course they didn't care about the customer because they didn't delete the data in the first place so the hackers released all the data so some enterprising person got all the data found out that somebody in his hometown was on this site wanted some 500 bucks to not release his name the guy was so embarrassed he ate a bullet committed suicide because the data wasn't exposed so when you have data retention policies you are understanding what data you have in your possession and what harm it could do if it's released to somebody besides your company so you need to think through these problems because you're not only protecting your company you're protecting the customer control 401 encrypt the data why would you want to encrypt the data make things harder yeah again you're trying to protect the customer here and the company if you're sending sensitive financial transactions across the internet that aren't encrypted could be leaked or captured by a man in the middle or something like that right deploy antivirus software to all systems affected by malicious software typically windows right not necessarily mac or linux but it's getting there and now iot devices there's a company that focuses on looking at iot devices and looking at changes to iot devices the carbon black people their product is really good if you're running a lot of iot devices i suggest you look at the carbon black product why would you want to deploy AVS software what is AVS software what is the virus doing in the first place so you click on a link and you open it up in your browser and the virus is loaded to your machine what does it do okay but generally what's happening it's looking for a vulnerability in the browser implementation right so now we're looking at the hacker's playbook or the want of an ale right you haven't patched your browser so the virus is actually able to get into the browser it executes its payload the payload runs it escapes the browser's sandbox and gets to your host now you've lost not only the application but you've lost the host this thing phones home the control network and now there's a human on the other side of this thing that's got control of your host what does it do so explore your network see what other damage it can do in the case of the not petrovirus it launches automatically and it captures your entire environment in less than 5 minutes kind of trace every single host on your environment if you get caught with that one so horrendous virus and you really can't stop it so explore your network it's that good it's probably state-owned in the original version there's a great there was a if you're interested in that virus and you want to know more come talk to me afterwards there's a great story about that so you're trying to prevent you're trying to protect the application by running AVS now there are two types of AVS software there's signature base you need to run both I want to tell you you have to run both signature base firewalls signature base AVS only capture what has been known in the past it doesn't capture what's in the future the behavioral base will capture what's in the future so you need to go after that okay why would you want to have a environment and process in place to make sure that you are updating your virus software your AVS software yeah to try to stop them anyways to stop as much as you possibly can how many people update their AVS software every day once a week once a month never I'm serious you've installed it and you've never upgraded it you need to do it every day you need to set your AVS software to get new signatures every day generate audit logs why would you want to generate audit logs in an environment besides filling up your hard disk with random pieces of information yeah yeah so think about the want of a patch once somebody gets on your system what do they do they've escalated to administrator or root in a Linux administrator or windows system what's the next move yeah they need to move and I don't know the guy's password perfect you want audit logs to make sure that you can actually find out what happened and possibly where the attacker went next you want to be able to follow how much they were able to capture of your network so audit logs are very important and this is for every single system in your environment not just the web servers at the front end but every single system we've just had all kinds of problems with this thing I'm really sorry about this so every single control in the PCI DSS like this and you basically look at every single control and you say what am I trying to protect what is it that I'm trying to do so you can understand why you're implementing that specific control so let's say you've done this and now you're thoroughly confused because there are 240 controls that you have to implement in a system that's never had security applied to it besides going to the bar and getting drunk because you're going to be overwhelmed for the next six months implementing security controls and your boss is going to go crazy and everybody in your company is going to have problems because you're going to shut stuff down when you didn't mean to I'm being very serious about the latter part getting drunk doesn't help just gives you a headache in the morning yeah so there is the PCI has put out a very good starting guide it's called the prioritized approach the PCI DSS prioritized approach it takes all 240 control and it groups them into six categories and you implement category one first which is essentially your network control they deem those the most important and then you go to AVS and then you go to log in and then you go to things like IDS intrusion detection intrusion prevention system and then policies which are the game plan for the war right go through all of these things in the prioritized approach in order to put in a security practice at your company how many people in this room are actually dirty posture at all how many have measured that how many have measured that against something besides just kind of waving your hand in here okay a quarter of you okay so for those of you who haven't or you feel that maybe the measurement wasn't quite as rigid as you'd like go through the PCI DSS and mark down what you've got in place using the prioritized approach it will give you a risk score it's an Excel spreadsheet that you download and you can say it's in place or not in place and be honest with yourself if you don't have the full implementation as required by the PCI it's not in place you can kind of fudge it a little bit and talk to your boss about how you should do some of these things but the prioritized approach will give you a good basis of measurement that you can do yourself it doesn't require a lot of expertise to do it's a very easy thing you read the requirements do we have a firewall does it have a denial at the bottom have we restricted inbound access have we restricted outbound access right everybody does a good job with the inbound but most people don't limit outbound connection the attacker gets in because somebody clicked on a link you know in an email infected the machine and they're able to go out without regard if you limit outbound connections you can stop the attacker from getting out you can sequester them inside your network you basically neuter that specific event so do you have these things in place if you don't you don't have the control in place or the requirement in place measure your implementation against PCI DSS you will be surprised what you find that we're going to do better than others okay let's see what else was in the talk that was not that you didn't see I think that's about it I'm really sorry about the technical issues my machine had all this stuff on it and just crashed kind of just trying to put it together questions from people yes sir the current version of the PPI is 3.2.1 it's gone through three major revs and about a dozen intermediate revs along the way over the past ten years it's gotten harder they've raised the bar they keep adding requirements who talked about segmentation you did young man so before if you were a I'll give you an example before if your service provider and you were using segmentation controls you only had to have it tested once a year by a pen tester they've now added a requirement that you have to test it twice a year at the audit stage and then six months later because what they want to make sure is that the segmentation controls that you put in place that they point in time and the audit are still in place at the mid-year so they keep raising the bar it's going to go through a major rev probably in another two years they typically do revs every 24 to 36 months so sometimes the three series came out four years ago so the four series is probably another two years away or so but they keep increasing the bar they are dealing a lot with mobile technology that's the big push right now and viruses the virus virus malware has gotten very smart very very smart mostly because what happens is a government backed agency in some country ours included will develop billions of dollars with 200 person teams that develop this malware and then they release it on a target and they don't quite sequester it well enough and it escapes in the wild and it affects some commercial company by accident that happened to be a service provider to the place they were trying to infect the service provider calls in a commercial response company who discovers this thing and because wow this is really cool we have to write a white paper about this and here's the code and we need to make sure that everybody knows about this virus now and once they do that the entire world has this virus that's super super smart and the hacker comes along you know or the script kitty really comes along and he puts a new payload in it so instead of going and I don't know causing problems with the fissionable material centrifuges in Iran it's now installing crypto mining systems and they're launching it against these commercial companies to do crypto mining that's the new thing why take over somebody's machine and ruin their business when you just steal about 20% of their cpu cycles and mine crypto currencies so that's the new model and then protect the system so nobody else can do it so once they're in there your systems get bolted down because these people go around they make all these system changes they can't come in and displace them so they protect their turf kind of like a little it's a war so crypto miner team A or crypto miner team B once A gets installed they don't want B to get installed so they protect your systems and they monitor your systems for attack and then if that attack happens they fight back against it so yeah it's why pay for power when you can get it for free from somebody else and then mine crypto currency so yeah it's gotten better you know they're making things a lot harder for people to pass for companies to pass which is great for me but it causes capital expenditures for people yeah yeah no it's very relevant it's very relevant the biggest thing about the PCI is it's a floor it's a floor specification it's the minimum requirement that you should be doing in your company to protect your environment it's not the ceiling you can go and do a lot more things than what the PCI says but it's a great floor measurement system and if you don't have security if you don't have a security posture or a stated security posture take the PCI start putting it in place even if it takes you a year or two to do it you're going to be way better off at the end of that exercise you know even if you get 50 to 60% of the controls in place you're going to be better off than if you had nothing right so it's a very relevant place to start and to measure again yes sir I put a router in I use a firewall that has multiple ports on it I use a PSense and I put it on a box that had 8nix you can download PSense it's an open source firewall system it's very easy to configure very good and you can put it on a little I got a pizza box I bought on ebay I think for 200 bucks and I uploaded PSense on it and I have multiple nix and I basically if it's a firewall it's a router by default so I just created a PSense box with multiple nix so if I want to stand up a test network I've got a nix to plug it into so both I'm not going to recommend a brand because all the brands have different pluses and minuses it's very different if you're trying to deploy for a single machine in your house versus a corporation with 10,000 nodes so what you want is you want to make sure that you run both a signature base and a behavioral base so most of the behavioral base they're looking at what they model the machine and then they try to figure out what behavior is different oh it's going someplace that's never gone before on the network throws an alert for somebody to go look at if you get penetrated it's going to cost you 3 to 6 million dollars to clean it up no no no you don't know you've been penetrated yet there are two types of system managers those who will admit they've been penetrated and those who don't know they've been penetrated okay so assume you've been penetrated because you probably have you know it's just an it's just an idea of how far somebody has gotten in your network because you have people who click on links on all kinds of sites in your business unless you're running a really really tight network policy right so assume that you've been penetrated you probably have it's just a matter of how far they got so getting buy-in so what I tell executives in a company is how much energy and how much money have you personally put into this company and most of them will tell them tell you years and lots of money I've staked my career and my retirement on this and then you say well how much do you spend on insurance every year for this company and they'll give you a figure and you can say well we should at least spend that much on our cyber security because that's what cyber security is it's insurance what you are never ever going to stop somebody from attacking your network and getting in what you're trying to do is to notice them quickly so you can shut them down and remediate the problem as fast as possible so in all of the things that you're trying to do in a cyber security policy in your posture you're trying to make it harder for somebody to come in your system and disrupt your business you can go on google and you can search for any number of examples there was a guy who upset a customer and the customer happened to be a fairly skillful software engineer and so the customer started to penetrate the network and he spent about four months crawling through this guy's network and then on a three day weekend he erased every single machine in the company the guy came into his company that he had been building for 15 years everything was gone including his online backups for the last 10 years because he never took them offline so it's an insurance policy you want to make it harder so they go someplace else if somebody really wants to own you they're going to own you if the chinese army wants in your system they're going to come in they're going to spend time profiling you they're going to get in what you're trying to do is to prevent the drive by so is it easier to knock over a bank that has all kinds of alarms and gate controls you knock over the 7-11 easier to knock over the 7-11 than the bank so you go knock over the 7-11 that's what a lot of cyber criminals do they knock over the easy stuff the ones that are targeted for walmart and all these other places they spend years crawling through a network and doing the orchestration very slow I watched one hacker five months he moved three machines in five months he would come in about every two to three weeks and he would execute one command and then he would leave and he wouldn't come back and then random time he'd show up he'd execute one more command and he was just doing this to avoid all of the ideas that we had in place unfortunately he made one mistake at the very beginning and we caught him just before he was able to monetize something we turned it off because we were watching we were using him as a canary where were our holes and we had a plan in place that every time he moved we would figure out in our test network how to block him and fix that hole so when we pull the trigger we block all 55 points once in our deployment so you can learn a lot from somebody who's smarter than you okay I'm being told that I'm done thank you very much ladies and gentlemen if you have questions come and talk to me can you guys hear me wait for one more minute and start okay let's get started good afternoon everyone thank you for taking time joining our session so this is my first time at scale I've been so great to attend so many great presentations so how many of you have here zero trust security it's better so how many of you have zero trust security if you have please raise your hand I see a few hands very good how many of you are actually moved towards a zero trust security model are working towards it well good go to see you guys you are actually making the world more safer okay so my name is Mujeeb Buhab this is better okay so my name is Mujeeb Buhab I have my colleague here in rehabilitation we both are from Verizon media we have two of us TechCrunch, AOL and many other brands so in today's digital world we need to defend against modern adversaries to protect our consumer data and resources and that's this topic about zero trust security with Athens so in this session we'll ask for what is zero trust security why it's important what is Athens and how Athens enables you to achieve zero trust security so let me start with the quote from John Kinnebuk came up with the concept of zero trust security so in the digital system why trust is treated as dangerous vulnerability because trust is also an exploit the bad actors out there can leverage the trust we have placed on our network to do the bad things so let's take a look at it so the traditional 10-20 years old of network perimeter model we have like a DMZ or perimeter network it's like a cast of to protect all our resources we also have like ADC network load balancer or security zones those are like a thick and thin walls within those castles to further protect our resources with so many massive data which is happening regularly we know that this legacy our old network perimeter model that's once we pass can use a beacon or a stolen password or the attacker can send that all the way through the mail and gain access as a legitimate user to the network we also know that once attackers breathe the network they can also move laterally to more valuable assets which are not well protected within our network so it clearly shows that this notion of trust but verify the perimeter model is no longer secure as once we pass so zero trust notion is the never trust and always verify so basically assumes that relying on the network security model is not the only way to protect our resources so exactly the zero trust zero trust is a security concept unlike the traditional belief of trust but verify it's centered on the belief that organizations should not automatically trust within their internal or external network everything should be verified before granting access and thus any and all trust among the participating entity has to be explicit and limited since zero trust is treating the surrounding environment as inherently untrusted it mandates application of certain core principles the principles like traffic encryption whether any communication between our ways of client or components within the network must be encrypted right the second is always authenticate anything and everything trying to connect your systems and authentication itself is not enough you need to authorize authenticated principle whatever action authenticated principles trying to perform on your resources is authorized and the trust should be the dynamic the trust entity gets it should be very limited duration lastly the least privileged access whether it's organization or software or hardware whatever the privileged gates should be least privileged to perform the action is trying to perform it so we talked about this core principle one of them is obviously authentication so want to spend some time on service identity and authentication authentication is a way of verifying that proving the claimed identity right take a look at that identity first before we dive a little bit on authentication side so there are two type of identity humans and service what I'm calling service here is anything other than the human a service can be a workload or a job or a piece of a code or even a service itself so getting back to the human side user authentication is a pretty much well known problem there are many identity providers so humans get authenticated using username and password with a multi-factor authentication biometrics so it's a well known solve problem pretty much most of the identity provided today by default do this in your trust kind of security however how do we trust instance of service running on them claiming an identity how do we trust in the world of man in the middle of the talk or DNS cache parsing or off spoofing the service running on them claims who it claims to be right so we need to have a strong authentication for all the services as well there are many different way we have been doing the service authentication the stronger authentication solution basically is that X1 and certificate based authentication so in the X1 and certificate based authentication the service uses a digital certificate when it communicates to other servers basically exchange the certificate between those two servers and we are not authenticating each other the X1 and certificate is basically digital certificate based widely accepted international X1 and public key infrastructure standard to verify that public key contained within the certificate represents that identity represents right so the stronger you know that X1 and certificate based authentication is much stronger the couple of years at Verizon media when we decided to move towards of X1 and certificate based authentication we also wanted to have quite a lot of additional requirements to before we jump on to that switch everything to X1 and base authentication the first thing is you want to make sure that every client out there in our infrastructure has a unique identity in the form of X1 and certificate the second thing is once the client has an identity we can do a stronger security by doing a mutual TLS authentication so most of you are familiar with the TLS SSL thing right so the client is basically authenticates by making sure that whatever the server representing the certificate for example mail.yao.com sports.yao.com so the client is making sure that it's talking to the right server by verifying the certificate similarly we want to make sure that every service also authenticates all the client so client has to represent its own identity in the form of X1 and certificate so that server can authenticate by trying to connect it and we also want to make sure the certificate is pretty short-lived usually when you buy a certificate from public certificate authority like Digi said usually get the certificate for anywhere between 6 months to 2 years but one of our requirement is that you want to make sure the certificate is very short-lived basically duration of an hour or days and not months and one way to that you can buy an excellent certificate and use that for all your services but we also wanted to make sure that every instance has a unique certificate so buying one client certificate from certificate authority and use it to thousands of instance where that service is deployed we want to make sure every instance has its unique certificate and obviously we need to boost identity automatically and since the certificate lifetime is very hourly or days we need to automatically make sure that certificate get trust as well and lastly we also wanted to make sure we do some authorization on the certificate which we'll cover shortly so with all these requirements couple of years back when we looked at the community open source community we couldn't find any open source product which we can just use it there are a lot of frameworks about zero test security certificate based authentication but there's no product exist which meets all these kind of requirements and that's why we built Athens Athens is an open source system it basically provides two major features one is service authentication and authorization in service authentication it provides secure identity in the form of X1 and certificate to every single workload service deployed in the modern environment whether it's a private cloud or a public cloud and the second major feature is that it also provides a fine grained role based authorization system so we know that Athens can do this authentication which will issue identity it can do authentication and also do the authorization but the advantage here is that how do you bootstrap this identity to automatically secure weight automatically to all our instances that's where we build a solution called Copper Ockers within Athens it's a generalized model for authorized service provided to launch other service identity in authorized way through a callback based verification model so with this model Athens has been integrated with some of the open source provided like OpenStack so in OpenStack every OpenStack virtual machine or a bare metal when it gets bootstrap it automatically gets identity again in the form of X1 and certificate similarly in AWS AWS easy to instances or AKS or ACS the containers gets own identity Kubernetes same thing if you are using Kubernetes also integrate with the Kubernetes where containers can get all the identity whatever service gets deployed on the container of the parts we also integrated with a couple of other open source product which we open source from Yahoo like screwdriver screwdriver is a CI CD system basically every single job or build within that builds a CI CD environment can get its own identity and you can use that identity to for example if you are publishing an artifact to a Docker container then you can authorize that only that particular build to publish it similarly if you build need to build a CI CD system then you can authorize instead of authorizing entire common build environment so there is a talk tomorrow on the CI CD whatever in room number 105 if you want to learn more about it please stop by so that's that's the bootstrapping so let me invite Henry we will go over in more detail about how the bootstrapping works how does authorization works we will also cover some other use cases thank you so let's go through in detail how we are bootstrapping our instances whether they are running within our open site environment whether they are running in AWS or within Kubernetes and how do we provide ex-finance certificates to all of our workloads running within bread and media so before going through details how we do the bootstrap let's spend couple of seconds on the data model so I'm not going to go through the full data model right now because I'm going to come back to this slide during the authorization phase first we want to look at the first couple of icons on the top of the data model so first we decided we don't want to have a single flat namespace for every single product within our company you have separate products they have separate roles they have separate policies so you want to be able to separate them into separate namespaces and this allows us when we do the centralized authorization model for a product only care about a very specific subset so for example you have sports and weather and they have different roles and policies whether it does not care about roles and policies defined for the sports product it only cares about its own product so therefore we separate everything within domains there is a dotted line going from domain to itself because every single domain can create a subdomain so obviously you have a product domain sports you can actually create subdomains which are complete separate entities you can separate your production staging, development all your environment into separate subdomains and for a subdomain it is concerned within the concept of Athens it is a complete domain separate from its parent it can have its own set of administrators its own set of roles and policies so that allows you to right away to separate all your environments you can give your developers admin rights over your developer domain you only give your SCs so that each domain can have a very specific set of services and this is what we are going to focus on so we are going to allow each what Athens does it allows each service to come up within our ecosystem and have its own excellent certificate identity and services and users together there are our principles and we talk about the users obviously users are not managed by Athens however users are referenced in Athens so what Athens provides is basically a simple interface called an authentication authority where you can implement to integrate your authentication system within this ecosystem so for example we are using Octa and Duo for our authentic users we have a plugin for that infrastructure that we allow to authenticate users the default implementation available in open source is basically a simple Unix user authentication but I'm assuming in your environment you could use something else and you have to write that and users come up with our principles so now let's focus on services more specifically so when we talk about services Athens is not the source of the service right we do not bootstrap instances some other infrastructure in place is responsible for it for example if you are bootstrapping instances bare metals or VMs within OpenStack OpenStack is the source of truth for that OpenStack knows exactly what it's doing it's bootstrapping an image it knows exactly what is the IP address what's the hostname what type of image it's running and all that information if you're running within Kubernetes same thing if you're bringing up a pod we don't know about the pod but the Kubernetes itself knows about it if you're bootstrapping something within AWS same concept so what we do initially so let's say step one on top so you as a user you go to talk to your provider whatever interface they provide and you want to launch an instance so for example if you're OpenStack you're using Nova command in AWS maybe you are using your cloud formation scripts or Ansible or you're actually using the console so you go to your provider and you say I want to bootstrap an instance so the provider in this case does its own authentication verifies that the authorized principal in step one is authorized to launch instances and it goes ahead and bootstrap an instance the requirement from Athens is when you bootstrap an instance that instance has to have some type of an identity and that identity again it's different for every single environment so in the case of OpenStack OpenStack provides an identity document that is signed by its own private key and when an instance comes up the instance has the identity document along with the signature in the case of AWS we don't really need an identity document because AWS already provides the metadata along with the signature that is signed by AWS's private key so we can automatically instead of depositing a document with the signature we can actually use the AWS API to extract that metadata so the end result is we bootstrap an instance by the provider and the instance comes up now it has an identity in the form of an instance document however, now the challenge is how do we get this into an X5.NET certificate so the requirement from our side is you have to have some type of an agent running on a box this is in step number three and what the agent does the first thing it does it generates a private key one thing which you mentioned was we wanted that every single instance to have a unique identity because it allows us for the auditing perspective if something is compromised and we know exactly what certificate it was used for or compromised we can audit and go back and know exactly which instance it was we're not generating here a single instance certificate and then distributing that across every single instance in this case we want every single instance to generate a private key and it generates a CSR based on that private key so it generates a CSR and now the actual integration with Athens happens so if you look at step four it makes a call to our Athens system and says hey can you register this instance as part of the registration process it actually sends the CSR it sends the attestation data in which case it could be the instance identity document along with its signature plus some other information that whatever the client needs for its attestation so now Athens ITS service received the request in step four now it needs to validate to make sure this is valid so first of all we do a simple authorization check with that in that step number five so in this model what we want to do is to say it's the owner of this domain has authorized this service to be launched by the specific provider so the use case for this is so we have lots of providers we have OpenStack we have Vespa we have ScrewDiver we have AWS and Kubernetes all of them can bootstrap instances but what happens if I'm only running with an OpenStack I will never run through Kubernetes and Kubernetes provider gets compromised I don't want Kubernetes to launch instances that belong to my domain so you as the domain admin you define an authorization policy in Athens that says I'm only authorizing that OpenStack provider can launch my instances so if Kubernetes is somehow compromised or AWS provider is compromised they will never be allowed to launch instances for your domain so we do that authorization check in number five we go great so the domain admin has authorized OpenStack to bootstrap this instance and now we're doing the actual callback as I mentioned Athens is not the source of truth in this case for example OpenStack is the provider and on OpenStack is the source of truth for it so we take the access station data that was given to us by the client and we make a callback to a registered endpoint that was within Athens to OpenStack and we say here we just got this request this instance with the following UID is trying to bootstrap the CSR it provided here's the domain and here's the service name and here's the corresponding IP address in the certificate can you validate that this is correct or not so every single provider its responsibility is to validate that information to verify that yes indeed I'm about to or I just bootstrap that instance so for example in AWS we only allow you a couple of minutes for that to happen and if it's valid it comes back with a successful response back to the TS and says yep I just bootstrap that host all the information there is valid so the TS once it gets a successful response it talks to our certificate signer which is basically a simple PKCS 11 interface supported certificate sign demon we have the private key for our Athens in cloud HSM if you're running in AWS or if you're in another type of HSM if you're running on-prem we sign the certificate we get back an external certificate and that external certificate that was unique to that instance is now dropped on the host and that's step number eight so now we have an instance that was brought up and now it has a unique external certificate on the host from that point on any service, any application utility running on that host can now use that private key that was already generated on the host itself along with the certificate that was given to it by Athens DTS service to talk to other services within the company and establish mutual TS connection so this is great so we did the initial bootstrap process but as Mujib mentioned one of the requirements for us was we want this to be very short lived so how do we automatically refresh them so now the refresh process is very similar to what we had before so the agent running on the host itself now does the exact same thing generates a new private key generates a CSR based on the private key and does the same dance again it goes to the TS and says hey I'm trying to refresh this instance is this valid or not for the refresh operation we do a couple more authorization checks do we go and say well has the domain admin changed his or her mind are we still authorizing this provider to launch this instance or to refresh this instance and if the answer is yes we actually do a couple more checks to go and say have we actually received another request by possibility from the same UID and it's asking to refresh and that we're doing that to handle the case that somehow if someone compromised the host and stole the key in the certificate and is now trying to refresh it from a different box we keep track of at every time for the specific provider UID what was the serial number that we issued last time and the expectation there now is when you come back for a refresh you must use the same certificate that we gave you last time so we do that authorization check and that's valid and that's that's for the diagram if it's correct then we go back to open stack or to the provider in this case and we ask is this still valid is this instance still running within your framework or you know we should not allow this to continue on so open stack obviously has access to all of its instances it verifies it it goes yep I just built that instance with that specific UID still running so it returns back our successful response we do the same thing we go back to our certificate signer we get the HSM to sign our certificate and we return back to a certificate we return back to our host so now our host does this every day gets a brand new private key a brand new certificate and it continues on running with those to make an MTLS connection with other services obviously when we talk about certificates we have to talk about revocation lists and this is one area where we spend quite a bit of time to see how do we make sure that if there's any need to revocate a certificate how do we support it and as we talk to more people more experts on the subject what we kept hearing was instead of focusing on revocations whether you want to support CRL or SCP or SCP stapling a better solution for us is try to lower the number of days your certificates are valid so instead of focusing on CRLs just go down instead of like 30 days go down to 15 days go down to 70 issue a certificate that's only valid for specific a single day or maybe even number of hours that some of our paranoia have suggested and with that result is what you want to do is instead of focusing on CRL you want to make sure your applications automatically reload those certificates so for example as part of our open source we have a library we have the library available for Java so for example it actually instead of creating the SL context yourself you actually say here's the private key path and here's the certificate path the SL context library automatically creates an SL context so for example if you have Jetty and you can say here's bring up the server with this SL context however in the background we actually automatically monitor those two files and any time those change we automatically reload them into SL context so now you have a service that is running with a latest set of certificates every day they are refreshed and does not require you to restart your instance just to reload the certificates so now we have all of our services running and automatically refreshing their certificates every day let's quickly go down a little bit more detail to see what kind of certificate we issue and what kind of information we have in the certificate so first of all is how do we identify our certificate the issuer is the Athens CA in this model as we said because if you notice we had our own certificate signer and we had our own HSM we are acting our own root CA so within the organization when you have service to service communication the one thing we recommend is instead of trusting your every single possible public CA for inner service communication you only need to trust one CA which is the Athens CA because you will only accept requests from other clients and services running within Verizon media and they will have a certificate that was issued by Athens application so Athens CA is the issuer and the second one is this one is the validity so in this test example it's only valid for 30 days how do we identify the service that is using the certificate so primarily one is we use the CN field in the subject obviously the first part is the C C state and organization are pretty much could be hard-coded for your organization the OU field is the cluster that was responsible for attestating for this instance so we know that this instance in this case we looked at Athens dot sinker and if you look at our data model Athens is the domain name sinker is the service name so I have a service called sinker within my Athens domain and that was bootstrapped within the OpenStack cluster one obviously in OpenStack you might have multiple clusters and every single cluster has a unique identity so now I know that I have a service it got bootstrapped and it's running within cluster one in OpenStack by looking at the subject we also support the SPIFI URI so if you look at the SPIFI URI if you have an application this is actually used by our Kubernetes team because they have like you know Envoy and most of their solutions are based around SPIFI URIs so instead of looking at the subject for the identity of the service they actually look at the SPIFI URI so in this case we have the domain service agent is the hard-coded string in between separating domain and then the last one is the sinker so now I have a service using based on the SPIFI URI I have a sinker service running within my Athens domain next we have the usage so by default we issue a certificate that is good for both client and service you as the provider have the control over to say I'm going to test for this instance however I only want Athens to issue a client certificate or a service certificate only not a client certificate so by default we allow you both however you as the provider have the capability to tell ZTS only issue a client certificate and obviously we automatically generate a DNS name for this service so the way it works within our organization is we have Athens.cloud let's you know this is actually not a real one but for example we have yahoo.cloud and then every single subdomain it's on domain delegated to themselves based on their product name based on their product so for example in this case the Athens domain what's called Athens so there's Athens ZTS Athens.cloud is actually delegated to that product and then we generate another entry called sinker that Athens ZTS Athens.cloud has a DNS name for this service so if you bring up a service everything will be assuming that is registered in your DNS it will automatically routing to that instance next one I talked about oops next one I talked about is like when we're doing a refresh we always validate to make sure that the same instance is not asking for two certificates simultaneously so here we're actually using the URI field in the certificate to say here I have an instance ID again I have OpalStack cluster 1 is my provider and that's the UUID that was assigned to OpalStack to that instance so by looking at that we can keep track of if that instance ID is using the same certificate when it's refreshing so at the top of it we have the serial number so we keep track of it in our database and next time when the refresh comes in we validate that the serial number that was presented to us for a refresh matches to what was previously issued to that instance and then obviously we have the IP address in the request so once we're done with the authorization slip now we can actually do the authentication phase now we can actually do authorization let's go back and look at our data model again so we have we talked about our domains completely separated for each one of our products we talked about services and users that end up our principles now we need to do authorization and Athens itself provides a role-based authorization support so what that means is as a domain admin you can define set of policies on the left side of the diagram and the policy includes set of assertions or rules and the rule says if you can form a specific role then you can execute a specific action against a specific resource so if you look at the Athens model so Athens owns the services because they're defined within Athens it owns the policies and roles because policies and roles are defined within Athens and then we define what principles can assume the roles those are also defined in Athens however what we don't have is we don't know what the actions and resources are you as the product owner you just reference them in Athens and you want Athens to carry out authorization checks based on that data so for example if you're a database server your resource could be a specific table or you can even go and be more granular and say I want to authorize based on a very specific role in which case your resource would be a role and then on those resources if you're talking about a table you can define what the actions are as the resource owner you define and then you just reference them in Athens so now let's look at the two authorization models we support so the first one is what we call centralized authorization control model and the centralized authorization control model is you always go to Athens and you ask the authorization question so in this model the advantage here is you're doing very little work on your side in your application to actually implement authorization all you have to do is make one CMS and you get a response back so let's look at a very specific example obviously in this model where this is useful when we talk about centralized authorization model your service cannot be handling several thousand requests a second it can't handle thousands of requests a second because you cannot afford for every single one go talk to CMS over MTLS to say is this allowed or not this is a very good case where you have a simple configuration a small number of requests a second and you don't care about latency and you say great if it takes another hundred milliseconds to go to talk to CMS that's fine I don't care so in this case let's look at the policy first so we create a policy called heap manager manager heap memory manager policy and we have a rule that says grant set set is our action to heap memory manager role on maximum memory setting so in this case maximum memory setting is a resource that only makes sense within my own product I'm the configuration manager I know what that means to me my action is set so I say anybody who can assume the role heap memory manager role is allowed to execute set on that setting and then obviously I have a role and I have a member of that role and it says who's authorized to assume this role so in this case there is a service called config-manager in the sports domain so I add that member to the role in this case so let's look at on top so I've done the step one already I've configured my Athens domain with a specific policy so now the service config manager goes to the configuration manager in step two and says hey I want to set this setting to eight gigabytes configuration service manager we're doing MTLS because remember config manager is a boost service Athens identity and we did MTLS and we verified that yep the identity in the certificate config-manager and we figured out what it should try to do it's in my own application I know it's trying to set a setting so I go to dms in step four and I say this force.config-manager principle authorized to execute set on a very specific resource excuse me so obviously it's a role-based authorization system so what dms needs to do is to verify what role you can actually the dms goes to its data it goes what role can this service identity assume so it can assume a role called heap memory manager role and then it goes to the policy and goes well does the principle who can assume this role are allowed to execute set against a very specific resource and the answer is yes so we return yes to the so now let's look at look at a very specific the other case where we have decentralized access control in a decentralized access control obviously is we need to support a use case where we want to be able to handle several thousand requests a second so we can't afford go to dms to ask for that information every single time if you remember in the centralized model the important thing that we need to do is to say excuse me what role that was the most important part because we are a role-based authorization system we have the policy based on the role name so we need to know what role you consume and this is where GTS comes into play so GTS is our token service and it takes all the roles and policies defined within our infrastructure and in indexes based on the principle name so you as a caller you go to GTS first and you say I'm trying to access a specific service what can I assume within that service GTS does the lookup and goes yep you can assume the following set of roles obviously we talked about list privilege access you need to know which role that you want to assume if you know that then the recommended approach is you go to GTS instead of saying that hey I'm trying to access any roles I can assume within the specific domain you can go be very specific and assume a very specific role that grants me access to that service and this is where we are able to accomplish list privilege access so we do the same thing so we have a role in policy so let's go through this example now so we have the same policy very similar policy at the bottom we have a role that says who's authorized to assume that role all that information is set up so now the support config manager wants to execute the operation against our secret manager service so first it goes to access secret manager GTS looks it up and goes yep you can assume a role called project secret role and actually generates a role token and sends it back to the service think of it as a very similar to the concept of an access token so you have an access token and we support both models either a role token or actually a role certificate instead of a token you can actually get a certificate issued for that role and the identity running with support config manager now actually has either a role token or a role certificate and now instead of going with its own service identity to the secret manager it actually goes with the role token that was issued to it so now the configuration manager knows what role the principal can assume all he needs to know is is that role authorized to execute a specific set of actions on a specific set of resources this is where the whole distribution of products when we talk about a little bit earlier comes into play in this case the sports or no sorry the secret manager service does not care about every single product defined within Athens it only cares about its own domain its own set of policies which is very a very small subset so we have a utility that automatically pushes and pulls the information from the TS service onto the box so we have all the policies now are being pulled from the TS service onto the box and then we have a simple library that loads all the policies into memory and now all you have to do is a simple authorization check within your processes space that says is this role authorized to execute the action against the resource and now you can actually support several thousand requests of second operation because all that operation takes a small amount of time negligible if the authorization is concerned because all that information is running within your application process space you're not making any more REST calls to the TS service and this is what we call our decentralized authorization model the next thing I want to talk about is temporary credentials so as we were thinking about this and we were expanding REST and media to actually utilize both private cloud and hybrid cloud well we have all of our AWS accounts they're all we need some credentials to access AWS services AWS itself recommends using temporary credentials because they're very time-bound they're small they only allow very specific actions and but how do we get AWS temporary credentials onto our instances running on-prem in our private cloud obviously one option they say is like well you can have static credentials locally and you have to make sure they're properly rotated when somebody leaves the company if they have access to those static credentials you want to rotate them so instead of dealing with all these rotation problems the solution was how can we utilize Athens and the authorization policies it provides to actually obtain temporary credentials from AWS so this is a very unique feature that we have in Athens as well in addition to our service authorization model is where we allow ZTS to run within AWS and it assumes temporary credentials on your behalf assuming you have authorized that service to assume that role and it returns those AWS temporary credentials back to your application so in this model we have this set up first so you set up ZTS and so you say my specific service is authorized to assume the following role in IAM you have to configure that says you are authorizing for that IAM role to be assumed by ZTS service and then when the request comes in you have an application running on-prem it makes step one it makes a request to ZTS ZTS has all the authorization checks goes to STS AWS STS and if the authorization is valid gets the temporary credentials returns them back to the application and now application can use those temporary credentials to talk to AWS service could be S3 or whatever other services running within AWS and the solution again here is they're completely temporary we're using Athens to manage all the authorization in a single place so as far as the other thing is concerned you can go look at our Athens data and see exactly who's accessing which AWS temporary credentials from which domain other use cases obviously all we talked about so far are available as part of our open source product but I wanted to also talk about quickly a couple of other use cases often have and these are not open source these are not open source however I want to give you some ideas how you can actually utilize them you can use Athens for more use cases within your organization very similar to AWS temporary credentials we actually have Athens acting as our IDP so all of our logins to our AWS accounts are actually handled through Athens as well Athens is the IDP you create roles and policies in your Athens domain that says what roles are to assume you associate your Athens domain with your AWS accounts and the end result is you now have roles in Athens that say exactly what roles they consume in AWS and they can login without creating user accounts in AWS so what we've achieved based on the last two features is now we have our AWS accounts without any static credentials without any users being provisioned they're all being provisioned on the Athens site as far as roles and policies are next one is our batching access very similar when we login to our AWS accounts we're not logging in as very specific users we have a very unique solution within our company called SHCA which provides SHCA credentials based on Ubiqui so we get our SHC credentials we go to our batching demon and we're trying to login a very specific instance within AWS we go back to Athens and does the authorization check it already authenticated based on your SHC credentials and goes okay your user ID SHCA and are you authorized to access this specific instance does the authorization check and if it's valid you're actually allowed to login to the instance again no user accounts all the access is based on authorization data within Athens so I'll give it back to Mojib to wrap up very real quick just wanted to list out all the principle the zero-trust principle I was talking about earlier that traffic encryption so with Athens issuing a certificate it doesn't matter whether it's a private network or the public network that we can make sure that every single service or service communication is completely encrypted using those X1 and certificate now obviously Athens is issuing client certificate that we can make sure that client gets authenticated by doing this and with the fine-grained access control it provides authorization can be also done and since Athens also provides this certificate with a very short wave like one hour in some cases one day so that that trust is pretty much a dynamic with the limited duration and lastly with the least privileged access which I talked about that with fine-grained access control it then provides with the least privileged access as well so with that let me close with the quote from J. Johnson he's a former U.S. Secretary of Homeland Security the cybersecurity is a shared responsibility and it's bounced down to this in cybersecurity the more system we secure the more secure we all are so why the cybersecurity is a responsibility because we all are online if there's a vulnerability found in one place we all affected some places right so it's all our responsibility to make sure that we build our system which is less vulnerable to the cyber threat and that's all we have it thank you so if you have more questions if you want to learn more about Athens you can visit Athens.io and obviously there's a link to Slack channel as well so feel free to think or Slack us so we could answer anything about Athens any questions you guys have so we have written documentation how you can spin it off in AWS AC2 what are the different components of Athens we also have a way how you can build the Docker thing by working on putting together an image as well and more videos and documentation ok the first question is that we decided to go with other model of not going with the certificate revocation so certificate revocation comes into place where you have a longer certificate so our certificates are pretty short the only revocation I was talking about is that it's a unique and corresponding private key and try to copy that and try to reference the certificate then we know that the previous request for this serial number came from this IP and now it's coming from the different places so we basically don't allow them to reference the certificate go ahead anything the second question is identity provider are you talking about that user identity user authentication right I don't know both some other identity provider you probably don't need to use the Athens the authentication for user we're talking about yes it's a plugable system so you can have whatever identity provider for user authentication you can plug it in you can do for the user authentication remember that Athens issued identity for the services right so not for the real human user we do have a certificate for the user as well but basically you authenticate against whatever identity provider you have whether it's different providers you have yeah we were working on OAuth back on Iwanda I don't know what that is so the primary use case what we have to be honest with you is primarily for internal usage so for external usage what we have is specifically for authorization part we're working on adding OAuth provider supporting to Athens so that way your external partners can basically all they can get is OAuth access tokens from ZTS and then they will be part of your ecosystem as far as authorization is concerned however the requirement there is for authentication part the expectation would be like you would have a probably separate authentication system that handles all your external partners and not necessarily all your internal users so the question was if if there's any work being done for GCP for now no that's one of our desires is like in the open source community somebody will pick up on that right now obviously we have a lot of effort on the AWS side because we do utilize AWS within red and media itself but so far we haven't looked at GCP but I don't there's absolutely nothing that would stop anybody to actually integrate fully exact same functionality within GCP or Azure for that matter actually that Athens itself as we open source in a year and a half back it's a fairly new open source community so the question was like once the client gets its own identity certificate if it goes through Athens no because at that point you just do a regular HTPS connection your client uses the client and client private key and the certificate to establish a mutual TLS with your service directly so Athens is not coming to play in any shape or form in that process all we do is we give you a certificate you own the private key we give you a certificate from that point on you're on your own to make your client connections that's the video of this one authentication yeah we use the name and password so all this can be hacked into keystroke walking fishing and all that stuff and it has to be authenticated at the application level with the experience certificate with authentication that all things happen at the TLS layer itself that the server side gets terminated if you don't present a wallet as Athens is your certificate so it doesn't even go to your application it gets terminated at the TLS layer the application can extract a subject name to get the identity of the service name then further can do authentication based on either the 380s or something else in one of the slides you are mentioning about the process of how the authentication happens that is a stage where the Athens system talks to the provider about registering some information you know what exactly is that it needs on the provider side correct so if you go back to for a second so the only requirement is we're not dotted line when we go not the dotted line but the line that goes from ZTS to back to the provider so in that model that's the important thing that you have to do you as a provider if you want to be part of the Athens ecosystem you have to provide you have to write a new service that actually needs to implement two endpoints one is slash instance one is slash refresh and those are the ones we call whenever we get a request to either register a new instance so in that model the way it works is we will get the request from the client which will contain your attestation data whatever it is instance document signature we don't control that as far as Athens is concerned it doesn't understand what that is or what the format is for example in some cases some providers are using JWTs some are using some other JSON format ZTS just takes that information along with hey I just got the request from this IP address from with the following CSR request with the following attestation data we send it back to the provider provider you are the provider you have to implement those two endpoints to be part of Athens and you have to register the endpoints within Athens so when it comes into Athens and you say I want to bootstrap a host for provider open stack cluster I go look up the registered endpoint for open stack cluster one provider make the call slash instance and go here's the data it's your responsibility now you've got a request from Nova saying hey bootstrap that instance with that UUID you verify all that and then you come back to ZTS and go yep that's a valid data that's the primary source of truth part it's the callback mechanism that you are responsible for telling ZTS that the instance is valid you have to go ahead and issue a certificate because Athens itself doesn't have that knowledge correct yes there are two endpoints if you look at the documentation it's called we'll make the call again over mutual TLS we'll make the call you do the work you value the data come back with either a 200 that's a valid or 403 or whatever some other error code HTTP status code that is considered as invalid every single provider we don't necessarily provide a plug in we provide you like I have a sample example code there that's how you would implement a instance provider model using JWTs for example it's a sample code that I would expect but it can't be a plug-in because it's basically one endpoint that we make a call from the ZTS standpoint it's a one rest call to some other service but you still have to implement it but every single provider logic is different I don't know exactly what your attestation data is or how you're going to validate that those are the providers already integrated so these are two as we can point you to the open source code for those things they're pretty much in the industry like OpenStack Covernment AWS that someone can contribute them some of those are completely internal solutions so they're not available the bastion stuff so there's some the FHCA stuff that was based on the Ubiqui I believe our parent organization is planning to open source that bit so if that's available then we'll have the appropriate documentation outside how to integrate with the Athens as well thank you so much one more question thank you so much see how are we doing with sound does this sound good? too low? too high? too low we need someone from the media crew to manage wherever the thing is over there what about now? well I guess I'll speak a little bit louder okay so we have extra setup for this talk because there are just too many devices and windows that's finished updating well it decided to do it right now actually it hasn't finished yet it is finished that's alright we don't need windows just yet but we need to know that it works otherwise there will be a different problem okay it has finished updating and it does work alright so we only have one demo God against us the other ones are favorable alright we check the windows machine we're good cable first I am is on alright usually I take questions at the end because of the amount of stuff in this case I also take questions as we go because I may manage to confuse you so if you're confused ask a question if not there is more coming you may want to hold on a little bit longer alright I think we're all set let's let's get this going so so a little bit of introduction so I have the privilege of I've had the privilege of spending basically my entire career in free and open source software save a short exception for academia and I currently work as the product management director for CEP in the storage business unit of Red Hat previously I was the Ubuntu server PM at Canonical and if you turn the decade I was the dreaded systems management star at SUSE the talk has nothing to do with my job I'm a manager these days but I was an embedded developer so this is sort of my definition of fun obligatory disclaimer for a hardware talk is that we'll most likely break some hardware while playing with it and it will come out of your pocket no liability if you follow our instructions grab your toe or bring about the end of the world or simply break your device which is far more likely so you've been warned with great power comes great responsibility don't do anything you should not just and don't limit yourself to just obeying the law since this is a hacking talk be nice let's get going we have 59 slides and two demos we have to hustle a bit so this talk is all about abusing the security assumptions we have made about USB devices a security analyst posted the dissection of the most impressive device here a couple of years ago this is actually one of the smallest devices so I went around and started shopping for all sort of random things the fanciest device here is in the Intel box every time we open it it plays the Intel Jingle which is a horrible abuse over there trademark since there is something truly evil in there but I guess we are having a little bit of fun as this is a pretty dark talk dark talk once you take the some of it let's start with something much more benign before we go dark and deep this was something that was on sale on think it was this Black Friday there is a whole class of devices called Anoya something you would lose in the cubicle of someone you don't particularly like and every once in a while it would make a chirping sound maybe a very five minutes just to annoy them or a squeak or something no idea where the sound comes from this device is the computer version of that it is actually nicely designed not something you have to fiddle with software to configure here there are actual physical switches it's a USB device to keep with our theme with the dial for the time delay we actually have a physical one but in a room of this size it doesn't make sense for me to show it to you let's use the pictures you select the time delay and it will flip the caps lock tap the keyboard or move the mouse or all of the above this is perhaps the most benign of these things Garrett Mace designed a smaller stealth USB caps locker about ten years ago for April's Fool where you could build one for eight dollars using a digit spark if you do not want to fiddle with proto board yourselves this is what Garrett's version looked like this is less polished than using the digit spark but it fits almost entirely in the USB slot as you can see in his picture on the right it uses a nutmel AVR tiny 45 chip and has a great educational value as you learn how to run a USB stack on bare hardware there is no operating system here carrying out this attack is hideously simple attack goes in quotes in this case obviously you can do it without particularly privileges you just stick it in the back of the computer of your victim and when they log in it will have access in some cases without logging in because it's a keyboard an HID class device to be precise so it is a keyboard and or a mouse this this story is innocent enough but things get nastier from there the HID class of the USB protocol defines human interaction drivers human interaction devices to be precise the driver is one for all these devices keyboards mice game controllers and a variety of low bandwidth devices like mag stripe readers which are actually mapped to keyboards these are all HID devices in the USB world thermometers RFI readers barcode scanners even UPS batteries sometimes show up as HID devices at Princeton they actually briefly taught a class a random hardware into an HID device they would basically go to a junkyard and the students would all pick a random piece of electronics and they would turn it into an HID device as their assignment for the class here is a Hewlett Packard keyboard in honor of a friend sitting in the audience that is the prototypical HID device one benefit of a well-defined specification like USB HID class is the abundance of device drivers available in operating systems on the downside are the inherent trust keyboards are extended and the possibility for USB devices to change their type or announce additional sub-devices or probably actually I think it's not device that's proper term in the embedded track they taught me the right terminology very forgetting it right now in any case the physical device can provide multiple logical devices and there is a proper USB terminology and this can change dynamically combining this with USB's default behavior of accepting any device that connects to it if there is a driver well we can start looking at how we can exploit this so the first idea is Tomo Tomo is a very small device also not meant to do anything evil and it is basically the size of a UB key if you know what a UB key is a factor device that basically fits inside a USB port and just produces a known second factor Tomo is a USB platform to prototype things like UB keys the author explains that while a UB key costs around 50 bucks a similar device can be built with two dollars in parts and his intent here is to drive down the cost of that kind of device but you don't have to build UB key substitutes out of Tomo you have a microcontroller you can build whatever you want the only thing is what do you have you have a Cortex M0 so you're pretty much writing to bare metal you don't have an OS 8 kilobytes of RAM 64 kilobytes of flash and it's also the first open source hardware association part certified out of Australia I narrowly missed meeting with the author at LCA two months ago so close but not quite this is available from Siege Studio so you don't have to build one yourself you can just order one premade and it has way more flash than a DigiSpark 64K instead of 8K for about the same price you don't have a whole lot of software or complex stacks here but if what you're doing is USB and has a two button two LED UI you can basically build it here this could build a device fitting inside the USB port entirely on the other hand there are much better platforms for an attack we will see one with four cores and eight gigs of storage later but this is interesting to prototype this kind of Ubiqui-like application limited interaction you could definitely build an Anoyatron type of this platform it also seems like completely interaction free devices are not that interesting or at least not as malevolent I guess they are interesting only when they things are interesting only when they cause trouble so let's look at something that will cause trouble so I plug in the Anoyatron and have access to your keyboard either because US trusts keyboards inherently and lets them in or because I leave that plugged in and then you log in and when you log in the device gets in you usually need both of these things to occur again physical access both necessary and sufficient condition to pwn the device we have an entire industry of pen testers that are paid to assess your security and the gold standard for them is a keystroke injection tool disguised as a USB drive which is this the rubber duck the USB rubber duckie is the original keystroke injection attack tool this is almost ten years old although the hardware has been revamped while it looks like a USB drive to us it acts like a keyboard when talking to the operating system and typing over one thousand words per minute specially crafted payloads written in a custom scripting language mimic a trusted human user while entering keystrokes at superhuman speed it is named this way because if it cracks like a keyboard then it must be a keyboard you have the shell you can just type your way to success you don't offer overflow or some other strange attack you have the shell and if you don't have the shell in Windows R and you get the shell so seems simple solution this is a full rubber duckie kit in its current version first and third are the parts to assemble it into a standard some drive case well standard a few years ago now they don't look as much like that anymore the second is to enable keystroke injection into a micro USB device so this is effectively enabling HID attacks on Android smartphones and then we have a USB adapter to load payloads into the micro SD card so you put the micro SD into this adapter and then to your device and then you load the attack and then you put the key the ID you put the SD and then into that closing on the hardware on the left is the micro SD card slot that conveniently lets you swap a library of pre-made payloads without having to load them every time just have a library of cards for whatever attack you want to use that day the replay button is in the center of the shot and this is convenient during development to avoid having to remove and replug the card all the time your fingers will thank you because the edges are pretty sharp but it's also good on the device so that you don't stress it and just press the button to rerun the event as if it had been plugged in the use of the button can also be redefined although this is not accessible from outside the case and there is an LED up top that lets you know when the payload is running the LED is multiple colors so you see usually green when the payload is running correctly red if there is a problem and again this is closed on the flip side we have the Atmel AVR CPU power in the device and some support electronics depends on the left side should be a JTAG header but I have never seen them used so I'm not 100% sure here are the specs the system is AVR powered while it is not an Arduino in practice it resembles it in many ways the internal flash and RAM make board bring up simpler of course if you're an embedded developer you know what I'm talking about and the potentially unlimited mass storage via the SD card interface make it extremely flexible in the kind of payloads you can carry the LED flash can flash in multiple colors as I mentioned signaling execution state or error during development there is JTAG and GPIO access and I think those are the headers at the head of the board and this combined with the standard that meld a few bootloader make it a potentially moddable platform also I don't think I've seen that much hacking in terms of modifying a rubber ducky itself because it's not a particularly cheap device we usually see clones being developed instead so look it's a keyboard this is how Windows sees it this is how the USB some drive looks when you actually plug in not that anybody is looking this is a Windows 10 system at the latest update level all that Windows is going to do is that it's going to warn you that it's configuring an existing driver the first time you plug it in you may see an overlay it doesn't ask for permission doesn't ask for anything but there is a pop-up saying I'm setting up the device so by now we have all understood that we have a rogue keyboard to do our bidding but what can we do with it there are three primary attack vectors to be concerned with the first is file exfiltration that is copying files to a remote web server the ducky is a keyboard you don't copy files to a keyboard so the keyboard the ducky is instructing the system to copy files to a remote drop which is probably a web server somewhere in some bad part of the internet or in some anonymous place that can't get back to the attacker the Windows environment exfiltrating domain credentials and passwords even if encrypted is the default target so they get whatever credentials are in memory for Windows and they get the domain credentials if they can even if they are encrypted because they can brute-force them later on their own time and you can always get the wifi passwords in the clear if you have ever bothered to look in Windows you can just ask and Windows will tell you so why not you can you can steal those a third one is initiating a reverse shell and the ducky can start a reverse shell in about three seconds beating any human user doing this the reason why I'm in front of this audience with a Windows machine is that all this stuff usually targets Windows so I had to build a little dedicated setup in my lab with these Acer machines they are they are netbooks they are about a hundred dollars because I needed some machines that could be infected by all these horrible things that we're going to see today and I didn't want to compromise my actual machines so these are not very good to do anything with but they're convenient targets so it has taken me quite a bit of time to build the lab in terms of their imaging capability I learned something about Windows backups which may be useful but the reality is that some of these devices like the ducky have a company behind them they have a reputation to protect you can kind of trust what they're selling but a lot of these things are just being sold on eBay or Alibaba by unknown vendors literally and there could be anything on them so some of them are even known to phone home to unknown servers for reasons that so far haven't been determined in any case to show this we need an OS change so Windows here we come we're going to start with an attack that is cleverly named Satec Astronomy where is the ducky which is basically the simplest thing you can possibly do with a rubber ducky and I preloaded the I preloaded this ducky with my attack so we don't need to do anything we'll just plug it in into a USB according to the law of quantum USB it's always wrong and you see that flashing and that was basically the ducky doing its thing starting a browser and taking us to to a pre-configured site so this is the easiest thing that you can do with a ducky pop up a webpage of your choosing this is what they actually tell you what professionals tell you you can use to teach your staff never to trust USB drives you basically go to their machine and put in this thing and show them what happens and this is not that bad you're just showing the webpage and then you can use the same type addressing of windows so I didn't even have to say open a browser I just said windows run and the URL of a page on my personal account at Red Hat just downloading and gif reproducing something from the movie sneakers if you are familiar so me it's not on scale but this is a simple part now let me show you how you actually load an attack into a document so let's do it we have to take the document out we take the micro SD card out we install the micro SD on the on the card adapter so that instead of showing like a keyboard it shows like a mass storage device we plug it in so that we have a drive now this is cute for those of you that get the movie reference but we're done with that now as you can see there is a file called inject bin I have another attack here which is called hello scale we are going to see how this is done so we need to go to I think it's duck toolkit so the exploits are written in a language called duck code which is a scripting language for keyboard input it includes pauses modifiers like control or windows key windows key string input it's all very simple but not in a naive way it was designed to be easy to use it has really great usability and it needs to be compiled with an encoder there are multiple encoders the tool chain is generally available and I think a large part of the tools are open source even though the hardware isn't there are Java encodings and there are there is an online encoder that I'm trying to access here but for some reason is not loading the source code for this attack is this one I think this is the source so REM like in proper DOS is a comment delay means wait wait one second because delays are very important in a rubber ducky exploit the ducky is the capability of start firing keys at the moment it's inserted in the slot but the host system may not be ready the operating system almost certainly won't be so you wait one second to let windows catch up with the fact that there is a new device the other thing that's important to remember is that you are typing blind this is sort of like spearfishing conceptually you have to make reasonable guesses depending on the other side OS and application wise so you can guess that there is Excel you can guess that there is a Windows you have to assume there is somebody listening in this case we're assuming windows and we're assuming anything since windows 95 which is not that unreasonable we're going to add this to the encoder and we're just encode payload and if there are no errors we can download it from here since we don't like to be at the mercy of the internet I have it already compiled over here and there we go that is all that it takes to create create an attack on a ducky there are of course additional steps that you can take you can optimize this attack you can obfuscate this attack so that it's not clear what's going on in there but these are the basics then we put the key back into the we put the SD card back into the ducky we plug the ducky in and it's going to execute again and it's going to type hello scale as our program constructed so those are the basics the duck language is the language to define how you're going to type things basically it's very straightforward quite easy frighteningly so and it has been very successful in the street why this is basically a standard for penetration testing so we go of windows and we go back to our slides see what what else is in our horror gallery so in an unprecedented feat of cinematic accuracy the rubber ducky was apparently featured in an episode of Mr. Robot who knew Hollywood could get hacking right even once this is a rubber ducky plug in this guy in and wait 15 seconds then yank it thanks to a tool called Mimicad it will pull all cash passwords and the domain and store it on the drive pretty correct I'm not sure what happened there but it's right this is what the hello world payload looked like these are backup slides in case those device decided not to work so we're going to skip them clones as I said there is a rise of the clones according to what I shall hereby call the walley law of ecosystems in an open source system in an open system success will bring you what commercially are freeloaders it is a good thing it's a good sign for a healthy business and here the result is that is that there are clones of of the duckie using similar tool chains or clones of it the micro duck is a stealthy HID injector that fits in the usb a port itself like other things we've seen so far while Thomas's project is a $3 clone of a duckie built using the Digistump Digistpark platform and a tool called duck to spark into the duckie community uses after a decade this is a very popular tool so what do we do about it what you do is something like this this is basic usb security a device called a sync stop as a cio you can buy a bucket of these in order that everyone use them don't worry they still won't this is a device that cuts the data lines of a usb connection preventing data siphoning at charging stations which is now known as the juice jacking attack so if I'm charging anything at the airport's usb charging station it goes through this so I'm getting only power if I'm plugging any of these rogue devices into my system to charge it it goes through this so there is no data connection period good luck hacking me through the voltage also I'm sure at some point they are not there just yet so that seems like basic prophylaxis but unfortunately it's not very practical because it's saying that someone that's providing you power cannot provide you data that's okay but for a lot of devices we want data so what's the solution here there isn't really a good one a more customizable approach could be built with a device like usb-safe 2 conceivably you control the firmware so you could choose what behavior to respond with you could decide to filter on the class of the device the idea of the device or some other property the current device protects from access the current formal the device protects from excessive voltage limits current to preset levels and most interestingly sets in software the data protection mode and that's why I'm saying that this could be the basis to build something more interesting it is an interesting idea it would take a serious amount of effort if the value can be demonstrated and you would need to add a usb stack in here this is pure hardware there is no place running a usb stack this is not a trivial exercise and you would have to mod the hardware but pretty much every time I speak about the subject someone asks me what is the usb stack the usb stack is also interesting to monitor what is actually going on in a usb chain as it has independent leds for data and power lines you can see them here at the top the usb on the usb a side of the board up top and bottom the board is a two layer sandwich so the leds are in the middle layer the single bottom up top in the middle controls the level of current and the pass-through mode in either click or press and hold that's basically the entire of the interface while the voltage is preset to 6.5 volts it will shut off if voltage exceeds 6.5 volts as I said one is a toggle the other one is a press so one button is the whole UI and the modes is canceled on the back by the way I don't think this will protect your electronics from a deliberate usb killer type attack at 200 volts but it will definitely help while developing your own electronics where it could be some flakiness and it stops over voltages at least up to 15 volts another device getting even fancier here is called PISA this is a similar thing but instead of being a pass-through this is a usb you have three buttons and a tiny OLED screen basically this is a tiny Linux computer geared towards providing more flexible storage than a plain hard wood it is built on a pi zero so it cannot provide usb on the go behavior switching between host and device but it is still quite interesting it comes with a kit of screws and you combine it with the pi to produce a working computer it's a super flash drive built on the computer and you can see some versions also include a 3D printed cover to make it more robust for everyday use I have not found an application for this yet at least not an evil one but I have found a good one it has interesting demo applications like press button one provide the arch Linux ISO press button two now it's in the and obviously it is more powerful and has more storage but it has the same button plus usb interaction model obviously supplemented with a tiny screen so it can actually talk back to you so let's go back to the what we were saying about evil things we left it at the ducky for pen testers the solution is to get in that's what they're paid for they have a contract to do it put the log in in the morning there they are besides your machines are not patched we know that so it's too easy but it gets worse at some point someone realized that different classes of usb devices have different inherent permissions for example a storage class device may get different access from a network card enter bash bunny the grown up sibling of the rubber ducky can do the on the go behavior that was just discussing but on steroids this usb stick can show up as a storage device and a minute later it becomes a network card then it becomes a keyboard or a serial port it keeps connecting and disconnecting to change the driver it talks to as needed because different drivers have different permissions and different capabilities and different implicit trust what is absolutely amazing to me is that in our industry where basically no one can tell you work for Apple or your product sucks usability wise these security products are amazing they are actually really easy to use I don't know why but it's pretty impressive for me the spec on this hardware are also absolutely impressive quad core CPU half a gig of RAM eight gigs of storage the interesting additional attack vector enabled by the bash bunny is to show up as a network device plug in advertise a very fast route with fantastic metrics you can do this so that the right network driver is already in windows so no permission required windows will set it up automatically and now you can get the system to route all network traffic of the machine into the spec network where you can inspect it it's going through your device before you forward it back out through the normal interface so it actually reaches you are inspecting all of the machine's network traffic but no one monitoring the corporate network ever sees what you're doing because it's happening on the machine on local host and these explosives have already been written for the bash bunny so you do not even need to put in the effort to code them which is scary basically you're planting into the fault route and if you're nice you let the user see the internet because you don't want them to notice you're there you're going to make the connection a little bit slower but you can see everything that's going on a different take on the theme is the Cactus WHID it includes a wifi access point so your HID attack is no longer blind we're looking for all intensive purposes to a hardware remote shell note the case it opened without breaking so they ship you the device opened but once you assemble it it makes it harder to inspect but the fact that they ship it open is great for our pictures so you can see that the wifi part of the device is the inevitable ESPA 266 and to facilitate the process of weaponizing USB gadgets you can write or you can wire another the new attack vector here is that you can get a USB powered knickknack like a plasma globe or a Nerf gun turret or whatever wire the data lines to this pass through the power from that device to this device so there is power and then the attacker will mail this as a present to the target here is a plasma globe from big company name that I bought and it's basically social exploit without the social you just mailed it to them so beware unexpected devices in the mail there are the new hardware enabled spear phishing ok this is already bad enough but fear not it gets worse this one is actually rather outrageous and it is making me lose faith in the concept of computer security as a whole I bother hacking you with a device if I can hack you with a cable from the picture it looks like a big cable but it is actually a rather tiny cable and yeah it has something at its end this is something this something is actually a GSM phone it gets power from the USB connection this is the phone with an infinite battery the device itself is marketed as a location cracker usable in cars where a thief uses a USB cable as a tracking device this is nonsense the device of poor tracking precision you would never find a car and it makes no mention whatsoever of tracking on its packaging so is this a tracking device to protect your car no it's a high speed data charging cable so I want to highlight how the packaging is also designed to be opened and closed without trace which is not the plastic up and then you put it back in and why is this because you have to activate the phone to be able to use it as a new tracking device you have to put in a SIM card and register it with some phone network so you need to open the packaging but obviously the hacker needs to deliver this to the target in pristine condition so the packaging is designed to do this it gets worse didn't I say it would it does you can buy this monstrosity for 9.85 on eBay free shipping of course are you worried yet okay let's take a deep breath what in the heck is going on here this is not only evil it is also cheap you would think that if you use this to spy on someone at least in the United States you would go to jail for unauthorized recording or wire tapping but the device is effectively just a phone in a ridiculous form factor so it is almost legally legal for Amazon and Alibaba to sell it Corey Doctorow calls this trickle down surveillance because this is apparently a low quality low cost copy of a device the NSA had on their leaked tailored operation catalog code named cotton mouth so apparently leaking such secrets is not such to the use the cable itself looks like this ribbon a USB cable with a USB A connector on one end and a micro USB on the other some folks have pictured it next to an Amazon basic charging cable and the size of the connector is frankly the same later on you can come here at the front and see a cable and judge for yourself but I think that pictures make it look more impressive than it is it doesn't look that much when you actually see it doesn't call out for size until it is fielded for use and the casing is super-loot shot as one does the cover slides off to reveal a small board this is a GSM listening and location device hidden inside the plug of a standard USB data charging cable it supports 850 900 1800 and 1900 megahertz GSM it's a quad-band GSM phone the cover of the USB A SIM in the slot the end cap of the USB A plug locks back in place so folks don't find out accidentally there is a phone in the cable you have to event them it can do a bunch of things it can track the location of the cable it does not have GPS so it won't give you too precise a location we're looking at minus and precision with a more densely populated area likely delivering better resolution it has a microphone it can listen to what you're saying it has a collection of AT commands so that you can configure it it can send text messages to tell you where the cable is you can text to the cable to change its mode so you can listen to what's going on the chip is a MediaTek CPU that does not have a published spec sheet it is believed to be a chip designed for low cost smartwatches there is a serial port that shows the initial bootloader sequence but once it gets past the bootloader it is no longer configured so the kernel doesn't show what it's doing this device has undocumented commands that nobody knows what they do and it does send packets to places in China that we do not exactly understand but presume are used for the geolocation service at least we presume never ever plug this device into any computer unless it's a secured lab facility here's a view inside the case as best as we can tell USB lines bypass the device itself and go back to the cable the mini SIM card reader is in the center the device is quite finicky about SIM cards I spent the whole week piping that and on the right you can see the electric microphone commonly found in any standard USB charging cable not on the flip side we find the pad to the serial port conveniently labeled this is always a plus if you're an embedded developer the media tech the media tech is in the center so we're in a basement the demo gods could fire the game stuff for a plan for a demo so we will walk you through what you have seen inside the problem I ran into is that AT&T shut down its C2 network I thought I could get around this using C-Mobile but despite multiple attempts in the last few days I could not get the device to go in the C-Mobile and of course I'm not sure why yet it's somewhat hard to debug as a headless unit first I thought it was a limited antenna and a limited transmission power but going outside left and right didn't help either then charge pin and that was also not the problem so here is what you did but this hacker is a friend she will get a text message back reallocating the phone to a city in the world in the United California including a link to a page on a site called gpsky.net showing a map of the approximate position of where the cable is with roughly one mile actually another interesting text message that our hacker to send is one one one one four one this activates the mode in which the implant will call the arena back it's the noise at this location exceeds 40 dB which definitely does right now perhaps a conversation is underway or a talk is underway and the device would call her she would receive a phone call listening on the stuff a security analyst named Mitch dissected the implant last year and published a long report lots more details in his write up so in closing don't obviously doing any of this is in violation of US law requiring two parties to know a conversation is going on or both parties having to consent to a conversation being recorded or probably a bunch of other things please don't do anything like this it is a crime in most US states clearly we have rules about this not being acceptable but what is changing is that folks willing to break the law can do so with very low barrier to entry namely nine bucks on Amazon 25 bucks on Alibaba whatever it is before you had to go to some black market purveyor of illicit goods and to find out who they were in the first place drive up some strange place in New Jersey and purchase or the like now you can go in Alibaba and have them express delivery to you I'm not impressed this is not what the future should look like so let's go back to the security aspect we were joking about the fact that it was easy to break in but realistically this is physical access if you have a device you can do things to it we can also do it over wifi fine that hardly matters what matters is that this brings the lost usb in a military base people are trained to shoot on site when they see a usb key but in industrial settings people are much more lax I am amazed that this sort of thing exists and is being produced on this sort of scale I paid almost nothing for this device something to be on the lookout for if you're working on a sort of operational capacity in a security sensitive industry let's put it in economic terms or in MBA terms let's thumb it down a little things change when you drop the cost of doing something by a factor of 10 think of the raspberry pi we went from $300 computers to $30 computers and then the pi zero brought the right showed the right thinking by doing that on almost a next drop again with the price of the zero this kind of thing is dropping the cost of data exfiltration or surveillance or covert activity or industrial espionage to the point where everyone can afford it I'm not going to say that we do not have bigger problems but this is not nice cheaper industrial espionage is not on the positive column of progress this is why any place that has any kind of real security should be pouring glue in their USB port there is no other reasonable solution you can put locks in the USB port attackers will find a way to break those or your employees will break them out of convenience the only real solution to make a USB safe is not to use it barring that you need to work to limit the blast radius of a breach the only reason why USB security is not a bigger issue today is that we are so bad at network security that it is easier and cheaper to exploit your remote level this is not a comforting thought but it's true that's the only reason why we're not screaming about USB a reminder use a sync stock or any other brand of these things if you charge out of anything that's a USB charger that is not your own this is a reconstruction of a tweet that was posted last year at scale scale 16 last year someone ran to their session and forgot a charger in the main hall you know the hallway track where we all say hi this was on the wall the conference twitter lit up with folks thinking this was a honeypot so perhaps there is some hope after all that's a slightly positive thought this is what I have this is my contact information if you want to send me questions comments any criticisms or whatnot and now we're up for questions there is actually supposedly a microphone but I will repeat the question I used the AT&T SIM when before the network failed and before the network was shut down that was fine now the only G2 provider in the United States is T-Mobile so basically that's the only option right now I haven't managed it also some smaller virtual providers that sell you SIMs to basically power data loggers that one would also was fine actually tries to disconnect when I connect to the car over Bluetooth it says do you want to allow a mirroring of your phone box the phone actually is better security than the laptop which doesn't make sense thank you ok this seems to work so thank you for coming here I will slowly start my name is Mikhail Ritski I'm working for the top top level domain registry and I will be talking about why we are making open source routers and how we are making them and what we are putting into them but first who we are and how we actually got into this business we are check top level domain register legally we are kind of association of companies but those companies are actually competing with each other and the only thing that they agree on is that we need stable and working internet stable check domain and basically general good word piece and stuff like that so that's what we are doing we are basically run like a nonprofit in general and apart from taking care of check domain we are doing some open source development as well historically we are doing bird which is BGP routing daemon we are doing Knot which is not which is DNS server and resolver we are also doing some propagation of open source ideas and stuff like we are doing books IPv6 check translation of ProGate we published LibreOffice writer book and we are trying to teach people how to handle internet and we made a TV series about that that's running in Czech public television so we are trying to do a lot of good stuff apart from that we are also running Czech national ceasar team and one of the stuff that we are doing is open source Wi-Fi routers how it all started well we had this interesting question how safe are home users from from attackers on the internet and we wanted to figure out how many attacks is average Joe facing every day at his home is there any attacks at all if you don't serve any services if you are not listed in any domains if you don't have a web server mail server and stuff like that if you just use your internet connection to browse the internet is anybody targeting you to figure out the answer to that we created a project and the point of this project was to actually create some kind of security probe that we will give away to people they will install it at their homes and it will collect data about the attacks at them so automatically collect firewall locks prepare some honeypots for attackers and try to figure out what are those automatic attacks about and as we developed this probe it was hardware probe and it actually turned out to be a router and it had a wifi and because we wanted to make it nice and yeah we wanted to make it the right way we made it open source we made the software running in there open source one of the obvious things that you have to do if you are giving people some hardware device you have to make sure that they have security updates all the time and one of the very ethical thing to do is to give them a root account on the device right because they want to be in control of the device that they are running at home so we did all that we had some interesting hardware for that and we started our secretary search we got some results from the secretary research and we published them we went to conferences and we were talking about the threads which users are facing and people started getting interested in the router itself because it turned out that nobody does stuff the right way at least most of the people do it the wrong way so people started asking that they would like to have the router as well so we actually created commercial routers over the time that people could actually buy and that were meant as a router and not as a security project and on those routers people can still participate in our secretary search and help us to collect data so how do we do stuff? free and open source software that is the obvious choice I guess nobody will oppose it here we are apart from that we know that we can't compete with really small and cheaper routers and we also want to run plenty of services on our devices so we have to have enough resources so we are not trimming it down we are trying to make routers that that make sense apart from that obvious thing is security updates DNS sake validation because we are in DNS business we are giving people route accounts and we have some extra features so why open source? is there really any other way how to do stuff? yeah if you don't like doing stuff then yeah probably but why to trap yourself so it makes things much easier to get started there is plenty of software that you can easily integrate and you get plenty of benefits people can contribute and fix their issues people can do whatever they want and everything is great right? it has few issues there is plenty of software so the tricky part is which one you are going to integrate because people want everything especially their favorite text editor that just three people on the world are using and you haven't integrated it yet how is it possible? it's been out for three weeks but in the end some people will actually contribute and will package it and will send it to us and in the end they will contribute other tricky parts about doing stuff the way we do it is we allow people to do whatever they want and quite often it happens that people actually believe that they are much more capable than they actually are and they believe they know what they are doing without actually knowing what they are doing so they can easily break stuff in really creative ways and then complain that we did it wrong and that's kind of okay but tricky part is they try to rewrite parts of your system and then when update comes they realize that it got updated and their changes disappeared because they wrote half of the stuff without telling package manager about that and now it's gone so those are actually tricky problems that you don't have if you lock people out and at some dark moments I actually understand why the companies do it but then I'm using the device by myself so I don't want to be locked out another tricky part is upstreaming we try to push stuff upstream but sometimes it takes longer than everybody would like and yeah we are working on it slowly we have to make sure at the same time that we have deployed work and at the same time upstream wants to have everything polished and working but we can take some shortcuts for the stuff that we are deploying it doesn't have to be that polished but it has to be working fast and really working so now for the interesting part I would like to talk about software that we are building on top of the software that we are running on our routers that are helping us that is helping us to make something that our users like first thing we are running open source Linux distribution based on Linux obviously it's called OpenWRT it's generally meant to be distribution for routers if you bought some router and tried to play with it you might already encountered it it can be flashed to normal routers you don't have at least some of them if you are lucky typically it's optimized for routers that don't have as much resources as we have typically some megabytes of storage and megabytes and typically it's flashed in a way that you have a highly compressed file system that you flash and then you can install just one or two packages in the remaining space and then your memory is full and because you have a slow CPU it's it can't do much anyway so you are happy with what you managed to get inside we are doing it a little bit differently we are based on OpenWRT we are taking all the packages and all the work that is done there we are contributing back but apart from that we created our own web interface that is much simpler than the interface for OpenWRT OpenWRT is targeting people that are that are tinkering with their routers so they expect some level of technical skills we are trying to make something that is really easy for end users to operate and we are trying to use the web to make web interface that allow people to use advanced features even easily like setup OpenVPN server or guest network and stuff like that and compared to OpenWRT we are not doing highly compressed images but we are installing stuff as every other distribution out there on per package basis which is cool and nice unfortunately it has some drawbacks because OpenWRT doesn't do it that way so sometimes we run into some troubles that Upstream is not facing because they don't have to care about migration from package to package and we are doing some integration bits and tweaks here and there because we can afford to lose a few kilobytes of RAM if we get interesting functionality out of it so that's one of the parts that we do the other part that is interesting is butterFS as I said we are not doing highly compressed file system we are using ordinary butterFS it's the coolest file system out there for Linux and the coolest part about that is that it has snapshots so we went crazy with them we are doing automatic snapshots before every update we are doing automatic snapshots every week we are letting people do manual snapshots and our factory as that button actually takes you depending on how long you press it either to the previous snapshot or to the factory snapshot or refresh the whole router from USB steak or something like that so we are trying to integrate snapshots quite heavily and it helps our users quite a lot Suze is doing similar stuff with OpenSuze that's where we get the idea apart from that we found out that we have plenty of resources and some people would like to run their application that they are used to running from their big OS and sometimes the OpenWRT applications are a little bit too trimmed down so we have LXC on the router and some people are running buyers distributions even multiple of them and yeah it sounds crazy in some in some cases it makes sense in some cases it's totally crazy but yeah if people want to do that let them do that now one of the things that came from our security research and actually got separated from the router is one of the services that we are running it's called Honeypot as a Service and the story behind it is Honeypots are cool right everybody wants to have a Honeypot at home and to lose some attackers and see what they are doing trick them and be mean to them and stuff like that but it's it takes some time to set it up and when you set it up then attackers will start to come and yeah you need to somehow visualize what they are doing you need to make sure that your Honeypot is secure and up to date and stuff like that and you can't just let it be there and hope that nobody will break out so we came with the idea that we are actually running Honeypots on our servers and behind plenty of firewalls is like it so if somebody breaks out it can do no harm but we are offering them as a service so you can register on our website and download a client that will basically just open up a port on your device it can be router or server or whatever and provide attacker to our Honeypot and in the providing there is no such danger as running full Honeypot the risk is much lower and we will collect the data you get some nice visualization on top of on our servers and apart from that all the data that we collect are available for download from all the sessions from all the users around the world so you can download sessions from that other people collected and contribute back with your sessions so if you want to see what people are doing in Honeypots it has Nick CZ and you can download various sessions there we are trying to do something similar with the other part as I said we are collecting not only Honeypots but we are collecting firewall walks and we have something we call Minipots it's kind of Honeypot it pretends to be a service but the only thing that it does is it asks for credentials and then close the connection and it pretends to be Telnet or HTTP proxy or something like that and we are just collecting credentials that those attackers are trying to use most favorite one is system system so this is something that we are working on we used to have a system that did that on our routers but it was tied to our routers we never expected that people would be so interested in helping us collect all those data but with HAAS we found out that they are really interested and people are contributing from their servers so we want to help them to allow them to help us collect those data and then we will publish them again in some form on our websites for everybody to download and see apart from that we are providing those information to our CCID team and they are doing some research on top of it and the results they share with other CCID teams around the world apart from helping us and getting some visualization you are helping the world if you join us now other interesting parts that we can do thanks to being open source and doing everything open source is integrating Suikata Suikata is IPS IDS solution works with network flows allows you to look into what the packets actually mean can lock the information or can directly block the flow and there is open database of rules for Suikata that you can use or you can write your own rules so what can it show us actually for unencrypted communication it can show us pretty much anything it can decode the HTTP traffic and you can take a look what's in there and you can match individual fields if it's encrypted it's harder but even before encryption starts there is some handshake you can get print of information from certificate exchange so you get at least to know what server it's trying to connect and you get source IP some MAC address length of connection amount of data transferred and stuff like that and you can write alerts for stuff that interests you and that gets locked so what we decided to yeah you can do that what can you use it for you can monitor devices you don't trust so basically all the devices and see where they are trying to connect and what they are trying to do online you can try to detect some suspicious anomalies you can use the rules that are available the open source ones and watch for alerts the open source rules contains for example virus malware traffic rules so they know how some malware behaves and they are able to traffic to detect that kind of traffic you can write your own rules to detect and you can try to block some traffic so just to show you what you can get from TLS connection this is basically encrypted connection but at the beginning we get the information about certificate that server was presenting so even though we don't know what was happening afterwards we know who issued the certificate which servers it is from and stuff like that so if you are watching for the traffic you don't see just IP addresses you see the names and the names that are meant to be so even if there are multiple multiple virtual servers running on the same IP you can distinguish between them and you can write some simple rules yeah in this example I am writing the Facebook but because they are sponsored I should be nice to them so think about some other company that is evil not like Facebook and you want to block them so you can use the information that you get from the analysis to actually block the traffic so you can block the traffic based on information that are part of the SSL certificate and not based on the IP address because some of these services have plenty of servers around the whole world and it's hard to get maybe even impossible to get the list of the IP addresses that are there but people are trying to connect to those services based on the names so you can detect from DNS traffic the IP addresses and because they are trying to connect hopefully in secure and encrypted way then you can just check the certificate that is presented by the server and see what is the name of the server that people are trying to connect to and then you can block that one so we took what Suicata offers and we wrote some integrations a bit on top of that we call it Pakon and basically it stores all this information and aggregates it and presents it in some in some way that people can understand so it collects DNS requests and matches them with IP addresses it also collects the stuff from TLS certificate that I was just talking about so in the end you get a list of your home devices and the services they were trying to connect to not IP addresses not reverse reverse DNS entries but the stuff that the devices think that they were connecting to so that helps to actually understand what your devices want and where is your fridge browsing when you are not at home and stuff like that and then you can try to talk to the fridge and explain that it shouldn't go to China and search on Baidu and stuff like that another interesting part that people are doing with our routers is NextCloud yeah, it makes good sense NextCloud is solution if you want to have your own cloud self-hosted it's solutions if you care about your privacy allows you to store your files calendars contacts and other stuff locally on the device that you trust and yeah, router is a natural fit because it's ultimate self-hosting it's not in data center that you don't have it under control it's locked at your home you are root you are managing it and part of our security research is that apart from collecting all those data that I talked about is that we automatically generate a list of bad guys out of it and when we found out that somebody was attacking too much we mark him as a bad guy and distribute this list of the bad guys to all our routers and they block them so if he tries to SSH to two routers so he will be blocked on all of them so we have some extra added security for your hosting of Nextcloud so we have some plans with the Nextcloud we actually created a device or we are working on creating a device that will have that will target that we already have some easy VPN support we have automatic updates that's the basic stuff that you want from security security updates right and we have Nextcloud already integrated currently from CLI we have some simple web UI to format the drive and mount it and we are working on some better integration as well and another thing that I have here is a TV head end that's one of the little bit crazier usages that people are using our routers for well we have a USB 3 there and people still use TV for some reason even nowadays so they decided that they will turn their router into DVR they put the DVBT dangle into one of the USBs USB drive into the other one and with the help of TV head end they are able to actually record what's going on in TV save it to the hard drive we have Samba so they can share it to the TV mini-DLNA to stream it and they are happy and they have new DVR from their router so basically what we are doing is router that is combined with kind of home server and people are using it for everything and they are coming up with new and crazy way how to do stuff so thank you that was what I have in slides and since we have some time we can go for the questions or I can show you for life comparing OpenWRT with IP tables OpenWRT is a Linux distribution so it contains plenty of packages and part of those packages is IP tables and they have few scripts that will allow you to in more human-readable way not since admin-readable but human-readable way to set up the firewall rules but if you already speak IP tables language you can just drop your IP table rules into separate configuration file and it will load it so in OpenWRT you still have full IP tables but there is also some simpler configuration that is supposed to be more convenient for average average people okay how would I compare us with Cisco or other routers basically we don't have ASICs for doing stuff but we have we have 3Gb interfaces on SOC on CPU we have a switch that can do VLAN tagging by itself and we have dual core 1.6 GHz CPU so we have plenty of computing power inside the CPU so unless you go crazy with Sericata it's enough for normal usage and if you go crazy with Sericata depends how crazy yeah and the price around 300 euros okay another tricky question are they available in US we are working on it I hope that I would have answered but yeah we need to do different certification for US no CE FCC and we found out that it's much trickier than we thought because the documentation is completely different and we have to produce plenty of new paperwork that we didn't have to do before and we have to go through some company that will certify us and it's taking a little bit longer than we hoped and they always come up with we need to produce and and stuff like that so we are not officially here yet but there are some people selling those on eBay and stuff like that whether it's our OS compatible with other hardware in general yes the tricky part is in the OpenWRT as they are trying to trim it down as much as possible you have kernel tailored for the concrete device so that's what we do as well so for for the packages you can probably take them and use them but you would need a different kernel and we currently produce binaries for ARM v7 and ARM v8 and some weird PowerPC that nobody wants to know about okay I would just go quickly to the house just to show you what's the output from our honeypots if internet will work yeah, English and some global statistics and you see who's attacking the most congratulations you are there and as I said we have the data and you can go for a specific month and download a specific day and get all the sessions that were recorded and let's see if I will log in and if my yeah if my honeypot is still running do you have some questions in the meantime yeah how many data can we push on how many sessions can we put on house or through the router on a box well with nothing we just tried to hyper and gigabit is fine with with with suikata when we went through when we applied some rules and we did nfq and we looked at every packet we got down to I'm not sure how much but something like half of this gigabit but you can use plenty of tricks to actually speed up because one of the things with suikata if you are running it most of the traffic is encrypted nowadays so what what interests you is just the beginning of the of the communication and then you don't need to look at every packet because it's encrypted so it doesn't tell you anything anyway so then you just need to take a look at the beginning of the communication and if the communication is long going then it doesn't slow you down actually so here you can see my recorded sessions you'll see who was attacking what he was trying to get inside with and whether he succeeded and this guy looks like he put in plenty of commands so let's take a look at what he did so he tried to run some stuff some weird stuff more weird stuff and you can browse it and yeah that honeypot that guy never went to my router he just thought that he managed to get in sometimes in those sessions you will see that they are they didn't realize that they are already in and they are trying to login again sometimes you see that they have syntax error in there sometimes you see the same guy going in over and over again and trying the same stuff without realizing that he already tried it so yeah it can be fun and you also get statistics for every device that you connect for my home router it's for some reason hated by French guys yeah that's the fun that you can have with honeypots without risking anything actually well we displayed here like that but we are also sending it to our CSET team that's the guys that are coordinating the security threat alerts and they are doing security research and contacting other teams around the world and they sometimes find some interesting stuff like new types of attacks new worms that are going through internet and stuff like that but apart from that all these data are available online and you can download them and if you have some clever idea what's to do with those data and how to find some new way of some new way of finding and researching figure it out catch the bad guys and yeah save the world yeah as the thing that I'm showing right now you can download it it's actually available on pip it's available in openSUSE the whole thing okay so yeah I will finish just this thing you can download and install on your servers and we will be happy if you contribute with more data regarding our distributions our distribution with all our customizations and stuff like that it's available on our GitLab and yeah you can yeah you can download it there and compile it for your router we are currently in the process of cleaning up all the stuff that we have accumulated and trying to upstream as much of it as possible but we are probably not going to upstream our easy user interface because it's written in Python and you probably are not going to get Python into router that has just four makes of storage so it doesn't make sense but we are working we will have a separate feed with our software and we already have a separate feed with our software and our integration bits so even if you don't want to have all the customization that we are doing into the base openWRT you can just use our separate feed and compile packages that are interesting from that any other questions? no then thank you and see you around if you have some more questions tomorrow I will be hanging around and usually I am hanging around at open to set boot at Expo so thank you