 Hello everybody. First of all, I'm going to be talking about security. We had a bit of a scheduling snafu, so there was a flip in the schedule. So I hope you stay, even if you weren't here for security. It's a pretty cool topic. But I wanted to thank everyone at RailsConf and the speakers who helped us swap things around so that we wouldn't have to be right up against Justin Collins and his own security breakman talk. So thanks to RailsConf for facilitating that. Today, I'd like to kind of talk a bit about security, how to defend against security attacks and propose a series of ideas that are built around a concept of, let's try and attack them or defend against attacks at a higher level abstraction, similar to how metaprogramming is programming at a higher level abstraction. I want to investigate a possibility of defending against attacks at a higher level than we currently do today. So let's start by talking about the anatomy of a security attack. There's two sides to a security attack. One is that you have a vulnerability, someone broadcasting that they're gonna be out of town for two weeks. And the other side is you've got some kind of attacker who's gonna look at that and figure out, I can do something with this. The reason I pull it out like this is, the way we generally handle security today is that we defend against vulnerabilities or we defend against attackers. So how do we defend against vulnerabilities? Well, the general way we do it today is we have a lot of processes and policies in place. We audit all of our code, we look at all of our frameworks, libraries, gems that we use, we keep track of them, we watch lists of security vulnerabilities, CVEs, and we make sure that we're up to date on the latest software. But obviously, as anyone who has done like a Rails 3 to Rails 4 migration, it's not an easy process to stay up to date. So you end up in this sort of cycle where you find out new information, that there's a vulnerability, you patch, you test, you deploy, you test it in production, and this tends to take a decent amount of time. On top of that, it's fun to rag on PHP at Rails. PHP, as an example, has 24 vulnerabilities on average in their existence for the core platform every year. Unfortunately Rails, not even talking about Ruby, but just Rails, has a similar story. Over the existence of Rails, there's been on average seven vulnerabilities, CVEs attributed to it every year, and that's actually had an uptick in recent years into the double digits. And the reason this is problematic is not just because there are vulnerabilities, but because every time there's a vulnerability, it takes a lot of time to go through your cycle, to patch it, to update all the source code. Like in the PHP case, if you think of the site wordpress.com with tens of thousands of blogs, they've got a real challenge on their hands if once every couple weeks, they have to deploy out a new security patch and update all of these blogs. So it becomes a question of how fast can you spin this wheel? How fast can you turn over your patches? But even if you can turn over your patches quickly, that still is only addressing the vulnerabilities that we know about today. There's a study that showed vulnerabilities that are bought on the private market, maybe the black market you might say, sold to attackers or maybe government security organizations. On average, they remain private for almost half a year before anyone has detected them and reported them publicly. Then on top of that, the frameworks, the libraries, the gems, wherever the vulnerability is has to be updated, released, then that has to filter out to all the people using it who have to notice it, update, patch, test, deploy, test, and so on. So it's a serious problem for the vulnerabilities that we don't know about, but attackers know about. But then on top of that, there's all kinds of vulnerabilities that are sitting in source code that no one knows about yet, but they're just lurking there for someone to find. A lot of vulnerabilities you look at them and you trace back when was this introduced and it was two, three years in the past. It's just been sitting there. So the story about how we defend against vulnerabilities is kind of fraught with perils. And because of that, people turn towards defending against attackers as well. So how do we defend against attackers? Well, one of the main ways we defend against attackers is using something called a web application firewall. And unfortunately, you don't get a mini Harrison Ford in a box to fend off the attackers. What you do get is usually it comes as a black box piece of hardware. You plug it into your data center. You spin it up and it looks for patterns of things. But to really make it work optimally, you have to configure it for your application. So you end up in screens like this. I think this is like a lamp stack. And you're having to go through all the different routes that your application has locked down, like who should be able to access those routes, when they should be able to access them. It takes a lot of knowledge to be able to operate these types of firewalls in an optimal manner. So on top of having bought expensive equipment, you now have to have consultants or hire on security gurus yourself to make sure that you're using web application firewalls correctly. And on top of that, they only work at the networking layer. They're not actually inside of your application. So they look for patterns of things. Patterns of things that look like SQL injection, that look like cross-site scripting. But they don't actually know what's happening in the application. So at best they're making very intelligent guesses. Because of all of this, how difficult it is to configure and how they have to make a lot of intelligent guesses, you tend to hit a problem with web application firewalls. And to demonstrate what that is, I'm gonna, we're gonna go on a field trip. We're gonna go to the Castle Gayard. This is a castle in Normandy, France. And in the year 1203, it was actually under the control of the English king. The French king wasn't very happy about that. So he wanted to lay siege to it. So it was a very complicated, multi-step process. But one of them involved, how do we get past these rock walls that you see here? And the way they did that was, they found there's something called a garter robe. And a garter robe is a fancy term for a medieval toilet. You sit on the top of the wall, you do your business, the waste falls down outside of the wall. And so what the French army did was, in their finest hour, they scaled the garter robe and entered the fortress, entered the castle through the toilet seat that was unguarded, because why would you bother? And once they were inside, they surprised everyone else. It took over the castle, opened up the drawbridge, let all the attackers in, and the entire defenses of this fortress were toppled over through a toilet seat. So the reason I mention this is because when we're talking about defending against attackers and looking at patterns and configuring web application firewalls, it's very hard to configure every point where an attacker can get in. Attackers use extremely sophisticated scripts and tools that spider through websites. Just like Googlebot does, they look through every link, they try to find every page that's possible to access. And then they look on every page, and instead of Googlebot trying to index the content on the page, they look for buttons and forms, places where they can interact with a website, and they try to inject anything they can think of, SQL injection, cross-site scripting, anything that they can just try and see what happens. So there's very sophisticated tools on the attacker side. If you have a tiny hole, that may be all it takes for them and their tools to find in a short period of time, insert themselves into your program and tear down your defenses. If someone inserts themselves into your program, gets a pass your defenses that you have, that's called like a false negative in the security field. There's a opposite side of these types of problems called a false positive. A false positive is where you have some kind of defense that is alerting you so frequently for things that aren't actually threats or attacks that you become numb to them. Or in the worst case, you're actually blocking legitimate traffic. Legitimate user is trying to use your site. There's something called alert fatigue, and that's where if you have too many alerts popping up at you from whether it is a security product or a DevOps machine or you're watching performance, if there's too many alerts, it can be hard to take actionable steps from them. When there's a security hack, you generally, or an attack, even if it's defended, you generally wanna do something about it. It could be that a user has their credentials compromised and is being accessed from all over the world. You probably wanna reset their password. Or if there's some kind of breach in your template rendering, you may wanna look at how they're getting passed and injecting code into your site using cross-site scripting to patch that up. But if you have too many false positive, it's too hard to figure out where the problem is and to address it. So trying to, our tools for defending against attackers also have problems, just like our tools for defending against vulnerabilities. So what can we do? Well, I spoke earlier about the anatomy of a security tech and how it's comprised of a vulnerability and an attacker. But in reality, in this case, a thief, he's not gonna enter your house and sit on your couch and watch TV for a few hours and then leave. He's gonna do something. Probably gonna steal all your jewelry, all your expensive property. It's this exploitation that is damaging. It is kind of creepy if attackers get into your house, but the taking of property or the messing up of your house is what's really damaging to you in the long term. And so when it comes to a security attack, can we think of ways that we can try and defend against the exploitation rather than just the attacker or the vulnerability? So that that lens, that gives rise to this idea that I have of meta security. Securing against the exploitation that an attacker is trying to carry out. So an example, if you think of the famous Indiana Jones scene, he's in a temple, he sees a gold statue. Now it could be that this statue is sitting there for the purpose that anyone can come along and maybe pray to the statue, admire its artistic qualities, whatever it is. So it's hard to filter out attackers. I mean, Indiana Jones looks like an archeologist, not that he would steal the statue. And the statue is inherently vulnerable sitting out there in mid-air on a pedestal. So instead of trying to defend against the attackers or the vulnerability of the statue just being there for the taking, there's a booby trap weighted to the weight of the statue. So when Indiana Jones goes to swap it for a weighted bag, it doesn't work and the booby trap is fired. So this is an example of trying to defend against the actual exploitation rather than the attackers or the vulnerabilities. So how can we take this concept and apply it to web security? So I'd like to talk about two classes of exploitations. The first one is SQL injections and the second one is cross-site scripting. So let's look at how SQL injection works today. Now, this is one way you could write a query in your application. It's a pretty terrible way to write it where you're interpolating user input into a query string. But it does the job for this illustration. So let's say you've got this statement and someone comes along and the user ID from their browser comes back as five. This is how it's meant to work. We're finding the user whose ID is five. But let's say if you've got a hacker who's trying to do interesting things, he might say, well, my ID is a string five or one equals one. And when you put that into your query, that returns true for every record. So now you're returning every record of every user in your database. Maybe you're rendering that out to the browser. Worse is if it's the string five semicolon drop table users, in which case now you've deleted all the user records for your site. Not cool. This is kind of a trivial example. You're doing some string interpolation, but there's other examples in the real world that you might think should be fine. Rails active record should be escaping things for me. Here we're calling the delete all method. And we're passing in a string that will be turned into a where clause. It's still a little trivial because you're still creating a string from user parameters. So, you know, I probably shouldn't do that. Here is an example of again being able to delete every user in the table. But then we talk about things like the calculate method where you can sum all of the values in a table. And by using a carefully crafted query, instead of asking for the sum of all prices in all orders, you may be asking for the sum of all ages of all users named Bob. Here's an example where you can combine how rack turns a query parameter into an array and is passed into exists in such a fashion that this will actually cause the exists method to always return true no matter what. And in this case, we are passing in a parameter that depending on how the structure of the table set up, we're able to turn all of our users into administrators. And the common thread at least among these last three is that we're passing a value, a variable directly into active record. It's very easy to think like well active record should be handling things for me, it should be escaping things, but it's just not the case. So that's how SQL injection can happen. I'd like to talk about cross site scripting and how that works. So cross site scripting is when you as an attacker manage to get your code to run in someone else's browser on someone else's website, a crazy idea. It's kind of convoluted how it works, so I'm gonna give an example. Let's say that you are signing up for a social networking site. So you go through, you populate this sign in form, sign up form, and once you log in somewhere on the page, maybe in the upper right side of the page, it'll display back to you your username, your first and your last name. Now let's say that there's a vulnerability in the last name field. So I'm gonna try to put a script tag in there that's gonna call the alert method, pop up a dialog box. So what's this gonna do? Well, when we log in after we've signed up, it's gonna try and render our first name and our last name. And if we've got a cross site scripting vulnerability in here, the last name might be injected directly into the page as raw HTML code. If that happens, now when we go to render the first and last name, we get our first name, it's just some text, but then we get a dialog box. So you might say to yourself, okay, big deal, I've just harmed myself. Every time I log in, I'm gonna see a dialog box. That's not cool, but it's just on me. But then let's say that there's a social networking site. Someone goes and posts a message. I go and I start to add comments to this message. Well now, associated with those comments is my first and last name. And they get rendered alongside it. So if someone else pulls up this post in their own browser, when it renders it, it's gonna render my first and last name. It's gonna do it twice. I left two comments. So now I'm pestering them with two dialog boxes. Okay, that kind of sucks. I'm creating dialog boxes all over the internet, but it's still not that big of a deal. The problem is if I can run a dialog box, I can also try and get a session token from your site. Now a lot of times that these days in major site session tokens are locked down, but there's still a lot of sites out there that do not do a proper job of locking down session tokens. So if I get a session token in my code, I can then send it using Ajax to my own server. Now I've got a session token for a logged in user on a social networking site. And I can start to create content as though I am that user without having to know their password and go through the login details. That session token is all I need. And that sucks for a social networking site, but if it's a bank site, I could start transferring funds. I could do some really fun stuff. So how does this work? How do we get to the point where this has happened? Well, to explain that, we have to talk about something called string.htmlsafe. It's a poorly named function that makes it sound like we're gonna take a string that could contain some unsafe HTML code. And we're going to pass it to HTMLsafe, which is gonna make it safe for us. And that's actually now what it does. What it does is it takes some text, it calls HTMLsafe on it, it returns to us that same text but wrapped in something called a safe buffer. A safe buffer says, I vouch that anything inside of the safe buffer is safe. So if we take that safe buffer and we append some other text to it, it's gonna make sure that it escapes that other text first. So now it's gonna append some script tag that is in a regular old string. It's gonna escape that first and create a new safe buffer. On the contrary, if I take one safe buffer and I take a second safe buffer and I append to it, it's gonna append in cleanly without any escaping because we got two safe things already. We've got two things that are already vouched to be safe for by someone. All right, so how is this actually used for any purpose? Well, when we're doing Rails rendering, it uses safe buffers to append things together so that we don't have cross-site scripting attacks. So it starts with an empty safe buffer. We go through, we take parts of the literal HTML code that's in the template, we call HTMLsafe on it to wrap it in a safe buffer if we're vouching that it's safe and we append it to our rendering buffer. Then we go to an expression. This title method is something in my app and it returns a string. Happens to have some input that the user put in. So I'm an attacker. I try to put in some cross-site scripting code down at the bottom to raise a dialog box but I'm thwarted because the title method returns a string. When that string is appended to the rendering buffer, it gets escaped. So we're all good. We append the end title tag just as before. Now we come down to here, a different helper method that's called inside of an expression. It's a JavaScript include tag helper. And in this case, we're gonna actually have an HTML tag, a script tag, and we want it to get through. So the JavaScript include tag itself is special in that it returns a safe buffer. It is vouching for what it returns as being safe. So when it gets appended to the render buffer, it goes through and it goes through unescaped. Finally, we add the end of our head tag and this is how a template gets rendered together. This is how the safety mechanisms work to prevent cross-site scripting. So you might ask yourself, how do you get cross-site scripting in the real world if we've got all this awesome, complex, Rails security doing things for us? All right, so this is a lot of code on a slide. I don't, you're not expected to actually read this but what this represents is code that's in a gem out in the real world. It's a gem that helps you put a bootstrap UI into your application so that it generates a flash message, a little banner message that you might put at the top of one of your webpages. So you've got an application, an error condition occurs, you wanna put up a flash message. So you say, I'm gonna create this message, user ID, and then there's this user provided input, does not exist. So if they provide the ID of five, that's okay. We create this banner message, looks right. Let's say that I'm a hacker and I try to put in some cross-site scripting again, another alert box. Well, what happens is, similar to how I showed the Rails rendering before, the message is a string. It gets passed through, appended into safe buffers and it gets escaped. Everything's great here, what's the problem? Well, someone came along, they had a different app and they wanted to provide a link inside of the banner. So they added a link to helper. And when they ran it, they were like, oh wait, what's going on? It's not actually rendering the link as HTML, it's escaped it and now I can't have a link. So they said, oh, I see what the problem is. I've got a gem, this bootstrap gem. It's got these lines of code, lines 17 to 19, where it's creating a content tag, a div to hold that message. And it's passing in this message into this content tag helper. The message is going straight in as a string and it's being escaped as it's being appended to the safe buffer for the div. So they said, oh, I think something's wrong here. I'm gonna change it so that it now represents, it now passes in a safe bufferized version of the message. So they go through, they're so proud of themselves, they get the link to pop up inside of their banner message. Everything's great and this got shipped and up in the bootstrap gem. But here's the problem. The first guy comes along, he updates his app to use this latest gem and now he finds that someone, an attacker comes along, tries to put some malicious script in as the ID parameter and now it's no longer being escaped because the gem was wrapping everything in a safe buffer. So all of a sudden, they didn't change any code, they just updated a gem and a cross-site scripting vulnerability was added. What should have happened is the person who had the link should have themselves vouch for the contents of the message and said, you know what, I'm creating a link in part of this text, it's safe. I need to vouch for it, add HTML safe here, not at the gem layer. But this is so hard to get right. It's very hard for people to have the knowledge and all the understanding in every case. You may have like interns who are new working on code baseness, their first real programming job. How are they going to be, how can we expect them to understand the nuances of where HTML safe should be? So that they're not introducing cross-site scripting vulnerabilities to the internet at large. So how can we fix this? Well, with the cross-site scripting, we go through our processes where I ask ourselves, is this, are we actually vulnerable? Yeah, it looks like it, is it pretty important? Well, it's our homepage. Is it accessible by the internet? Ah, man, yeah, it looks like we gotta do something. So now we gotta go through our cycle of patching our code, testing it, deploying it, testing it in production. Not very much fun, taking our time away from building the awesome new features that our customers want. How can we prevent SQL injection? Well, it's very simple. You just need to memorize a long list of rules. When you're calling calculate methods, you have to make sure that the arguments that you're passing in are valid table names. And always use hashes or arrays when calling delete all, destroy all, where. Always use hashes when using find by. Never use hashes or arrays when using exists, though. There you need to turn it into a string first. Never pass user input into group joins, order, reorder, pluck, select, having. And lastly, don't ever try to use find yourself unless you're a security guru because it's got like 10 different ways it can be called with all these different options, each of which has its own rules. So, okay, you learn all of that. You audit all your source code and your boss comes around and says, oh, that's great. Okay, once you're done auditing our stuff, can you audit all our dependencies too? All our gems, Rails itself? Rails has had SQL injection vulnerabilities. I think they had three last year. Okay, you've done that. Now, can you teach everyone else around here about security because we do have a new intern who's starting next week and we don't want him to add anything that could create a vulnerability. And on top of that, we decided we're gonna add a security team. Don't worry, they're gonna review every code change, but we've got two engineers now. So, if one's on vacation, there won't be a bottleneck. We've got 40 people committing code, but I'm sure that these two guys can keep up with reviewing it all. That's not a very good solution. So, let's go back. Let's think about how can we apply these ideas of meta-security? How can we defend against the exploitation rather than the vulnerabilities themselves? So, let's think about cross-site scripting. Where can we actually have cross-site scripting? Let's say we've got a template here. Well, the places where we can have it are where there's expressions. That's where a user, a hacker, could try to provide input to your site that includes some scripts somehow, that could somehow get back to another user to be run. So, the question we ask ourselves is, should there be any script tags here in any of these expression tags? And it's a good question. I mean, how do you know? I mean, we as humans, we can look at it and say, well, the JavaScript include tag helper, that should probably have a script tag, but I don't even know in the bottom one when we're yielding out, what is that yielding to? I don't know, maybe you should have a script. Well, let's say that we start to wrap the HTML safe method. I mean, it seems like everything that is going wrong is somehow going through HTML safe. It's being misused. Okay, so we've wrapped it. What can we do with this knowledge? Well, now every time HTML safe is called, we can look at it and we can ask, where's this being called from? If it's being called from a known good location, like a Rails helper, like JavaScript include tag, we can probably be pretty sure that the right thing is happening. It's the, you're not actually injecting a script tag directly in, you're asking Rails to provide you a script tag with some content. If you're going that far, you're probably the developer writing the app doing that. If we're not being called from a known good location like Rails, then it's very likely there should never be a script tag there. Again, if you have to hard code in a script tag, not using the Rails asset pipeline, that's cool, you can do that, but you really ought to be using a script tag helper, the JavaScript tag or the content tag helper. So we can look for script tags and we can make sure that we escape them first and that should help cut down on the possibility that a gem in the wild has been updated with a cross-site scripting vulnerability because someone threw an HTML safe usage in there at the wrong spot. Let's talk about SQL injection. So this is the same examples that we had before, where you've got a query that you're interpolating user input into, like the worst way to do things. But even in this case, we know that when we execute this query, there's a specific structure to the query that we expect. So when we execute it with a number, it should have a structure that looks something like this where there's a letter that signifies every token in the query and it ends with the number one, meaning like we're looking for ID equals some number. Well, if later on we see a structure that is different, then there's a high chance, a high probability that something funny is going on. Again, in the drop tables case, you get a different structure in the end. You even get a semicolon. So clearly there's two queries that are being executed here. But okay, we can find out what the structure of a query is. But how do we know what is expected? So we have to be able to know like what the app should be doing before we can filter out what shouldn't be happening. Well, you can think of it this way. Every query that gets executed in your code is executed from some stack trace at the top of which is deep inside active record, actually calling out to MySQL or Postgres or SQLite or whatever it's doing. But as part of the stack trace, it includes your application code as well. So here I've got a line in my own test app where I'm querying the car records, looking for cars of a certain make and model. So whenever that line of code is executed, I end up at a stack trace once it finally ends up being executed all the way called down to the database, that stack trace will always be the same. It'll always run through these same lines of code. So we can start to learn that. We can say, okay, I see a query coming in from a specific stack trace. I can learn the first time that the query should look like this. It should have this expected structure. So now when future queries come in, we know what the expected structure is. If a new query has the same structure, it's okay. We let it through, everything's good. But if it comes through with a different structure, we block it. We say this is obviously bad and we respond with a 403. So this is how we can block against the exploits of a SQL injection, even if you do the worst thing possible of interpolating query strings and executing them directly. So going back to summarize, it's good to defend against attackers and to defend against vulnerabilities. We should always be staying up to date as much as possible. But realistically, vulnerabilities are always gonna exist. Attackers are always gonna be out there looking for those vulnerabilities to do something bad. So we need that third level where we're defending against the exploitation itself. That's what's really gonna allow us to process requests, handle things even before we get a chance to patch something for a zero day vulnerability that we see in the wild. Now the reason I'm up here talking to you today is because I'm with an awesome team at Immunio. What we're trying to do is apply these meta security concepts to a whole host, a whole series of exploitation classes. SQL injection and cross-site scripting are just two of them. We're actually running inside of your application. We're watching your queries, we're watching your templates being rendered. We're looking at the headers of the requests coming in. We're watching for people brute forcing your login requests. And we throw up mitigation strategies where we block SQL injection attempts, where we slow down logins from specific IP addresses where we see brute forces coming from, or we throw up captures. We're taking this model and applying it as everywhere that we can get our hooks in into a Rails application. So I wanna thank RailsConf for giving me this opportunity to talk to you. I want to thank you for coming, especially given the time change and everything. And I'd be happy to answer questions here. And we're, we actually just announced this week at RailsConf. This is our big unveiling. We're taking beta signups. We'd love to hear from you, get a sense of is this product doing a good job of exposing you, alerting you to the threats that are coming in against your servers. So we encourage you to come find me. Even after the talk I'll hang out in the hallway. Come find us down at the booth in the exhibit hall. We're still here until the end of the day. And chat with us. We'd love to have you try out our stuff and defend your app. So thank you. Yeah. Right, so he asked what is, let's say the persistence and the learning behind the SQL injection, where you have to learn a stack structure to a SQL statement structure. The way that we address this is the first time a query is executed from a given line of code. We learn the structure. After that it gets locked down and we make sure that all the future structures must match. So to a certain extent, we don't have a learning period. It's just the first time that that line of code is executed. But beyond that, once we've learned it, we communicate it to our backend service which helps disseminate that to all the other application processes you have. Because most people run, whether it's Unicorn, Puma, the fact that you have multiple servers, processes, all of those need to get the information about what are expected structures for SQL queries. So we have a backend service where threat information is sent, but part of the data we send is just what are expected SQL structures for given lines of code so that they can be disseminated to all of your application processes. Yeah, that's a really good question. So the question was what if there's a possibility that you build up a query and it could have different structures and then there's a different line of code that actually executes that query. And we do see that at time to time, whether it's because people are building it up manually using string interpolation, or because they may be generating a query using the A-roll layer, A-R-E-L, which is this library that ActiveRecord uses under the covers to generate a query. So you can do that. What we're doing to address that is, A, it's a minority of cases, so many people won't hit that. B, when you log into our UI, you can see these types of false positives where we say, okay, the second time it came through, it had a different structure. But when you're looking through it, we can then have a, we have a button that says, this was a false positive. When you click that, we learn that structure as well. So if there's, say, five different forms in which a query could have a structure for a given line of code, you learn that a few times and we take care of it. I'm gonna wrap up, but I'll be in the hallway if you have any other further questions, so thank you.