 Two tales from the crypt. Today's tale features Rubazzle.com. A growing startup that sells awesomely. Rubazzle's MacBook cases, like the one you're looking at. Business has been brisk, locally sourced, fresh coffee is flowing. But last Monday brought more than just blinged out MacBooks. Little did our poor developers know how terrifying the day was going to be. It was the Monday before RailsConf. At the Rubazzle Chicago headquarters. Hey man, how was your weekend? It was okay. We had a little outage, but, you know, we survived. We got a couple things we had to talk about at stand-up today, though. Cool. Hey, I just got off the phone with a couple of customers and they're telling me that they're getting laptops ordered that they never ordered or they're getting emails that they never ordered laptops. Man, people are being so stupid sometimes. They probably just ordered something and forgot about it. Wait, when did they say this happened? They said the orders started coming in last night. Oh, well, like we just updated the order UI. But, you know, we tested it pretty well and I think it was reviewed by a lot of people. So I bet it's not a bug. I bet the problem exists between the keyboard and the chair. People have said that about me before. But I'm going to say that maybe it's not. Why don't you take a look, please? Okay, sure. We'll get right on that. Hey, wait a minute. I'm getting more emails. There's lots of people having this issue. It looks like maybe it's more widespread than we initially thought. I mean, I got a lot of my plate right now. We got a release heading out. I don't think it's that urgent. I mean, why? Like two people got affected. It's probably not a big deal. Well, based on the email I'm looking at, it's up to like 20. And I think this is a significant issue. Plus, all those orders have lots of stuff on them. It's not like they're ordering one case. It's like 5, 10, 20. So I want you to get on it. I mean, okay, fine. We'll take a look. All the tests are passing. I'm clicking through. Looks like everything is working as expected. I don't know what's going on. Hey, can you give me the usernames of the people that were affected? Maybe I can take a look at their accounts. Sure. Let me just check my email here. It's abedraatrubazzle.com, presidentbeefatrubazzle.com, and mcondedrubazzle.com. Oh, wait a minute. That's strange. All the user requests, they're coming from the same place. What is this? WowSoDogeMeme.com? I guess they're like, that must be the common referrer here. I wonder what this is. Looks like there's some credit card numbers in there, too. I mean... Look at that. That's kind of funny. It's Doge Ross. Oh, come on. Let's not get distracted by some silly dogs. Wait. Hold on. What's going on here? I didn't place an order. What? Okay, wait a minute. Something really strange is going on. Everything looked fine, and then I visited this Doge meme site. I clicked through some images, and all of a sudden I got an order confirmation. Okay. Okay, this is strange. All these complaining users had this in common. And... Okay, I don't know. Well, go back to the site, and let's look at the page source. Let's see what they're doing. Okay, let's pull this side up real quick, and we'll take a look. Hey, wait. Why is there a domain in the JavaScript in the page? Weird. That request, it's going to the order endpoint. But someone would have to be logged in to even complete that request. Yeah, that's strange. I'll tell you what. Let's go over the last changes from the release. Let's see if we missed something. We'll take a look. Wait, wait, who did this? Yeah, that was me. I was having trouble setting up the new JavaScript UI, and every time I tried to process an order, I got some weird error about authenticity tokens or something. So I just looked on Stack Overflow, and they said just comment out this method, so I tried it, it worked, and I went with it. Wait, what, so did you figure it out? I mean, you could say we figured it out. That comment above the comment that he made? Yeah, that's the thing that says don't comment this out. So, I mean, that little protection forgery method is the thing that actually prevents what's happening right now. It looks like that site also had a cross-site scripting vulnerability, and somebody just pasted this in here. That launches the attack every time somebody views the page. So every time someone goes to that page, it tries to submit an order on our site? Yeah, so let's say somebody's logged in through Basel.com, and they go to BaselDogeMeme.com. All of a sudden, it's gonna make a request, and they're logged in, which means the browser has the right cookies, and it's gonna make that request on behalf of that user. Okay, okay, this sounds crazy. How do we fix it? We uncomment the comment. But if we do that, it's gonna break all my changes. Yeah, we're gonna have to roll back the UI, and then we're gonna have to fix the JavaScript, right? It should send that CSRF token along with the request. It's probably good instead of having our entire site affected by this. Okay, I'll start the rollback, but our release team isn't gonna be too happy with us. Neither am I, and our boss is a dick. That was a pain, but I got the release rolled back, and I made sure that you can't make those requests without the authenticity token anymore. That's awesome. That's good news. But here I've got an email from our council. It sounds like they've been contacted by the authorities, and credit cards that were used to buy Rubazzle cases are now being used fraudulently in other places. So I've never really been through this before, but I need you guys to start looking at this. Dude, we got a release going out. Like, I mean... It's been one of those days, I guess. Wait a minute, though. We encrypt all our credit card data. Okay, this is not possible. Well, I'm gonna pull up the logs again. Okay, I'll take a look and see who's been on the server. Hey, can you take an email to the team and see we've had one of our regular vulnerability scans and see if there's anything that came up? Yeah, I'll get in touch with Team CISSP right away, but I don't know, I need you guys to keep working on this. Okay, I mean, I'm looking here. I don't see any weird access. There's nothing... I mean, I'm the only one who's even logged into this machine. It's patched. This is strange. Yeah, that's weird. This is kind of strange, too. I'm looking at the logs, and I see a lot of requests coming from one specific IP, and it seems like they're looking at sequential order numbers. Okay, so what do we show with an order? Well, we don't show the credit card number. Of course. Wait a minute, though. It looks like I can actually look at other people's orders. That shouldn't work, should it? You mean like if I'm Veronica, I can see my order, and then I can see someone else's order by changing the ID in the URL? So you're just incrementing the ID? Yeah. Well, we probably should fix that. Wait, wait, wait, wait. You mean that a user can see another user's orders? Yeah, that's kind of what's going on. Maybe we should fix it so that people should only see their own orders. How do we do that? I don't know. I heard something about a library called CanCan and something called Pundit. Maybe we should try one of those. Sure, I mean, I guess we could probably make sure that we scope those queries, too. Maybe it's just like the current user is only allowed to see their own orders, right? As I heard back from TeamCISSP, our PCI scans passing green check mark, everything's cool, so they don't know anything. Well, I mean, we found out that we're not restricting our orders the right way, but we don't show credit card details on the order, so I still don't think it's us. Well, maybe we can refocus on the issue at hand and not stop worrying about orders if it's not related to orders. I don't know, maybe we'll look at the logs again. Yeah, sure. So this is interesting. Seems like our endpoints that get hit the most are the ones with .json after them. Maybe we should look at that order controller. I'll take a look, yeah. Wait, what's a controller? Yeah, uh... Look, we don't restrict what parts of the model get sent when they ask for JSON. So if I put this in the browser, it just shows me everything. Oh... Oh, wait, awesome. That's it. We found it. Awesome. It's not exactly how I'd describe it. Well, I mean, now we can fix it, right? I mean, now we just restrict the output of the JSON when people request it. Yeah, and if we get can-can in there, then we'll make sure people can't see orders that aren't theirs. But wait a minute. I thought this data was encrypted. Well, technically it is. When we save it, it's encrypted. But the application itself needs to decrypt it to use it. So it looks like maybe, you know, people are just using the application to decrypt the data because that's what it's supposed to do. Yeah. I think I'm going to go home and rest up for RailsConf. Yeah. Before you go, can you check out this-this thing from our social media guru? Wait, we have a social media guru? Yeah, I guess someone mentioned at Rubazzle in a tweet with Pastebin, and it said it has a-a bunch of our users' information up there. Um, we also got an email from somebody that said that our server had been hacked, and that's how those passwords got out. Um, seems like kind of a big deal to me. Well, I'm looking at it. It's just a Pastebin with emails and MD5 sums. Um, I mean, it's probably bogus because we don't use MD5 to hash our passwords, right? Um, actually, uh... Uh, no. You're kidding, right? Well, so last summer, I tried to get the team to upgrade to Bcrypt, but we ended up having to de-prioritize it for this, like, AJAXy Rubazzling feature. That dropped on the floor. Well, that AJAXy Rubazzling preview feature was, like, the cornerstone of that release. It was a big deal. Wait, so are you saying that these might be legit? I guess I'll start checking them against our users' table, and I don't think I'm going to have time to make the RailsConf. Wait, do you have access to production? Uh, yeah, well, the credentials are in database.yaml, so I've just been using those. Seriously? I mean, what happens is, like, maybe somebody got access to our source code and just logged into the database. Wait, are you saying that there are hackers in our network? Do we need to call the FBI? Let's lock this place down. Hold on, hold on, hold on. Let's not get crazy. Let's get crazy. I am the network people. We'll figure out and see if there's any weird access. Let's just double-check this. I want you to take care of this. Okay. All right, well, you're doing that. The credentials, I mean, it looks like they're straight from our database. They're even in the same order as our table. Oh, that's not good. Do the network people know what was going on? I'm looking here. It looks, I mean, everything seems fine, actually. Um, no database access from outside our firewall. Maybe you should check the logs from the website again. Yeah, I mean, I'll take a look. Um, yeah, let me pull it up right now and, wait, this is weird. What, did you find the hackers? No, I'm seeing a lot of 500s on the order update page, though. Okay, I'm looking at the controller. There is a database query for right here for looking at the qualifying orders for our frequent root basilar rewards program. I see that, too. It looks like something strange there. Um, I don't know. What about it? It looks fine to me. Well, I'm usually suspicious when there's string interpolation in a query. Um, I mean, look, I mean, look at it. We're just dropping that user's reward code straight in. I mean, remember when I showed you that Railsqli.org site last week? Yeah, I remember you showing me that, but I still think it's safe. I mean, that value, we don't even get it from the user. We set that in our admin panel. It still seems pretty shady to me. Um, well, I'm looking at the user model, and we even have a validator for that field, so I'm pretty sure we're safe. Well, I mean, technically, yes. But if somebody just put a new line in there, it actually completely bypasses that validation. Uh, really? Uh, I mean, I didn't realize that when I wrote it. You know, sometimes I think Ruby's regular expressions are kind of weird. But still, I don't get that value from the user anywhere. I'll fix it, but I don't think that's our issue. Well, if we had moved to Rails 4, like I said we should have, this actually would have never been a problem. It wouldn't even be able to start the app. I mean, I'm still worried, though. I'm going to check out the user update action and make sure there's nothing weird. I don't see any reference to the app attribute here, but I'm going to take a look anyways. Yeah, I really don't think that's the problem, so let's just keep looking. Permit in the update method. Uh, in the update method? Oh, right. Yeah, remember we started using strong prams last month? Yeah. Yeah, I was seeing some weird issues, so I went on Stack Overflow and I read the answers. They said just use permit. I tried it. It totally worked. So, I mean, what's the problem? The test passed. Of course the test passed. So, basically, we're saying anybody is a model, including the one that we're using in that SQL query. Man, I screwed this up again. I'll get the query cleaned up. It should just take a minute. Okay. So, I just went to look through some of these logs and I actually see some of these exceptions from the attacker and I found some interesting stuff. I put something together. Check out this request. Okay. What the heck is that? It just looks like a big huge mess to me. Well, I put this curl request together based on some stuff I saw on the logs. Okay. But, I mean, what is it doing? Well, so what we're doing is actually setting that rewards code via the mass assignment bug I talked to you about, which shouldn't be possible, but it is. But instead of sending a normal rewards code, I just went ahead and put my own query in there. Okay. I'm still not quite sure how you're doing that, but, I mean, after you do that, what's it matter? Okay. So, here's what we look... Here's what it looks like. We're actually not escaping that rewards code. So... Oh, okay. I think I see... What's this... Okay. So, we take the value and we interpolate it and then they're breaking out of the quotes and adding their own query? Yeah, absolutely. I mean, basically, the result is that when the query for orders with reward codes is actually doing a union with the user's table and user email and password right out of the database and into the form. Actually, pretty clever. Okay. So, you're saying we had a SQL injection where we were putting the model attribute into our query, but I thought that was safe because the user couldn't set it anywhere, but then we had a mass assignment issue which allowed them to set it, but the only way they were able to get the actual SQL past our validator was by using a new line? Yes. This is exactly what happened. Basically, it was a giant, you know, set of cards that collapsed on top of each other. Man, that's like... that's a bunch of things. I can't believe someone figured that out. Hey, I guess we should call the lawyers and get everyone's password reset. Yeah. Man, this is really a bad way to start RailsConf week. You're definitely not going to RailsConf. Oh, come on. Fellas, yesterday was awful. Really bad day. We need to figure out how to not let that happen again. What can we do better? Well, I mean, at least our Rails version is up to date, right? Yeah. I remember that crazy bug in the XML serialization that let anybody just do whatever they wanted on people's Rails apps. Yeah. The bug got announced and within a day there was a meta-sploit module for it. Yeah. What on planet Earth does that even mean? Oh, well, what it means is anybody who has that little module is going to push a button and do whatever they want on anybody's server. Oh, I'm not feeling well. Oh, man. I guess it's time for a post-mortem. What did we do wrong? Man, it was like a ton of stuff. We didn't protect that model update with strong parameters. That's right. And when we did, we shared the whole table via JSON. Right. And then we... Well, I turned off the CSRF protection so that I could get that JavaScript to work. Yeah, that was pretty bad. Well, we also messed up the anchors in the model validation, which in turn allowed the app to build queries that resulted in SQL injection. Right. And we were using MD5 for our passwords and we knew that was wrong. Yes. Oh, yeah. Don't forget about the fact that anyone can see or edit anyone else's order. Right. And we were storing the card information in that same table, but we were pretty sure it was safe because it was encrypted. Yeah. Whoa. Back from the dead. So now how can we change our processes so that this doesn't happen again? Personally, I think we need a little more training. I mean, I wasn't aware of a lot of these issues, but last night I started reading the Rails Security Guide and the OWAS Rails Security Cheat Sheet. There's a lot of good info there. I mean, we should probably start reviewing our code before we ship it out to you. Well, that slowed us down. I mean, we got to hit our dates. I mean, it's probably worth it. I mean, this is a pretty bad day yesterday. I guess you'll take your point. You know, I was also looking into some security tools. There's this one tool called Breakman. I was reading about it and it looks like it will check out our code and help us find these security issues like the ones we saw. So we can just run it against our code? Does it come in? Yeah. So we could probably drop it in CI then, right? Oh, yeah, that would probably be a good idea. I heard something about this thing called Bundler Audit, too. It helps you keep track of security issues and then your dependencies. Yeah, we should try that out. Okay, it sounds like we need to be more up-to-date with Rails security issues, best practices for Rails apps. Definitely sounds like we need to do code review. Yeah. Training. Maybe get some automated tools in place so that we can catch ourselves if we make mistakes. And maybe we even need to get some partners to help us review code, do an audit to kind of make sure everything is okay. That's probably a good idea. Yeah, I guess so. Wow. Rubazzle.com. Yeah. So this is back to you. Heed our sad story. Otherwise, your app may suffer the same way as what we at Rubazzle.com did. So thanks so much. Obviously, these were jokes, but they weren't too far off from the truth. The stuff that we talked about, the three of us have all seen throughout our time securing, auditing, changing Rails apps. So we just kind of modified the stories we had to fit our lovely Rubazzle company. So my name is Aaron Bedra. I work at Groupon, and I've been working in Rails security for a long time. And I've really seen some fun stuff, including almost everything we talked about here today. My name is Justin Collins. I work for Twitter on the application security team. I'm also the primary author of The Breakman Tool, which I so sneakily promoted in our skit. Oh, thank you. And if you're interested, I have a very small stack of The Breakman stickers like this after we're done. Grab one before they run out. I'm Matt Konda. I do a lot of code review and training. Don't work on any prominent security open source projects yet. Other than Rubazzle.com, which is in Bitbucket, you can go look at it and see the vulnerabilities. These are all real. We tested them all in sequence. The curl is actually kind of cool. And that's kind of... Yeah, so we wanted to give a little bit of room at the end of the talk for questions. Obviously, we had some pretty contrived examples, but there might be something like this that's affected you guys. Obviously, don't speak too far out of school if you can't. But if you have questions, feel free to ask. Now's a great time to start a discussion. Third-party security recommendations. I do, actually. Maybe we should mention them afterwards. Yeah. So, yeah, there are some really great firms that do your security review and actually do Rails-focused security review. Oh, yeah. So database credentials ends the source code, right? Don't check in your database YAML file. Also, don't check in your secret for your sessions. You need to sort this somewhere else, right? So there's lots of different ways to do it. You can go all the way... You get really hardcore and use an HSM, store everything encrypted, and use the HSM to decrypt and retrieve it. There's a lot of infrastructure overhead. So, you know, I mean, putting in a separate repo, a private repo that only production has access to, automation, of course, ChefPuppet, those kind of things are super important there. Yeah. I mean, at Twitter we have some system that someone very smart built, and basically those all go into a separate repo. Only a few people have access to that, and then they're copied to the server when the app is deployed. Yeah, there are actually quite a few tools out there. You know, you can get the bigger commercial tools, app scans, those kind of tools to run against your code. Actually, one of the things I like to do is if the build passes, have the build deployed to a staging environment and kick off a dynamic scan against the app as part of the build pipeline. You can't do it every release. It gets a little backed up, but for actual builds that are going to go to production, you can have that build pipeline that makes sure the dynamic scan doesn't find any high vulnerability issues. It's kind of a collection of tools, but actually you can run headless. You can build some extensions to make burps that run headless and do basically a spider. It will take all the results and send them to a scanner and then scan the site. It's not 100% going to find everything, but it's going to kick off some of the most common attacks against your site, and automating that as much as you can, making it part of your build process is a really good thing to do. Can I add something here? I would recommend taking the breakman and the dynamic analysis plus the dynamic analysis plus a human level of interaction. Frequently dynamic scanners won't find important issues. Anything related to business logic, they won't ever find. They can find like brute, SQL injection or cross-site scripting a lot of the times, but they don't find more complicated kinds of vulnerabilities that can be really important and even easy to exploit. And they're noisy. Any time you see an exception, make sure somebody sees that. Don't let exceptions fall out of the wayside. Don't ignore them. Don't let your application be so noisy that you stop looking. So exceptions usually are the sign. Maybe you screwed up, but every now and again that could be some real issues. If you see SQL errors in your 500s, take a really close look because there could be some bad stuff going on. Yeah, if you want. You should look at the guys from Etsy who have given a bunch of talks about doing really neat monitoring, looking at requests and basically flagging anomalies. Yeah. And the way they have it set up, it's like those anomalies are usually cross-site scripting. Yeah, the behavior of your users is really important. Find out the normal behavior of your user and when somebody doesn't act like they should and you start to trail some interesting stuff. Every time a log aggregation is a big deal, if you have logs, make sure they're all in one place. Like Splunk and Log Stash and Sumo Logic and all those different providers. Get your logs in one place and start doing queries. Look for patterns, look for behavior that doesn't fit, and then you'll find some interesting stuff that way. I guarantee it. List for vulnerabilities. Ruby on Rails mailing list is super important to be on, but also there's kind of a catch-all. That list has basically all the open-source projects will aggregate inside of that project. It's run by Red Hat and MITRE and they're basically collecting everything. I don't think it's a CVE that has an open-source project label that's going to get blasted on that list. I mean, there's other ones to look at. I mean, if you're using a framework, a big one, there's probably a list for it. Your OSes as well. But that OS Stash Security mailing list is probably one of my favorites for kind of a catch-all. I really like Bundler Audit. It pulls from the OS... What is it? OS VDB. Oh, yeah, yeah. A lot of times, if you're just looking at vulnerabilities that get reported, they're not necessarily relevant to you. Like, oh, some gym that's like no one uses and someone found something in it. Or maybe some people use it, but it's just not relevant to you. But if you're running Bundler Audit and you have it, you can set it up to automatically pull in the latest, and then it's going to be looking at your dependencies and telling you, hey, one of your dependencies has a problem that's been reported. Yeah, so the question was, how do you prioritize CVEs, issues, that kind of thing. That's a really, really hard thing. I mean, so at the end of the day, you have a business to run and the business has to function. So if you have a release pending, what's the risk that somebody's going to exploit this thing before you can get a patch out? If the feature you're about to release is going to drive up revenue by 10x and the CVE is not a stop the world thing, release. I mean, it's about being pragmatic. It's not about stopping the world for security. The point is security should help you move forward faster, not slow you down or stop you. I think usually what I've seen anyway is generally you can determine pretty quickly if it potentially affects you or if it definitely doesn't affect you. There are CVEs that come out and you're like, well, okay, but we're not using that. We're not using that feature. We're not using that. You still want to keep up with the releases, but maybe you don't stop everything and patch it. Maybe you patch it as you get to it. Also, it's good to have somebody on the horn. If there's somebody that you're going to work with on security issues, give them a ring and say, hey, how bad is this? I mean, it was really hard, you know, last week, maybe before last with Heartbleed. Nobody really understood how serious it was until everybody said how serious it was, right? It was like, oh, this thing's bad and the world kind of stopped for a minute and then all of a sudden, oh, wait, this is really, really bad and then everybody started patching. So it's good to have somebody you can call and ask, right? That same company you work with through those 30-party auditing. You'll call you. That's right. The thing I would just add to that is I always recommend that you have a process for responding, not just like, it's not about only how severe is it, but certain categories or a certain set of features are going to make you push your app today. Other sets of problems are going to make you push your app this week. Another set are going to make you push your app this quarter, right? And maybe if you collect a bunch of those, and then you get to decide as an organization which are the things that push you into which bucket, because practically, it's pretty obvious that the XML deserialization one we referenced that was like a remote ODE, I can get remote code execution on your server was pretty serious right when it came out, right? So everybody should have patched that as fast as they humanly could, right? But a lot of them are a lot muddier. I think that's probably why you asked a question, right? Yeah. I want to say one other thing. There's been a lot, like, two questions from almost everybody I talked to who said, oh, you're doing a security talk. What? And the first thing they ask is, well, doesn't Rails basically handle security for you? And the other thing is, like, oh, you know, what's the worst thing in Rails? Right? Like, what's the worst, single worst security thing I need to worry about in Rails? Right, or the most common? Yeah. And, like, my response is, like, every single time I've looked at a real, practical, real-world application, there have been tons of vulnerabilities in it. Yeah. Right? Like, it's not practically, I can't tell you, like, go fix the CSRF thing and you'll be good, right? And a lot of times they're chained, like, we tried to demonstrate. So the one thing may not seem like a big deal, but when they, in concert, they become a big deal. And so this is, I mean, I'm hoping that you take this home. They're like, this is real, even though Rails does a great job of doing encoding and a lot of really good details. We ran into something fun while we were putting the presentation together with the Regex, right? The online Regex by default. Yeah. Right? So it's getting, it's always improving and that is one cool thing about Rails, but you can't, like, depend on it to do security for you. Upgrade, though. I mean, yeah. Those of you who are still on Rails 2.3, okay, it's time to have a talk. I mean, I know it's tough. It's going to take you a long time. It's going to take you months to figure it out. But you've got to upgrade. One of the things that's, it's so important is keeping up the date because if there's a big deal and there's no patch for it, it's nothing you can do, right? I mean, you've got to stay up to date. The Rails core team is doing a really wonderful job and you find them, thank them for doing such a good job and they're paying attention to security for real, right? When issues come in, they patch them, they announce them. There's a great response from the Rails core team on security issues. But you've got to keep up the date. They can only fix the problems. They can't fix your app. Right, and don't just update when you feel like you're vulnerable to the issue. Yeah. Because we had a situation where there was an update that came out and we're like, yeah, like, you should apply this update to this app that definitely applies. And then they're like, oh wait, but we didn't apply the other updates and this update relies on that update. So it's always easier if you're just up to date and you should be aware anything before Rails 3.2 is not supported anymore. Yeah, I mean, it's about keeping patch. Put a card in your sprints or iterations or whatever you, however you do it once a quarter or sooner if you can, right? Just update the current version of Rails. Just keep it as a thing you do every so often as part of your process and make sure it's in your backlog, make sure it gets pulled, make sure that is done all the time. It's easier to do small increments than go from Rails 2.3 to 3.2. Yeah. Alright, thank you.