 Okay, thanks everybody for coming to this particular track to listen to our talk. My name is Trevor. This is Egypt. We work on a project called Metasploit at Rapid7 and today, thank you very much, yes, several Metasploit fans in the audience. Today we're going to talk a little bit about building a binary protocol client in Ruby. First we want to set a little bit of expectation. This is a complicated topic. There's a lot we could do here and this room is filled with people from all sides. There's probably some brand new people in here, probably a lot of people who have been writing code for a long time and probably some super-adept awesome people on the right side of the bell curve. You got to try to make a talk that everybody's going to be able to get a little bit out of. Today we're going to go through what a binary protocol is, how you can analyze it, how to relate some of the wire shark info you see to a spec that you've got from the internet, how to create blobs of binary data, binary strings, and then we're going to talk a little bit about a client that we've been building in-house. It's open source to deal with the SMB protocol and some of the challenges there and how we've dealt with those. We talk about protocols a lot. What is a protocol really? It's basically a communication structure that's known in advance, like two boxes or two pieces of hardware or something, two entities need to communicate. They're going to need to know in advance what the data that they're getting means, semantically. I like this part of the definition that actually comes from the telecommunications thing in Wikipedia talking about it, the variation of physical quantity. As an aside, we all deal in abstractions all the time, so it's kind of fun to start thinking about the fact that at a certain point all of this becomes physics. Right now I'm talking and then something's going through some wires and then you're hearing vibrations in an error medium, and that's how there's a physical change happening and that's why you're able to hear me speak. The reason two machines can know about each other speaking is basically because electrons are moving someplace or there's pulses of light on a wire or whatever. That's just an aside. I just thought that was an interesting piece of this definition. There's two basic kinds of protocols. There's text-based, which we should all be familiar with, HTTP, things like that. Then there's binary and we're going to talk a little bit about two of those today, Apple push notification and server message block. And these things are opposites. A text-based protocol is really designed to be read by people. I have colleagues, I'm not this good, but I have colleagues that can just open a net cat socket and bang out an SMTP interaction from memory. And the other box you're talking to will understand that. That's all you need. You can just type out some text and it's going to go. A binary protocol is intended to be read by a machine. They're created to make something as a communication interaction between two devices or entities absolutely as efficient as possible. So we all have probably looked at HTTP traffic before in some purpose-built tool. I actually bought a laser pointer, which I'd never had before, so I'm going to try it. Ooh, see? Yeah, right there. All right, so we know that there are portions of the data that you see in something like this that comes straight out of the protocol. Some of the stuff you see in an HTTP inspector like this is just going to be nice stuff that this purpose-built tool hands you to better understand the protocol. And some of it comes straight out of this. If I were to take a command line interface like Curl or something like that, I could grab basically that same key value information. I could run a command line command and it would give me exactly what's coming back over the wire and I could take a look at that and I could see it. It wouldn't be this pretty, it wouldn't be all purpose-built like this, but I could still make a lot of sense of the protocol. So what would it look like if I decided to do a command line command to grab some stuff from some binary data? Well, it looks like that. So it's super easy, right? No, not so much. And by the way, my images here are from the wire because you have to, right? I mean, how could you not put things from the wire? And thanks to Sunny for pointing out that they should all be from the wire. So yeah, clearly it's hard to make sense of this when it's not text, right? It's raw. So how do we do this? Like, what's our tool that sort of gives us that nice, like, Chrome-esque interface for kind of messing around? Wireshark. I hate when people do this too much in talks, but I do like to sort of get a sense of the room like, can you raise your hand if you had to use Wireshark as part of your job before? Oh, wow, all right. So I should maybe fast forward through a lot of this. But for the, like, 10 of you that didn't raise your hand, this is Wireshark. Wireshark is great. It's the way to analyze protocol, to analyze packets, supports hundreds and hundreds of protocol dissectors, about 800 and something. So chances are there are no protocols that you've ever heard of or will ever have to deal with that are not kind of already in Wireshark's world, right? So in this thing, we can see, like, the whole packet right here. These are all the hex bytes of the whole packet. And then we have clicked on, we're talking about one interaction of SMB, which I think is, yeah, it's a create file request. And then here we've clicked on a portion of the packet, which is the SMB header. And now that piece of binary data here is represented as hex bytes here inside. So it's like right in the middle of the packet. So that's really cool. We have kind of this same sort of, you know, nice, hydroveial triangles and hierarchical data. The binary blob of data that has been captured by Wireshark has been decomposed into meaningful things. And you can use Wireshark to sort of understand what's going on. Wireshark is your way of connecting a protocol spec to, in some reference implementation, maybe you need to make your client for your own purposes like we do, and we'll talk about that in a minute. But it's your way of kind of connecting those two things up together. You can see, okay, here's what the spec says the status should be. And here I can confirm that in Wireshark. Incidentally, you could also do the same thing if you wanted to with HTTP. It's not nearly as useful as using something like Chrome. But there it is. This is the same request. This is the chunk that actually outlines this big blob of HTTP. And you can see over here sort of everything that's in that 200 request. And then this is the hex bytes before they've been converted. But once the system has sort of decoded them, it's just text, right? It's not that unprintable binary data like we see here. Another thing that I should mention about Wireshark, just in case you don't know, is that this is one portion of it, right? But every layer in the network model, the OSI network model, it is going to be adding its own thing, right? And it's kind of reversed here. So you see SMB, then above that you see the session layer. Then you see the TCP part, then the IP part. Then above that you see the Ethernet frame. Wireshark really does help you kind of grok networking itself. It's kind of nice about it. And so, like I said before, it's important to understand that when you're looking at a packet, every bit of this is going to be represented. And that's important because on the wire, yeah, another wire one. Thanks, sorry, I just had to do it. So okay, so now we understand kind of like what kind of things we would use in order to try to analyze a given chunk of data that we've pulled off the wire and how we can kind of make sense and match them up with a spec. So let's take a look at what a spec actually looks like, and we'll do it for an easy binary protocol, i.e. not SMB, which is almost the dictionary definition of a not easy binary protocol. And we will look at something that I would imagine a lot of people in the stream are familiar with, which is Apple push notification. Whether or not you've ever written anything to deal with APN, if you've got an iDevice, you have certainly received a push notification at one point or another. Incidentally, I couldn't find anything from iOS 7, so I got your little throwback here. There's a super simple protocol, it's asynchronous. You're just firing off from your server a group of these things, right? So you're like, yay, the giant's won the baseball game, or, hey, somebody shared a nevernote note with you or whatever, right? Whatever the semantics of your server side of your app needs to be, you can interact with Apple's high speed backend and send your data. So what should that data look like? What does a push notification look like? Well, when you go and you look at the documentation, you'll see this handy diagram. And what this diagram is showing you, really, is a collection of pieces that have semantic meaning within a big chunk of binary data. And that binary data is what you're putting out on the wire, and it's just decomposed on the other end, right? The same way that Wireshark takes an SMB packet, and it uses its disector, and it knows that it's certain byte offsets, there's certain pieces that mean certain things, that's kind of what it's showing you right here. Okay, so this is saying, all right, we're going to send a bunch of notifications at once in a batch, and each item is going to have like a certain kind of structure. So what belongs in there? And you're not really going to be able to read this, but it doesn't really matter so much. This is kind of like a table of what those pieces should be. So you know, okay, the first 32 bytes, that's the piece, that's like the iOS identifier for your phone or for your iPad or whatever, it's your unique string put on at the factory. And then the next chunk down is 2K of JSON, up to 2K of JSON. And then at an offset a little further, there's four bytes that represent just an arbitrary ID. So if Apple has to bounce this back to your server, you can look it up and know which notification failed and whether you need to resend it or whatever. After that, there's four bytes that represent a date. Then after that, there's a byte that represents priority. The point is, each piece of this that's enumerated in this spec is going to be pushed into a structured piece of binary data. Well that's cool. So now we know that we can take a look at a spec and we know that we're going to be constructing a chunk of binary data that's going to have meaning on both sides. But how should we do that, right? I mean, we want to know how to do this in Ruby. This is a Ruby conference. And we know that we need very fine-grained control over what's actually getting put out there. And so effectively what we need is a binary string. So how do we make binary strings in Ruby? That you can make them and you can read them and write them using, or write them and read them using array pack and string unpack, right? Which basically are the similar sort of mirror images of each other. You let you create and decompose strings. And when you do things with them, basically it kind of looks like this, right? So here we're opening a file and then we're reading two bytes and we're unpacking it with a certain encoding and we're getting some semantic meaning out of this, right? Where we have name, width, height, et cetera and we're kind of getting what we know to be some data that represents a rectangle. But this is not something you could really build a client on top of very easily, not this way, right? This is sort of completely imperative. It's ugly, it's hard to keep from repeating yourself. Everything is hard coded. The size of the byte offsets are coded, the type of encoding you need to do, right? So we need structure, we need repeatability. So if we want structure and we want to be very, very tightly packed and want to be able to define the sizes of every piece of data, really what we want are structs, right? Which are kind of the original structured data type. This is not from the wire, but it's fun. It's like, I think it's fitty, right? It's just easy to look at or it's blinking a little bit. And so actually, you may or may not know this, but in MRI, all Ruby objects are just instances of different kinds of structs, right? There's R object, there's R string, R array, blah, blah, blah. So you're using structs all the time in Ruby, even if you didn't realize it. But this is not like the struct, like sort of anonymous Ruby class thing, right? This is a C struct. So this is what struct looks like in C and this is obviously not a real packet, especially the name should give it away. There's an ID and a sequence number and it's just two integer fields, right? So in C, what you would do, you make one of these guys and then you instantiate it sort of with some values, sort of setting ID to one, sequence number to 80, cool, arbitrary, and that's how it would be set up in C. So this is my big blocky definition, but what actually matters is that this is a physically grouped list of variables to be placed under one name in a block of memory. There's that physical thing again, right? This is actually in RAM next to each other. These are adjacent. So we have physically adjacent ones and zeros tightly packed together. We know exactly what's gonna be in our little chunk that we've named with a struct, right? So in other words, a packet is just a struct, cool. We still need to know how to make these in Ruby, right? Because we don't wanna use packet on pack. So there are several libraries for this, but the one that we've chosen to use, which we think is probably the best one going so far, is called Bindata. And it lets you do all of this stuff in a nice Ruby-ish way. Classes, DSLs, et cetera. And it gives you a declarative way to deal with your structured binary data. So remember this one right here, where this totally undry thing that we just totally don't wanna use that much. Well, we'll take that same kind of thing. Here we have the semantics of this idea of a rectangle, right? We're reading out binary data that we know at certain offsets, meetings, name, width, height, et cetera. We'll do it this way with Bindata. Then we have a thing that looks quite a bit like a Ruby class. These pieces here effectively look like accessors, right? I mean, they look sort of like attributes, rather, of a Ruby class. But what they really are is defining pieces of, I mean, you can see these are identifiers for C data types, really, right? So this is giving you the ability to define fields that are tightly packed together. And under the hood, it's gonna go ahead and do all that pack and unpack stuff for you. But you don't have to worry about it. You don't have to reason about it. You can have, semantically, a rectangle class. And when I open from an IO and I read in some bytes that are known, you get this read method for free from Bindata record, I read in some bytes that I know are a rectangle. As long as everything matches up and the offsets are all good and those fields are able to be unpacked that way, then fine, you've got a rectangle. You can just immediately read it in that ver. We can use it. So now we're able to deal with this stuff in a nice Ruby-ish way. So let's recap, because now we're kind of at the first part of the talk. We're almost done. So now we know how packets are structured. We understand how we can deal with a spec and match it up to the things that we see on the wire when we're looking at maybe a reference implementation or capturing some traffic to try to figure out how to build a client for this stuff. And now we have a way to produce binary structures on the wire in a repeatable way. So we have the structure to be able to build ourselves a client that is concerned entirely with the semantics of the protocol itself and not with the implementation of reading and writing things out on the wire. So now we're gonna move into a little discussion about Ruby SMB and I'm gonna hand it over to Egypt. Ruby SMB, specifically a new binary client for Microsoft's SMB protocol. And it's new for a lot of reasons we'll get into shortly but it's free and open source under a BSD license. It's freely available. It's still in a very, it's still in flux. There's no 1.0 yet. We're working towards a stable API but we're not quite there yet. We would love your help. It is a new pure Ruby implementation of SMB written to be socket agnostic. So when you write something to the wire you need a socket to do that. Usually that's something that Ruby provides for you that the operating system provides to Ruby that eventually gets down to a thing on the wire that pulses light or moves electrons. But when we're dealing with sockets we have certain restrictions that most situations don't need to deal with. So we'll talk about those shortly. And Ruby SMB is intended to cover both SMBv1 which is the older protocol and SMBv2 and 3 which are very similar to each other but very different from SMBv1. But first what is SMB? What do we do with it and what is it for? It stands for server message block which really means nothing. But it's the main protocol. The thing used by all Windows machines for something or another. It's used for file shares. It's used for printer sharing. It's the thing that Active Directory uses for communicating all of those policy things that your network administrator pushes down. So you have a domain controller that tells all of the workstations that this is how someone should log into you. These are the settings that you need to have. These are registry settings that need to be set for all of our software to work. This is where you get updates from. All of those sorts of settings come through SMB. It does a lot more than that. It also handles service creation, registry reading. It does everything basically. Anything that a Windows machine does on the network, it does over SMB. There is an open source library that does SMB called Samba. It's been around forever. A lot of it is reverse engineered and big portions of it are generated C. And I don't know how much C you folks have read. But reading C is hard to begin with when someone is intentionally trying to make it easy for you to read. When it's generated by other code, it sucks. So SMB is a family of protocols. It's not just one thing. I mentioned the older and the newer, but it started out as SMBV1. It was designed by blind committee. There were many companies involved in its creation and its evolution. It started out as SIFS, the common internet file system, intending to be a simple way to share files across machines. Largely pioneered by Novell, later picked up by Microsoft. Several other companies got involved as well. And it ended up being a lot of similar things and bolted on ideas on top of deprecated cruft. So there are a whole bunch of packet types that are intended to be used for this thing, but oh, we don't use that thing anymore. We use this new thing. And that new thing does exactly the same thing as the old thing, but with different pieces. And well, that's not the best way to do it either, so we've got another one that does exactly the same thing. So there's lots of cruft in that old protocol. And further, it's not really just one protocol. SMBV1 is a collection of dialects, which means that Novell had their version of SIFS and Microsoft created their version, Embrace and Extend, and then Destroy. And everyone wrote their own version and as new operating systems came out, Microsoft extended further. And so the version of SMB spoken by NT4 is quite different from the version spoken by 2000, is quite different from the version spoken by Vista, et cetera. So the old SMBV1 is very prevalent. It's everywhere, it's on all sorts of things, but it's also very fragmented. The newer SMBV2 and later three is designed more with simplicity in mind. So whereas SMBV1 had hundreds of packet types, it was a combinatoric explosion of this thing plus this thing, and then maybe also that thing, you ended up with hundreds of packet types. SMBV2 is designed specifically to get around that problem because that is really terrible to deal with. So there are 20 packet types in SMBV2. Further, the documentation for SMBV2 is astronomically better. So when the EU mandated that Microsoft must document the protocols that they use for compatibility with other platforms, Microsoft had no one working on SMB at the time. The team who had built that protocol had moved on. Some of them were still working for Microsoft, but many of them had moved on to other companies. There was basically no one who understood how that protocol worked, and they had to reverse engineer their own protocol in order to write that documentation. So when they tried to design a new thing to replace SMBV1, they started with real documentation. The documentation for SMBV2 is very thorough. It's great. It describes all of the constants that you need. It describes all of the packet types, and it strives for simplicity. So that makes our jobs a lot easier, but only for SMBV2 because like we said, we have to support SMBV1 as well. Part of the biggest problem, or one of the biggest problems that we face with Metasploit is that we have these old protocols and these old systems that are still hanging around after years and years. It's still common to find XP boxes in basically every corporate network. And worse, every so often you'll still find NT4, which sounds awful, but when you consider a company trying to make money, they have a thing that sits in the corner of a closet. No one's touched it in years, and it's still making the money, so they just leave it alone. We're just not gonna worry about that thing. We'll just let it sit there and make us money. Hey, speaking of making money, how many ATMs run on Windows XP? All of them, like literally all of them. Security support for Windows NT ended last year. Just so you know. Yeah. Wee! So why did we specifically make something new for these old protocols? There's already stuff out there. There are several gems that exist that add thin bitings over Lib SMB, for example, from the Samba project, or kind of awfully, they shell out to SMB client, which sounds like a great idea until you do it. But why did we need something new? So to understand that, we need to understand a little of what we're going to do with this stuff. Metasploit is a penetration testing tool. It's used for exploit development as well, but for our purposes, if you're writing an exploit for something that goes over SMB, you don't want to have to worry about how SMB works. You just want to be able to say, make this pipe and then do something evil to that pipe. I don't care how you make it, and I don't care what protocol it's actually going over, just do the thing for me. Metasploit is open source, and so we need an open source client. Specifically we're BSD, so we can't use anything GPL because it will infect our code base. And an important point here, Metasploit is audacious as hell. We go into a network and we knock stuff over, we kick over our sand castles and we lick cupcakes. That's our job. So we need something that is able to follow the spec for as long as necessary, but deviate whenever possible. So in a situation where we're writing an exploit for an SMB server, we need to be able to follow the spec up to the point where the server has a vulnerability and then we need to be able to deviate, do something that the protocol doesn't normally do to reach that vulnerable code. And importantly, Metasploit is written in Ruby, which is why you probably care. So to elaborate on that point, pen testing involves networks primarily, and when you compromise a machine, you get to see all of the things that that machine can see because network machines often see several network segments and that doesn't always mean they've got like an extra network interface, they're not plugged into two different networks, but it means that if you're behind net, a corporate network has a box that can see the corporate network, it can also see out to the internet. Those are two different network segments. So if I compromise a machine inside the corporate network, now I can see everything that that guy can see that I wasn't able to see before from the internet. So we can compromise stuff on the inside as well. This is called a pivot, and this is kind of what it looks like. We have a firewall in the middle. I can talk to the dude in the middle, but I can't talk to the two at the top, or the one at the top and the one at the bottom. I can't talk to them. So if we compromise that guy, now we're on the same internal network and we can talk to everything on the inside. We have a slightly more complicated setup, in this case with a PCI network for payment card industry, that's regulatory mandated that they must have a separate network for everything that deals with credit cards. So if you've ever had to take payments from someone, you're familiar with how this setup has to go. And in that situation, that top box is our jump box that is the only thing that's allowed to talk to the PCI network. So if we compromise him, then we can compromise the backend. But all of that requires being able to speak protocols across each of those guys. So why do you care specifically as Ruby developers? You know, this talk is about clients, it's not about hacker stuff, but to be able to get to those services on the inside, to be able to route our packets through there, we need to have clients that understand the vagaries of routing through those compromised machines. So, Medesplate network clients must create their own sockets. We can't rely on the operating system because the operating system doesn't know how to get there through our compromised machines. Most Ruby protocols handle socket creation for you with something like TCP socket. So things like Net HTTP, Net SSH, those rely on Ruby, which in turn relies on the operating system to create those packets. We can't do that because we have to be able to go through those pivots. So our solution to that problem is to abstract the two apart. We have dispatchers, which are things that send packets. And I don't care what the client is, it does something with binary, I don't care, I just take a packet, I turn it into a binary string, and I put it on the wire, wherever that wire happens to be. So the socket that actually gets me where I'm going are dependency injected. And then we have a client, which is the thing that understands the protocol. That sort of holds the state of the protocol and tells us what dialect are we using? We talked about dialects before with SMBV1 has dozens of them that have varying semantics of what every particular piece of a packet means. So then this is a little bit of the code. And you'll notice here, it's fairly simple. We're first making connection, and then we're passing that connection as a dispatcher into our new client. And then that client can do high level operations on the protocol with negotiation, determine what dialects are available and it can authenticate and do other high level things. But notice in this code, we didn't have to make any packets. We don't care about the binary string that represents those packets. We don't have to care about how the packets get on the wire. We don't have to care about the response or how to read it. That all happens in the client, the client knows. So when you're trying to read a file from a remote machine, you can use the client and just say, make me a new file. I don't care how you do it. I don't care what system is actually sending the packets. Just give me a file and I can read it like a regular Ruby file object. So the idea is we replace everything that needs the operating system with something that can go through our own socket system. So in the future with this design, the plan is to be able to speak SMB V1, V2, and V3 with all of the important dialects along the way, including encryption. You'll be able to route through things like net SSH or really anything else that can give you an IO like. So if you want to route an SMB connection to some server behind a firewall through SSH, the SMB client shouldn't need to know how to make an SSH connection. It shouldn't need to know how to do authentication. It should just know, I have a thing that I can give it binary data and it will put it on the wire in the right place. So the point of having this separate socket system is so that you can do that. So now hopefully you understand a little bit about how binary protocols work in general. You can relate a little to how a spec turns into binary junk on the wire and how to build that binary junk in Ruby. So and with any luck, you've seen where to go if you want to contribute to an awesome new SMB client. Any questions? All right, thanks guys. Thanks everybody.