 hosted a tour guide for the next 45 minutes. My name is Mark Bates. That's me there, patient enough, proof of identity. I've been a Rubyist since about 2005, so longer than some, but not as long as, say, Max. I'm a freelance consultant for Boston, so hiring, I think is what that particular goal points at. I'm a musician in case you couldn't tell. And I'm also an author. And this last point is actually pretty important to today's talk, as I am the author of Distributed Programming with Ruby. That's published by Addison Wesley, came out this year. Definitely worth looking into and reading, if you're interested in distributed programming. I know it's a plug, but I have to do it. That publisher can kill me if I didn't. And with that said, if anybody follows me on Twitter, know that I'm going to be giving away some free books today, which is also pretty exciting. And there's just very simple rules to this little game. You got to tweet me at Mark Bates. And during the show, I've got about five images in my slide deck from TV shows and movies. So it's the first few people to tweet me those five TV shows and movies win a copy of a book. Pretty simple. After the Q&A, come up, see me. We'll check Twitter together. And if you want to look at a few, you got to copy the book. So, what are we going to talk about today? Well, distributed programming in general is a big topic. Can't cover all of it in 45 minutes. That should be kind of go without saying. So I've chosen three of my favorite subjects of distributed programming. And we're going to kind of cover those three areas. Most three areas are RMI, which is remote method invocation. If you're a Java person, please don't go running towards the back of the room. It's not that bad, I swear to you. In the book, I do a little comparison of RMI and Java and RMI and Ruby. And the Java one's like three pages of code to say hello world. And in Ruby, it's like three lines. So it's much more fun and really cool in Ruby. We're also going to talk about messaging queues, a very hot topic. Several people have talked about them already. And we're going to kind of go into what they are and some little more detail into them. And we're also going to talk about background jobs. These were the three that I managed to actually get into 45 minute presentation. I wanted to talk about MapReduce, I wanted to talk about web services, but there's just not enough time. I'm really interested in those subjects. I highly recommend the book, it covers all of them. Okay, so let's talk about how we're going to do this today, right? We're going to talk about the who, the what, the when, the why of each of these subjects. Not so much the how, I'll talk about that in a minute. What I really want you to do is walk away from this talk knowing what a message queue is or what a background job is, when you should use it, why you should use it. And then who are the kind of big players in each of these areas. It's very important, a lot of people come to these conferences and you sit in the back room and you listen to the talk and you think, oh, there's fucking cool technology. And you go home and you use it and you're not using it the right way. And it's not that you're doing it wrong, you're not programming it wrong. You're just not using it when you should be using it. So it's very important to understand when you should be using these technologies and why you should be using them, not just because they're cool technologies. And that's really what I want you to pull away from today. So I know you're probably saying to yourself, the how is why we're here. We want to figure out how to do these things. So what's the point if we're not going to talk about the how? And again, it's really about the when and the why. Very important, we will look at code, we will talk about the code to help illustrate the points about each of these technologies. But to go into them into detail that they need to be covered in, it's too much for 45 minutes. And I've seen too many talks where it's just code after code after code and flying through it and you don't really get anything out of it. So take this away, again, check out the book perhaps. A lot of great in depth coverage there to get into the how of all of this stuff. Okay, this by the way is image number two for those of you keeping track of the free books going. We're going to talk about remote methodification now. Some of its aliases, as you might know by, are PRP, DeRuby, Distributed Ruby. Those are a few of the ones here. So what is it? Well, the name kind of gives it away. It gives you the ability to call methods remotely. Talk the server, client, talk together, call methods, serialize data back and forth. Pretty cool stuff as you'll see. It's in the standard library so no additional packages needed. Very quick to get set up, you can get set up right now. You're fast enough you can actually copy down the code that I'm gonna put up in these slides and actually run these examples locally. They're that easy to get up and running. So it's also very, very easy, very, very powerful to use. As you'll see, it's really cool stuff very quickly. It is also extremely dangerous. And we will talk about security here in a little bit. I'm not gonna go into great detail but I will show you a very quick way to actually wipe out an entire server's file system in three lines of code using this. So there's definitely an edge of danger here and you wanna heed that as you go forward. So why do we wanna use DRB in our applications? Well, it's very fast and easy to get set up and running. Built into the standard library, no problems there. Almost zero configuration, you'll see this in a minute. I'm using straight DRB, almost zero configuration. That's not including SSL and that sort of stuff but just to get up and running in your Dev environment pretty quick. No need for any of the standard web services stuff you use. No need for controllers with JSON or Apache or nginx configurations. You fire up two standard Ruby files you're often running. And most importantly, when would you use? You use a closed environment. You don't wanna offer this up to the public. You don't wanna say, hey everybody, my DRB port is port 8080, feel free and start calling remote methods on it. You don't wanna do that. You wanna do it internally behind firewalls in a closed environment. As you'll see when I show you that security example, why that is important. Okay, so here's a quick little graph that kinda shows what a typical round trip would look like when using remote methodification. We have a client here, right? The client sends a message to the low world server saying, say hello for me. The low world server says I got that message. I know what to do with it. I'm gonna respond with a string below world. It serializes up that string, sends it back down to the client. The client then deserializes that string and does whatever it wants with it. And then serialization is very, very important to what we're gonna talk about and it's how this whole thing works. Everything is serialized between these two and the deserialization, as you'll see a little later, is also incredibly important. So let's look at a quick little demo here. I need to build essential authentication system. I've got 10 to 20 apps. They're all these big apps and every one of them has usernames and passwords. What I really need is a system where I can send the username and password to it. It's going to check it against the database or whatever way it checks to verify those username and passwords correct. Then it's gonna send back a hash of information about that user to me and then I can do with it as I want. And if not, I wanna send some sort of message that says it's been a security breach. So let's look at how we would do this using DRV. Obviously, we could do it with cast systems that are already built but this is kind of a little more fun. So we have a user service class here. It's a standard Ruby library. We're sending Ruby class, nothing special about it at all. On this class, we have an authenticated method. It takes a username, it takes a password. We run it against this incredibly complex algorithm for determining whether they're valid username and password. If they are, we send back some details about this person as a hash and if not, we return a string that says it's been a security breach. What's really interesting about all of this is down the bottom of this file here and unfortunately people in the back probably have a hard time seeing it, there's one line that says start service. We give it a user, we give it an IP address and a port and a copy of the instance of the user service class. So we're gonna buy the instance of this user service class to that IP address and port so that other people can then call off the authenticated method on it. That's all we've done. We're gonna run this in a little file and instead of sit there and wait for people to send messages. So our customer service app that we're gonna build for this, very simple, the customer service reps just need to know the username and password so they can retrieve your details and say oh yeah, your email address is marked at markface.com. So we're gonna connect to that IP address and port that we specified earlier. Get a copy of the user service back and then our loop here, we're just gonna keep asking for username and passwords. Then we're gonna call the authenticated method on it on that user service we got and pass it to username and password. That's honestly that's all you need to do to actually get DRP running. So here it looks actually running. In the top panel we have our user service in the bottom panel we have our client. So I'm gonna type my username here markbakes and my password of bad passwords, see what happens. And I have a security breach which is exactly what I want to happen. I type my correct username and password this time I get the hashed on the bottom here. The string of the hash, the inspect of the string. The inspect of the hash or other. Print out the bugs, my username, my email, my created app date. Everything works great. Through to the top here you can see that I'm printing out that I'm trying to authenticate the user. That was it. There was no third server I had to run. No third file I had to kind of implement. No Apache settings I had to do. I now can, I can now use this hash in my client to do whatever I want. I get the email off it, print to the screen, send them an email, change their email, do whatever I want with this hash. And it's there. So what happened was I asked the remote server, authenticate this here is using a password. I serialize that, send it to the service. The service then found the hash, serialized the hash, send it back down to the client. The client deserialized it, created the hash object and then gave it to me. Now I want to get a little more fanciful with it. We want to actually get, return a user object because that's what all of our apps use. They're going to use a user object. So we're going to change our service here just slightly. Instead of a hash, we're just going to return a user object instead. That's it, let's see how it's changing our service. Here's what our user object looks like. Pretty straightforward, pretty standard Ruby code. And particularly to note down here is our 2S where we're going to print out a nice active record S type of two string on this. Okay, so if we're going to run our fancy user service again this time. And we're going to type in Mark Bates and our password and password. Let's see what happens. Okay, well we got a response. We didn't get a security breach but we didn't get a DRB unknown object. And the question is why do we get a DRB unknown object? And the last example, we got the hash and the hash worked just fine. We should be seeing the 2S of our user object. And the answer is very simple. When we serialized up the hash, deserialized it on the other end, Ruby knew what a hash was. It had the class definition of hash. It does not have a class definition for user. So how do we solve that? Well, we can solve it by creating a class definition of user in both environments. But then it's not very dry. Or we can maybe put it in the gem and bundle up the gem and install the gem in both places. Not the greatest solution either. But Ruby has our backs here. So what we can do is we take our user class and we can have one line, the DRB on dump line. And for those of you who've ever seen this line before and wondered exactly what it does. When I was writing the book, I was really diving into the source code of this and looked up this module to find out what this module does. And all this module does includes one method into your class, the underscore dump method. And all that method does is raise an exception. That's it, that is all that module does includes a method to raise an exception. So what happens is when Ruby tries to deserialize this class up and send it down the wire, it gets the exception and says, okay, I don't know what to do with this. So what I'm gonna actually do is send down our reference to this object. I'm not gonna send down the actual object itself. So it's a little like this. So our client sends a message to user search saying find the user with the IDM1. User service says, okay, I'm gonna send it back to you, but I can't. So instead of actually sending that, this is DRB object that contains a reference and the reference ID of the actual user object which is now stored in the user server as opposed to sending that down to the client. The client then uses that reference as if it was local and says, okay, well, I need to print out the user name to the screen now. So it calls dot username on this potential user object. And what that does is actually sends a message back up to the server saying, here's my reference ID, here's the message I want. Do something with it on this side and then it serializes up and sends down a response on the other side. So let's actually see this work, this time over with our DRB and I've done it. And hopefully it will work, it's pre-recorded. So user name is in my face. You never know what demo is, you really don't. Okay, great, so we got what we expected, we see the nice, beautiful two-story down at the bottom. So it did work, which is fantastic. Now there's two things wrong with what I just showed you. And most importantly, I'll actually start by saying, don't do this, don't ever do this. And the reason is like I said, there's two things wrong with it. The first thing is it's incredibly slow. Every time I make a method call on this object, it serializes up that method call and any parameters you send to it back up to server. The server is now responsible for chugging on it and doing all the processing, not the client. Once it gets that information, then serializes it, sends it back down the pipe and then has to be deserialized again. This is every single time you make a method call. With the hash, we had a local copy, here we don't. So you're passing on these user objects and now everybody's just talking and overloading that main service, not a good idea. The second reason is security. We'll show you that in just one second. But one last thing I kind of want to point out here is you can see we're printing out, asking for email up here at the top service, which was in my user class. And I wanted to point that out just so you saw that it was actually executing on the server side, not the client side, when we called self.email and the inspect. Okay, so this is slide number three of images. So this is the danger we were talking about. Right? When DRB encounters a method on a class that it doesn't know of the client, it sends that message up to the server and says, hey, can you do me a favor and execute this method on your side because I'm hoping you have a copy of it. And the server says, yeah, I'll try my best and I'll do my best for you. So what happens if you take an undeath method that is on, say, every object in Ruby? You undeath it on the client side. What's gonna happen is it's gonna make that method call on the remote side. So if we have a class, a little Ruby file that looks like this, and we get our DRB object, and we undeath instance of AL on that, and then we call instance of AL with rm-rf star. Do not run this code, by the way. Anybody ever run this code? Please, I really can't stress that enough. I've never actually run this code. I'm taking on faith this code works. So please do not do it. So we undeath instance of AL, then we call it on the remote object. It doesn't know what to do with it, so it calls back up to the server and it says, oh, I know what instance of AL is. I'm gonna run it. And then it runs the system command and destroys our entire file server. Not a good idea. Please do not run this. Ruby does have our back. There are flags you can set that prevent the entire family of the vows from running. And I highly recommend you set them on every single DRB file you ever write. Regardless, you only need to set it once, but set it on every single file because you can never have it set enough time. That's what I can really say about that. Okay, so that's kind of DRB in our mind in a quick little nutshell here. There's a lot more you can do with it and I really recommend going deep dive into it. When you set up a radio servers using Rinda, now all of a sudden you can have automatic service discovery so we can get rid of these nasty IP addresses in ports and actually start looking for objects and checking objects out of the service and putting them back in again. A lot of really fun stuff you can do with this. But again, there's always that caution and I can't caution you enough just how unsecured this can be. Every instance, every object you share and every object you send out a pipe, any method on it is now available. This is not a web service where you say, here are my half dozen end points. You can only call these and then I am going to control them and I'm not going to call a valve on any user programs you set through or at least you shouldn't. I'm sure somebody out there is and is in for a surprise one day. You're not in control of that anymore. I could call any method and I could undef any method. This is Ruby. You can do whatever the hell we want with it, remember? So really be very, very careful with it. But it is a lot of fun when you use a really cool shit with it. And again, the book goes into a lot of examples and a lot of cool stuff you can do. Definitely check it out. Okay, so who are the big players in this area? Well, DRB and Render are really the biggest players. There have been a few others that have tried to kind of fall off the line. There is a little gem I wrote called Distribunaut which wraps a lot of this kind of stuff and does a lot of cool magic where it automatically looks up objects and namespace and finds them in services and locks them and checks them out and it's also cool stuff. But if you got to use it, I would definitely recommend checking that out. Okay, so message views. This is slide number four, I think. No, I was just about to say his name in the speed of delivery. But I'm not going to. So what are message views? Well, they're also known as asynchronous message views although there are synchronous message views but they're not the norm. They're active message queuing protocols, big protocol for them or AMQP. So what are they? Well, I think the name kind of gives it away. They're a stack of messages to queue. You put messages into it, messages come out on the other end. They typically run on a FIFO system. So first in, first out. You put message at the top, the bottom message comes up. Obviously when it's processed. Fire and forget. There is typically no interaction between the person putting the message on there, the queue and then the queue sending back some sort of message to that person. You typically put the message on the queue and then that's it, let the queue do what it does and maybe it puts another message on another queue that you pull off later. But that's not really that kind of communication. Very fire and forget. So when and why do we use a message queue? Right, well they're extremely fast and efficient. Incredibly fast. Most of them, like RabbitMQ is written in Erlang. So it's incredibly fast. Reliable delivery. They all offer reliable delivery. You can turn off persistence on the wall. But if you turn off persistence, they're incredibly even more fast. But you kind of lose reliable delivery. They are immediate. So typically the message itself is not actually stored. You send an object to the database, you say here's my object, store it, I wait for a response from the database saying it's been stored. With message queues you say here's the message and then you leave. You don't get a response saying it's been stored. It doesn't say hold on a second, let me just make sure that it's a good message. You just take the message and you walk away from it. So very immediate in terms of that. And typically there are several workers or processes per queue that are constantly kind of being pushed all these messages and they're constantly working. And this is an important difference when we look at background jobs in a little bit. It's one of the things that differentiates background jobs and message queues. Here you would say you have three or four, 10 processors all working on one queue. And they're also very good for inter-process communication. So you have several apps. You want to put a message on the app. AppA puts a message on there for AppB. AppB comes along, picks it up and says oh thank you. This person's registered so I set up an email now. We'll do something fun like that. Okay, so I'm gonna do a little demo. For this demo I use Starling, which is a Ruby-based messaging queue. It was originally written by this little startup in California called Twitter. They use it for a while. I don't think they're using it anymore. It's a fun little message queue. I wouldn't really recommend it for high-skill production because it is all Ruby-based. But it's very easy to get up and running, get started with just why use these demos. Typically I use RabbitMQ, which is fantastic, but it's a lot more work to kind of get up and running and to kind of do some of the examples. So I thought simplicity was best for practical demonstration. So let's see what we got here. So we're gonna connect to Starling server. Now all we have to do is run the Starling command line to the Starling server. We'll show you that in a second on a particular address and port. And then we're gonna ask you for something silly to say. And then that message, whatever you say, is gonna get shoved onto the login queue with a timestamp. So basically a little logging system here. So imagine we have our application every time a user comes to our app. We're gonna send the request information just popping onto this queue. And then we're gonna process it in our logger. Later we're gonna do analytics. We're gonna do all sorts of heuristics on it right then and there instead of just running into a file. So here's our login queue. It's very simple. We're just going to loop through the queue. We're gonna flush the queue. So every time they're messes in the queue, they're gonna get popped up into this block. And we're gonna print it to the screen. That's it. That's all we need to do to get Starling kind of up and running and going and connecting to our login server, a login queue. So that's a good example. At the top here, we're gonna start Starling. Very simple. Get rid of that. We're gonna start our log worker. Just gonna sit there and wait for stuff to go in. Now it's gonna ask me a question. Something to say. Yes, I'm gonna say hello, we're in the conf. And then boom, it shows up right there at the top. The time stamp we put it in, which obviously was a few days ago, Wednesday. And then another one down the bottom. So that's it. So don't imagine the power of that to do some really cool stuff. In this case, I'm using logging as an example. Logging is a great example of message queuing. Cause we're always wanting to do stuff for our logs. Why not pop them on there very fast? Probably faster than writing to a disk. When you've got all those files, oh and everybody trying to write to that same file, putting stuff on a queue can be very, very fast. And then let it do processing and formatting and whatever you want to write to a database or create a pie chart on the other end. All right, so show you a little more interesting example. We're gonna create a number queue. And from this number queue, we're gonna send it spaced limited numbers. We're gonna add those numbers up in our queue. Pretty simple, pretty straightforward. If you don't know what this Ruby code is, I can't help it. So here we go, start client, what numbers you wanna add and it's gonna set them on the queue. Here we go, we're on our number adder. Do some numbers. Two and two is four. One, two, three, four, five, six, seven, eight, nine. Should equal 45. Here we go. So here we're doing something a little more interesting. We're giving it kind of a message. We're saying, do this thing, add these numbers together. So you can do some fun stuff with it. You can't really send complex objects, but that's probably a good thing. You know, I probably could have created some sort of number adder class and sent that along with all my attributes. I can also send an array. It can deal with basic stuff. You really wanna try to stick with your very basic class types when dealing with messages. One, because if you create these large objects, you're really starting to feed the point of that immediacy. Just take what you have and send it as quick as you possibly can. Don't create large complex objects. Let the processor do that. Just send over the string. I've sent over a string of numbers that I grabbed. I didn't split them. I didn't send over the read. I didn't spend the time on my inside to do it. Use that immediacy to your advantage. Let the processor do that work for you. That's what it's there for. Okay, so who are some of the big players in this area? When we talk about RabbitMQ, which I said is my favorite one I use all the time. I just love it. Along with the AMQ PGM. Fantastic. Starling, very good for just getting started. One player to stuff today. Starling is really easy to do and get out of the planning very quickly. An active MQ. I don't really have any experience with it, but I've heard lots of really good things. So, check it out. That's your thing. Last image that your tweet's in. Background jobs. By the way, if you guys don't know who this is, I mean seriously, there's something like, I got the first four. I am totally lost with these two. Honestly. Just, you'd be surprised, I guess. It's actually my son's favorite show, which is The Rage, which is really sad. Anyway, so no one else says on background jobs here. Offline tasks, background workers, you probably heard them refer to as that. Just play a show of hands. How many people actually use background jobs in their apps today? Okay, so a decent amount of people. That's good. You should be. So what are they? Well, they're dedicated workers to a dedicated task. I apologize you put that on there. And that's a big differentiator from a message queue. Message queue, bunch of processes just sucking on information and doing something with them. Here you're actually bundling up information about a very specific task, and that could be a whole bunch of information from instance variables to message IDs, user of object IDs, to whatever it is you need to do, large images, whatever you're bundling up and saying, here process this when I tell you to process it in the background. It typically use priority and run at base queues. So instead of first in, first out, you wanna tell it, well, this worker, this job is more important than this other job. The example I always like to use is every site, most websites will send you a welcome email when you register with your confirmation like you click on it and confirm your email address. And a lot of sites will also send an email when say you have a to-do that's overdue, right? Well, what is more important? Welcome email or to use overdue? And obviously the to-do is overdue, is the most important of those two emails. Okay, obviously it's a welcome email. So you wanna shove that in there as urgent and run at now, and the other one you might wanna say, you know what, kind of low priority if it gets delivered at midnight tonight, like I wanted to, that's great, it gets delivered at 12 o'clock, five, I don't really care, if it's something more important, they're also fired and forget, but without the kind of immediacy. The reason they're using, without the immediacy is you see in a minute, they're typically database backed or stored by some sort of service somewhere, typically a database. And you have to kind of build the object before you save it, before you give it off to the queue. Okay, so when and why we use this? Well, here's a few good examples, right? You wanna process large files. That person uploads that 10 megabyte avatar, that's not gonna be, you need to cut up some thumbnails for it. You don't wanna do that while a person's waiting, right? You want that to happen offline, and then show them a little shadow avatar while your wall is processing, and then start populating your avatar whenever they go around the site. Offload long running tasks. One of my favorite examples of this is every site typically has the invitation system. You paste it on your friends' email addresses, type them a little message, and then you spam them with links to the site to get them to join, because you get that referral bonus or something, right? So that works great, you know, you put in 10 email addresses, find, it's gonna send out 10 emails. Not the best, I mean, what happens if the S&P servers down, or is some other reason why a site's getting slow? What happens if you type in 1,000 email addresses or 10,000 email addresses, right? You're gonna sit there and let the person wait, then click the submit button, and then it's gonna sit there and send out 10,000 email addresses, emails? That sounds like an awful user experience to me. Why not take it, shove it in the job, and let that job then sit there and spend its cycles, sending out 10,000 email addresses, and retrying again if one of those addresses fails or the S&P connection is down, and just return back to the user and say, thank you very much, we sent your emails. They'll be done the way they are. All right, I said they're typically database backed, so they're very easy to usually get started with, you usually have a database in your application. And again, if priority is important to you, then this is definitely a win over a message to you. And also great for recurring tasks. I know you can say, well, we've got cron, we can just run a cron job every night of midnight to do my to-dos or every hour to calculate up how much the person has used in bandwidth or every five minutes to pull down that hashtag from Twitter. Yeah, you could write cron jobs to do all of those things, but what if you don't have cron? Or you're on Broca, right? Broca, you have to pay for an hour of the cron, and I'm gonna get that five minute Twitter cron going. Recurring tasks, very good. They also will sell me, you can set them to self and read to you. They'll retry for you if there's a problem. It's a great solution to that cron problem. And writing cron jobs kind of sucks, not gonna lie. So let's do a little demo. I scratched my head trying to come up with a demo to show you that demonstrates long running background processes and stuff like that. So what I came up with was a little news reader app. It's gonna be a killer news reader app. Everybody's gonna go flop to it by the end of this tutorial. And what I'm gonna do is I'm gonna create a, add a URL, that's one go, and aggregate all the articles from that news feed. Pretty straightforward. I have this active record class here. It has many articles. And then on my app to save, so every time I save this, I'm gonna go and get my articles here. I'm gonna aggregate them using this aggregate method. So I connect to the URL that I have, I parse the feed, I then loop through the entries of that feed, and then create my article objects, and then my server gets a little sleepy after it does that, so it takes a little second break, and then does the next one. So if you have 20 and 30 articles, it's gonna take 20 and 30 seconds minimum to run. That's not including the length of time it takes here. It actually connects to the site. Length of time it takes here, it pars the body, loop through them to the database creation here, and all that fun stuff. So let's actually see what happens when we try to do it from a user's perspective. I create my Netflix news feed here. I'm gonna get the new releases. I click create feed. You can see down here it's processing. And we're still processing. And this is just a great user experience. And this is wonderful. What happens if that news feed timed up? Then my session, my browser's gonna time out, or if it takes longer than 30 seconds or a minute, my browser's gonna time out, and then everybody's gonna freak out, they're gonna have to create, so you get that error message, and I can create it again, and maybe it's gonna do it again. It's just not very good. If we look down here at our log, we can see that it took a total of just over 25 seconds for that to actually return back. Now granted, they have, here are their movies and their articles rather up here in the top. I believe that's Jonah Hex, so obviously a good one. But it's there now, but now every time they say this, this is what's gonna happen. It's gonna take the 25 seconds to get a response back to the server, and that is just an awful user experience. So how do we use a background job? Now this could obviously have been an image or anything. So how do we make this work? Well let's change our feed class around just a little bit here. Instead of the aggregate method, we're gonna call this inque worker class. All that's going to do is call this aggregate worker class and inque it and give it the idea of the feed. That's it, that's all it's doing. So now when I click save, it's going to inque worker and complete. It's gonna come back to user. Nothing else changed, but here's my aggregate worker. We're using a delayed job under the covers, which is a great system for doing background jobs. And using a gem I work called DJ remixes that gives you unique workers, re-enqueable workers, hot-toed supports, hooks, really kind of cool, fun stuff we're using.