 It's really an experience for the groupie. I'm Jesse Stover, and a lot of people tell me they don't recognize me in this picture. This is my third picture, just even with the person in the picture, that's me. I work for Shopify, I work for Canada, and I also have published these e-books kind of on point and top of the day. So, what are you going to be talking about? You're going to be talking about technical talk, you're going to be talking about school, you're going to see stuff you can do with groupie. It's often thought of as the debate of, like, C programming, which can do so many things, that we can come through a groupie. And you're going to be talking about these guys before, right? So a few years ago, 2009, I was a Rails programmer. I had a group on Rails, a groupie, and I came across this blog, one of my friends, about Unicorn, a web server, a groupie that does a lot of unique stuff. It was a good time for me because I was ready to learn something new. A lot of Rails, maybe nothing about slow-level stuff, didn't have a full science education. So Ryan suggested checking out Unicorn, linked to the main source file, which looks like this. I opened the documents kind of like, well, what did I do with this? It's really long, a lot of documentation, a lot of methods. I read through it and I realized there was a lot of methods I'd never seen before, a lot of stuff I didn't understand. The more I read, the more I started to press. Because even though it wasn't like the normal Ruby code I'd seen, there was a certain confidence, there was a lot of elegance, this guy was using stuff that was also powerful, but that I hadn't ever seen. So if you're looking at Ruby documentation, looking at manual pages, thinking of some books I started to figure out, what Unicorn does, how it works, I want to share some of that with you today. So when did you talk about Unicorn? If you've ever heard of it, it's a web server for your app guys. Can I see hands, if you've ever deployed an app on Unicorn? Cool, lots of you. Now hands, if you've ever opened the source code. There are a few. Any you can produce? Okay, I don't see any. Okay, one. So I'm not a contributor, I have studied and learned from the source code, but I can't take credit from the project. The rule of the website, Unicorn is, our app needs to be server for fast clients and units. But today, we're going to arm most of it, and just so I can teach you some stuff, talking about the way managers use processes, but it's a little bit special. The first feature I want to talk about is called pre-working. Before we go there, we should talk about forking, and make sure we understand what that is. So when I say forking, I'm talking about the work system call. This is the manual page for it. You can do, for example, C code in the description. I'll zoom in on the most important part. Work creates a new process. So let's do it in Ruby. The bottom after the screen, I'm just showing you the list of IRB processes. There's very little you saw in the list. Just imagine the process ID's. Then I'll call work, and you say, get back at the process ID. There's two IRBs in the list. And the good thing you'll notice is that I kind of have two return values there. There's one right after the fork, which is the number, and there's a nil that I got highlighted. As far as I know, fork is the only Ruby that's in that returns place. I don't need to returns two values in an array. I need to returns two times. The reason is the fork creates a new process. All the processes now, they both return. They're both writing the standard out. That's why you said it's new return values. So because it returns twice, you're creating things like a challenger understanding of these nodes. So say it fork. One thing else that they think, and if I run this, it looks like it's true and false. I always have to think this stuff is dangerous, but it's just a matter of understanding what's really happening. So let's do it again. It's not getting an issue. It's the true part and the false part. I'll print process IDs as well. You lose your difference. So let's do it again. And as you can probably guess, you're going to see the same output, but both of those blocks have different process IDs. So it's kind of interesting, even though they're the same source file, they're writing this on each other in code, they're running it set of processes on the system. So I'm going to revise this statement. Four doesn't really return twice. Returns once, but in two processes. So where it was one, there are not two going to be parallel. And the if construct is kind of weird. It's usually how you do it in a scene. The first thing we do is we have more alligates, solutions, more alligates to DSLs. So the way you normally explore is to get a block. We get, we're evaluating the child process, the newly created process, you have to be able to touch it. And the last thing I want to show you about work before we continue is how it shares everything. So we create an array one-to-five, the full process, and that process is one-to-five in the array. And if you look at what I got, in the color process, it's going to see what the effect is on the array. It's a normal Ruby scoping, something happens out of block to an instance variable and it would affect the same variable in that instance. But in this case, you run it, you see that the two have different arrays. This is because when you work with processing, there's something you're asking the operating system to do. So when you work with processing copies, anything in memory, copies, all the scripters, the copies would be virtually free. There are two movies where there's one and since the processes can't access each other's memory, they've got separate copies. So let's add one more thing here before it creates an exact copy. We'll talk later about how it can optimize this, but from your perspective, it's an exact copy. This is the code we just saw. And you may think, oh, maybe this is a trick of instance variables or something, this is kind of weird. If you change this to globals, it's the same effect because effectively the code inside the block is running in a totally separate process, cannot be affected by the other code. Process weight there is just so that the original process waits for the child to finish its work to make sure that we've deleted something from the array. So we learned about forking. It creates a new process. The process is a copy. We can use that to understand what pre-forking is all about. And pre-forking is really cool. It's really cool because it leverages what the operating system has to offer. It pushes as much as it can onto the operating system. And it's not a new idea. The fork man page that I showed earlier, if you scroll to the bottom on my Mac, the date is 1993. And I mean the fork API has really been unchanged for 30, 40 years. It's a very old idea. Apache in some configurations can do it, same with Passenger, Nginx, and Unicorn. Unicorn didn't invent this stuff. I'm just going to give you a quick, like a pseudo code example of how Unicorn boots up. So if you were to open the Unicorn gem, go to the HTTP server class. These are two methods you would see among lots of others. The start method, which gets called, that's basically the entry point. So what does it do? It opens a listener socket and loads your rack app or your Rails app. Then however many workers you want, it forks them and they enter a loop. Now we've got two concurrent processes running in this loop, right? Two workers. Each of them gets connections off the socket. Process clients calls the rack app. I'm going to explain this in a bit more detail. I want to show you a diagram. So this is the master process. The socket and the rack app in memory. It forks. Now it's got two workers ready to handle your app. Two parallel processes. Now, you might look at this and say, okay, what have we really got here? Two processes ready to handle the app. Why didn't we just, like with Mongo or some other server, you could just boot up two at the command line? What's the real gain here if we're having to copy everything over? But thanks to fork and thanks to something called copyunright, fork can actually be faster and use less memory than booting these things up by hand. And I mean, two processes, not exciting. At Shopify we have 70, like 70, parallel processes per machine. So the savings is really magnified. So copyunright, what is that? Well Ruby has a funny history with copyunright, but it's a kernel feature such that when you fork a process, it doesn't actually copy all the memory. As long as it can, it lets them both reference the same part of memory. And if one of them decides to change that, then it gets its own copy. But think about how much of your stuff, how much stuff in your rack app never changes. When you load Rails, when you load Gems, when you load your models business logic, you don't modify that stuff at runtime. So the parent process should be able to load that, fork the workers, they can share that memory. You never need to be copied. So you could save a ton of memory that way. But if you're using MRI 1.8 or 1.9, which probably a lot of us are, it's not copyunright friendly. So it actually kind of like disables this feature. And it's thanks to the garbage collector. The garbage collector you may have heard uses a mark and sweep strategy. What that means is it looks at all the objects in memory and marks bits on them saying which ones need to be swept. So it's effectively writing all over the place. So when you fork a process, copyunright saves you the memory. As soon as you're in the garbage collector, all the savings are effectively nailed. So if you're using Unicorn, if you're doing a lot of forking and you want to make best use of resources, MRI 2.0 has solved this. Ruby Enterprise Edition, that's one reason why it exists in the first place. The bottom one is a version of MRI with some 2.0 stuff back ported like the copyunright friendly garbage collector. And the way that these guys get around it, they still use mark and sweep, but instead of writing all over the place, they store all the bits in one little part of memory so you can end up preserving all that shared memory. So back to this example. Quick pre-forking was awesome. Why is it awesome? Well, because if you want to spin up a bunch of parallel workers for your app, you can save a ton of memory and do it a lot quicker than spawning processes yourself. Like, when I say yourself, I mean, like, trying to do it 70 times from the command line, everyone's competing for resources. In this way, the master loads the app once. The fork is pretty quick. All the workers can share the same piece of memory. And theoretically, you should get tons of savings. That's one reason why it's awesome. The other reason is I was talking about memory, but I also mentioned that in a fork, open files are shared, sockets are shared. So the master opens a listener socket. It doesn't take any connections. And each of the workers, as part of its loop, they accept connections. And so this is the accept system call that they use. And the kernel does the job of making sure that each connection only goes to one worker. Things are load balanced nicely. Things are cued fairly. You don't need any kind of a proxy in front of Unicorn to talk to each worker. It pushes all that down to the kernel, and the kernel handles it nicely. So yeah, pre-forking is awesome, because you can get faster boot time. You can save more memory, get great connection load balancing features from the kernel. Let's do another Unicorn feature. It can replace an instance of itself without losing any connections. This is like a zero downtime restart. The most obvious example of when you want to do this is if you want to, you know, deploy a new version of your app. You write some new code, you're ready to put a new app online. So this gets triggered by you send a... Like if you wanted to deploy new code, you would put the code in place. You would send a user to a signal to the master, which would tell it... Here, we'll look at the code. Send the user to signal which would trigger this method. Again, this is pseudo code, but if you look inside the Unicorn HTTP server class, you'll see a method called reexec. And you see the two components here are fork and exec. We haven't touched on exec yet. It's another system call. Stitched from the man page, exec transforms the calling process into a new process. Let's do it in Ruby. I print high, exec a command, print by. So when I run this, I'm going to see the LS output scroll by. If I go back, I can see it says high, but it doesn't say by. The reason is because what that exec did, it printed high, then it called exec, which effectively blew away the Ruby process. Replaced it with LS. So LS did its thing and it exited, and that was it. Ruby is effectively gone in that case. Replaced. So how does Unicorn replace itself? Well, what's happening here is fork's a new process off the master. It grabs the arguments it was called with. This is a list of file descriptor numbers that it's listening sockets are listening on. Then it calls exec. The listener socket file descriptor numbers is important because just like you can share a socket through fork, you can share a socket through exec. This is at the point with the most processes. This is what you'll see. The old master has forked on the left a new master. That new master has called exec on itself with Unicorn. So we're going to load up a new process, load up the rack app as before. This time, instead of opening a listener socket, it's just going to reopen the one it got passed from the old master. So from the outside world, the socket never closes. On the inside world, there's only one socket too. If you ask the kernel, there's only one socket. In this diagram, there are six handles on that socket. It can all work with it, but there's still just one. So when the new master and its workers are ready, you kill the old one and you end up with the same kind of process tree as before. In this way, when you deploy new stuff, the socket gets passed down the tree. The tree grows and shrinks, but there's never any effect on the inside world. So this is the URL for Unicorn. You should definitely check it out. Besides just what I talked about, there's lots of other cool features. I talked about pre-forking and how it does restarts. If you want to look around at it and see what other kind of stuff it does, the self-pipe trick is pretty cool. Really, there's lots of cool stuff in there. So what did we actually talk about? We talked about some Unicorn features, but the concepts really built around fork and exec. So if you only take away one thing today, let it be this statement, fork and exec. Because fork and exec, fork copy the current process, exec transform it. It's at the heart of process spawning in Unix everywhere. All these methods in Ruby, process, spawn, system, back takes P open. If you look at the source, they all do fork and exec. So there's no magic to them. That's how they do it. This is process spawn from Rabinius. Among other things, it does fork and exec. It just does one example. And fork and exec, it isn't just for Ruby. It's for lots of programming languages. And even for your shell. When you do in your shell, curl Google and send the output to a file. If you're implementing that shell in Ruby, you might do something like this. You fork the shell. You set up the environment that the command needs. In this case, you redirect standard out to a file. You might also set an environment variable. You might point standard out to a pipe instead. And then you exec the command. So the shell waits. Makes you wait until the command is done to do the next one. So fork and exec is very... It's really at the heart of Unix. So I said I would talk about these guys, too. The guy on the left is Ken. The guy on the right is Dennis. Ken is the original author of Unix. Dennis is the original author of C about 40 years ago. About 40 years ago. These guys obviously collaborated a lot. They really have credit for the original Unix Beards. Now, Unix Beards are kind of a funny thing. Sometimes, like in the case of these guys, you might think it's wisdom, respect. Sometimes you see a beard legged out and think that the guy spends too much time in his parent's basement. I never really gave it too much thought until I read this a few months ago. The Unix beard is really an extension of the philosophers beard and the academics beard. And I thought that was so on point to how I see them. When I read stuff that they wrote years ago, they weren't looking to change the world or to make a lot of money. They were doing research. They were looking to find truths about programming, about operating systems, about language design. So I have a ton of respect for these guys. But not everyone has to grow a beard. The guy on the right is Linus Torvald, obviously a Unix hacker and no beard. So I certainly wouldn't want to encourage everyone to grow a beard. If you feel like doing it, that's great. But you can have a figurative one, too. So I'm going to leave off with a quick story. I've got a three-year-old daughter. We like to read this book. It's a Dr. Seuss book. It's about a young guy. He's learning the alphabet A to Z. He learns all the letters. A is for apple, B is for bear, Z is for zebra. An older kid comes along and says, oh, you don't know much. Let me show you something else. It shows him these crazy Dr. Seuss letters and Dr. Seuss words. But in the end, the younger kid discovers some stuff the older guy didn't know. So my challenge to you is if you've learned something new today or if you've seen something you haven't seen before, if you spend most of your days doing web programming, try and find out something you haven't found before, look in Unicorn. It's a good place to start and help me learn some more stuff. I don't know. So I'm a bit early, I think, but that's the end. I'm interested in my books. You can get a discount code there. I've got a few copies to give away if anybody wants one. I think I've got time for questions. Thanks. So in Unicorn, do you have any comment on the term handling? Do you have to do it in the comments? So Unicorn swaps the signals the same way engineX does? Yep. Do you have any insight on why that is in Unicorn? To be honest, the only thing we've ever seen about it is that there was a commit saying, let's do it like engineX does. So I guess you'd have to ask someone who knows more about engineX. I saw another hand at the back. The question was, why is it necessary to fork a process in order to exact a different one? Are you asking why there isn't something that puts those two together? So the question was, fork seems to do two things, creates a new process and inherits the environment. What can you do if you don't need the environment? So I'll have a two-part answer. The first is, there is something called POSIX Spawn, which gets you what you want. You POSIX Spawn give it a command, and it creates a new process that doesn't inherit the environment of the old process, which makes it much quicker and spawns a new process. It's something that is not supported everywhere. It's not supported in core Ruby. There's a gem for it, POSIX Spawn, I think it's called. And the other reason that I'd say is, I think a lot of this stuff, like I said, fork is a very old concept. And if you look at a lot of system calls, when I look at them, I see people who were, like, basically designing the system they needed to write shells and to write programs they needed a long time ago. And that stuff is still useful today, so it still sticks around, and it's the most widely available. But it is written for times past. Yes? You're running 70 processes. Do you all use RE, or do you just have, like, 11 billion RAM? Both. We used to use RE. We switched to 193, like, stock 193, and we only saw, like, the new VM saved so much memory that we only saw a little bit of a loss. But we're now using that patched version in production, which has different speed patches, as well as the copy and write patch, and it's been wonderful. Yeah? You mentioned a couple of versions of that, what about the other applications? Actually, is there anybody here who knows if Rubinius is copy and write friendly? I couldn't find out. The JVM doesn't support fork. So in its efforts to be cross-platform compatible, like on Windows and stuff, Windows doesn't support fork, JVM doesn't support fork. Rubinius does, but I don't know if it's copy and write friendly. Okay. I'll take you right for it. Yes, back there. The question was, is the backwards, is the back ported stuff from 2.0 stable? We're using it in production and Shopify. I use it for development every day. It's been stable for me. Yeah, so I was just saying that JVM is heavily encouraged using threads instead of processes. The question was, isn't that a better model for what needs to be done? It really depends on your needs, right? JVM is great with threads because the JVM is great with threads. MRI, for instance, if you're doing threading, thanks to the global VM lock, threads aren't really happening in parallel. Multiple processes, you do get parallelization and you do get shared memory, which is big savings. So I can't say that one is better than the other. It depends on what you're deployed on. It depends on what your needs are. Of course, the best thing to do is to test and use metrics. Yeah. Great point, which is that if you're using fork, you don't have to worry about shared state with threads or locking or synchronization. It's a much simpler approach. Yeah. Favorite system call. Favorite system call. I'd have to say fork just because it's so useful. I mean, it's been said that it may not do what you want every time, but I find it so useful in so many places. I know there's some funny names, but I can't think of any right now. Anybody else? Yeah. Do you find yourself sending signals randomly on the command line now before you learn how to use my system? Do I find myself sending signals randomly on the command line before I knew it? Now that I know about signals, maybe not randomly, but I know how to... I know what kill-9 means now. Who else? Yeah, Adam again. Do you find yourself using any IPC mechanisms? The question was, do you find yourself using any IPC mechanisms besides sockets or pipes, I assume, stuff like shared memory or maybe POSIX cues? I never have in Ruby. I think there might be gems for some of that stuff, but not in Ruby core. So if you want to communicate between processes, you can use pipes or sockets, but if you want to do memory mapping or shared memory, I'm not sure how to do it in Ruby. I never have. Yeah? Yeah. How does the kernel determine which one gets the first number served as a power line? Yeah, so when they all call, except, it's a blocking call, so they're all waiting, and I presume that, you know, the next one next in the queue, but I don't know the kernel source that well. Who else? That's it. Okay, thanks, everyone.