 Again, my name is Thorsten Ball, as you just heard. I'm a software developer from Germany. I program in Ruby, C, Go, JavaScript, and that's on the left, that's my GitHub name on the right, that's my Twitter name. I work for a company called Flink, and the title of this talk is Unicorn Unix Magic Tricks. That's quite mouthful, so let me explain what this means. Unicorn is a web server written in Ruby for rack and rails applications, and when I first encountered Unicorn, when I first used it in production, I thought that it was magic. And that's because Unicorn had all these crazy features. It had this master worker architecture, where you had one master process controlling a lot of worker processes who do all the work while the master process just does some housekeeping. And it also had this crazy feature called hot reload. Now hot reload means that you can spin up a new master process with the new version of your application while the old master process is still up and running. Now you have two master processes running, both serving your application, and as soon as the new one is ready to go, you can kill off the old one and the new one takes over. And that looked like magic to me. Also Unicorn had this crazy amount of signals. It has a file called signals in which it documents all the different signals, like TTYN, TTOU, user one, user two, and the only one I knew was quit. So the rest of that was just black magic. And another cool thing about Unicorn is a feature called preloading. I didn't know what it was. All I knew was that Unicorn somehow preloaded my rails application, which took a couple of seconds to boot up in memory in the master process. And then when you spawn up new workers, it does that in under a second without loading anything. And well, the last drop was Unicorn has a philosophy file. I mean, which project has a philosophy file? But as it turns out, it's all just Unix. I duck into the Unicorn source code. And after learning a bit about Unix, I realized that everything Unicorn does is just using some basic Unix principles. And most of you know Unix. You know the command line. You know the shell. You know pipes. You know crap. You know redirection. You know signals, maybe. But there's this whole other side of Unix. Let's call it the developer side from which you can use Unix to give your applications more power. You can use your operating system to do work for you. And in this talk, we're going to look at Unicorn and how Unicorn uses Unix. Because Unicorn uses a lot of Unix tricks to compose these cool features. And we're going to do that by looking at a few, let's say, basic Unix tricks. And we're going to see how Unicorn uses them. And at the end, we're going to look again at stuff like hot reload and preloading and signal handling. And we're going to see how Unicorn uses Unix. So the first block is fork. Fork in Unix is basically everything. Every process that runs on your Unix system was started by fork. Well, the first one was started by your bootloader. But every process that came after was started by fork. So what is fork? Fork is a system call. As you can see, fork has a tool behind its name. That means you can find the documentation for fork in section two of the Unix manual. That's the man to command here. And a system call, well, a system call means that you can ask the kernel to do something for you. A system call is a direct connection to the kernel. You don't have the direct access because you have to write some CPU registers and stuff like that. But most languages offer some wrappers around system calls. Now, fork splits your process in two. If you have a process running and you call fork to system call, you're asking the kernel, hey, kernel, can you please split my process in two? And when fork comes back, you now have two processes. You have a child process and a parent process. And what's really interesting about that is that the child process inherits a lot from the parent process. The data, that means the code, all the Ruby code that's in memory. The heap, that means the memory layout, all the objects, all the classes, and the stack. So how do we use fork in Ruby? Well, as you can see, you just use fork. You use fork and you pass it a block. You don't have to, but it's in this example we're going to do this. So we're using fork, we pass it a block. Everything in this block is going to run in the child process. And the parent process is waiting for the child to die. That sounds kind of hard, but well, under Unix, if you don't wait for your children to die, they turn into zombies. And that's a fact. So as you can see, we save the return value of fork into childpid. That's a new process ID of the new spawn child process. And in both processes, we print out a bunch of stuff. And if we run this, you can see that the numbers match up. The parent process ID in the child process matches the process ID printed out in the parent process. And as you can see, the childpid is nil in the child process. That's because fork is kind of weird when you use it, because it only returns in the parent process. In the child process, it returns nil. That means you can use fork and check the return value to see if you're in the parent or in the child process. Now if we were to put some sleep statements in this example and run it again and then use a tool like PS or PS3 where we can see our processes, we can see that we now have a parent process, the top one, and a child process. And the process IDs match up. And it's really that easy to create a new process in a unique system. You just use fork. And in Ruby, you just use fork. So the question comes up, how does Unicorn use fork? As I told you before, Unicorn has this crazy master worker architecture. But actually, it's quite simple. This is a little snippet of Unicorn's source code. And you don't have to read all of it, because the main thing is right in the middle. You can see the call to fork. And what Unicorn does is it calls fork. And if there's a return value, that means we're in the parent process. It saves the child process ID so it can keep track of the running processes. And in the child process, it just calls the worker loop. Because the code is shared between the two processes. Now the child also has access to the same Ruby code. Now after this function ran, you have one master process and 16 worker processes. If this loop ran for 16 times. And all the child processes are doing is running the worker loop and then exit. That's pretty easy, right? That's the main thing Unicorn does to start up its master worker architecture. And it's just a call to fork and checking the return value. And now you have 17 processes running. So let's talk about pipes. I think a lot of people here know about pipes. It's one of the most beloved and famous features of Unix systems. And the pipe basically takes the output of one command, like for example in this here, grab, takes the output and uses it as input to WC. Now I think a lot of people think that pipes are just the feature of the shell. You have to use the pipe character, but that's not true. Pipes can actually be used outside of the shell. If you use the pipe system call, you suddenly have a pipe in your process, in your program. Because the pipe is nothing more than two file descriptors. You have a read-end and you have a write-end. And if you ask the kernel nicely, hey, can you give me a pipe? It returns two file descriptors. And you don't need a pipe character or anything. So if you have a pipe in your process, you have two file descriptors. The file descriptors are nothing more than just numbers. So if you look at C code that deals with pipes and with files, they deal with numbers. They write to numbers and they read from numbers. That's because they look up the number in the file entry table in the kernel. And that means pipes have the same API as files. You can read to them and write from them. And the great thing about pipes is, since they are files and files are inherited by child processes, pipes are inherited to child processes. So let's have a look at an example. In Ruby, we use io.pipe to create a pipe which somewhere down below uses the pipe system call and we get a read-end and a write-end. And then we spawn a new process with fork. We close the read-end because we don't need it. And then we use the write-end and put a message in it. Then we close the write-end again. Now in the parent process, we close the write-end, wait for our child to die, and then we read from the read-end. And if we run this, it behaves exactly as expected. And now we have communication between two processes. That's pretty simple. You just use the pipe and fork off, and then you can talk between the processes. A lot of Unix servers do this. They use pipes for IPC, inter-process communication, and Unicorn does this too. And Unicorn uses pipes a lot. If you tell Unicorn to use 16 worker processes, it's going to use 16 pipes. It's going to use a pipe for each worker master relationship so the master can talk to the workers. And it's the same principle as we just used in our example. And Unicorn also uses another pipe, which it uses just in the master process to talk to itself, because writing to and reading from a pipe is blocking, and it's a file and has a great API. We'll get to that later. And Unicorn also uses a pipe in a really clever way if you run Unicorn as a demon. Because if you run a Unix process as a demon, it has to fork two times to detach from the terminal it's controlling. Now what Unicorn does is it opens a pipe, then it forks two times, now it has grandchild. The grandchild tells the grandparent as soon as it's booted up, hey, I'm ready to go, you can quit now. So the grandparent waits in the pipe until a message comes in and then it quits and the grandchild takes over in the background and now we have a demon process in the background. So let's talk about sockets and select. Sockets are, let's say, everything that has to do with networking in Unix. If you want to open a new connection, that's a socket. If you want to listen for a connection, that's a socket. If you want to do UDP connections or UDP messages and TCP connections, that's a socket. And select is a system called that allows us to monitor and handle sockets. So let's have a look at sockets first. As I said, everything in Unix has to do with sockets. The Unix networking API is called a sockets API because if you want to do any networking, you have to use sockets. You have the TCP sockets, UDP sockets, Unix domain sockets, raw sockets, streaming sockets, SCTP sockets and as it turns out, sockets are just like pipes, files. They have the same API again. It's just the number you can read from and write to. Now the system called socket returns a socket, but that's not enough. If you're going to write a Unix server, you have to do a little bit more than that. So this is the basic socket life cycle. A Unix server has to go through when it uses networking. It has to use the socket system call that means, hey, Colonel, give me a new socket. You can specify the family. Is it IPv4, IPv6? You can specify the protocol and so on. After you get the socket, you tell the Colonel again, hey, Colonel, please bind this socket to a port, like 9,000. And after the Colonel got the bind call, you tell the Colonel, please listen on this port and give me any incoming connections. What you're doing after is called accept. Accept means, hey, Colonel, if there's an incoming connection, please give it to me and I'm going to handle it. So accept is a blocking call. And if you think about this, that's a little problem. If you have more than one socket, how are you going to handle them if accept is blocking? If you're accepting on the first socket, you can't call accept on the second one because the first one is still blocking until a connection comes in. And this is where select comes into play. Select is another system call, which is pretty old. And select allows us to monitor file descriptors. You can tell select, hey, please watch these two sockets and tell me when one of them is ready for reading or tell me when one of them is ready for writing. And select two is blocking, but now we solve the problem with accept being blocking by using select because we can now monitor two file descriptors. And if we use this in Ruby, we can build something like this. In this example, we're creating two sockets, both TCP sockets. One is listening on port 9999 and 8888. We use the socket system call, which is somewhere down there below the Ruby API. We bind it to port and then we call listen. We call listen with the parameter call 10 in this case, which is the back lock queue. That means the kernel keeps up to 10 connections before dropping connections if you can't handle them. And after this, we call fork five times and go into our own worker loop. Now in this worker loop, we call io.select on socket one and two. And io.select is going to return as soon as one of the two sockets is ready for reading. And as soon as that happens, you can take the socket, the first one in the readable sockets because io.select returns arrays, three arrays of readable, writable and error-prone sockets and we read from the connection, close it again, and start again. So now we have five processes doing all the work while the parent process is just sitting around waiting for, well, the children to die eventually. If we run this, as you can see on the left, we're running our server. And on the right, we're using a sophisticated TCP client called Netcat, and we sent some complicated messages to our server. So we're going to send two messages on port 9999, two on 8888, and then another message on the first port again. And as you can see on the left, the first number here is the child process ID. And every message we read was read by a new child process because the kernel does the load balancing for us. If you use select, the kernel decides which process gets to have the socket first. So now we built a brief forking TCP server in 23 lines of Ruby. And what we just did is basically what Unicorn does. Unicorn boots up. It has a master process. The master process creates a new socket with socket, then it binds the socket to a port you specified, and then it calls listen. And then it forks off, let's say, 16 times. And then the worker inherits the listening socket. Each new child process inherits the same memory layout and the same file table entries the parent process had. So the parent process just has to open the socket, and the workers just have to fork. And the workers then call select and accept on the socket, and the pipe which is connected to the master process. And it's actually quite easy. It's not a lot more code than we just had here on the screen. And what the worker then does is it calls select, and as soon as the connection is ready, it calls accept, reads the request, like get slash an HTTP request. It reads the whole request, passes it to your rack or Rails application, reads the response from your application and writes it to the connection, closes it, and then starts the worker loop again. And Unicorn uses this small interface. It's not a lot of code, and that allows Unicorn to even have multiple listening sockets. You can tell Unicorn that each worker should listen on a different port. The worker 2 can listen on port 8000, worker 3 can listen on port 9000. And it's pretty easy to do if you use select. So let's talk about signals. You've probably seen this. You use kill on the command line and send the signal with the number 9 to the process with the ID 8,433. Now that means you're telling the kernel, hey, kernel, please deliver the signal with the number 9, which is the kill signal to this process. And the kernel then goes, takes the signal and calls an interrupt on the new process. Now there are a lot of signals. I think in Linux 3.2, that's over 30 signals. And the great thing about signals is you can actually tell the kernel what you want to do as soon as a signal arrives. And that's called signal handling. And as I said, you can define signal actions. That means you can say, hey, kernel, if the user 2 signal arrives, I want to write something to a log file. If the quit signal arrives, I want to quit. And if user 2 or user 1 arrives again, I want to do this. And you can also ignore signals. Not all of them, but you can ignore most of them. Because most of the signals have default actions. The default action for the quit signal, for example, is to terminate your process and save a core dump. And you can also redefine signal actions. That means the default action of quit is to quit your process and save a core dump. But instead you can tell the kernel, hey, if I receive the quit signal, please shut down all the workers here, write something to the log file and then quit. But as I said, there's a little problem you can't catch or ignore the kill signal. The kill signal is still there and the stop signal. So you have a sure way to kill a process you can't control anymore. You can't ignore or redefine the signal action for kill. But if you're going to use signal handling in Ruby, we just use trap. That's a pretty easy API. You trap a signal and you pass a plug to trap, which is your signal action. Now, what you're doing here is you tell the kernel, hey, if I receive this, please do this. So we tell the kernel, if I receive the kill signal, print this. If I receive the quit signal, print this. And if I receive the user one signal, print this. We then print our process ID and sleep for 100 seconds so we even have a chance to send some signals to the process before it exits. Now if we do this and we run our server on the left side and then send some signals with the kill command on the right side, you can see that everything works except the kill signal, which, well, you can redefine the signal action for kill. And Unicorn does a lot with signals. Unicorn redefines a lot of signal actions and does some pretty cool stuff with it. Unicorn has its own API just with signal handling. Some of them are pretty, let's say, in convention because the quit signal doesn't do anything different than it would normally do. But there's some pretty cool features hidden in Unicorn. The thing is about signal handling that signals are asynchronous. That means a signal can happen anywhere in your code, anywhere where you are. And signals can happen a lot of times. So if you have a signal action that deletes a file, what are you going to do if the signal action happens a second time while you're still in the first one? Unicorn has a pretty great solution for this. And this is it. As soon as the master process boots up, Unicorn creates a pipe. That's called a self-pipe. It's a pretty famous trick in Unix land, the self-pipe trick. And it also sets up a queue. Now the signal handlers Unicorn defined for the quit signal, for example, they do nothing more than just write the name of the signal to the queue and write to the self-pipe. Because the workers ignore all the signals, and the master has to do all the signal actions. And it does that to handle the asynchronous nature of signals. And the master is in its main loop. It's sleeping, right? But in its main loop, it sleeps by doing an io.select on the self-pipe with the timeout. That means it calls io.select with a timeout of five seconds, so it sleeps for five seconds. But it wakes up as soon as one of the signal handlers writes to the pipe. And then it starts again in the main loop, reads the signal, and can handle it on its own term. So now we talked about sockets, pipes, signals, select, system calls, and so on. Is this magic? All of that stuff was pretty easy, right? There was just small basic tricks, and they're all pretty known in Unix. So now that we know about these tricks, let's again have a look at these crazy Unicorn features. Preloading. As I said before, preloading means that the master process preloads your Rails application, all the objects, all the classes, and all that stuff in memory, so that when you spawn off new workers, the workers don't have to spend the time again to load the Rails application. And I think you can probably imagine how that works by now, because it's pretty easy. The master boots up, and it defines a lambda that loads your application. It uses a rack builder and eval to do that, but it works. So it uses a lambda, and when you call the lambda, it loads all the application into memory. Now if you told Unicorn to preload the application, as soon as the master booted up, before forking, it calls the lambda, and the whole application is suddenly in memory, right? And now the workers fork off. That means you have 16 child processes with an exact same copy of the master process with the exact same application in memory without doing any work at all. Unicorn doesn't have to do anything because the Unix kernels do the work for him. So as you can see, that's pretty easy. And with that in mind, I think it's pretty easy to imagine how this works. Unicorn allows us to scale the number of workers by sending signals. Let's say you have 16 Unicorn processes running, and you send the TTYN signal to the Unicorn master process. Now suddenly you have 17 workers. How does that work? Well, the master process again uses trap to catch the TTYN and TTOU signals. The signal handlers again write the signal name to a queue and write to the self pipe to wake up the master. Now the master wakes up and sees that it received a new signal. And it sees that it received the TTYN signal, so it increases the internal number of worker processes and then just calls the housekeeping method to adjust the number of processes. And the new iteration of the main loop. So we've looked at this before, but now you can see how it works. The variable Unicorn changes on the TTYN signal is the instance variable on top, worker processes. Now if this one increased, all Unicorn has to do is call fork another time and save the process ID. And then it returns. Now it has another worker that's doing the same work as all the other workers before. But what happens if you want to decrease the number of workers? If you send a TTOU signal, well, this is the other method maintain work account which is called every time Unicorn goes through the main loop again, the master process. And if the difference between running processes and the processes, it should be running. It's not negative. It soft kills the workers with the number that's too high. As you've seen before, each worker gets a new number in the loop and then it's just easy to find the workers that are too much and kill them off. Now it uses the method soft kill here and you can imagine that it probably sends a signal to the worker processes and they quit. Well it doesn't. Unicorn actually uses the pipe connected to the worker processes and writes the quit signal to it. So the worker still has a chance to finish its work before again it calls IO.select, reads the new signal from the master process and then shuts down. And that again is just a combination of signal handling with trap, forking and maintaining a bunch of different process IDs. But now my favorite feature of Unicorn, hot reload. Like I said, hot reload, zero downtime deploy, hot swapping, whatever you want to call it, it offers us the possibility to spin up a new version of our application while the old one is still running and have the new version take over. So this is a little bit more complicated but not too much. So let's have a look at how this works. This is the scenario. The master and the worker processes, they're happily doing the work. The master is doing the housekeeping. The workers are reading from connections, calling the application, writing responses. Then suddenly the master receives the user to signal, which means hot reload. So the signal handler again writes the signal to the queue and the master wakes up from its sleep because the signal handler wrote to the self pipe. Now as soon as the master wakes up from the sleep, it sees, oh, I received the user to signal, I'm going to do hot reload now. So it calls the method reexec. You don't have to squint your eyes. I'm going to walk you through it. And this is what happens inside the method. The first thing it does is a sanity jack. It checks if we're already hot reloading, if so, return because we don't want endless number of unicorn processes booting up. Now it then writes its process ID, the pit, to a pit file with the new ending dot old. So we can find the process ID of the old master process and send the quit signal to the process. And it then calls fork. That means the old master now calls fork and has a new child process running. In the child process, which is the same code, it checks which sockets are listening for new connections at the moment. And as I said, sockets are files. Sockets are nothing more than a number. So it goes through all the different sockets, gets the number, let's say seven and nine, and writes them to an environment variable. That sounds pretty hacky, but just wait a second. So unicorn now has written the socket numbers to the environment variable and it closes all the unneeded sockets and files. That means all the pipes, the self-pipe, and all that stuff. The new child process, the new soon-to-be master cleans up all the stuff. And then it calls exec. Exec is another system call that turns a process into a different process. That's what your shell does, like the bash shell. Because if the bash shell only knew about forking, it wouldn't be any useful, right? It would just make copies of itself, but you want to start whim. So what bash does is it calls fork and then it calls exec with vim as its argument. The kernel, in turn, turns the process into vim. So now you have created a new process. So the old master forks and the fork, the new child process calls exec with the original arguments that unicorn was called with, like please run in demon mode. This is the path to my configuration file and so on. So the new process starts up with new application code, because that's the feature, right? It starts up, it reads the application code, because you specified preload application. Now the master checks if there are any numbers in the environment variable. And it finds that there are two numbers, seven and nine. And since it's a child process of the parent process, it inherited the file table entries of the old master process and just uses io.4fd to turn the numbers into socket objects. And the cool thing about this is that normally, if you spawn up two workers listening on the same address, you get the error address already in use. You've probably seen this before. But unicorn avoids this problem by just sharing the socket with its children. They already had access to the sockets. And now they just turn them into objects. And everything is ready to go. The new master spawns of workers, and the workers call select and accept again on the listening sockets. Now you have two sets of master processes running. The new one is already doing its work, but the new application code, and the old one is also still running. And now you can send a quit signal to the old process which shuts down, and the new master process takes up. And that's it. Just forking, signal handling, and knowing about what child processes inherit from their parents. So this is my conclusion. It's not magic. It's just unics. All unicorn does basically is it knows about unics. It knows what forking does. It knows what sockets are. It knows that sockets behave just like files, that you can inherit them. And the combination of these simple tricks, signal handling, forking, keeping track of process IDs, sockets, select, accept. Well, in combination, they make up for really, really great features without unicorn having to do all the work. As I said before, IO select takes over and does the load balancing in form of the kernel for the process. And that's pretty, pretty cool, in my opinion, because it's just a combination of small clever tools to make up great features. Now, you might think, why? Why should I, as an application developer, learn about this stuff? I'm a Rails developer. Why should I know about system calls, sockets, pipes, files, file descriptors? I write web pages, right? Well, I've got a few reasons. And I think debugging is a really important reason. If you know about this stuff, about system calls, suddenly the error message can't open too many files, makes sense after you open a new network connection. Because you know that a network connection is a socket, is a file. And if you know about this stuff too, you can make creative design and architecture decisions. You don't have to rely on threads anymore. You can use processes. But only if you know about this stuff, or you know about how networking is being done in processes. You know how you can talk to other processes over the network. And another point, somewhere someone said, it's always good to know one more level of abstraction beneath the one you're currently working on. And I agree. I agree. I think Learning C made me a better Ruby programmer, because I could imagine what the Ruby VM does. And I think knowing about this stuff makes you a better application developer, because sometimes you've got to have to push this stuff into production. And most likely today, it's a unique system. And if you know about this stuff, you can fix your bugs more easily. And well, concurrency is another big issue, too. Concurrency is a huge topic still. But everybody just talks about threads. What Unicorn does instead is use processes, and really, really well. Unicorn performs really good, just by using processes, by using fork, without having to write any additional code. And well, the last point for me, the most important one, is the realization that all of this stuff was not magic. I looked at Unicorn like some black magic that some wizards in some basement have written, with green text and black terminals. Well, it turns out it's just a bunch of system calls. And once you know about them, you can use them. And you can have the operating system do work for you. And it was so much fun realizing that I could write software like this. I could write a web server with all these cool features, just by learning about how the system works and how I can use it. And well, that's it. Thank you from me to you. And if you want to talk about this stuff, feel free to talk to me. I love to talk about low-level programming, Unix, C, Ruby, and so on.