 So I know there's a keynote before this morning, and then there was a break. So I'm assuming you had time in the break to rest and recuperate, because we're about to do some really hardcore routing this morning. So I'm going to try it again, because I need all your energy, because I got 123 slides. So I need your energy to help me out. All right. How's everybody doing? Better. All right. I'm going to wait 20 more seconds, because there was a break, and then nobody came in, and now a whole swarm of people are coming in. So I'm just going to wait, let everybody get settled. It's like a good time to start. OK. Good morning. My name is Vaidahi. Hello. I'm Vaidahi, as I just stated. And I am an engineer at Tilda in Portland, Oregon, where I work on Skylight, your favorite Rails profiler. You may have noticed a booth. That's our booth. You should come by and say hi. Skylight is a Rails app, but it's also not my first rodeo when it comes to Rails. In fact, I've been working with Rails for about four years, and I'm pretty comfortable with it. I really love the framework, and whenever I work with it, I feel like I'm coming home. It's super organic and comfortable, and I know it pretty well. But just like other frameworks, when you get into the groove of working with something and following the Rails way, you can be pretty productive without thinking too much about what's actually going on. It's super comfortable, and you know it very well. And this seems to hold true, except of course, in the few instances, or sometimes many instances, where you just veer a little bit off the happy path. So for me, veering off the happy path usually means I'm trying to figure out how something works, or I'm stuck debugging an error, or for some wild reason, I am in the source code of Rails, and then I'm very lost. But generally, going off the happy path for me means I'm gonna stumble upon some sort of abstraction. So even though I feel like I know Rails pretty well and how it functions, there are some abstractions where when I find them, I'm like, I thought I understood this, but it turns out I actually only understood one layer. And for me, a great example of that is the Rails router. So quick show of hands, how many of you know how the Rails router works, like the nitty gritty? Yes, all right, good. I was looking for that hand, I was like, there's gotta be one hand in here. That's Aaron, by the way. He wrote the router, so. It's good that his hand's up there. So it's okay that you didn't have your hands raised, because I also didn't know how the Rails router works, and to be honest, even though I spent all this time to try to write this talk and learn about it, I still don't know everything. But I think today we can try to investigate a little bit about what we can learn about how the Rails router works under the hood, and hopefully we'll have some fun along the way. So, here we go. We're gonna recap some routing basics to start, just to refresh our memory. It's early in the morning, it's day three of RailsCon, I understand, we'll just work our way up from the beginning. So the router basically allows us to recognize URLs. It lets us navigate throughout our application. It's the go-between of our application and the external world, because it's what handles incoming requests. The router, you can kind of think of it as the post office of your application, which means you can think of the incoming requests as all of the incoming mail that has to go to the correct places. And correspondingly, responses are outgoing mail. But we're just gonna worry about incoming mail today. The router is responsible for making sure that a request ends up at the right mailbox. So when we say mailbox, what we really are saying is the right controller action, because that's where all requests eventually wanna go. When Rails receives an incoming request, the router is the one that has to decide which controller to send you to, and then the controller is gonna be the one to take it from there. But the router is kind of like that middle person, because the controller can only get what the router sends it. So, to refresh our memory, some handy commands, we can see all of the possible addresses, and when I say that I mean like controller actions, we can see all the possible addresses or mailboxes in our app by using Rails routes. This command basically just shows you all the list of available places, available end points that a request can end up in your app. The definition for this, of course, comes from the routes RB file, which every Rails application will have. So, here's an example of a simple Rails, or simple routes RB file. The controller name and the action correspond to the route in the form of a string, and you can route to a route, you can use get, post, lots of different things. You can also use resources, which is actually not one route, but many routes. Just some syntactic sugar for you. So, that's basically our routing, and that's probably what, you know, early on when you're working with Rails, your introduction to the router will be, and that's fine, because that's really the basics of what you need to know to get started. Okay, cool, I can buy all of that. But it feels like there's a lot of magic, like things just end up at the right place, and who knows how, and we're just gonna write some strings, and all right, things just go there, I don't know how, except we can find out how. It's really easy to accept that the Rails router does all this magic for you, but it's also worth taking a little bit of time to investigate how it does this magic, and we can familiarize ourselves with little parts of the router to try to break down these abstractions. So, the first step to investigating how the router works is, of course, by identifying exactly where it is. You have to figure out where the router comes into play in terms of the life cycle of your app. So, if you wanna figure out how something works, the best rule of thumb, look at the docs first. So, if you look at the Rails guides, it turns out there's a command called Rails middleware, and this outputs the entire middleware stack, which is in use within your Rails app. If you run this command, you'll see that a bunch of things are being output, but all they really are just the middleware in the order that it's executed in your app. Pay attention to the thing at the bottom of the stack. It might come in handy in about five slides. So, before we go any further, in case you forgot what middleware is, that's okay. It can seem kind of confusing and scary, but all it really means is it's a rack app that takes another rack app as an argument. This is an important distinction because you can have rack apps that are standalone, which means that they aren't initialized with other rack apps. A good way to differentiate these two things is rack apps that are standalone are often referred to as rack endpoints. All right, but I forgot what rack is. Oops, let's quickly refresh what that is too. So, a rack app is any app that responds to call, so a proc is often used in this example, and all it does is return a status, headers, and body. And all the middleware we just saw in that middleware stack, well, those are just rack apps, which means you know they've got to take in another rack app and respond with this. And if you don't quite believe me yet, I have a little pro tip that you can try. This could really be a talk in and of itself, but here's a little teaser. Next time you're in your Rails app, try replacing some of your controller endpoints because controller actions are actually just rack endpoints. So, if you don't believe me, try doing this and see what happens. All right, but we're not gonna talk too much about rack today. Let's get back to our middleware stack because we were trying to find out, we're trying to investigate where our router comes in. So at the end of our middleware stack, we see something called application dot routes, which means it's the last piece of the middleware stack. And the fact that it says dot routes means something important since we're looking at the router. And in fact, it is actually very closely related to the router. We're just about at the router now and since this is the end of the stack, we need to figure out what this routes thing actually returns. So if we do some more investigation and just search in Rails, which is always fun, you'll find that there's actually a file called the Rails engine and it has a definition for a routes method. And this routes method returns an instance of all of our routes. So now we know that this is how the request actually gets through the middleware stack into the router. The question of course now is how does the router route it? Well, let's go back to our routes RB file. It's something we're familiar with and there might be some hidden clues there. So how does the request that comes in to a router get sent to the right one of these? Well, the naive solution and maybe even on first glance, it makes sense to just work our way through all of the routes in our app until we find the right one. We could maybe even just iterate through and write a loop and we could use a regular expression and just check the request and see does it match this route? Does it match this route? Does it match this route? That is an option, of course, right? You can do this. It's not the best because as you'll notice, you might have a lot of routes and your if statement's going to get very long and there's already regular expressions in there and it can be just generally confusing and hard to read. And the fact that there are a lot of if statements is one problem, but another issue is that as your routes file grows, this is not going to scale that well. Most of our Rails applications are going to have a lot of routes in them and this is probably not the best solution for figuring out where to route our requests. In fact, let's say we have N number of routes and N could be anything, depends on the size of your application. This is actually going to run in linear time because in the worst case, you could be looking for a route that doesn't even exist and you're going to check through every single route in your route file and end up empty-handed. Another way to describe this is O of N which just means linear time. As the number of our routes increase, the amount of time that it's going to take to search through them also increases linearly. So this isn't great. So you'll start to realize how painful this initial solution really is when you start to compare it back to that post office example that I was talking about early on. Imagine that the post office just looked through a long list of addresses every single time it got an envelope and it would just check to see if your address, if the address on the envelope matched the one that it was looking at and as you can imagine, that's a very long list. It's not efficient. The post office is already pretty slow getting my mail to me. This would just make it way slower. I'm not a big fan. Who knows how long it would take for our requests to get to the right place? It's just a lot of unnecessary work and it's going to be slow and inefficient. Where's gotta be, hopefully, a better way. What we really want is to just narrow down which routes we're looking at and instead of iterating through our whole list of addresses, instead of looking through all of our routes, what we really want is to narrow it down so that we're not spending time searching for things that are definitely not gonna be a match. There's definitely room for improvement in our strategy here and guess what? We are not the first ones to realize that this could be improved. Turns out, someone already has. Namely, tender love. In fact, I should really thank Aaron for his work on Journey because if he hadn't written Journey, I literally wouldn't be giving this talk because I spent a lot of time learning about this engine and it was really fun and there's some really cool, complex things happening there that I didn't know existed. So I'm really glad that he put in his time to create it. So wait, I just said something about Journey. What's Journey? I just said it's some sort of engine but I should probably tell you more about that. So Journey is a routing engine that, as I mentioned, Aaron wrote and it used to be a standalone library and it was merged into Rails around Rails 4. So if you take a look at Journey's readme, it can admittedly seem a little bit complex and intimidating but if we start breaking down what it's actually doing, the concepts aren't as scary as they might seem. So where does Journey fit into the router? Well, remember when we called that method Routes? Well, that Routes method returns a Routes instance which we can see creates something called a Routes set. All right, so this is like, we're actually getting to our set of routes. We're finally like inside of a router. We finally made it, hooray, this is so exciting but now I just have a new question which is what is a Routes set? I don't know if I have any more answers than I had before. So what's this set thing? Well, if we look at the definition of Routes set and now we're inside of action dispatch actually, we'll see that a set has Journey routes within it and this is the first time we kind of touch Journey's code. At this point, we're gonna actually take a break from Journey's code itself and we're gonna focus on the general concepts of what the engine does. That way, if after this talk you wanna go back and look at the code, you'll be well equipped to understand what it's doing and how it's working and you'll know where to start too. So Journey uses a computer science concept called a graph in order to make life much easier for our routes. Graphs are kind of like a data structure that you can't escape from when it comes to computer science and so I've learned that you should just embrace them and love them. So I love graphs, hello. They're all over the place in computer science and once you start to really understand how they work, you'll start to see them everywhere and then you'll start making graph jokes and people will be like what? Or they'll be like, ha, that's great. It really only goes one or two ways. So all you really need to create a graph is one node but usually you're gonna see graphs with multiple nodes and those nodes are gonna connect to each other with something called edges or links. Now edges can be either undirected or directed and for the purpose and context of Journey today, we're really only gonna be dealing with directed graphs. So let's go back to our post office for a second. This post office is gonna hang around but hopefully you'll see that the metaphor kind of continues and it helps make what Journey's doing a little bit easier to understand. So if you think about how a post office works, ideally if it's a post office that functions well, no post office is just randomly putting in envelopes into whatever mailbox. There's some sort of system and ordered how it functions and the system that it uses is very, very efficient in solving a problem that would otherwise not scale well. Specifically in a post office, the rules which is really the algorithm that the post office abides by for figuring out where to route your mail is by starting with the most broad part of the address first. Then we narrow down from country to zip code, actually two parts of zip code, then state and city and street and then your actual house address. There's some mechanicalness to how this works and it's kind of an algorithm in and of itself and the same is true with the Rails router. The idea of narrowing down and matching an address to help avoid unnecessary searching is the same idea that you can find within Journey. So Journey, as we know, uses a graph data structure to help the router figure out how and where exactly to direct a request by matching against the URL, which is the address of the request when it comes in. So we can imagine when Journey looks at a request, the first thing it has to do is figure out where on earth this request actually needs to go in the Rails file. That's how it needs to start its search. So for example, when we have a request that comes in like recipes ID, we have to start looking for a specific recipe. What we eventually wanna do is send it to the controller where we can look for a specific recipe by ID. But before we even can find the controller, it seems silly to even consider the root or articles or any of the comments route because we know for a fact it's not even gonna be there at all. And you and I know this intuitively because we can just look at the Rails file and be like, of course, this seems obvious. And of course, this becomes even more important because remember every Rails file where we have a resource we're actually narrowing down and eliminating a huge number of routes that we don't have to search through. So what if instead of looking at every single route like our original naive implementation, what if instead of that we did something similar to what the post office does? What we really need to do is smartly look at the address when it comes in and narrow down the country, state, city, street address of our request. And then we can send the request alone. But first things first, when a request comes in, it needs to be read. We can't really worry about sending it to the right address if we can't read the address. We need to make sense of what's on that request envelope. Wait. But how does the router read? That seems like a very big skill for a router to have. Well, we have to teach it. And it turns out that Journey reads similarly to how you and I would read. And in fact, the way that Journey reads is exactly how compilers will read code too. So when you read a sentence like the cow jumped over the moon, what your mind is really just doing is looking at this string of characters and you're looking at the capitalization, the punctuation, the spaces. And in your mind, you're dividing it up into coherent letters. And your mind takes those letters and builds it into a sentence that follows some sort of grammar so that this sentence has some meaning to you. The ability to break up this string into pieces that make sense and string it to follow a grammar is basically what Journey needs to be able to do. It's replicating what you and I can do as humans so easily. So if a human can do it, then obviously there's gotta be a way to program a computer to do it too. So this is what Journey does with the request address that's on every single request that comes in, the URL string. Except of course, in order for Journey to read this, it also has to know what a letter is. It needs to know what grammar is and then it needs to be able to group them together so that it makes sense of what on earth is on this request envelope. Journey has to perform a process called tokenization. Tokenization seems hard maybe, but all it really is, is the work of breaking one expression down into its minimally significant parts. There's a name for those little parts as well and those individual pieces are called tokens. Thankfully, Journey does have some help when it comes to tokenization. There's one program within Journey that handles the work of tokenizing. Well, okay, technically speaking, it's not exactly a program, it's a class. It's called the Journey scanner. I feel like this scanner is like one artisanal Mason jar away from being like a true hipster. So the Journey scanner actually inherits from Ruby's string scanner class. And the scanner can take any string and it's gonna follow a set of defined rules in order to derive tokens which are the minimally significant little bits of the expression. And it's going to do that from whatever input that we give it. So the input we're gonna give it is a request URL. And we can actually see the scanner in action by going into the Rails console and creating a new instance of this. And you can do this in your own Rails app. We can also see it do the work of tokenizing a string into individual tokens too. If you call next token on the scanner instance, you can see it actually splitting apart our request URL recipes ID. And you can see it split that request string into individual expressions or tokens. Journey scanner recognizes a handful of tokens including slashes, string literals, left and right parentheses, among a few others. And you can imagine that it would be important for Journey to be able to distinguish, hey, these are like the letters and the words that make up my string that I'm trying to parse. So all right, the scanners helped us out. It knows how to divide up a string into words or tokens. But that's only half the battle because words mean absolutely nothing unless there's some sort of grammatical rule to follow. The next step is for us to figure out what's going on with those words and make some sense of them. Well, Journey is just like us. They have to follow a grammar too. What you and I are intuitively able to do, Journey needs to learn how to do. And the way that Journey actually solves this problem is with some help from another class called the parser. The parser's job is to take those tokenized pieces, the tokenized string, and make some sense of it. The parser does this by utilizing yet another computer science concept called a syntax tree. Now, syntax trees might sound like, I don't know, very hard and I don't know, based in linguistics, which they actually are, but we're not gonna have to worry about that today. They're not as scary as they might sound and we won't go too much in depth. We'll just kind of cover why they're important in this context. So you might remember in elementary school, you probably at some point had to diagram sentences and you had to kind of distinguish different parts of the sentence to understand and learn grammar, whatever your first language might be. So you probably had to do something like this where you took a sentence and you turned it into a tree of the words in the sentence in order to figure out the parts of the sentence and how to structure it. And that's just what a syntax tree is. It's an illustrated version of the structure of a sentence. And it turns out, it started out in linguistics and then it moved on to education and educators now use it to teach grammar to kids. As it turns out, the parser creates a similar kind of syntax tree for journey and to help it make sense of the grammar of its language. So here's an example syntax tree that journey would generate for a route that corresponds to the show action for a recipes controller. You'll notice that if you read the tree from the bottom and start to work your way up the nodes and start to reduce the nodes as you work your way up, the combined expressions actually start to reduce until all you have left is a single expression at the end. And in fact, for every single route that we define in our routes to RB file, journey's parser is gonna create a syntax tree for it. And if you're curious, you can actually see this in action in the Rails console. You can output the syntax tree for a request and it'll turn into an HTML string and you can actually use I think graph viz to visualize it. All right, so now you have a bunch of trees, but like it still doesn't really help us combine all of our routes together. These are kind of like disjointed parts in the mail in the post office. We need like one single system to solve this problem. So what do we do with all these syntax trees? Well, you might remember our graph and this is where that comes into play. Journey uses a graph to route your request and that graph is made up of those syntax trees. Journey builds a graph of all your possible routes by combining these syntax trees and journey's code actually refers to this as something called a generalized transition graph or GTG. But for our purposes, all you really need to do is think of it in the context of a state machine. The state that we're really dealing with is the request URL that we're parsing and how far along the URL we've parsed. When a request makes its way to a router, Journey is going to walk this graph, one node or one state at a time to match a subset of the request URL. Basically, when it reads a request URL, one section at a time, it's going to walk down the graph and based on which nodes match the state it's currently in, it'll progress to the next part of the graph. If a node in the graph, which is one node of, remember, many syntax trees that have been combined, corresponds, that's how Journey knows to walk down the next path inside of the graph or rather within the graph. It'll keep walking down this tree until it finishes reading the address, until it finishes parsing the request URL string. And if it finds a node that matches the state of that string at that moment when it finishes reading, well, that's how it knows exactly which route that the string that came in with the request corresponds to. So if you really think about it, you and I probably could have done this super easily if we had just looked at the routes file and been like, oh yeah, obviously, we wanna go here. But what we can do in one glance and identify very easily, it's so simple for us to find the controller and action that correspond to this, Journey has to do all this work in order to mimic and replicate what we're doing. But it does it very fast, so it's okay. In order for Journey to do what we can do, it has to implement data structures under the hood. And in fact, what you can really imagine is that there's like a little robot that's kind of walking down the path with this request URL, similar to how there's probably someone in a post office who's like, all right, I'm gonna sort this in the correct way. We can imagine this little robot walking down this tree for us, walking through our state machine for us. And it's systematically checking the request URL string and deciding which path that it should take in our state machine graph structure. So another name for the same concept that I just described here is called a non-deterministic finite automaton or NFA. And an NFA is just a type of state machine and it handles a little bit of input at a time and decides, oh, how am I gonna proceed and transition forward? This is the current input that I have. This is the current state. I'm gonna make my choices based on what I have now and where I should go forward in the graph next. If you ever Google NFN, you end up on the Wikipedia page, do not be afraid, because this is all it is. There are basically two outcomes for our little robot who's walking down our graph for us. There's only two things that can ever happen. Either we'll find something that matches the state, the string request URL that we took in as our input or we don't. It's really a binary situation here. When we get to the end of our string and we're at a point in the graph that actually works that corresponds to a real route, then we know that there's a route that matches our request URL. And if we're able to find a route that matches, then that means we found an acceptable place, an acceptable controller in action to send the request on forward to. This is also referred to as an accepted state. And if there is an acceptable route to be found, then all we have to do, or all journey the router has to do rather, is just dispatch the request to the corresponding controller in action. So so far we haven't looked at any real journey code, but I think we're finally ready to make it happen. This is actually not technically code. This is their visualizer, but it's quite a bit if you look at it for the first time. It just kind of looks like this. And you'll notice it doesn't look that different from that colorful graph I had before. The same idea and same concepts. And hopefully looking at it now, you can kind of see some sort of similarities there. Here's an example of what that same NFA would look like if we were searching for a valid route. And this is an example with just four routes. So you can imagine how big it would be in other cases. So if journey is able to find a valid accepting state and it can dispatch the request to an actual controller in action, then it will go ahead and do that. But what happens if there's a request URL that's like garbage or is typo or just isn't a real URL? What happens to our NFA then? Well, in that case, you have an invalid unacceptable route and the term for this is a rejected state. You really have no choice but to leave the state machine because there's nowhere you can go. And probably it would make sense to like raise an error or indicate somehow that, hey, this is not a real route. Maybe you've made a mistake. So this is what the NFA will look like in a rejected state. You'll notice that when you reject out of the router, if you've ever run into issues routing with your Rails app, you might have run into an error like this. No route matches. And in fact, journey is the starting point for where your Rails app knows, hey, no route matches. And the error kind of begins from this point. As it turns out, you can simulate the same equivalent NFA that I just showed you there in your own Rails app. There is a command or there's a visualizer that you can run Rails application routes router visualizer and you can actually see your own internal NFA because it's now part of Rails, which is really can be a great debugging tool. Well, this has been quite a journey. But we did it. Like we figured out how the request makes it either to a valid controller action or just short circuits out of that graph. And now that we've kind of made it through the entire life cycle of the request making it into the router, hopefully you see that it's actually not as magical as maybe you initially thought. And in the process of trying to understand the internals of the Rails router, we actually learned a lot of other cool things along the way too. Namely, journey is a regular expression engine, but we learned that it uses tokenization, it uses scanning, it uses trees and graphs, and automatons all under the hood. And these concepts amazingly are not unique to journey. You can actually find them in lots of places including within languages, other frameworks, other parts of Rails, and in compilers, which is always fun. And now you understand the fundamentals of how these things work. So maybe when you see them in the wild, you'll remember seeing them in the router too. So I think Rails is great because you don't actually need to know all of this to start being productive. And that's an amazing thing. I can imagine trying to learn all this when you start out and just being like, I, what, automatons? All right, bye. This is too hard. And the great thing about frameworks like Rails and frameworks with abstractions is that all of the nitty gritty things have been abstracted away for us so that we don't need to think about them on a daily basis. And we don't have to solve those problems again and again. We just get to be productive, which is awesome. But at the same time, abstractions aren't something that have to be scary because if you're ever curious about how something works, you find yourself in the source code, you're debugging something, or if you're just like me and seem to go off the happy path all the time, abstractions aren't something to be afraid of. Chances are you actually can understand what's going on. You just might need to do a little bit of investigation to figure it out and to get to the bottom of what you're trying to learn. So speaking of abstractions, if you're interested in learning about abstractions and what's going on under the hood, if you like computer science or you just like learning how things work, you might like a project that I worked on recently called Base CS. It's a weekly writing series that uncovers a new computer science topic every single week. And there's also a video series for it now as well as the written series. And we also have a podcast where we introduce you to computer science fundamentals in a fun and friendly approachable way. I also have some podcast stickers with me today. So if you'd like one, come find me after. And you can find out more about this at basecs.org. I hope that, thank you all so much for coming, but I really, really hope that if nothing else, you feel empowered to understand how things work under the hood. People can sometimes tell you that you can't understand things or if something is really hard to understand then it's not worth learning. I fundamentally disagree with that. Everybody can learn everything. So I really appreciate you going on this journey with me. That's the last one, I promise. Thank you so much for your time.