 I just wanted to say thanks to Winston and to Anna and Alan and Jin and everybody for organizing this. It's been an excellent conference, I'm having a great time. This talk is about what we do at Discourse that is kind of moving away from the Rails conventions. I called it Off the Rails. It is also, there are also some crazy things here, so the name is pretty apt. A little bit about myself. I've been doing open source development full time for the last three years. I've co-founded Discourse with Robin and Jeff Atwood. I worked remote now for the last six or seven years. I haven't gone into an office. I've been, I'm fascinated with performance problems and squeezing the extra little last millisecond out of the app. In a previous life, I worked at Stack Overflow and my Twitter account is Sam Saffron. So a little bit about this Discourse thing that I just mentioned. You can see here, this is the picture of forums today. This is one of the most popular forums in the world. This is what everybody tends to use. I'm sure you've all seen this before, Google Groups. We have mailing lists, which are very hard to navigate, and Discourse came out to try and reboot this space and create something modern and nice that is very easy to use and has all of those features that we'd expect an app that was built today to have. We're strictly open source, unlike a whole bunch of other things. There's only one version of Discourse, the awesome version, and you can fork it and work on it if you want. This actually leads to this. We have a happy Richard Stallman, unlike the sad Richard Stallman that we had yesterday. So I wanted to talk a little bit about a couple of years ago, DHH had this blog post talking about rails as Omakase and he chooses all of the wonderful dishes and he's a sushi chef and if you don't like this, Unagi then don't have Unagi, but you have to be more or less thinking of those terms that he's choosing the things for you. Where I struggle with it is that analogy. It's very hard to explain what's going on in terms of that analogy. For example, we have to monkey patch all sorts of parts of rails to make it a little bit faster and how do I explain that in terms of Unagi? It's very hard for me to do that. So what we do is we bolt on all sorts of pieces to make our development life much more fun. We bolt on all sorts of pieces that are missing from rails that give us a better caching story, better profiling story. We've got all sorts of gems that we install that give us completely free performance benefits without paying any kind of cost. And they're not included in general in rails because they are MRI specific and rails in general will not take in MRI dependencies. But we have, there's no reason to be spending 2% of your time in the app in this dot blank call. Like literally we've seen those kind of numbers. So like we've installed these gems that just give us absolute free performance. We've got a big, big piece of rails that we add which is messaging. So we have a reliable message bus that we can talk between our processes and from the processes to the client. We've got a very interesting logging UI and we're an Ember app, unlike, so we don't use turbo links at this course. We use Ember instead. And we think of our app as a complete app. So we use Docker to deploy it. And our app is the nginx config. And it is all of the little bits and pieces that glue the deployment story as well as the actual development story. In general, like when you think about rails apps, then it stops at the point you run unicorn start. And all of the rest is the ops problem. But for us, it's a complete app from all the way down, including nginx optimizations and monitoring nginx and all of the, and dealing with logs and whatnot. That's what makes discourse. And the only supported way that, the only way that will support the open source installs out there is if they're using a Docker based install. So I'd like to give you a little bit of a demo. Let's see in production of how this thing looks. So this is meta discourse in production. This is my view of it. So one thing that you'll first notice is that I get these little tidbits at the side of the page. So I'm constantly aware of how performance is going on the website as I'm using it. And that means that I can't get away with stuff. If somebody has an n plus one query or something like that, it'll be automatically in my face in production. This is obviously something that only admins get, but that it is incredibly valuable. And I can just break down and look at the actual sequel that ran inside the app. I'm going to have to remember to log out because you probably saw my auth token and that's probably not a good idea. Let's ignore that for a moment. So with the sequel, you can actually get the back traces of each sequel statement. So debugging of why is somebody running this sequel is very easy. And you get timing for each sequel statement as well. And this is the rack mini profiler gem. The rack mini profiler gem comes with the ninja switch, which is putting question mark, PP help at the end of it. And that's the ninja switch of all of the magical stuff that you can do with it. So for example, if I want to see a flame graph, this is all in production now, by the way, just running on our meta instance. I want to see a flame graph of this page. So I'll run this thing. And then I can see these are back traces that are side by side in along across the timeline. So the little snippets of what is happening in the app. And I can zoom in and see where various stuff is running. I can see where active record is spending its time. This for example would be where we're executing some sequel and this would be active record overhead before the execution of the sequel. So I can look and zoom in and get good analysis of what's going on. Inside as well, like it's interesting. An interesting thing is that we've got integrated memory profiling as well. So I can run this thing, which I didn't test running, but who knows. Anyway, so I can see like for this particular request, I can see how many objects were allocated, how many were retained. I can get a breakdown of where they are. I can even see the locations of where certain memory was allocated. So if output safety caused like 387k of allocation and so on. And I can get the actual line where everything is happening and debug into it. So there's a nice big detailed report. I can see which types of objects and I'll go through memory profiler in a bit more. A last thing that is very cool about us is that we're living in the future. And in the future, if you need to debug your JavaScript application in production, you'll have source maps. So I can actually go to the user card view and I can say, okay, when hide is called, let's just drop a break point here and then I'll just bring up a user card. Whoa, hide got called and I'm inside here. And all of our JavaScript is obviously minified and bundled together. But I'm able to see now the actual file where this happened and debugging in production becomes much, much simpler when you have source maps. And I think that's about it for like the production demo. And I'll just go back to the slides. Okay, so the first piece I'm gonna spend some time talking about is this message bus gem. And this gem gives us a feature that there is some discussion of getting it into Rails 5 with Action Cable. But we've had this now for over two years and this is how it looks. On the server at any point in time, we can publish messages to a channel. And then on any of the other processes that are running discourse, we can subscribe to the channel and we can print it. So you can basically communicate between all of your processes at any point in time and deliver messages anywhere. Additionally, you can deliver messages to the clients, to the JavaScript app. So you can see on the top, you've got the Ruby code where it's publishing a message. And at the bottom, you've got the JavaScript code where you're actually listening for the message and you're acting on it. So conceptually, you can send out messages and then receive them on the client and actually restrict it so only a certain user or only certain group can get access to that or a certain client can get access to that. And you're guaranteed that when you post a message, it'll be received at the same, at a reliable ordering. And it will recover on the client if it happens to be, if the laptop lid is shut, it'll eventually catch up and get the old message. The big things about message bus that are a bit different from what I've heard about Action Cable is that it has multiple back ends for dealing with this transport. So we have polling and long polling that we support out of the box. And long polling is what is known as Comet, where you make a request and then to the web server and the web server just grabbed onto it and keeps it ready there until it's got data and then it'll ship it back to the client. It has security built in, reliable playback, which I mentioned, very, very efficient transport. So you can have like a hundred different channels, but it's all multiplexed inside Redis. And that means that the transport is very, very efficient. And multi-site support, which is very important for discourse, which is another kind of off the rails thing. We have to be able to host multiple sites on one process. And that means that they all get their own database. So we need to, when all of our components need to be able to work in that mode where there are multiple databases going on. And how is this magic done? And how is web sockets done in rail? So this is all the code you really need to know to understand how this works. So a feature that was added to rack a few years ago was something called Rack Hijack. This is supported by every big web application server that's out there. So Unicorn has it, Puma has this, and Passenger has this. Thin does not have this, because Thin did their own thing a few years earlier, and they haven't moved to this thing. But Message Bus also supports Thin. So basically what this little code example doing is a tiny rack app. Whenever it's called, it starts a new thread. The thread sleeps for a second, and then it returns something to the client. The one very important thing to notice is that there's this connection close thing. So, let's see if this, anyway. It does IO write connection close, which is critical to do when you're dealing with Rack Hijack. Because if you keep the connection open, then the next request that you get may not be for a hijacked request. It might have meant to have gone to the main web server, and you have no way of returning a connection back to the web server. So you always have to close connections. And you can see the numbers are pretty amazing of what you get. So you get 100 concurrent connections, so 100 different clients connecting at the same time to the server. They all sleep for one second, and within 1.08 seconds, they return back. So there's no blocking. Like if there was blocking, you'd have to wait 100 seconds to serve all of this. Now, you'd never do this stuff with threads. That's way too expensive, and Message Bus does not do that. But this is just to get the concept across of what Rack Hijack is. Because threads are expensive. You don't wanna spawn them like this. This has been used in production now for multiple years. Has almost no dependencies. The only jam dependency that we have is a dependency on Redis. Now this dependency can go. It's only, it's isolated to one file, and that's for the transport. And potentially, you could implement transport under PG, which gives you PubSub, or you could implement a memory transport. So you've got an app that's running single process. And this is the real cool thing about this is that all you have to do is include this gem in your Rails app. And then you've got this feature that you can build stuff on. So it doesn't run another process. It's all running in the same single Rails process that you already have. And then the big question is, what about action cable? Why do I need to look at this? Shouldn't I wait for action cable? And with action cable, there are a lot of questions that are unanswered at the moment. One big thing is this thing web sockets only. And I'll go through some of the issues with web sockets. What about this reliable message ordering? What if I close the laptop and open it? Am I gonna be able to catch up on all the old messages that I missed? What kind of other dependencies am I gonna have? Am I gonna have to bring in an event machine or celluloid? So those are the kind of worries that you have when I'm thinking about this. But really, we don't know. We have no idea what it's gonna be. But message bus has been used in production for multiple years now. So there is that. And one interesting thing, I noticed this the other day, like they had these new performance tools in Firefox Web Developer Edition. The bottom, on the same page, the first comment, it goes, web socket traffic monitoring in the network tab, please. Because one thing with web sockets that they haven't told you is that debugging these things is a huge pain. And all of the tooling to debug it is also very, very painful. We initially did support web sockets, but there were a few things. The HTTPS required was like a very huge one. And the thing is, if you're running web sockets over HTTP and not HTTPS, weird things can happen. And by weird things can happen, you can have all sorts of proxies that rewrite traffic and mess stuff up for you. For example, in Australia, Telstra Network decided that they don't like web sockets, but they like it enough to allow you to connect. But then they'll just store all the data. So you have a web socket that's open and you think that you're meant to be getting data from it. But the whole pipe is just clogged. They shut the pipe down. So you need to be able to recover from all of these kind of situations with web sockets and fall back anyway. So the idea with Message Bus is that we first solve the base problem of having reliable messaging. And then transport is added on top of that. And you can have whatever transport protocol you really want. And it's not starting at web sockets. I'm not against adding something like web sockets to Message Bus, if we had to. But that is a value add as opposed to the first thing that it does. And web sockets is just another protocol for transport. Now, if you have any questions, one thing that I did notice here that people are a bit shy to ask questions. So I made a site for you to ask questions, which I'll go through at the end of the talk. So you can all go there. And what that is, is a little chat app written on top of Message Bus. That you can all just go in and chat on. And hopefully it doesn't crash. It's about 120 lines of code to get kind of a very rudimentary chat app that gives you a presence. So you can see who's online and allows you to send messages. And you can play around with it. So if you have any questions during the talk, you can go there and just put the question in at the end of the talk. I'll go to that site and quickly triage it and try and answer any questions you have. Okay, the next gem I'm gonna talk about is LRU Redux, which is a piece that I know Erin wants to put into Rails as well. It's a least recently used cache. Now, one cool thing about Ruby 19 is the hash class is actually ordered now. Back in 18, when Matt's made the hash class, it wasn't ordered. But then it became ordered in 19. And we can depend on that feature to build a very, very efficiently recently used cache. And that construct becomes incredibly useful. So at the base, what LRU Redux does is gives you this. But the thing about it is that it is ultra efficient. This is as efficient as it gets. When I designed this gem, I looked at all of the other gems that were out that were doing the same thing. And the main criteria was to maintain as close to a hash API with the fastest performance. So this will perform probably about, I don't know, 30% better than the best other one out there that is also using pretty good practices. And we keep trying to squeeze any extra performance we can out of it. And having this, having a cache is very cool. So we have a cache and we also have a TTL cache. So you can have a cache with a time to live. We have thread safe versions. And yeah, the semantics mean this is Ruby 1.9 up. But hopefully none of you are using Ruby 1.8. Is anybody using Ruby 1.8? Nobody, this is awesome. All right, all right for no more Ruby 1.8, sorry Matt. We have to move forward. So now, so we have these two building blocks. We have LRU Redux and we have the message bus. And what happens if we marry these two things together is one example of what we do at discourse. You get a distributed cache. Now that is fantastic, right? Suddenly, you can set something in a hash in one process and read it out in another process. And it's a cache. So if it fills out, it'll start dropping off the old things. So all of that is done in about 50 lines of code. But once you have these building blocks, all of these other new magic things on top of them can be built. And that is incredibly useful for us to have a very lightweight cache like this that we can use across processes. And for us, we have to optimize for adoption of discourse and to be able to get adoption, we need to be able to install discourse on very, very cheap hardware. So we're constantly looking at ways of squeezing more grant into less memory, which means that we always have to be profiling everything. And that's why that demo that I had at the beginning showing these are all of the things that we do is so interesting. Because we have to do this to compete with these PHP applications that take nothing. Even though we provide a thousand other features, people still want to be able to install this on a $10 server. So I'll talk about this memory profiler gem that uses a new Ruby 21 APIs that Oman and Koichi built into MRI and give you a little crash course on it. So this is something you all could do right now for your Rails app if you wanted to. There's not a lot of code here. I'll make sure I publish these slides so you can kind of cut and paste this and run this yourself. But what this does is it measures how many objects you're allocating on boot of your Rails app and how many are being retained in memory after boot. And the answer is a lot. So it breaks stuff into two types of reports. You get an allocation report, which is the amount of objects and memory that was allocated within the block. And that may not have a large impact on the process memory, but it has direct impact on performance. The more you're allocating stuff, the more work you're doing, the slower stuff is. That's pretty simple. So if you allocate less stuff, you're going to be doing less work. And then it's going to be faster. So when we look at the discourse boot at allocation, we can see the staggering amount of bytes that we allocate and which files are responsible for allocating active support dependencies. Likes to allocate a lot of memory when it's loading up stuff. And interesting one there, which is number two, RubyGem specification. I mean, why are we allocating so much memory in specifications? Seems a little bit odd, I mean, do we really need this? And the MimeTypes gem, which is wonderful at allocating a lot of memory. We can then break it down and look at the strings that actually got allocated. And we can see that we love allocating new lines. So on boot discourse, we'll allocate the new line string 40,000 times. And we also love Ruby, so we allocate Ruby 3,157 times. And it's the same string, right? So that's part of the reason I was asking Matz's questions yesterday, of why can't the actual programming language solve this for us? But we can solve it ourselves, because these libraries are all open source. We can contribute, we can fix this, and we can follow new patterns to make sure that these kind of problems don't happen. And there are very actionable reports that we can then run afterwards and see, wow, I just fixed that. I removed those 1,000 allocations from that one line, or 20,000 allocations from that one line. So we can zero in on where it actually happens. Now, retention is something that relates directly to your long term performance and memory of your app. The more objects are in the heap, the more memory you're gonna take, the slower your GC is gonna take, and so on. So you do, definitely for us, having low memory in our application is paramount, because without it, people can't install on a one gigabyte digital ocean slice. So seeing retention, you can see where after the block run, where memory is stuck. And you can see we've got a bunch of threads, and each thread is taking a meg. So that they're stuck, bits of memory allocation that. So we have to be careful not to run too many threads, because every thread that we run is gonna be very expensive. We can see that the MIME types jam is retaining that amount of memory, like two megs of memory. So your process would be two megs lighter if it didn't have the MIME type gem, but it's actually a little bit more, because there's fragmentation and all sorts of other stuff, but that's how we can look at all of these various gems, and you can find, if I got a gem that's really consuming large amounts of memory, we've removed most of them from discourse, except for the ones that we have to keep, so there's not much for you to see here. But I'm sure if you run your app, you'll see a bunch of gems that it'll be very strange for you that they're holding 10 megs or whatever. We can also look and see what types of classes and memory retained by gem is very interesting to see which gems are the most offensive gems when it comes to memory retention. And now I'll talk through a little optimization under the discourse using memory profiler and techniques like that to give us a faster feature in Rails. So I'm going to be talking about the pluck feature. So if you're not familiar with Rails, what pluck does is it's trying to select these two columns into an array. So you want to get an array of columns back with the data. You don't care about anything else, just an array of arrays. Pretty simple. So I want an array of arrays that has two columns and ten rows. And to get that from active record, I have to allocate 286 objects and I have to allocate 25ks of RAM. Now that's, is that high? I don't know, let's check. So I look inside memory profiler and I see that the string product is allocated 28 times when I was not even selecting it. And then I see the string price is allocated three times. And I see that active record result is allocating this amount of memory. So I go, okay, so what if I had no active record? So if you notice the object count, so it went from 286 down to 44. So there's an enormous amount of objects less. And look at the memory, it went down from 25k to 3.7. But still there's something odd there. There are 21 strings allocated and I didn't even select a string. So you have a little think about that and yeah, well, you're selecting a price, but really the PGGEM used to return price in string and then you convert it on your side to an int, which is kind of silly. So last, Keneson PG018 added a feature that allows you to do this mapping inside the gem itself, which means we're doing all of this casting native and then we're able to only allocate 13 objects and 1.1k down from 25k of allocation. So that's an enormous difference from where we were when we started. So I decided to backport this new type mapper into active record and use it at discourse. The history was we have this forum of trolls that like trolling us and one of the things they did was they created a topic with 40,000 posts. And we have a pretty silly implementation in one place that is selecting the post number for every post in that topic. So every time you go to this page, you're selecting 40,000 numbers out of the database. Now, obviously we have to fix that, but I wanted to have something that gives me a bit of a stop gap and just improves performance for all of these kind of cases. So I built what I call Fastplug, which is completely backwards compatible with Rails 4.1 and 4.2, uses a new PG type map. It reduced the allocations down from 286 to 198 for that block. And the memory allocation went down. And it will not be backported into Rails 4.2, so you can forget about it. The only way you can get these kind of things is to monkey patch. And the reason it won't is because Rails is a big mature project. And proving that something like this will work in every single case takes a long time. So that is one of the reasons that we monkey patch. So we're working with Sean and Erin to try and get these fixes in. Sean has a patch that's using the type mapper that is going to come in to Rails 5. But we need to keep looking into things. I wanted something fixed today. So literally I came up with the idea and two days later it was in production. So that's how we have very good test coverage. So we caught all of the edge cases there. So you can borrow that if you want. You can see a graph, and this is where the graph is very interesting. On the left hand side you can see pluck. And on the right hand side you can see fast pluck, which is compatible. So at the beginning it's performing quite close, just a little bit faster. But then the more data you select, the more significant the difference is. So at some point it becomes 1,000 rows, it becomes two times faster. And 10,000 rows it becomes four times faster. And these are pretty narrow rows, so the wider rows would mean even more. Which brings me to this, why do we love Active Record so much? And this is why we love Active Record so much. This is the paramount example that explains why we love it. And we love it because we can compose stuff. If the color is green, then add this to the work clause. And if the max speed is that, add that to the work clause. And pass these little things around and build them up. And that is the big cell that Active Record has. And it's simple, elegant, flexible. It's not SQL. So if you love SQL, this is kind of a bit weird to you. And it is sadly a bit inefficient. So we need to work on making it more efficient. But we can do this by hand. We did this by hand. We were doing this for many years by hand. And this is how it looks. And this is terrible. We hate this code. This code is very, very horrible code. And or we're variables, and this, and all sorts of ifs that are impossible to understand. And it's incomprehensible. It's fast, risky, and yes, SQL. So for a discourse, sometimes we need to run SQL. So we built an alternative to do things where we need to run SQL, but we want it to be nice. And we want to be able to use it in a clean way. So I built this SQL builder object. It's only like 100 lines of code or something. It's not talking a large amount of code here. And it allows us to compose queries as well, like we were able to in Active Record. And it allows us to get objects out of it like we can at Active Record. So I can get car objects out of it, instead of getting rows that I need to cast and move around and whatnot. And it's yes SQL, it's fast, it's sane, it's tiny, and it's in our source code base. And when I say how fast is this? So I was running a comparison here. This is thousands of operations per second, two columns per row of basic object mapping. So we're just selecting objects out of the database. So when you're selecting one object out of the database, you can see that it's only 40 or so percent faster. So there's definitely room to improve it compared to the blue line, which is raw performance. But then Active Record is like less than a half the speed of what it would be in raw. At 100 rows, it's starting to get closer to the speed of doing it. And in 1,000 rows, we're actually beating, we're beating coding it by hand. And the reason we beat coding it by hand is because one demo was using the new type mapper and the other demo wasn't using the type mapper. So because we started using the type mapper, then we're catching up on performance. But the thing there is, it's not really a bug. That gray line at the bottom at 1,000 rows is the numbers for Active Record once you're selecting a large amount of data. And I'll tell you a bit of a story. When I was working at Stack Overflow four years ago, we had this problem of stuff was slow and we had to optimize performance. And there was this quote from my blog post that is still relevant today. So I started measuring and I saw that the query is taking 12 milliseconds and the profiling showed the block is taking 90 milliseconds. So there was all this time lost between what SQL was doing and what the app was doing. And this is what we did at Stack Overflow. We built this lightweight object mapper called DAPA. And it had one goal and it was to be the fastest object mapper that the .NET framework had. And that meant that we measured every single other framework out there. We compared it to every single framework out there and its goal was to be fastest. And use every single trick that it could come with to be fastest at this particular problem which is taking SQL queries and giving you back objects. That's it. And it just looks like this. You do connection.query.dog and you give it some data and it gives you back a list of dogs, basically. And I believe that Active Record needs something like this at the bottom. To do all of the work that it's doing with SQL. At the moment, there is no blessed way of running SQL statements and getting back Active Record objects that is efficient, that is isolated from the rest of the jungle that is Active Record and that is standalone and reusable elsewhere. And if we had what I like to call Active SQL and you had Active Record built on top of Active SQL, you could do stuff like this. You could select 10 car objects very efficiently and then go to Active Record and save them. So that is where I guess, if I'm looking at the future of where I'd like to see Active Record, I'd like to see it in this kind of state where we take the problem of object mapping and pull it out of Active Record into its own entity and make that really, really, really fast. Okay, I'm gonna move on to a different topic altogether which is job scheduling. I know it's a bit of a segue from all of this Active Record stuff. But at Discos, we've got this problem of job scheduling. We've got jobs that need to run daily, hourly, minutely. We've got jobs that we just need to queue to happen later, like send an email to this person, send an email to that person. So what we have is a little bit different than what in general people would have in their Rails app. And I think these concepts are very strong. We have three different ways of scheduling jobs. We have regular jobs. We have lightweight jobs. And we have background jobs. So background jobs you all know about at SideKick, you're all using it. Regular jobs, some of you may use it and you'd be using stuff like SideTick but we wrote our own which is ultra efficient and doesn't have a lot of the issues that a lot of these other gems have. We tried the other gems first but they just did not answer our problem. So you could look at that and extracting that from Discos if you're looking for an adventure. And we have lightweight jobs which very few of you have heard about. And I think that's very cool. So I'll expand a bit on it. So lightweight job, we're able to extend Unicorn and tell Unicorn to run something after it's finished with the request. So a great example of this would be, I need to track a view for the page. So what I do is I tell it look, defer, run this later. And then once I've done and I've given the request back to the client, I can run that little piece of code. And that piece of code is a piece of code that I don't need to run within my request. So it becomes a lot more efficient. And similarly, there are a bunch of stuff that we try to do between requests which is one area where we're struggling a bit to upgrade to two-two. We run the GC between requests, for example. So the GC work does not impact your requests in general. And it's all amortized and put between requests. Now, Ruby 2.2 does it a lot differently in that you get a lot shorter pauses, but it's spread out through the lifecycle of your app. So in general, that's gonna be a lot better for you unless you're used to doing stuff like this where you're just pulling all of this work and trying to run it between requests where it is not necessarily better for you. So we're still struggling with how to kind of marry these two things together. And other thing that you noticed that I was able to debug JavaScript in our big JavaScript app. So we have source maps in production. We use sprockets, so we're still using that, but we patch it to bits and get it to generate source maps for us. And that's how I was able to show you that demo at the beginning that was debugging through JavaScript code. And that means that any way that has discourse running, I can just debug the JavaScript, which is really, really, really handy. Another thing that we have, which I think is a building block that is a little bit missing, is the idea of an anonymous cache. So we've got the front page of discourse. It's 99% of the time it's accessed by anonymous and then 1% of the time it's accessed by a logged on user. So caching becomes very, very complicated. How are you gonna cache this thing? So we have rack middleware that sits at the beginning of our rack stack that takes care of doing that and makes a decision of whether we need to cache it or not. And then uses Redis as the backend to do that. And the performance difference when we implemented this was enormous. Like if you get anonymous requests, we can serve them at, you know, more than an order of magnitude faster, like 50 times faster in some cases. So the logic is tricky, which is why we didn't implement this in nginx at the moment. You could potentially do that, but it was fast enough to do in Rails and it's rack middleware and it's available. You can pull these things out of discourse and see how it's done. And another piece that I don't have too much time, but I'll go through is that we're optimizing for development. The development story at discourse is as important for me as the production story in some cases. Like if I'm hitting the wall while developing and not having a fun time, then I can't do work. So I built things to make my work a lot more fun. And that is Rack Mini Profiler. We use Better Errors by Charlie Somerville, which is really awesome to look at errors when they happen. Logs to look at the logs and have fast browser reload times, which I'll talk about, and rake autospec, which I'll talk about. And live CSS refresh, thanks to message bus, because we have this new building block. So when you look at rake autospec, some of you may be familiar with Guard, but this is a little bit different. And I'll show you why it's different because I think everybody should see this because it's cool. Okay, here we go. So we have 5,000 specs at discourse. And they take five minutes to run. And I wanna start working on something. So I don't know. I wanna work on the user spec. And what I really wanna do to the user spec is I'd like to break this here. Ah, so my spec runner figured out that I was, I saved that file and now it's running the specs for that file. I didn't need to tell it to do anything. It just did it. And it told me that, and now it's running all the specs for the file. Oops, something failed. Ah, okay. I know, it was a different file that I did it. So maybe I'll go to the user controller file and I'll try and edit it here and I'll fix it here. But this remembers now that I was, that I had a failed spec from before and it'll try and run it to see if my change on this file fixed that file. And okay, now it's time to continue. Let's not do this anymore. And now it'll run it and it'll continue. So that's how I live with specs. I don't have to run any commands to run my specs and tests. They're constantly running. I can use puts debugging if I need to. So I can just at any point in time figure out what's going on, output variables. Did I, I didn't undo it. Anyway. So it becomes much, much simpler to debug. And another thing that I've got during development and production for that matter, I don't know if I showed you, but we've got this application called Logstar. So I can see at any point in time, this is rack middleware as well that you can include. I can look at all of the errors that I've seen and get back traces. I can see environment as well of who made this request. And this is also useful in development because then while I'm developing, I can just have another browser window that shows all of the stuff that Rails is running. And the big advantage over the console window that you have now is that I get back traces for everything. So I can see a back trace of every statement as it's happening. So for debugging, that makes it much, much simpler. And another thing you may have noticed in development. Discourse is a huge, huge JavaScript application. We've got 900 JavaScript files. But I just pressed reload and it was real fast. So what you're probably thinking is, ah, yeah, you're just cheating. You can't debug this thing. How can you debug this thing? You've probably got it all in one file and that's why it's fast, which is true. But we can still debug this thing. So I can go to the user card template or I can go to anywhere I want. Anyway, I can go to this user action file. So I've got all of these files that I can see, but how did we pull this little magic trick? And I'll go through that in a second. Additionally, I've got mini-profiler in development so I can see why stuff is taking so long. And I'll go back to the presentation and show you some of the tricks here that make that happen. I think I, yeah, I'll start it again. Cool. So this trick of the fast debugging is something you all, if anybody's got, who's got a big JavaScript app? Raise your hand. And who has two second reloads in development? It's awesome, a few of you. And who's able to debug in those two second, who has one giant file? Everybody that has two second development. So this is how we do this trick. We have 9.15, so that's more than probably all of you, which is a lot. And now, if we didn't have our hacks, you can see it would have taken us 30 seconds to load that page. Now, that's not really acceptable. That's why I've got that emoji, which we need. We need this emoji, just saying. So the general thing we do is we take debug and we turn it true to fast and everybody's happy and it's fast, but we can't debug. But here's the magic trick that we use. There is this feature in JavaScript that allows us to eval a statement and then at the end we give it a little comment saying what its logical name is. So we go and wrap every single snippet of JavaScript in an eval and that's how the wonderful debugging works there. So this is supported everywhere. Like if you've got I11, Firefox or Chrome, you've got this. So you can debug your JavaScript today. It's very easily back-ported into the Asset Pipeline so I hope we can include this in Rails fairly soon. And it's fast. You saw how fast. And now Robin built this so that is a big thanks to Robin for building it. So I guess the conclusion here that I wanted to give for the talk is that you can choose your own adventure. We can make Rails faster. We can make Ruby faster. You can experiment. You can add gems. You can bolt on stuff that will make your application work a lot better. Hopefully you learned a few interesting concepts and you can take them and extract them from discourse or use them yourself. And it's very important to keep having fun and to experiment while we're doing all of this. And these are the things that I showed and not necessarily things that nobody's come out with this magical version of Ruby that is suddenly gonna erase all of the inefficiency from the libraries. I'm afraid that's not gonna happen. So you could be dreaming and Koichi can go and make Ruby two times faster but he's not gonna make it 50 times faster. The libraries are what need love and the libraries are things that we can work on today. We don't need a new version of Ruby for that. So we can make our application faster. We can make our frameworks, consume less memory and we have the tooling to do it. And I'm just gonna go to questions now which means I'll go to the chat and see if there were any questions there. So let's see. Let's see if this still works. I'll go in. Oh, wow. There are a lot of people in the chat room. Nice, how do you check for online offline state? Okay, so I'll just go through this. How about server-side events? People keep putting questions here. I can't read. Stop it. Okay. Testing, will it crash? Will it crash? Did it crash? Having a not-so-lightweight job running a lightweight queue. Okay. This is a good question. I'll repeat it. What would the potential consequences of having a not-so-lightweight job running in a lightweight queue? How do you define it as lightweight? Is it as long as it doesn't impact the result of the request? So we have to be totally careful. If you put a real long-running job in a lightweight queue, you're gonna mess stuff up big time because you're gonna choke your Unicom worker and it's not gonna be able to process stuff and the Unicom master will go and terminate it and it's just havoc will ensue. So what we decide is like, there's a very strong performance criteria of what makes a lightweight job. Ah, cool. It's moving. How to be more handsome. Eval is dangerous. Can passenger run lightweight jobs? That is something that we've mentioned with Hung Lee. He's talked about, I've talked about that before. He doesn't have a proper story yet for dealing with forking of lightweight jobs, but I think he'd be open to adding something like that. My recommendation would be to open a discussion with him directly on his mailing list. Okay, is there any reason to keep Node.js and ROR? Are you looking at Puma instead of Unicom? Okay, two questions. I'll try and answer these two questions. So do I need to keep Node.js? We actually ship discourse with Node.js and we ship it with Node.js because we use Aglify.js, using Node.js to compile our assets because it's a lot faster than doing it via the Ruby process. But I love the idea of having one process that can do everything. Because when you have one process that can do everything, it's very, very easy to deploy. If I need to scale the amount of message bus stuff, then I can just easily just deploy the same app on another five machines and put an HA proxy rule that just farms the message bus traffic there and it's all done. I don't need to think about deploying multiple different types of apps. I've only got one type of app to think about. So I think that with stuff like message bus, definitely if you're using it for this, there's no reason to bring another tool into the stack that you need to learn. You can use Ruby to do these things. How about server-side events instead of message bus? Instead of, so server-side events can work, but it's just sugar on top of what long polling is anyway. So at the bottom, you're just running long polling. So you're not really going to win much. You could potentially use server-side events. In message bus, you'd need to implement an extra backend for it. I'd be completely open for a pull request that adds that as long as you measure that it's actually helping. And I'm not confident that it would help that much because at the end it's just an HTTP request that's hijacked just like the long poll. How does distributed cache compare to Redis or Memcache? And the difference between that is that you don't have to go to Redis. When I have distributed cache and message bus, I am not going to another service to figure out what the data is. I'm going local in my process. So it is much, much, much faster. There's no I.O. when I go to look stuff up in the cache. And that's what makes it magical that, you know, you've got these high-performance structures without needing all of the, needing to have used Redis immediately to get that data. Do you have any recommendations for profiling Rails boot time? My app currently boots in 30 seconds and I would like to understand why. And I'll publish my slides and my recommendation would be run memory profiler because that will first of all give you a good picture of where stuff is allocated. And the other thing that you should run is run stack prof which I'm unbuilt to see where time is being spent when you're loading your app. The, I guess the key is that little baby script that was in my thing. You can use that just to boot and then you can measure that little chunk of code. So it's only like four lines of code to boot a Rails app and then you just wrap it in whatever you need to figure out what it's doing. Now there are lots of tricks that you can do. Like you can just do a binary search to see which gems are causing the problem if you need to. All sorts of tricks like that. DHH, how do you think? I don't think DHH is on the message on this list. I'm not sure. Flame graph view showed you as impressive. Is there something similar graphically speaking for object allocations? Interesting. I need to talk to you but I'm not sure. I'm not, like how would you present the information in object allocations is a hard thing. Like what it does is it displays stack side by side. So if you can give this script the stack side by side it can render it for you. So as long as you can give it that information I think it would be able to be used for memory if you can represent it conceptually like that. I have a Backbone.js app wrapped in a Rails app to run it. Every time I deploy it takes forever to compile assets. What can I do to speed this up? Look at our rake task that we have for asset pre-compile. It uses Node.js directly instead of using Aglify.js. What we found was that Aglify.js was taking six times slower than using via Ruby, was taking six times slower than doing it via Node. We don't know why there's a Rails Google Summer of Code working on this problem and hopefully we'll get to the bottom of it. In the meantime you can use Node to do that. And you can use the discourse source code to figure out how to port that into your rake task so you can do that. Will Logster Jam be able to detect an error containing within begin and rest? I don't understand H&M's questions. I'm not sure. Oh cool. Anyone suffers from PG duplicated statements when provisioning extra dinos on Heroka or when I'm skipping that question? Why did you use Rails for discourse? Oh, this is a good question, especially half of the function I have in the browser. So when we started discourse, the two contenders were Node and Rails. And we went with Rails, we went with Ruby because Ruby wasn't cool. Which is, it's cool not to be cool. Because Node was cool then, but we didn't know where it would be in a year or in two years. And Node had 50 frameworks to choose from and we didn't know which one to choose. Express was kind of winning, but we weren't sure about that. And on the other hand, like Rails was well established, it's a framework that everybody uses. We know it's gonna be around. Personally, I love Ruby more than any other programming language. So it just fit very well for us. And that's why we went with it. Can we invite DHH to RDR? We'll make sure Winston invites DHH to RDR. And we'll see 2016. Are there any thoughts about shipping discourse metrics via StatsD? And generally, how do you feel about straight line? I didn't mention this in this talk, but we extensively use StatsD. And I recommend if any of you haven't looked at StatsD to have a look at StatsD. And we ship metrics from all sorts of spots into StatsD. And then we graph them with a tool called Grafana, which is some nice UI sugar on top of graphite. And I think that is critical when you have an app doing stuff like graphing memory over time. Like how do you know that you've got a memory leak? And the shape of the memory leak makes a big difference. Like if your memory is just going up in a slow trend, then it means one thing. If it's spiking every 30 minutes, it means another thing. So by having these long-term graphs, you're actually able to figure out what it is that is making your app leak memory. And it's just that visual representation is really important. So I think graphite and having long-term graphs of StatsD is very, very critical. And I actually created a Docker container that you could all use to just play around with graphite, which you can just launch with Docker run if you're familiar with Docker and just start sending metrics to and graph it. So I took all of the little bits and pieces that you need and put them in a container. And I think we're out of time. So I'm going to say a big thank you. And if anybody has any questions, hit me up, and I'll try and answer. I'll leave the chat on as well, so I'll check it again in a few hours. If there's any questions, I'll try and answer. Cool. Thank you.