 Hey, you guys ready? Thank you guys so much for coming. This is awesome. I was really, when they were putting together the schedule, I said, make sure that you put us down in the caves of Moria. So thank you guys for coming down and making it. I'm Tom. This is Yehuda. When people told me I was signed up to do a back-to-back talk, I don't know what I was thinking. Yep. So we want to talk to you today about Skylight. So just a little bit before we talk about that, I want to talk about us a little bit. So in 2011, we started a company called Tilda. It's a shirt. Leia made me self-conscious because this is actually the first edition and it's printed off-center. Well, either I'm off-center or the shirt's off-center. One of the two. So we started Tilda in 2011. And we had all just left a venture-backed company. And that was a pretty traumatic experience for us because we spent a lot of time building the company. And then we ran out of money and it sold to Facebook. And we really didn't want to repeat that experience. So we decided to start Tilda. And when we did it, we decided to be DHH and the other people at Basecamp were talking about being bootstrapped and proud. And that was a message that really resonated with us. And so we wanted to capture the same thing. There's only one problem with being bootstrapped and proud. And that is in order to be both of those things, you actually need money, it turns out. It's not like you just say it in a blog post and then all of a sudden you are in a business. So we had to think a lot about, OK, well, how do we make money? How do we make money? How do we make a profitable and most importantly, sustainable business? Because we didn't want to just flip the Facebook in a couple of years. So looking around, I think the most obvious thing that people suggested to us is, well, why don't you guys just become Ember Inc. raise a few million dollars, but your business model mostly prayer. But that's not really how we want to think about building open source communities. We don't really think that that necessarily leads to the best open source communities. And if you are interested in more in that, I'd recommend Leia Silver, who is one of our co-founders. She's giving a talk this afternoon, sorry, Friday afternoon about how to build a company that is centered on open source. So if you want to learn more about how we've done that, I would really suggest you go check out her talk. So no. So no Ember Inc. Not a lot. So we really want to build something that leveraged the strengths that we thought that we had. One, I think most importantly, a really deep knowledge of open source and a deep knowledge of the Rails stack. And also, Carl's, it turns out, is really, really good at building highly scalable big data systems. Lots of Hadoop in there. So last year at RailsConf, we announced the private beta of Skylight. How many of you have used Skylight? Can you raise your hand if you have used it? Okay, many of you. Awesome. So Skylight is a tool for profiling and measuring the performance of your Rails applications in production. And as a product, Skylight, I think, was built on three really, three key breakthroughs. There were three key breakthroughs. We didn't want to ship a product that was incrementally better than competition. We wanted to ship a product that was dramatically better, quantum leap, order of magnitude better. And in order to do that, we spent a lot of time thinking about it, about how we could solve most of the problems that we saw in the existing landscape. And so those breakthroughs are predicated, sorry, delivering a product that does that is predicated on these three breakthroughs. So the first one I wanna talk about is honest response times. Honest response times. So DHH wrote a blog post on what was then the 37 Signals blog, now the Basecamp blog, called The Problem with Averages. How many of you have read this? Awesome. For those of you that have not, how many of you hate raising your hands for the presentations? So. So for those of you that- Just put a button in every seat. Press the red button if you have. So if you read this blog post, the way it opens is, our average response time for Basecamp right now is 87 milliseconds. That sounds fantastic. And it easily leads you to believe that all is well and that we wouldn't need to spend any more time optimizing performance. But that's actually wrong. The average number is completely skewed by tons of fast responses to feed requests and other cast replies. If you have 1,000 requests that return in five milliseconds and then you can have 200 requests taking 2,000 milliseconds or two seconds, you can still report a respectable 170 milliseconds of average. That's useless. So what does DHH say that we need? DHH says the solution is histograms. So for those of you like me who are sleeping through your statistics class in high school and college, a brief primer on histograms. So histogram is very simple. Basically you have a series of numbers along some axis and every time you're in that number, you're in that bucket, you basically increment that bar by one. So this is an example of a histogram of response times in a Rails application. So you can see that there's a big cluster in the middle around 488 milliseconds, 500 milliseconds. This isn't a super speedy app, but it's not the worst thing in the world. And they're all clustered and then as you kind of move to the right, you can see that the response times get longer and longer and longer. And as you move to the left, response times get shorter and shorter and shorter. So why do you want a histogram? What's the most important thing about a histogram? Well, I think it's because most requests don't actually look like this. Most endpoints don't actually look like this. Right. If you think about what your Rails app is doing, it's a complicated piece, right? It turns out Ruby turned complete. You can do branching logic. You can do a lot of things. And so what that means is that one endpoint, if you represent that with a single number, you are losing a lot of fidelity to the point where it becomes, as DHH said, useless. So for example, in a histogram, you can easily see, oh, here's a group of requests and response times where I'm hitting the cache and here's another group where I'm missing it. And you can see that that cluster is significantly slower than the faster cache hitting cluster. And the other thing that you get when you have a distribution, when you keep the whole distribution in the histogram, is you can look at this number at the 95th percentile, right? So the way to think about the performance of your web application is not the average, because the average doesn't really tell you anything. You wanna think about the 95th percentile because if it's not the average response time, that's the average worst response time that a user is likely to hit. And the thing to keep in mind is that it's not as though a customer comes to your site, they issue one request and then they're done, right? As someone is using your website, they're gonna be generating a lot of requests. And you need to look at the 95th percentile because otherwise, every request is basically you rolling the dice that they're not gonna hit one of those two-second, three-second, four-second responses, close the tab and go to your competitor. So we look at this, here's a crazy thing. Here's what I think is crazy. That blog post that DHH wrote is from 2009. It's been five years and there's still no tool that does what DHH was asking for. So we, frankly, we smelled money. We were like, holy crap. Why is the slide not green? Yeah, it should be green in dollars. I think he has the dollars, the make it rain effect I should have used. So we smelled blood in the water. We were like, this is awesome. There's only one problem that we discovered. And that is, it turns out that building this thing is actually really, really freaking hard. Really, really hard. So we announced the private beta at RailsConf last year. Before doing that, we spent a year of research spiking out prototypes, building prototypes, building out the beta. We launched at RailsConf and we realized we made a lot of problems. We made a lot of errors when we were building the system. So then after RailsConf last year, we basically took six months to completely rewrite the backend from the ground up. And I think tying into your keynote, Yehuda, we were like, oh, we clearly have a bespoke problem. No one else is doing this. So we rewrote our own custom backend. And then we had all these problems and we realized that they had actually already all been solved by the open source community. And so we benefited tremendously by having a shared solution. And so our first release of this was really very bespoke. And the current release uses a tremendous amount of very off the shelf open source projects that just solve the particular problem very effectively, very well. None of which are as easy to use as Rails, but all of which solve really thorny problems very effectively. So let's just talk, just for your own understanding, let's talk about how most performance monitoring tools work. So the way that most of these work is that you run your Rails app and running inside of your Rails app is some gem, some agent that you install. And every time the Rails app handles a request, it generates events and those events, which include information about performance data, those events are passed into the agent. And then the agent sends that data to some kind of centralized server. Now it turns out that doing a running average is actually really simple, which is why everyone does it. You basically can do it in a single SQL query. All you do is you have three columns in database, the endpoint, the running average, and the number of requests. And then so you can, those are the two things that you need to keep a running average, right? So keeping a running average is actually really simple from a technical point of view. I don't think you could do it in JavaScript due to the lack of integers. Yes, you probably wouldn't want to do any math in JavaScript, it turns out. So, so we took a little bit different approach. You could, do you want to go over the technical architecture? Sure, so when we first started right at the beginning, we basically did a similar thing where we had a bunch, your app creates events, most of those start off as being active support notifications, although it turns out that there's very limited use of active support notifications. So we had to do some normalization work to get them sane, which we're gonna be upstreaming back into Rails. But one thing that's kind of unfortunate about having every single Rails app have an agent is that you end up having to do a lot of the same kind of work over and over again and use up a lot of memory. So for example, every one of these things is making HTTP requests. So now you have a queue of things that you're sending over HTTP in every single one of your Rails processes. And of course you probably don't notice people are used to Rails taking up hundreds and hundreds of megabytes. So you probably don't notice if you install some agent and it suddenly starts taking 20, 30, 40, 50 more megabytes. But we really wanted to keep the actual memory per process down to a small amount. So one of the very first things that we did, we even did it before last year, is that we pulled out all that shared logic into a separate process called the coordinator. And the agent is basically responsible simply for collecting the trace. And it's not responsible for actually talking to our server at all. And that means that the coordinator only has to do this queue, this keeping a bunch of stuff of work in one place and doesn't end up using up as much memory. And I think this ended up being very effective for us. And I think that low overhead also allows us to just collect more information in general. Yep. Now after our first attempt, we started getting a bunch of customers that were telling us that even the separate, so the separate coordinator sort of was a good thing and a bad thing. On the one hand, there's only one of them. So it uses up only one set of memory. On the other hand, it's really easy for someone to go in and PS that process and see how many megabytes of memory it's using. So we got a lot of initial complaints that said, oh, your process is using a lot of memory. And I spent a few weeks, I know Ruby pretty well. I spent a couple of weeks. I actually wrote a gem called allocation counter that basically went in and tried to pinpoint exactly where the allocations were coming from. But it turns out that it's actually really, really hard to track down exactly where allocations are coming from in Ruby because something as simple as using a regular expression in Ruby can allocate match objects that get put back on the stack. And so I was able to pair this down to some degree, but I really discovered quickly that trying to keep a lid on the memory allocation by doing all this stuff in Ruby is mostly fine. But for our specific use case where we really want to be telling you you can run the agent on your process, on your box, and it's not gonna use a lot of memory, we really needed something more efficient. And our first thought was, we'll use C++ or C. No problem, C is native, it's great. And Carl did the work. Carl is very smart. And then he said, it is now your turn. You need to start maintaining this. And I said, I don't trust myself to write C++ code that's running in all of you guys' boxes and not segfault. So I don't think that, that doesn't work for me. And so I noticed that Rust was coming along and what Rust really gives you is it gives you the ability to write low level code, I'll loss your C++ with manual memory management that keeps your memory allocation low and keeps things speedy, low resource utilization, while also giving you compile time guarantees about not segfaulting. So again, if your process is randomly started segfaulting, because you installed the agent, I think you would stop being our customer very quickly. So having pretty much 100% guarantees about that was very important to us. And so that's why we decided to use Rust. Just keep going, keep going. So we have this coordinator object. Basically the coordinator object is receiving events. So the events basically end up being these traces that describe what's happening in your application. And the next thing, I think our initial work on this, we use JSON just to send the payload to the server. But we noticed that a lot of people have really big requests. So you may have a big request with a big SQL query in it or a lot of big SQL queries in it. Some people have traces that are hundreds and hundreds of nodes long. And so we really wanted to figure out how to shrink down the payload size to something that we could be pumping out of your box on a regular basis without tearing up your bandwidth costs. So one of the first things that we did early on was we switched using protobufs as the transport mechanism and that really shrunk down the payloads a lot. Our earlier prototypes for actually collecting the data were written in Ruby. But I think Carl did like a weekend hack to just port it over to Java and got like 200X performance. And you don't always get 200X performance. If mostly what you're doing is database queries, you're not gonna get a huge performance win. But mostly what we're doing is math and algorithms and data structures. And for that, Ruby is, it could in theory one day have a good JIT or something. But today, writing that code in Java didn't end up being significantly more code because it's just algorithms and data structures. And I'll just note something about standardizing on protobufs in our stack is actually a huge win because we realized, hey, browsers, it turns out, are pretty powerful these days. They've got, they can allocate memory, they can do all type of computation. And protobufs libraries exist everywhere. So we save ourselves a lot of computation and a lot of time by just treating protobuf as the canonical serialization form. And then you can move payloads around the entire stack and everything speaks the same language. So you save the serialization, the serialization. And JavaScript is actually surprisingly effective at taking protobufs and converting them to the format that we need efficiently. So we basically take this data, the Java collector is basically collecting all these protobufs and pretty much it just turns around. And this is sort of where we got into bespoke territory before where we started rolling our own. But we realized that when you write a big distributed fault tolerant system, there's a lot of problems that you really just want someone else to have thought about. So what we do is we basically take these, take these payloads that are coming in, we convert them into batches and we send the batches down into the Kafka queue. And the next thing that happens, so the Kafka is basically just a queue that allows you to throw things into, I guess it might be considered similar to like something like MQP, it has some nice fault tolerance properties and integrates well with Storm. But most importantly, it's just super, super high throughput. So we basically didn't want to put any barrier between you giving us the data and us getting it to this as soon as possible. Which we'll, I think, talk about in a bit. So we, so basically Kafka takes the data and starts sending it into Storm. And if you think about what has to happen in order to get some requests. So you have these requests, there's maybe traces that have a bunch of SQL queries and our job is basically to take all those SQL queries and say, okay, I can see that in all of your requests you had the SQL query and it took around this amount of time and it happened as a child in this other node. And the way to think about that is basically just a processing pipeline. So you have these traces that come in one side, you start passing them through a bunch of processing steps and then you end up on the other side with the data. And Storm is actually a way of describing that processing pipeline in sort of a functional style and then you tell it, okay, here's how many servers I need, here's how I'm gonna handle failures and it basically deals with distribution and scaling and all that stuff for you. And part of that is because you wrote everything using functional style. And so what happens is Kafka sends the data into the entry spout, which is sort of terminology in Storm for the streams that get created. And they basically go into these processing things which very cleverly, cutely are called bolts. This is definitely not the naming I would have used but they're called bolts. And the idea is that basically every request may have several things. So for example, we now automatically detect M plus one queries and that's sort of a different kind of processing from just make a picture of the entire request or what is the 95th percentile across your entire app, right? These are all different kinds of processing. So we take the data and we send them into a bunch of bolts. And the cool thing about bolts is that again, because they're just functional chaining, you can take the output from one bolt and feed it into another bolt. And that works pretty well. And you don't have to worry about, I mean you have to worry a little bit about things like fault tolerance, failure, item potence, but you worry about them at the abstraction level and then the operation part is handled for you. So it's like a very declarative way of describing how this computation works in a way that's easy to scale. And Carl actually talked about this at very high speed yesterday and some of you may have been there. I would recommend watching the video when it comes out if you wanna learn more about how to make use of this stuff in your own applications. And then when you're finally done with all the processing, you need to actually do something with it, you need to put it somewhere so that the web app can get access to it. And that is basically, we use Cassandra for this. And Cassandra again is mostly, it's a dumb database, but it has high capacity, it has some of the fault tolerance properties that we want. We're just very, very heavy right. Like we tend to be writing more than we're ever reading. Yep. And then when we're done, when we're done with a particular batch, Cassandra basically kicks off the process over again. So we're basically doing these things as batches. So these are roll-ups is what's happening here. So basically every minute, every 10 minutes, and then every hour, we reprocess and we re-aggregate so that when you query us, we know exactly what to do. So we sort of have this cycle where we start off, obviously in the first five, in the first minute, you really want high granularity. You wanna see what's happening right now. But if you wanna go back and look at that from three months ago, you probably care about it like the day granularity or maybe the hour granularity. So we basically do these roll-ups in the cycle through the process. So this, it turns out building the system required an intense amount of work. Carl spent probably six months reading PhD theses to find theses, to find data structures and algorithms that we could use because this is a huge amount of data. Like I think even a few months after we were in private data, private beta, we were already handling over a billion requests per month. And obviously there's no way. Basically the number of requests that we handle is the sum of all of the requests that you handle and all of our customers handle. Right. Right, so that's a lot of requests. So obviously we can't provide a service, at least one that's not, we can't provide an affordable service, an accessible service if we have to store terabytes or exabytes of data just to tell you how your app is running. And I think also it's problematic if you store all the data in a database and then every single time someone wants to learn something about that, you have to do a query. Those queries can take a very long time. They can take minutes. And I think we really wanted to have something that would be very, where the feedback loop would be fast. So we wanted to find algorithms that let us handle the data at real time and then provide it to you at real time instead of these like dump the data somewhere and then do these complicated queries. So, hold on. So this slide was not supposed to be here. It's supposed to be a rail slide. So, whoa, I went too far. Okay, we'll watch that again, that's pretty cool. So the last thing I want to say is perhaps your takeaway from looking at this architecture diagram is, oh my gosh, these rails guys, they have completely, they jumped the shark, they ditched rail. I saw like three tweets yesterday. I wasn't here, I was in Portland yesterday, but I saw like three tweets that are like, I'm at RailsConf and I haven't seen a single talk about Rails. So that's true here too. But I want to assure you that we are only using this stack for the heavy computation. We started in Rails. We started, we were like, hey, what do we need? Ah, well people probably need to authenticate and log in and we probably need to do billing. And those are all things that Rails is really, really good at. So we started with Rails as basically the starting point and then when we realized, oh my gosh, computation is really slow, there's no way we're gonna be able to offer the service. Okay, now let's think about how we got there. I think notably, a lot of people who look at Rails are like, there's a lot of companies that have built big stuff on Rails and their attitude is like, oh, this legacy, terrible Rails app, I really wish we could get rid of it. If we could just write everything in Scala or Closure or Go, everything would be amazing. That is definitely not our attitude. Our attitude is that Rails is really amazing at particular, at the kinds of things that are really common across everyone's web applications, authentication, billing, et cetera. And we really wanna be using Rails for the parts of our app. Even things like error tracking we do through the Rails app. We wanna be using Rails because it's very productive at doing those things. It happens to be very slow at doing data crunching, so we're gonna use a different tool for that. But I don't think you'll ever see me getting up and saying, ah, I really wish we had just started writing the Rails app in Rust. That would be terrible, right? So that's number one is honest response times, which it turns out seems like it should be easy, requires storing an insane amount of data. So the second thing that we realized when we were looking at a lot of these tools is that most of them focus on data. They focus on giving you the raw data. But I'm not a machine, I'm not a computer. I don't enjoy sifting through data. That's what computers are good for. I would rather be drinking a beer. It's really nice in Portland this time of year. So we wanted to think about if you were trying to solve the performance problems in your application, what are the things that you would suss out with the existing tools after spending like four hours depleting your ego to get there? And I think part of this is just people are actually very, people like to think that they're gonna use these tools, but when the tools require you to dig through a lot of data, people just don't use them very much. So the goal here was to build a tool that people actually use and actually like using and not to build a tool that happens to provide a lot of data you can sift through. So probably one of the first things that we realized is that we don't want to provide, this is a trace of a request, you've probably seen similar UIs using other tools, using for example the inspector in Chrome or Safari. And this is just showing basically, it's basically a visual stack trace of where your application is spending its time. But I think what was important for us is showing not just a single request because your app handles hundreds, thousands of requests or millions of requests. So looking at a single request statistically is complete, it's just noise. And it's especially bad if it's the worst request because the worst request is really noise, it's like a hiccup in the network. Literally the outlier. It's literally the outlier. So what we present in Skylight, it's something a little bit different and it's something that we call the aggregate trace. So the aggregate trace is basically us taking all of your requests, averaging them out where each of these things spends their time and then showing you that. So this is basically, like this is like the statue of David. It is the idealized form of the stack trace of how your applications behaving. But of course, you have the same problem as before which is if this was all that we were showing you, it would be obscuring a lot of information. You wanna actually be able to tell the difference between okay, what's my stack trace look like for fast requests and how does that differ from requests that are slower? So what we've got, I've got a little video here. You can see that when I move this slider, that this trace below it is actually updating in real time. As I move the slider around, you can see that the aggregate trace actually updates with it. And that's because we're collecting all this information. We're collecting, like I said, a lot of data. We can recompute this aggregate trace on the fly. Basically for each bucket, we're storing a different trace and then on the client we're reassembling that. We'll go into that in a little bit. And I think it's really important that you be able to do these experiments quickly. Every time you think, oh, I wonder what happens if I add another histogram bucket if it requires a whole full page refresh, then that would basically make people not wanna use the tool, not able to use the tool. So actually building something which is real time and fast and gets the data as it comes, it was very important to us. So that's number one. And second thing, so we built that and we're like, okay, well what's next? And I think that the big problem with this is that you need to know that there is a problem before you go look at it, right? So we've been working for the past few months in the storm infrastructure that we built and makes it pretty straightforward to start building more abstractions on top of the data that we've already collected. It's a very declarative system. So we've been working on a feature called Inspections. And what's cool about Inspections is that we can look at this tremendous volume of data that we've collected from your app and we can automatically tease out what the problems are. So the first one that we've shipped, this is in beta right now. It's not out and enabled by default, but there is behind a feature flag that we've had some users turning on and trying out. And so what we can do in this case is because we have information about all of the database queries in your app, we can look and see if you have n plus one queries. You can maybe explain what n plus one query is. Yeah, so people know hopefully what n plus one query is, but it's the idea that you buy accident for some reason instead of making one query, you ask for all the posts and then you iterate it through all of them and got all the comments. So now you, instead of having one query, you have one query per post, right? And what I'd like to do is do eager loading where you say include comments, right? But you have to know that you have to do that. So there's some tools that will run in development mode if you happen to catch it like bullet. This is basically a tool that's looking at every single one of your requests and has some thresholds that once we see that a bunch of your requests have the same exact query. So we do some work to pull out binds. So if it's like where something equals one, we will automatically pull out the one and replace it with a question mark. And then we basically take all those queries and if they're the exact same query, repeated multiple times subject to some thresholds, we'll start showing you, hey, there's an n plus one query. And you could imagine this same sort of thing being done for things like, are you missing an index, right? Or are you using the Ruby version of JSON when you should be using the native version of JSON? These are all things that we could start detecting just because we're consuming an enormous amount of information and we can start writing some heuristics for bubbling it up. So third and final breakthrough, we realized that we really, really needed a lightning fast UI, something really responsive. So in particular, the feedback loop is critical, right? You can imagine if the way that you dug into data was you click and you wait an hour and then you get your results, no one would do it, no one would ever do it. And the existing tools are okay, but you click and you wait, you look at it and you're like, oh, I want a different view so then you go edit your query and then you click and you wait and it's just not a pleasant experience. So we use Ember, the UI that you're using when you log into Skylight, even though it feels just like a regular website, it doesn't feel like a native app, is powered all of the routing, all of the rendering, all of the decision making is happening as an Ember JS app and we pair that with D3. So all of the charts that you saw there in the aggregate trace, that is all Ember components powered by D3. So this is actually significantly cleaned up our client side code. It makes reusability really, really awesome. So I'll give you an example. This is from our billing page. The designer came and they had a component that was like the date component. And the- Seemed really boring at first. Seemed really boring. But this is the implementation, right? So you could copy and paste this code over and over again everywhere you go, just remember to format it correctly. If you forget to format it, it's not gonna look the same everywhere. But I was like, hey, we're using this all over the place. Why don't we bundle this up into a component? And so with Ember, it was super easy. We basically just said, okay, here's a new calendar date component. It has a property on it called date. Just set that to any JavaScript date objects. Just that you don't have to remember about converting it or formatting it. Here's the component, set the date and it will render the correct thing automatically. And so the architecture of the Ember app looks a little bit something like this where you have many, many different components, most of them just driven by D3 and then they're plugged into the model of the controller. And the Ember app will go fetch those models from the cloud, from the Java app, which just queries Cassandra and render them. And what's neat about this model is turning on web sockets is super easy, right? Because all of these components are bound to a single place. So when the web socket says, hey, we have updated information for you to show, it just pushes it onto the model or onto the controller and the whole UI updates automatically. It's like magic. And it's like magic. And when debugging, this is especially awesome too because I'll maybe show a demo of the Ember inspector using this. So yeah. So that Lightning Fast UI, reducing the feedback loop so that you can quickly play with your data, makes it go from a chore to something that actually feels kind of fun. So these were the breakthroughs that we had when we were building Skylight, the things that made us think, yes, this is actually a product that we think deserves to be on the market. So one honest response times, collect data that no one else can collect, focus on answers instead of just dumping data and have a Lightning Fast UI to do it. So we'd like to think of Skylight as basically a smart profiler. It's a smart profiler that runs in production. It's like the profiler that you run on your local development machine. But instead of being on your local dev box, which has nothing to do with the performance characteristics of what your users are experiencing, we're actually running in production. So let me just give you guys a quick demo. So this is what the Skylight, this is what Skylight looks like. Let me send this. There we go. So the first thing here is we've got the app dashboard. So this looks like our 95th percentile response time peak. Maybe you're all hammering it right now, that would be nice. So this is a graph of your response time over time. And then on the right, this is a graph of the RPMs, the requests per minute that your app is handling. So this is app-wide. And this is live, this updates every minute. Then down below, you have a list of the endpoints in your application. So you can see actually the top, the slowest ones for us, where we have an instrumentation API and we've gone and instrumented our background workers. So we can see them here and their response time plays in. So we can see that we have this reporting worker that's taking 95th percentile, 13 seconds. So all that time used to be inside of some request somewhere and we discovered that there was a lot of time being spent in things that we could push to the background. We probably need to update the Agony Index so that it doesn't make workers very high because spending some time in your worker is not that big of a deal. So then if we dive into one of these, you can see that for this request, we've got the time explorer up above and that shows a graph of response time, again 95th percentile. And you can, if you wanna go back and look at historical data, you just drag it like this and this has got a brush so you can zoom in and out on different times. And every time you change that range, you can see that it's very responsive. It's never waiting for the server but it is going back and fetching more data from the server and then when the data comes back you see the whole UI just updates. And we get that for free with Ember and D3. And then down below, as we discussed, you actually have a real histogram. And this histogram in this case is showing, so this is for 57 requests. And if we click and drag, we can just move this and you can see that the aggregate trace below updates in response to us dragging this. And if we wanna look at the fastest quartile, we just click faster and it will choose that range on the histogram. I think it's fastest third. Fastest third. And then if you click on slower, you can see the slower request. So this makes it really easy to compare and contrast, okay, why are certain requests faster and why are certain requests slow? You can see the blue, these blue areas, this is Ruby code. So right now it's not super granular. It would be nice if you could actually know what's going on here. But it'll at least tell you where in your controller action this is happening. And then you can actually see which database queries are being executed and what their duration is. And you can see that we actually extract the SQL and we denormalize it. So we normalize it so you can see exactly what those requests are, even if the values are totally different between them. Yeah, so the real query courtesy of Rails not yet supporting bind extraction is like where ID equals one or 10 or whatever. Yep. So that's pretty cool. One other thing is initially we actually just showed the whole trace but we discovered that obviously when you show whole traces you have information that doesn't really matter that much. So we started off by, we've recently basically started to collapse things that don't matter so much. So you can basically expand or condense the trace. And we wanted to make it not that you have to think about expanding or condensing individual areas but just you see what matters the most and then you can see trivial areas. Yep. So that's the demo skylight. We'd really like it if you checked it out. There is one more thing I wanna show you that is really freaking cool. This is coming out of Tilda Labs. Carl has been hacking, he's been up until past midnight getting almost no sleep for the past months trying to have this ready. I don't know how many of you know this but Ruby 2.1 has a new stack sampling feature. So you can get really granular information about how your Ruby code is performing. So I wanna show you, I just mentioned how it would be nice if we could get more information out of what your Ruby code is doing. And now we can do that. Basically every few milliseconds, this code that Carl wrote is going into the Ruby, into MRI and it's taking a snapshot of the stack. And because this is built in, it's very low impact. It's not allocating any new memory. It's very little performance hit. Basically you wouldn't even notice it. So every few milliseconds it's sampling and we take that information and we send it up to our servers. So it's almost like you're running Ruby Profiler on your local dev box where you get extremely granular information about where your code is spending time in Ruby per method for all of these things. But it's happening in production. So this is, so this is, we enabled it in staging. You can see that we've got some rendering bugs that's still in beta. And we haven't yet collapsed things that are not important for this particular feature. So we wanna hide things like framework code, obviously. But this gives you an incredibly, incredibly granular view of what your app is doing in production. And we think this is an API that's built into Ruby 211 because our agent is running solo level because we wrote it in Rust. We have the ability to do things like this. And Carl thinks that we may be able to actually backport this to older Ruby's too. So if you're not on Ruby 21, we think that we can actually bring this. But that's a TBD. So I think the cool thing about this in general is when you run a sampling profile, this is a sampling profiler. We don't wanna be burning every single thing that you do in your program with tracing. That would be very slow. So when you normally run a sampling profile, you have to basically make a loop. You have to basically create a loop, run this code a million times and keep sampling and eventually we'll get enough samples to get the information. But it turns out that your production server is a loop. Your production server is serving tons and tons of requests. So by simply taking a few microseconds out of every request and collecting a couple of samples, over time we can actually get this really high fidelity picture with basically no cost. And that's pretty mind blowing. And this is the kind of stuff that we can start doing by really caring about both the user experience and the implementation and getting really scary about it. And I'm really, like honestly, this is a really exciting feature that really shows what we can do as we start building this out. Once we've got that round work. So if you guys wanna check it out, skylight.io. It's available today. It's no longer in private beta. Everyone can sign up. No invitation token necessary. And you can get a 30 day free trial if you haven't started one already. So if you have any questions, please come see us right now or we have a booth in the vendor hall. Thank you guys very much. Thank you.