 I want to just quickly start by telling you a little story. A couple of years ago, anyone here open raiders fan? I'm so sorry for anything. So I'm not an open raiders fan either. I actually am the most pathetic football fan. I'm here at Jeff's fan, which means I've got about 114 years before the contract with Joe Nagan, the devil, because they convinced me that Super Bowl III expired on the four-raider winning season. But a few years ago, a friend of mine came and actually had raiders play-off tickets. You can remember that there actually was a time when the raiders were in the play-offs. And he said, why don't you come with me? It's in the Coliseum, and we're in the Black Hole. If you're familiar with the Black Hole, you're not familiar with the Black Hole. We know those pictures of open raiders fans, and everyone's wearing basic pink, has machetes, and guns. That's the Black Hole. Yeah, so he knows. So here I am at Jeff's fan going to stay in the Black Hole for a play-off game. And so what he did was he gave me a Brent Bolivnikov jersey, and if you don't know who Bolivnikov was, he was a famous wide receiver for the Grogon. And so I got there, and everyone thought I was the greatest. They thought it was really terrific, and they were high-fiving, and the raiders were doing something terrible, and secretly inside, I was cheering. But outside, I would put on my sad face, and everyone was consoling me. And it turns out they're all really great guys, as long as you're wearing open raiders. The reason that I tell you this story is that Lou Cerny was supposed to be here today. And Lou, unfortunately, is definitely ill. I was evidence by this email that I'm going to put up here that just sent me... Let's see if I can actually get it to show them. It's a little hard to do that on the phone. So, Lou sent me an email that said, Thanks for something in for me. Kyle's Luke is getting stuck with Rockingroll Lullaby at the last minute. I have no idea what that means. So Lou lives around New York, and is very smart technically. And he has sent me to you, not the founder of New Relic, and not particularly smart technically. So my name is Bill Platt today, and I'm the vice president of business development for New Relic. So, anyway, with that apology in advance, I'm going to take you through some of what New Relic can do, what Arcade can do, and I'm going to talk to you a little bit about how we use New Relic on New Relic to make sure that it's really highly performed, because it is a giant rail tap, and it does get a lot of data sent through it all the time, and we're trying to keep it coming. And so we use this tool. We don't have enough to manage our production servers. Does anyone hear? In fact, I know some people who raise their hand. Does anyone hear I've reviewed Arcade? A bunch of people. Cool. For those of you who don't know the background, New Relic is a company that's mounted on the premise that having a web application that actually performs and having a real application that actually scales is a really good thing, and that you need tools to be able to make that show, to be able to make that happen. And so, we'll back to my presentation today. And so we created RPM. RPM is for rail performance management, and it is a production performance monitoring tool for rails. And like I said, we use this extensively, so we have a staging environment that we throw all our new code into, and we watch as the staging environment crashes here and there, and we fix things. But even more importantly, once things pass the staging environment, we deploy into production. And it turns out, you know, I can't do the Java world originally, and when you deploy into production, maybe, I don't know, every six months, every nine months, if you're selling a package application, you can get into a bank, and the bank can go off and they can deploy it in their one window here where they will have to put new code into production. So, let me take a quick show of hands. How many people deploy to production once a month? At least. At least once a month. How many shoppiles do people need to go to? How many deploy to production once a week? Once a day. I don't know what's an hour. Really? How about your constantly deploying production? No. Listen, we do have a questioner that we deploy five, six times a day to production. It turns out deploying to production and monitoring that production running application is really critical. And getting information back to your developers is really critical so that your developers can quickly fix bugs when they come up, because once something is in deployment, that's when your customers are going to get effective performance issues. So, most points to whoever can aside from you, as you know, whoever can tell me where that quote came from. It is a song. Who wrote it? I'll give you a clue. It's not a lullabar. That's Rockwell. Rockwell, somebody's watching. So, monitoring, monitoring production apps is important. It turns out that a lot of people around us are using monitoring tools to monitor production apps. In fact, there's a recent Rails hosting survey that you may have seen that basically says right now 50% of all people in the survey, and I think there were 1,500 respondents, that were playing after production, actually used tools to monitor in production, which is pretty cool, because it's not always obvious that your testing and everything else that has time will catch everything, won't catch every single problem. A lot of our customers use RPM for one particular purpose, and that is when there's a problem, having to know who needs to fix the problem. I didn't know if it's a database problem, with the problem subjecting all your customers, some of your customers, none of your customers. Is it an important problem? This is not an important problem. And overall, how do you perform this in your application, meaning your expectations of how well your application is treated. So, when you focus on analyzing the app tier of your application, the Rails stack, your code, model, finish, passenger, et cetera, which is good, because especially for those who are playing in the cloud, knowing how well the router is working, or how well the actual server is working, is not something that you necessarily have control of. So, the way we look at things is see how the app is performing, and see how the app is using all the other resources around the life of the database. So, don't put monitoring in the database itself. At least we're not doing that. There are great monitoring tools for my SQL room, for all the other databases that are out there. But instead, how's my app using the database? What queries is it making? How are those queries being affected, especially when I roll out the query? Bottom line is that there are a lot better tools out there than one of them. There are others. And they're better tools than weaning your log file stack. Because everybody loves to do that, right? That's the whole spread going from the bottom. So, I'm going to, before we get to Eric and Petra, I'm going to show you the Arcana. So, what you're seeing now is Neuralic Arcana monitoring Neuralic Arcana, our production service. This is our staging site, and I'm going to be using the staging site for the demo here. And so, let me see if I can do this. It's a little unintuitive, because it's not up on my screen here. If you do not hear the scope of Neuralic Arcana, we have about 1,500 customers that are constantly sending us data using Arcana to monitor their own applications, their own Rails apps. And that's all collected by what we call our collector here, here, which is at Engine.org. It's on our cluster, and it's eight posts or slices at Engine.org, and it's running 24 total modules. So, it's three modules per slice. And you can see some basic statistics on our dashboard page. But the most interesting one is here, and this is on a, you know, this is on a Saturday, and we're getting 19,034 requests per minute. What that means is that there are 19,000 agents out there sending a little packet of information back to us every minute that we're then capturing our collector here and writing for our database and letting you access where you are here, and we'll choose which is here. And we're doing that, on average, with a 38 millisecond response time. So, for those people who are new to Rails here, and I think that there may be some people, Rails absolutely can scale. I think we can show it. I know a number of other people here can show it too. In any case, with that volume of data, performance is really great because we need to be able to collect and manage all that data and then do the access. So, let's go into the collector's view. This is going to give you a short overview of the product because at least a lot of people don't know it. The first place you get to is sort of an overview page with a bunch of statistics like response time and throughput, your active record, does anybody know what ATT&CKS is? Aside from working with Shipman. Okay, one person. Well, ATT&CKS is a system management standard that's been out there for a couple of years. What we found was it's a really interesting metric for us. Here's what it does. It allows you to find one number of response time. And it says, this is a satisfactory response time for users of my application. And what that does is it allows you to then say, so what's a tolerable response time? Which is less than satisfactory and a frustrating one. And then it gives you a sort of one if all your transactions are above satisfactory. And it takes points away from you as you go down. What it allows you to do is prepare apples to apples application to application and also gives you a good response time. This guy comes and says the app is slow and you say oh well, it has a 35 full seconds response time. And the guy says, I know. See how slow it is? Well, this gives you a way to say no, we set an SLA and we set a number that says 35 full seconds and satisfactory response time. And so I'm achieving that. The other things on this page is CPU utilization, physical memory utilization. And then we have these lines here which indicate deployments. For us, we deploy I can barely read that. But we deploy right at that time graph. And you can do things like change the viewing window of course, etc. But I'm going to take you to errors. One of the things that we find is really critical for our developers when we make a new deployment is watching the rate of errors and seeing what types of errors are happening immediately following those deployments because they get an indication of problems in code that they didn't do for the production environment. And so what you're seeing here is the last 24 hours of errors in the error room. And it's actually pretty good that we're averaging about 4 errors per minute on our application on our collectors here. And then what we've done is we've come down here and we've aggregated the errors over this 7 day sorry, 24 hour concrete. And so we have a customer who will go to any place but the way before they were using RPF, the way they were handling error detection was they had an email go over to their inbox every single time that an error happened. And there were so many areas that all they did in box count and they watched the rate at which it was increasing and it was increasing really fast they got their app together and did something about it but if it was, you know, slow it was okay. So yeah, it's not you need to take it back right back. So what this helps you do in addition to aggregating errors is if you wanted to see more information about a particular type of errors we can go in here and it shows you the rails it gives you more information about what's going on with that particular error and then you can also look and see similar errors. You can share this error send it as an email to other people on your team and then what often happens with errors, you don't want to open a ticket so if anyone uses Lighthouse, we have an integration of Lighthouse that will automatically open a ticket and append all this information into your like, enter Lighthouse for you. Sorry to be looking back and forth but I wrapped my brain for an easy way to do this that there wasn't one. So another really interesting thing that our developers spend a lot of time looking at is transactions and individual transactions in particular. The dashboard that you saw earlier is all aggregated information. Traffic lights are all aggregated information and that usually tells a pretty good story and if in aggregate your application is performing poorly you need to do something about it to find that applications are performing really well in aggregate and yet you have customers complaining you have certain types of transactions or certain individual customers who are experiencing a really slow transaction and that's represented by the penguin of course, me and I just got to be me. In general everybody is the same individually, things can be wildly out of whack. Can I make it back to that? So let's look at some transactions here. I'll take a second and I'll just air card and that's probably making it restarted again. In any case, while this is coming up what's the transaction series feature like is it goes to your application production as it's running and it says show me the slowest transactions every minute and particularly the transactions that are over in a second. Just show us each request and everything that and it will show you the actual SQL for those transactions and especially where pieces of the SQL are slow over 300 milliseconds the queries over 300 milliseconds will actually pull out the SQL explain as well. We found this invaluable in trying to figure out what's going on within a particular application I'm not sure what this is but I'll show a presentation if the card can work. This is now running on our production service which seems to be working just fine. Anybody familiar with Shopify? This is Shopify's app. Nobody's nice enough to let us guide who's dirty laundry. Unfortunately for everyone it's not that dirty although I'm not sure what happened to him during this deployment. This is a really good example of why you need a production monitoring system. You would find out instantly in the alerted that something drastically happened when it was pushed out and then you can go and find out exactly what was going on and maybe we'll use this application as a further example for this entire thing. But let's go into transaction traces and see what this transaction looks like. What we're seeing is that 0.25% of all these transactions in this application exceeded 2 seconds and some of them exceeded them in a big way this one was 656 seconds. Let's check one of these things out Let's try this one with no fire in our mouth without the suspension. So what I've done is I've drilled into that into this particular hole This is smoking This is not a sequel transaction There's no sequel involved This is probably not a sequel I've never seen the inside of this application before So here's another application There's a request, there's some other things going on in here and what we show you is in order to have the slowest components both the total time and exclusive time is take out the amount of time the rails execute the rest of the time the rest of it took and then if we go into the SQL tab it's going to pull out for this individual transaction every single piece of SQL that comes called and you can see it's a significantly long transaction just at first glance there's not a whole lot that can be tuned here in terms of each individual transaction but it's here that if we had something that was taking a long time we'd see this all the time where somebody mis-indexed a query or has written that like joining a query or something like that I mean if these things can take care of them you can find that you'll be able to click on those seconds and see if there's one there's 159 milliseconds so we can click on that and work those in the back we get this all in production these deployments are really in their depths so let's go back and look into these deployments we find that deployments are again, as I said before it's one of the most critical pieces to keep your application running the time in the developer's life at least that's been told by our development team for our app that's the most risky and they're really not sure how it's going to happen and so what we try to do is for every deployment we'll capture data from an hour before through three hours afterwards and give you an idea of what's happened with that particular deployment and so in this case we have a lot of deployments that are still in the wild with us and here are all these deployments since late February and we can see you know, he had some that were good and some that weren't so good but in general it looks like he's using a little more CPU these days but a little less memory he's probably doing some optimization and then you can go in here and actually look at the detail you know, here's his license and you can copy that thing and you get a handle on what's going on with the deployment so again, I'm the foreign comment now I'm going to risk going back to our staging environment particularly interesting to capture to our deployments did you see that I don't know if you caught the errors but you can see it here we went from 22 errors per minute to 3.1 errors per minute on that deployment what I wanted to just show you is a way of tracking how higher applications are improving if you look here back in December we had a group of about 15,000 requests per minute so we're at least 25% more requests per minute now than we were then and the CPU was about 65% and we were using 73 megabytes of memory so you come up here we're using about the same amount of memory about the same CPU order it's about the same response time but now we're doing 25% more and we can track from deployment to deployment to see how that could place with the reference on and the romance on I'm not going to show this to you because I'm running out of time but we used some really nice scalability analysis all your throughput versus versus we got a show for you very quickly scalability analysis wraps for the last 7 days how's our database how's our database going so here's our interesting so here's our throughput at the bottom and response time rather than over time a lot of throughput versus response time and it's not exactly the graph that you want to see you want to see a nice horizontal line to see what response time is yeah this is an indication that you're making a lot of data at the low throughputs or there's something you can do to optimize our database so how do our developers use our NAND in particular well they use notes and they're constantly looking at the application and then they collect the data and they're passing it around to each other so one of our particular problems I'm just going to take you through one of the problems that we had the other day they're taking advantage of all these things that I showed you so we had a problem the other day we had one of our customers created a really great widget for macOS 10 that puts in your status bar puts a little green traffic light and you can click on and show it to your entire dashboard and then you have to log in to our again even more we used our data API and really the first time somebody had done something really fantastic with our data API what we found though was that we had some interesting problems happening on our staging site when all relative employees installed this particular widget the way that we isolated it I'll take you through because we created a node port so you see there's a node called staging slowdown so any graph that you have in New Relic has a little button on it that says after node and then you can have more core graphs and annotate them and so here's what we saw Blue noticed that our staging environment was getting a lot worse on 31st of March around noon and you can pretty clearly see where that was happening this is our AptX again our AptX score that I talked to you before and you can see that what's happening here is the number of transactions that were less than satisfactory in this case tolerable jumps dramatically so something is going on and that was its first indication that there was a problem in our production site so next he went and he said okay well this looks interesting with each of these deployments things are getting worse and so his suspicion was that this particular deployment done by Bill Kaiser not me I don't question the product well I had introduced something that was causing a problem and in fact when we were down here there was more corroborating evidence the yellow line here is throughput and so when you look at this deployment and you look at the response time going up and below this thing that's another indication of some kind of problem so then we started wondering is a page taking too long what was the model we then noticed that as soon as that deployment happened the database response time started to spike so something was getting the database a lot worse and had been a war causing extremely long periods to happen and so finally the development team within not too long since 1145 and the problem was first noticed at 1140 so here's five minutes after after the problem was noticed using these graphs they found the problem and it turns out that the accounts controller index which is a REST call that means that traffic light widget that I talked about was getting hit like every 15 seconds for every widget that was out there and it was a root yeah exactly it was a real problem so we all shut down our widgets and pretty instantly you can see once we turned off that API access for that widget response time dropped to almost not noticeable levels and we did that with our internet so here's an example of one of our developers in this case our founder was looking at the data that we're surfacing in a production environment and sending it then what he did was he said share the note and he said I know it's on the account and then you can send that note with all the data associated with it to everyone on the team and now you have some real actionable information about what you need to go and split and fix in your development environment so that you can quickly do a new control and you can see within five minutes that's exactly what happened found a problem happening isolated a problem made that quick determination of what the problem was fix the problem deploy it and then everything is back so we think this is a pretty cool tool we have about 1,500 customers as I said we're taking this pretty cool tool as well and for those of you who are not using it we even had a free version out there called Arcam Life and we started oh and one more thing a lot of the features that I was showing you were kind of higher end features and if you'd like to try those out we have a promotion code for this group it's L.A. Moody do copy that now you go to our site and sign up for Arcam Life which is free and type in that that promotion code you can see if they work for you so with that any questions yes where do your users hang out where do your users hang out where do our users hang out do you have a community well we have a couple of things that are interesting since you mentioned that we have a support form we hope users don't have to hang out there but it is a really great source of information and you know what important things the other thing so the other thing that we have is we sponsor a website called RailsLab RailsLab.com and on it you'll find a lot of podcasts and videos done part by us part by Greg Pollock of RailsMD some from NGR some from other sources Wolf Arnold did a couple on team building for scalable applications it's just a wealth of really cool information it's all free there's no registration and lots of people are hanging out on that site site too to learn more about in general how does Rails scale how do you make your app scale because it's good for all of us if everybody knows how to make really high performance Rails application yes you're already using RPM we'll be able to use that code to upgrade yeah absolutely so if you're already having an account for RPM and you go into your accounts tab and you click on change subscription then there's promo code box at the top don't change anything else just put in the promo code press a button and you'll be upgraded to your gold trial for 30 days and then after that it'll just revert to your previous subscription thank you yes did you say that site was Rails live Rails is one of the number of posts for as long as you want at $40 for posts per month Silver is $85 for posts per month and Gold is $200 for posts per month we have volume pricing and for anyone who's employed in a sort of cloud environment especially if you're conversing a lot of scaling up and scaling down we actually also have use-based pricing that if you contact us we can offer you the difference between those product categories the light gives you a 30 minute window again there's not a lot of historical reporting with light bronze seven days of data storage Silver is a month and then Gold is three months of data storage and then we layer in some of these advanced features like transaction tracing and the deep dive into compliance everything shows you the line where you're employed everything has the known scalability everything has the integration campfire and lighthouse things like the scalability analysis are a problem yes we're always looking at that the technology here is all Rails 100% Rails app but this could monitor anything with a framework to be able to know what we need to call the various business pieces of information so right now we're focused just on the Rails app but yes I think I can answer this yes we first launched this this product was launched at Rails conference last year it's not getting your a lot of development is going into this we have four developers but the thing that caught people's eyes at Rails conference last year were the graphs particularly the spinny graphs where the pie charts spin in but it's a charting package called fusion charts and it's not very expensive that's really cool other questions another hand thank you very much for letting us substitute for Rails and I hope this was valuable pull up on it