 All right, I think I'll get started. So thank you all for being here. I know you're probably or hopefully hungover and have been listening to people all day and your heads are getting really heavy. I hope, yeah, I won't bore you to death, but great to see you. My name is Tina Sorenson. I work as a solution architect with FFW, which is a really big global double shop with 450 people. I've been in IT since 1999, started as a CIS admin, and I've been doing a lot of development also over the years, or mainly that. That was back when we sold email addresses for like $700 a year and printed them in and VI by hand. I've been working with Drupal since 2008, mainly as a developer. And performance and scalability is my special interest area. I usually say that I like big, broken stuff. I've got a colleague back home that says I love him. But I used to work for Acquia also. Before I went to FFW, I used to live in Brisbane, Australia. And I got to work on, they have really big customers hosted there, and I got to work on some really, really big, broken stuff there and learn from a lot of people. So that's been fun. So anybody here from Asia Pacific? Hopefully not. There's one, two, three, couple. This talk, like four times, down and down the other side of the world. So if you were there, then it's pretty much the same stuff all over again, almost. So can I ask how many developers are here in the room? Oh, wow, that's great. And how many have built a Drupal site or done something configuration-ish? Great. So it's just so I don't get, if I start talking business, then you guys are going to leave me. So that's great. So what I'm not going to tell you about, I'm not going to tell you very much about server setup or Varnish, VCL, writing. That's what a lot of performance sessions are about, and that's awesome. A lot of other people are very good at that, better than I am. I usually, my main interest is when stuff is already broken, what to do then. I guess being developers, almost every one of you would have had a site that had pages that just take like, you can make dinner before the page load, or you need servers the size of eBay to even make the site run. Things like that. That's what I find interesting, and that's what I think is an area we need to get a lot better at, because Google is getting bigger, and be able to solve those issues. So that's what I'm going to talk about tonight. I'm not the best front-end person either. I'm not very good at making stuff pretty. I know a little bit about front-end performance, but not enough. But it's mainly going to be about application layer. Google is evolving. We all consider it enterprise ready, and have done for years now, I think. The sites are getting larger. It's delivered as software as a service. Often I've been a part of building a huge LMS platform, online learning management system, with only logged in users. We use it as a framework. The complexity is getting massive. So this is a whole new area that's getting bigger and bigger, and people run into more and more fun problems. So we can't be experts at everything. We have, just within Drupal, we have operations guys. We have DevOps teams. We have developers, steamers, designers, and I hope someday it's also going to be more common to have a performance dude or gal. But I think it's very much an area of its own. Sometimes when I've been around the world or worked with teams, developer teams, and business owners, then I might have done like a performance review for a site. And the customer will think, so what, we paid this developer team to build the site, and now you come and do a performance audit for the site, and you come up with like a list that's two pages long of things that needs fixing. So didn't they do their job properly or not? And I think that's a very wrong way of looking at it. It's a very different focus being a developer and looking at something just for performance reasons. When you're a developer, you are focused on things like business requirements and deadlines that were yesterday and just doesn't match the time that you really want or need to do things right. When you look at performance, you don't really care about all that stuff. You just go in there and measure about how long does it take? How many resources does it take? And it's a total different focus. I even took some of my own code a month later and cut like nine seconds off a feed import, just looking at it from a performance perspective instead of a developer. So I think it's two very different areas and you can do the best work when you do performance work if you have a developer team to collaborate with because it's like, when you look at things from a performance perspective, it's so easy to sit there and be like, oh, but why did they do that? That's silly, but it's not almost always the thing that looks silly, it has a reason. It's there for a reason. It happened in some context. Even if it's turning off caching, then it's most likely because somebody, it created a problem that caching was there. Something was getting shown wrong at the wrong time. So solving performance issues, a lot of the time or at least what I did before I started getting more interested in this area in particular was that I opened up Google and then I typed in Google performance. And then I would find like three or four pages that would give me a list of 20 items and that I then started implementing and that would be anything from enabling boost to looking at views caching or advanced aggregator module for CSS or whatever. I had no idea what I was doing or whether I was actually addressing a problem that I had or not, but I just did this stuff because how to find out what's actually happening in the application that I've built myself. So what I've, yeah, what I've learned over time is that the data is really what's valuable here. The thing with just starting to optimize from one end to the other is obviously not gonna help us. It's, you know, we just as lucky winning the lottery is solving the actual issue. We might hit something that helps, of course, so, well, we're all lucky sometimes. But to find out what's actually happening, what's actually causing the issue that gives us the visibility needed to actually solve it. It also gives us and whoever is paying for our time, often somebody is, the freedom to choose from which fixes to implement. If you can figure out what's happening, there might be 10 issues. Some of them are a lot of effort and might not make a lot of difference. Then it gives them the freedom to choose whether they want that implemented as well or not. The analyzed part is, it sounds like you have to be, like, been doing Drupal for 10 years in order to analyze data for what's happening. You really don't. So I'll show you a couple of screenshots later, but most of it, if you've built just a little bit with Drupal, a lot of stuff you can figure out, it's just your normal Drupal knowledge of what looks odd, what shouldn't be like that. Then there's the ability to be able to choose where to apply effort based on facts. Very often you'll find a lot of low-hanging food. I love those. That stuff that you can do really quickly that's really gonna make a change. I'm gonna show you some examples of that. And that's what I usually do. Prioritize low-hanging food and then whatever will give me the biggest payoff, but sometimes those are too expensive in the real world, of course. And then when you've done something, you need to start over again because every time you make some changes, then the wall is gonna look differently. If you had a site where there was no caching enabled, then obviously everything that you were looking at, all the data showing up, would show you that there's no caching enabled. And the next issues you can't really see in statistics and neurologic et cetera until you've removed that one. So what I often see when I've worked with teams and also what I used to do myself is that I used to start fiddling with the part of the application that I felt least secure about. So it would maybe be that feed that we implemented last year that was really buggy or we did really quickly. And then I'd start just working on that. Sometimes it's like the low-hanging fruit can be a surprisingly high-sell when I work with developer teams. It's like as a developer, we love code. That whole thing with, oh, but you just need to switch GD to image magic. That can take months to sell to a developer team. It's so funny and sometimes that means you can halve the PHP memory, it's just from doing that. But it takes less than an hour and it's a configuration. It's a click, it's not code. If I go out to a developer team and tell them that, oh, but you need to rewrite the entire caching layer, they're gonna be on it right now. So usually, not everyone, but very often it's like that. So that's nothing wrong with that. We do this job for a reason. But there really is a lot of good low-hanging fruit often when we work with this. That's my PVs, that's about collecting the data so that we know what we're doing, not just guessing at whatever part of the application we don't like. Solve it from the application, not just buying hideware or something like that. And then the penny-wise, but dull or stupid, that's what I usually, I've tried to say in a nice way sometimes to developer teams that we're gonna start rewriting the entire memcache integration that might give them a 1% improvement in two months when they had finished that and all the complications it would take with it, but wouldn't switch image metric and GD. So, I almost can't see this one. This is New Relic. Anybody, anyone of you guys work with New Relic? A lot of you do, so I love New Relic. When I do performance work, this is where I spend like 80% of my time, I think. It's, no, I like being geeky just as much as next guy or girl, but, and I could sit there with a XH proof and it would take me a long time to get an overview of every page on the site, a page type. But it's like New Relic gives you, it sort of combines a lot of different tools that you do use when you do performance into something with really pretty graphs. I love really pretty graphs and they communicate so well to whoever is in charge of the project and has the decision power. So, even if you don't know New Relic, then it's, there's a lot of functionality in there, but what you really only need, or the main priority that can set you very far to know is simply to look at the application and look at transactions. I don't know how well you can see this slide, but in the left side, if you open up New Relic, that's for anybody who doesn't know, New Relic is an application performance monitoring tool. So, it runs online and you can log in and see how your application is performing. And the main functionality you really need there that will give you a lot of visibility even if you don't spend two days figuring out everything that New Relic can do is if you go in under the application and then choose transactions in the left bar. That's a screenshot of New Relic I have here. And I just wanted to give you a quick idea of how easy it actually can be. Then if I tell you that these here, the next list that you can't see up here on the first screenshot, is different function names or transactions in New Relic terms. And they're all sorted by the most time consuming. And the one on top is called page manager node view page. And it takes around 50% of the application processing time in total. So anybody have a good idea of where would you start to look if you had to figure out why this one is running slow? If I give you one more hint, anything to do with page manager, that's something with panels. If you look through the code, just... Well, you go ahead. Yeah, so what's wrong on this one? That was a fact. What you can simply do is just take the function name and just search through the code base and figure out where that pops up. And then you can see probably what would be a good place to start looking. I'm not saying that is it, but it was in this case. This is a page or a site that doesn't have caching and no panel is caching. It's using panels really successfully. People like those pages clicking on them, but there's no caching on there. So half of what this application is doing is serving panels pages. Probably haven't been edited it for ages, but it's still computing them from scratch each time. This is another example of where I used New Relic that would have taken me days to figure out if I didn't have that. This was a site where all blocks on the site were enabled, but there was custom theming on there. There were custom TPL files, and in those TPL files it was calling all the blocks, so sort of both loading them and rendering them from scratch. It actually did that twice before it actually output any of them. So what was happening was that all blocks were getting, oh, and a lot of the blocks were views blocks with really heavy queries that weren't cached. So on each page load you would have like 50 blocks with really heavy database queries that were getting loaded or executed and not used for anything. And then the custom TPL file would kick in and do its rendering, and in there there were code loading those four blocks that actually needed on that page and showing them. So the entire site was running really poorly because those 50 blocks were getting computed from scratch on each page load and used for nothing. That would have been a lot of clicking around to find if I hadn't had New Relic because you need the combination of actually reading the custom theme and going to the blocks page and seeing that they're not especially configured. But that was the one on top, that was the application processing time before we fixed that. That was almost five seconds. On top of that, there was like 10 seconds front end processing time. So it was like 15 seconds load of page there. And the one on the bottom, that's 600 milliseconds. That's just disabling the blocks. That's like four and a half seconds in less than an hour's work that you could save on a page load. That's pretty cool. This is what it looked like when it went live. That was awesome. I love those screenshots, so. So examples of low hanging food. That was one with all the blocks rendering on all pages or getting executed at least. And actually also rendered, but not shown. Views, I don't know if this is just also an example of a low hanging food item, but have you guys ever looked at the difference between the different sections when you configure a view and what kind of SQL it produces? I'm by no means like I'm not a views contributor or anything. But there's different filters in there. And if you use contextual filters, those that are like arguments that you can give to the page, that usually produces like a left join. And when I've talked to developers that are like, I don't really know my SQL, so what am I supposed to do here if I, that thing with when you configure the view, you can enable that little gray bar and it doesn't really give them much knowledge if you don't know my SQL. But usually I've just told them that, okay, left join's are evil. That's really the main thing you need to know. And then as a rule of thumb, if that box, the gray box with the views query, that's building there that gets taller than like five centimeters, then you need to start looking at what you're doing. So, and that's a really good rule of thumb actually that can take you pretty far. But if you look at the views configurations as I started talking about, then the normal filters, if you for instance say, whether a particular item has this term attached to it, if you choose those just fields filters, those usually produce a where statement where the contextual filters, the arguments that usually produces a left join. So, there was this site that had views queries that were running, this should be running at maximum of couple hundred milliseconds, but this one was running at like 1.2 seconds. And it was running all the time and it was killing the database. So, and it was like free taxonomy terms that were getting checked. So, there were only three different arguments that were getting sent to it. But to split that up into three different views and not use arguments, but actually just filter on that field that produces a query that had where statements, which are awesome that they're a lot better than left joins instead of like eight centimeters of left joins. And that took off 80% of the query time. GD, anybody, any one of you guys, have anyone ever looked into what GD does, how it performs? Couple of guys. GD is really my main pet PV I think at the moment. GD is awesome. It's an image library. You probably all know what image libraries you know, you used to in Google six have to download image API and choose make a conscious decision of whether you wanted to enable GD or image magic. In Google seven, it's not like that because GD is running in PHP. So, for licensing reasons, then it's really easy in order to help people. So, they don't have to like download five different modules before they can even show an image. Then it's easier to just set GD as a default and I totally get that. It's just evil. So, I don't like it. GD uses a lot more memory than, this says image cache by the way, that's wrong. It should be image magic. Image magic is another image library and that runs server side and pretty much any default Linux installation comes with that. But it needs to be running server side so you can't put it as a default thing into a form. But image magic is a lot more memory efficient and it doesn't run in PHP memory. I've seen sites where just switching the image library from GD to image magic, which was a task that took like downloading one module and enabling that as a default image library, less than an hour, if even that. That took off like 240 megabytes of PHP memory. The maximum trace, that's a lot. The recommended memory limit is 128 or 240, so that's often when you put things on servers, they will be memory or there will be a definite amount of memory. So, just switching GD to image magic, that would be the same as saying you could half your server costs. So, that's pretty efficient. So, use UI. I don't know if any of, I never thought about this before, but it often puts like 80 megabytes of the maximum trace. It adds like 80 megabytes on top of that of the need for PHP memory. When I'm talking about the maximum trace, that's a tool that I'm using often that's called memory profiler. It's a tool that's written by Tim from Acquia in Brisbane, a former colleague of mine that's really clever guy. And that you can run and it'll sample the memory on the site on every 50th page request or something. You can run it for a couple of hours in production and get a feel of where's your memory actually going. It's the same that the development module can do, but you really don't want that running in production. So, that's how I found these things out by seeing, okay, running that module for a couple of hours, figuring out what are those pages, what's happening on those, the ones that show up with like, there would be three or four URLs that suddenly needs like 800 megabytes of PHP memory and the rest of the pages would be running at 60. So, what's going on there? And that would be, for instance, a designer doing what they're supposed to do and uploading a two megabyte image to a page that GD then has to scale or to render and to scale into different sizes for the different presets in Drupal. And that would take up like hundreds of megabytes of PHP memory. So, that's really cool info. Another site that had really its primetime event ever and it had one and a half hour of downtime. Looking into it, looking at New Relic again, things like that, we found out it had like 50 different panels and 50 different views, that's not it, but just one of them, just catching one panel and one view could bring the site up again. And that was a page with hundreds of millions of page views. It's really often just low-hanging food. So, another developer team I work with, switching GD to ImageMagic, tuning the server with APC, Memcache, DisablingViewsUIs and caching a couple of views. That actually allowed them to have to run four times as many processes on their box. That would be the same as buying four times as much hardware if they hadn't fixed that. So, you can get pretty big stuff with little things here. So, you're also all welcome to rewrite the Memcache integration and the cache layer, but I just love doing those little things first so that you can see how does the world look now? Is it really worth that amount of effort? So, the data, how do I get that? I look at symptoms, what's actually wrong? So, is it like you turn on the server, turn on the site and one and a half minutes later then the box dies? Is it because it needs hardware? Again, the size of eBay. Is it just really slow responses on some pages? What is the pain right now that the site is going through? Then the site profile and that needs to talk to often it would be a customer and the developers who build it. What kind of site is this? Does it have like 77 external integrations where it's waiting for answers? Or is it logged in users in 50 different groups with different permissions? What's, is there any special functionality on here? What's happening? And then the server, of course, I look at database logs, traffic, APC, how that one's, how Memcache is doing. That's also a very common one that gets overlooked that Memcache has too little memory. And then it's just dying because it has to push things out and override itself all the time. Application, there you can look at Watchdog, although I hope that Watchdog will be in just log, the not DB log, caching, settings, modules, and then I use New Relic. I actually mainly use New Relic just to get the overview. It often points me in the direction like the panels thing earlier. So this is again another New Relic screenshot and this is just to show you the really neat overview you can get when you click a transaction where you can see exactly how much of the application processing time is going where in a transaction. This one was a note page view where you can see that. A lot of it is going into the menu. And then you also, you can get if you have the Pro version, which yeah, damn them, they should pay me, but I really would recommend if you have a performance issue to pay for the New Relic Pro version. I don't work for them, but it gives you a lot of insight that you don't have otherwise. This is pretty much what you get from XH Prof, but it's right there with your pretty graphs in the New Relic interface where you can see exactly how much is going where and which functions are calling which ones. So a place I think you should not start is to go in and optimize Drupal because it's doing stupid DB queries and write your own because they're much better. They might be and you might be right, but there's a lot of different things to that. Often it's like, I think there's a lot of, a lot to be said about what I would call sustainable sites. So the less you have to fiddle around with stuff, the less you have to hack code or modify code, the more sustainable your site is gonna be and that's gonna be helpful in order to, that new modules you install that they're gonna work, that the next guy that starts working at your job is actually gonna have half a chance is figuring out what the heck is going on. Or the poor guy that has to solve performance at some time also can find what's happening. I think there's a lot about thinking about whether the complexity is worth it, how much you are putting up the entry level for the next person to work with the site. And hard code DB queries, I don't know, just in general, Drupal does a pretty good job at what it does. There's some really intelligent minds that's been over that code quite a few times before I hit it. So even if I have an opinion, then most of the time I found out there's really good reason things are the way they are. So double the hardware, well, that's a bit like peeing in the pants just to keep warm. So that's gonna work for a while, but it can certainly give you the kind of breathing room that you need until you fix what's actually happening in there, and that's a really, really good thing. But as a permanent solution, people do it. I've seen the most impressive server parks run really, really simple sites, but it's a very expensive, and it just feels wrong. And yeah, rewrite Drupal because it's stupid. Well, yeah, we can all bitch about that and feel we are wiser, but I doubt that we are. All right, some of you maybe. I don't trust myself to be, that's for sure. Or just drag everything out in JavaScript and fix it there because then it's not, well, again, it's a little bit like peeing in the pants to keep warm. You add complexity, you do data calculations where Drupal doesn't know anything about them anymore and you're probably doing more DB queries than the page load in itself at some point. That's the same with edge side includes. I think that's a really cool thing and it sounds awesome, I love the word. It sounds like I'm the techie, but the thing is that edge side includes kind of makes sense because if you only have like one or two things on a page, then if you just need to display my name, you don't need to do a full Drupal bootstrap to ask the database for my name as a user when I'm logged in. That's fine, but it's a bit like, if you do like 10 of those on a page, then you've done maybe 10 full or partial Drupal bootstraps in order to not do more database queries in the bootstrap you were already doing, which is just sort of silly. So yeah, you can end up with stuff that's actually running a lot slower and doing a lot more database queries with edge side includes, which was what you were trying to solve in the first place. Symptoms, we talked about that. The slow pages, database could be dying, the web server is dying, extreme hideware needs, which whatever it is, it means that you have to look in different places. So, but in any case, when you then start looking in those places and you have the data, then use that. Hunch is not enough. And then apply effort, where it corresponds to payoff. Performance is often a bit like web development where we say it's like 80, 20, if we can get like 20% of the site done in 20% of the time and vice versa. But performance can even be 99, one, if it's about enabling caching and switching GD to image magic and enabling views caching. Views caching is evil also. And then remember you need to maintain this. So the really, really big complex portion, Mercedes kind of code, or it just make everything pretty. Yeah, it takes a lifetime to make everything pretty. And complexity entry level versus gain, be nice to the next dude that has to work with this. So, sometimes I even choose a less performing option just to have it look neat and not having to modify modules. Not like massively, but it is a trade-off, I think. That's worth considering. So, other things that are really, that really makes a lot of difference for how a site is running is whether it's authenticated or anonymous traffic. I bet any of you guys work on sites that have a lot of authenticated traffic, like mainly only authenticated users. Yeah. When I worked at Acquia, I did a lot of hideware sizing for customers, new customers that came on. And often they had a hard time understanding what was happening when, you know, they just had hideware sized for a website that had like millions of millions of page views a month and that was a certain size. And then they had hideware sized for an intranet with like 10,000 users, so authenticated users, and that could be pretty much the same size. But that's, the thing that's different here is that when you're not logged in, then we can serve all, pretty much all the pages through Varnish. So, there's not gonna be a lot of requests that hit the web server. When you're logged in, then Varnish won't help you. Then, what we need in general when we're talking about performance that is to help the database and it's to do that with caching a lot of the time. Or caching is one of the ways to do that. So, when you're logged in, then Varnish, you can't serve full page loads with Varnish, but you can use memcache still. You can cache a lot of the site in memcache and use that, but it's not as efficient as Varnish. You're still hitting Drupal. So, yeah, it's a different caching method. Another thing I used to try to explain to people when they had a hard time understanding, it is a little bit confusing, the thing with, oh, it's an internet with 10,000 users and it needs the same hard wire as a page that has millions of page views. It's a bit like every request that hits the web server. It's using some sort of time and energy from the server. So, if you can serve pretty much everything without ever hitting the application, then you're sweet, then it's perfect. And that's what you can do with anonymous traffic. So, modules, whether there is like node access on there, that will disable block caching per default. There are ways around that, but it takes a little bit of extra logic and thinking things through before doing that. Organic groups, those are modules that also can be really bad for performance, but are bad for performance. It sounds like you should just disable them. Of course you shouldn't, but sites that have a lot of access restrictions on there or have node access modules on there or organic groups, things like that. They do have other performance needs. Organic groups doesn't have to, but if it, well, yeah, it does, but also it just adds a lot of complexity in general. And people tend to make fun choices for architecture on top of complexity as well. Custom code, the amount of custom code and how well it's written. If it's using caching at all or just doing direct database queries or something like that. And one thing I actually, I don't work at Acquia anymore, but one thing I still use when I do performance work is to simply upload a site at Acquia where they have this really neat insight report where you get a lot of tips for performance and SEO and all kinds of other stuff that I don't care too much about. But there's really good low hanging fruit or best practice general stuff in there. I think that, yeah, I know that, I think Pantheon has that as well. But that's a really good place to get started. Application caching, that's also a really fun one. Whether caching is enabled at all, it's amazing the amount of sites I've seen by now that still today has no caching enabled at all. Again, if we end up, if caching is not enabled then we end up having the website do all its work from scratch as if nothing had ever happened every time there is a request. So, and generally what it should be doing if caching is set up correctly is that it should really only be asking the website for something, when there's a request coming in that's asking for something that hasn't been shown before. And distinct stuff should really only be pushed from cache when it's been updated or it's new. So, and there is, you can run a caching report from views, blocks, panels, and maximum and minimum cache lifetime. I also look at that and I look at whether panels are cached or there's particular pages that are not cached by on purpose. Then I look at a lot of best practices, just basic stuff like statistics module, that's evil. I don't really know why we still have that. I don't know how to check whether it's in Google 8 but that one is writing to the database on pretty much every page load so that's gonna be really, really heavy. The database logging is also just evil in itself. I've seen that takedown pages all on its own. So, if you do just, if you have PHP notices enabled when you develop, you have, of course we do, that's best practice, we need to know what's happening. And then that setting also is like that on the live site and you are logging to the database. PHP throws a lot of notices all the time. So, you will have pages that can't be cached and that's taking a lot of effort, not really producing any content or anything that benefits the user. It's just writing to the database with log messages that are not necessarily relevant. Views, fields, rules, UI, that's also modules that'll add a lot to your PHP memory limit needs. Whether develop is enabled, that's not good for performance security or anything else and it shouldn't be on a production site and we all know that but we all forget it. So, and then do a full search. Again, the thing about trying to protect the database so that when we actually, the database is slow, it's a slow part of the application. When we ask it, we want it to be able to use all it's got to give us an answer. If it's working on all kinds of stuff it doesn't have to, it's gonna be slower. So if we are writing log files to the database, if we are using double search that can be replaced with solar search then it's gonna be slower. There's no need for that. So, if we are putting statistics there that really Google doesn't much better job at that anyways. So, it's about moving out the functionalities from the database that are not necessarily about serving content and that's better serve somewhere else. And then modules, I look at things like view slide, pager, so it doesn't have to calculate all the page splits of all the results at once. Fast 404, whether entity cache is on there. I've seen a site get like 30% improvement in page load speed just enabling entity cache and I've never seen it create an error on a page. Anybody seen, anybody ever had problems with entity cache? Oh, two people have. I stand corrected. So, but I'll say rarely then. But that's really neat module that almost always is a good thing. So, and then whether it's actually caching correctly. That was what I talked about before that cache you really don't want to update the cache or just expire things from cache every hour every three hours just because you've set that as a limit. You might as well just leave it. What we really want is for content to stay in there until it gets changed ultimate or that would be optimal at least. And that you can do with expire and perish modules so that the only time that you can set the cache limit like really, really high and then only go in and take things out of cache when there's actually something that's changed on that particular node. Views content cache is a module that will help support that in views so that views also expire listings, et cetera of that content type when something has been updated in that content type or a particular content type. So, and yeah, then use an APM. New Relic is awesome. I know there's others out there and they're probably just as good. I don't know. I don't think they are because I know New Relic, but no, I think it's very much about knowing the tools that you're using. I don't know if the others match up to par with New Relic, but I really think New Relic is a good one. And what I look at in New Relic, that would be transactions, the one that I showed you before on the fuzzy screenshot that nobody could see. So, that's the main thing that you would need to know to get started with New Relic. That's why you will find a lot of good information. But apart from that, then I look at latest show slow requests. I saw it instead of most time consuming than the slow ones. I look at the double modules just to see if there is something in there that looks odd. Again, that's about basic double knowledge that we all have. Things like the one with panels earlier. If you knew that page manager means panels, then you would probably have a hint like, really is panels supposed to use like 50% of the application processing time? Isn't that a bit much? Maybe I should just check up on that. Or if it's blocks, for instance, that's taking up 80% of the processing time. That's also, that's pretty good. That you only need like your own experience to see that. So, database queries also good in, you can also check in there. External requests can be really helpful. Sometimes that's actually, it's an external request waiting for some other system to reply, which is not gonna happen anytime soon. That's pulling the entire site down. We had ones like years and years back in Denmark, there was this ad service where it had to be set up so that the page or at least, I don't know if it had to or if we were just silly when we created the pages. We're probably just silly. But what happened was that the rest of the page wouldn't continue loading until we had a reply from the ad service and the ad service was really bailing out and seriously slow and there were like 15 ads on each page load. So it was killing 15 sites. So external request is a really good one to check as well. And then slow views. Again, this page with like millions and millions of visitors, just with no casting and either no panels of views, 50 each of them, just one view, one panel got it up again when it was broken and down and couldn't run. When I've looked at all these things, then I might go nitty gritty and sit around with the XH prof. So I like command line as well, but my life is getting, I'm getting too old for that, I think, if I can get pretty graphs. Custom code, these are some of the things that I look at. That would be things like variable set or cast clear all, cast clear all, definitely. I've seen code where the entire cache on the webpage where clear twice before it even started showing the page. You can set up, varnish all you want, but it's not gonna happen. So loops with no default. That's just bad code. If it never falls into any of the categories, you're gonna have an issue for error handling, for anything that's using hook in it, because that's gonna make it really hard to cache it. For hacked modules, I use the hacked module, the amount of custom code, amount of modules, whether they're actually used. So on the server side, I'll look almost always fast and foremost on memcache evictions just because that's a really common one. It's almost as common as caching being disabled, that the service will be set up with some sort of memcache memory and then it's like, oh no, we don't wanna give it anymore because then it's gonna cost money to do a dedicated memcache box or something like that. And yeah, well that's the best money spent ever if it's needed. So if you have a really high eviction rate in memcache, then it's almost always because it needs more space. So it starts like showing, like throwing out the other kids from the room because it needs to write before it's supposed to. And that's really, that's a lot of extra effort for memcache and it's really not gonna perform very well. And often people don't know because you need to keep an eye on the statistics for memcache and you really don't do that unless you have an issue or you often don't think of it. There is one really edge case scenario where there is something about memcache and how it allocates memory where you might have memcache allocated memory in a particular distribution of sizes or different sizes, boxes or whatever you would say with space for caching things to begin with and it never ever ever changes that. So I have actually once seen where evictions were not because it needed more space but it was because it had used up all the space it had of a particular size of cache. And even though it had like 70% free memory then it couldn't use it because it was all the wrong sizes where it couldn't fit in the chunks that it needed to. So that was a pretty particular case. All it took was to restart the memcache software and then it would start allocating memory again but I think it's actually solved in later versions. Anyways this is, I don't know why I'm wasting time telling you about a really edge case niche scenario just maybe then it makes me feel like that knowledge was worth something. But then there is APC statistics and percent of memory used in APC PHP that's also a very good thing to check whether there's enough but usually it'll come and bug you with errors if it's not enough. Viny statistics and headers, Viny stat, Viny log are just opening up the front page and going to the Chrome inspect element and looking at network and looking at whether the different items in there are actually cached or not. I was just looking at a page or site a couple of weeks back where I was like, oh yeah this site is awesome. It's, you know they have thought about performance. It's set up with CDN and with Varnish and all that stuff. And then I looked at the front page just to get a feel of so why is it running slow and look at the headers. There's not a single element in there that were actually getting cached. Viny was just a tunnel because caching was disabled. So it was like sending the traffic to the CDN for DNS requests then to Varnish where Viny could say, oh yeah sure. I know about that one. No I don't have it. It's not supposed to be cached. Ask the application like 200 times for each element. That was great. And yeah then the memory profile module I mentioned earlier to figure out whether there is or how much memory is actually needed. I've also come across sites quite a few times that had PHP memory limits set to two gigabytes but didn't even need it. So maybe it only needed 64 megabytes. So that's depending on how your server is set up or configured that could cause you to not get a lot of value for your hardware. My school query caching statistics whether that's you're actually getting value or you're actually getting things cached. PHP error settings. And then I look at logs also. PHP error log especially. Traffic reports if you have high number of 404s and you're not using fast 404 then you'll use a lot of views on that. Or just in traffic reports pages that shouldn't be there. Maybe something is I don't know calling some random page on your site all the time that shouldn't get cold. Google Analytics I might look at and that I actually rarely don't do it. But yeah, it might be good just to if you can't find it anywhere else just to figure out. So does Google think that the top pages are the same as I'm seeing in my traffic report? I saw a site once where the site was actually calling itself. That was really funny or it was funny when you found out but there was a blog that had some coding error in it. So the page was actually calling itself, bootstrapping itself recursively. That was a fun one. The weirdest thing that I, we found it and we fixed it quite quickly. The weirdest thing that I couldn't figure out was why New Relics showed that as a two hour, 22 minute page load. I was a bit like, how does it know when to stop? So well, later I found out that that was when the server went down. So that was made sense. So then there's the whole server tuning side of things and that's more on the upside but that's going in and figuring out, okay, what kind of hardware do we have here? Is it memory limited? Is it CPU limited? How many processes should we be running as opposed to how big is the server? And memcache size, APC size, that would be where I would call a colleague back in the day at Agria. Tim, usually the guy that wrote the lovely memory profiler module and have him tell me. Questions? Anyone? I've got like 16, 36. I've got like nine minutes left I think if you guys want to ask about stuff. Okay, you can do that. That's fine. If I've tried what? HHVM as an underlying PHP? Oh, the question is whether I've tried HHVM as a replacement for PHP. I don't even know what that is but it sounds awesome and I want to know more. So we'll talk later. Yeah, if you're going to say more than that it needs to go there because I can't remember and repeat all that. So, and otherwise you're not getting YouTube. Hi, so we've used HHVM in client projects, not all of them because it's not a standard PHP and it required quite a lot of testing to confirm that the client's projects were working successfully. And it does not, it gives improvement in terms of pure PHP performance that it gives all these nice things like just in time compilation and a lot of things. So in terms of PHP execution it's fast but in most of the web applications logic it's bound to external services like MySQL, memcache, third party services or whatever it is. So even though your PHP is going to run 0.1 second faster your overall page speed won't improve by much due to the external services and network delays. So if your main page bottleneck is CPU and it is not based on network communication but just processing loops and custom logic that needs to do whatever it is then HHVM might help you. Come, sir. Hi. Are you planning or are you aware of the black fire module, no not module, the black fire tool from essential apps about monitoring the app? No, sorry but I don't know that one. Another thing I need to look into. Teach me more. I've used New Relic a lot and have you ever found that sometimes New Relic seems to give you information that seems to be false? I mean like load times and stuff like that. So I've been monitoring my site for over a year for it with it but occasionally I find I compare it to other tools. I'm like all right I want a sanity check compare it to Pingdom tools, compare it to WebHTest, stuff like that. And what I get from those is like it doesn't always jive what New Relic sometimes wants to tell me. I think yeah, I think that comes down to statistics and numbers and location and yeah because if for instance if I use WebPageTest or Pingdom or something like that that's actually gonna ping the application from somewhere and it's gonna be really hard for me to compare those numbers. I don't know where New Relic is located but what New Relic gives me is that I can look at the New Relic numbers in a history containing other New Relic numbers. If I need to make sense of WebPageTest numbers then I will be pinging the site from Dallas or from Stockholm or from whatnot. So I need to do that repeatedly and then in the results I get there will be internet wins and weird lag and so yeah. No I don't find New Relic unreliable as that. When I've had something then usually it was me that needed to learn how statistics work. Like for instance at night time the cron job took out a lot of the processing time for fun reasons since nothing else was happening on there. People were sleeping so nothing wrong with the cron job. It was just the only thing running and it shouldn't be because it was running from doop. That's a different story but yeah. No I don't find it to be unreliable not as such. I don't see it as a precise form. I think it's really hard to get precise numbers like that but if when you can see the history then that makes sense. Anybody else? What strategies have you used to overcome AJAX requests using posts going through varnish? Or have you faced any of those issues? I'm pretty sure I have. I'm trying to dig through my mind. I remember something with that. It was killing us out and I can't remember what we did but I'm pretty sure I've solved that at least once or twice. I can't remember or is it even solvable? I don't know but yeah those are evil so. You have a solution for it. Oh please tell me. We have to rewrite some of the group way AJAX, J.S. But it wasn't the best way obviously. There was nothing really out there. Yeah. I can't remember. What? J.S. module. Somebody's saying that there's a J.S. module that would solve that. Okay. I can't even hear you. Stand up. The microphone's right there. We wanna hear you. The J.S. module gives a really good performance boost for all AJAX asynchronous requests either with post or anything. It was really helpful. It really worked for me. It really saved me. Thank you. Thank you so much. Anybody else? Yeah. The question is whether there's any tips to optimize a website where there's a lot of forms on a page. So and I'm guessing the thing is that well you can cache actually sort of, you can't even cache showing the form because you need the unique ID but you definitely can't cache probably the answer. Is gonna be different for each page or? Anyways, anybody have tips for caching around forms? I need to remember all that still. You know that, right? No, somebody's saying that all cache is, should be a solution for that. Yeah, please do. Thank you. We should have done this as a barf. So the workaround form caching might be possible through the old cache module. It will allow you to make asynchronous requests to actually get information for the form IDs and tokens. So you'll be able to submit the forms even though you have cached the whole page. So you cache the whole page and make additional requests to the backend to generate unique tokens. So you'll be able to submit the forms even though the page is fully cached. But it will be something that needs to be tested thoroughly because it's not a very standard solution. It's a contributed module but, sorry? All the forms will be, all the forms require a unique form token to be submitted and this is a security fix within Drupal 7 and above and old. So what you need is to have this unique token on all pages when they are displayed and this is why you cannot cache pages with forms if you want to work with the forms on them. And the solution there is to either disable the cache for the whole page or cache the page and generate asynchronously through Ajax code unique tokens so you can submit the forms. And there is no good work around. So you either do this through an AC tag so you generate them, yes, in varnish or you make this with a JavaScript. So these are your two options. Well, it's for any users, yes. If you want to cache the page and forms you need to generate dynamic form tokens either way. Perhaps we'll take that talk outside because I'm gonna get thrown out here in like two seconds. Yeah, but come catch me if you have more questions or say hi or teach me something I don't know. So thank you so much for being here everyone. I'm very happy to see you all. Thank you.