 Welcome everyone, and thank you for attending. I'll see you all at the pub afterwards. All right, welcome. So, my name is Nick Santamaria, and today we're gonna be talking about performance and scalability. Interesting topic. I'm a senior developer at Technocrat. I recently got my Acquia certified backend developer certification. Yeah. And you can find or contact me at any of those places. I'm always eager to hear from other Drupalers. It's great. So, just a brief outline of what we're gonna talk about today and what, you know, do the goals and sort of set the mood. So, what I was hoping to achieve with this talk is, you know, address site builders and developers who are, you know, building really cool, complex, rich Drupal sites, but maybe haven't had the opportunity or a feeling lost about where to go in terms of how to make them more performant, how to build them in ways that are gonna scale. And, you know, maybe some of you guys have had disastrous launches and you wanna know what you could have done better or, you know, maybe you're sitting there on your laptop furiously trying to do something right now. So, basically, it's gonna run through some basic, you know, what is performance and scalability, run through some common problems and solutions, give you guys some strategies for, you know, how you can architect your site to avoid these issues in the first place. Talk about infrastructure a little bit because that's a big part of it. Debugging performance and scalability issues. So, you know, if you have launched a bad site, how do you get things back under control? And hopefully, if we have enough time, I would like to open up the floor to just a discussion, ask questions, you know, specific issues that you've run into and, you know, see if we can figure it out together. And I would just like to say that, you know, if you have any questions throughout the presentation, you know, I'm a big believer in context, so just put your hand up and I'll ask you to ask you questions at the time rather than at the end. So, just the general introduction to performance and scalability. So, when we talk about performance, we're talking about how fast a single page, a page load or request can be executed. So, just one page load in isolation. When we talk about scalability, we're talking about how well your system maintains that performance as we go from one request to a thousand requests, 10,000 requests. As you increase your user base and your visitors, are you gonna be able to continue delivering that performance? So, the way that, you know, this is not based on any particular methodology, this is just how my mind works. I like to break up performance into back-end and front-end. So, components for back-end performance is PHP, obviously Drupal's built on PHP, so the main considerations I see is, you know, how much code is being executed? How many modules have you got enabled? How, you know, how efficient is your code? What's the code doing? Is it, you know, trying to process a huge lot of data on every page request? Is it, you know, if you've got requests out to external APIs, et cetera? It's also the database. So, how's your database designed? What's your information architecture like? How long are your queries taking to execute? You know, some, I've seen some websites with 50, 60 second queries. I mean, that sounds unbelievable, but believe me, it is absolutely happening on production sites. Yeah, briefly mentioned, API requests before, but this is a, you know, a big back-end API consideration too, because PHP, you know, it's difficult slash impossible to do forking, so every single process has to wait to be completed before the next process can continue. So, you know, if you make a request out, you know, like a post request to your payment gateway, and your payment gateway is under stress or taking a long time to respond, then your end users are gonna be experiencing that delay as well. And then caching. So, you know, Drupal, we love our caches, there's Drupal core default, or Drupal 7, uses the database caching, but there's other ways you can improve and extend that, so memcached, ratus, MongoDB, and for our page caching for anonymous users, we can do varnish, which is basically another web server that sits in front of your main web server, and proxies are requested if needed. So, in terms of front-end performance, the main components of that, I believe, are the network overhead. So, can I just get a show of hands? Who here generally hosts websites overseas, like in the US, or, you know, Europe, anywhere like that, yeah. Cool, so how much time do you think is being added to every single request by having it in the US? Let's say the West Coast. It's about 200 milliseconds per request, and if you think about a standard Drupal website, you've got the first page request to actually get the markup, then for every CSS style sheet, for every JavaScript file, for every image, and if you're on SSL for every handshake, you're making another 200 milliseconds added onto that page request. So, on a big, rich, heavy site, you could be adding four, five seconds on every page load just in network overhead. Another consideration is the number of requests. So, you know, Drupal does a pretty good job of managing this with its CSS and JavaScript aggregation, but, you know, we'll talk about some other ways that we can optimize that later. The other thing is payload size. So, you know, how much data is it taking to build your page? Why is it you can reduce that, optimizing your images? I see a lot of Drupal sites where the JPEG quality is set to 100. Yeah, I mean, you know, that's just as simple as a no-brainer. Just, I've found 83 to be great, you know, in terms of the quality of the image versus the size of the image. You know, minify your CSS and your JavaScript. With JavaScript, you might have to do that yourself because Drupal just aggregates it, doesn't actually compress it. And your markup size. So, you know, any of you front-end guys out there, you mean I'm sure you love this Drupal 8-tweak initiative and you know, really like trying to kill markup, I think, as Morton says. And the last thing I'd like to address in front-end performance is your JavaScript. So, you know, really rich JavaScript heavy websites, you know, can be quite slow and sluggish in your browser. So, you know, you want to have the considerations here of the number of scripts that you're actually running on the site. You know, are you running your script synchronously or asynchronously? And how efficient is your JavaScript? I mean, I've seen some pretty horrible things in there as well and a number of like, you know, services that offer these, you know, tools to integrate with them that are very slow. Add this. So, what is scalability? This is a quote from Amazon CTO that, you know, I think is poignant. It's really hard to retrofit scalability. It is so much easier and you save so much time if you're able just to do it from the beginning and have it as part of your planning. So, scalability is, you know, basically a system, scalable, if adding more resources allows you to proportionally serve more people. So, you know, will doubling your sites serve a number? Double the traffic it can handle. And I, at about 1 a.m. this morning, I put that nine women can't make a baby in one month, quoting, I don't know how it's relevant. But, no, I do know. So, essentially, just because you double the amount of resources you have available, doesn't necessarily mean that your site's gonna be twice as fast or be able to serve twice as many people. You know, you've gotta have the infrastructure in place to be able to do that. All right, so, with scalability, things we are talking about are your caching layers. So, Drupal has, you know, the block caching, your page cache, you're able to put reverse proxy caches in front of your web server to help alleviate strain on your web servers, opcode caching, so, you know, PHPs caching of your actual ARM scripts, and infrastructure. So, you know, are your web servers load balanced? Are you able to set up a MySQL cluster? And do you have high performance caching backend like Redis or Memcached? Is there any questions on things that are all clear? Cool. All right, so, common problems. This is a big one that I see constantly, and I've been responsible for a few of them as well. Uh-huh. So, too many modules. And it's very easy to do, because, you know, Drupal, the saying is, features are cheap, details are expensive. So, I mean, you just go to Drupal.org, download all these great country modules, and install them, and you've got this really great site, that it slows hell. And, you know, features is a big contributor to that. You know, how many of you have seen sites with, you know, 20, 30 features exported out? I mean, it's pretty common. So, this is the worst that I have ever seen, and it's not yours. So, this was a pretty crazy site. They had 24 core modules enabled, 51 custom modules enabled, and 72 exported features. The rest were all contrary, out of 365. Every single page request required 750 files to be loaded. So, you think about your views handlers, you think about Ctools, you think about module files. Yeah, and so, 10 to 20% of every page request was files being loaded to be executed. So, even with APC on, 10 to 20%. That's because APC does this stat call on every file. And, so not only that, but you've also got this additional CPU overhead, where, you know, for every module that you install, there's another iteration that needs to go through when you're invoking hooks. So, in this site, we'll have a 25,000 calls to module implements on every request. So, pretty crazy stuff. The solutions for this, I mean, it's not an easy problem. If you've got a site that you inherit a site, or you've got an existing site that has this sort of problem, it's really hard to fix, I mean, wait, can you just start turning off features, your clients, you know, that's not palatable to them. So, things you can do, consolidate features. There's two bits, I've sent off at least two emails to them. They had a script that, as part of their build process, consolidated all their exported features into one. I've asked them for that, but they haven't responded. So, I'll, you know, keep an eye out for that one. Custom modules, you know, try to consolidate those where possible, you know, it's great to have this, you know, modular flexible architecture, but, you know, you've got to make the call about where to draw the line. I see a lot of custom modules that have just got one hook in them. So, you might better just have a single glue code module that you put everything into. And the other issue is, you know, some things, you know, take, some features take, you know, three, four, five contrary modules just to get one little feature running that could have been done with 100 lines of custom code. So, once again, go to way out, maintainability with contrary or performance with your own custom solution. Another one I see is anonymous users being given sessions. So, you know, this is, I was trying to think of an example, but I just, I've seen things in the realm of this, in the wild, and it's, you know, it seems innocent, but if you don't know, this has a number of consequences. So, pages that have product at the start of the URL are never cached. Never, because there's no logic in there to say, you know, if the user's logged in or logged out. So, every single user is getting a session on that page. And that means that every single user who visits one of these pages, all their subsequent pages are now not going to be cached because they have a session. And Drupal, by default, won't serve a cached page to people with sessions. And then, as a next step, every single page request, their session data is being rewritten to the database. So, not only do you bypassing the cache, you're also going all the way to disk to update the person's session details. So, what's the solution? Just don't do this sort of stuff. You know, that would be my main recommendation. There's other things you can potentially do. I think there's, you know, not that I've done it, but there's session storage API module where you can store your sessions in Memcache and still use the page caching, get around that. But it is a lot of work, not a simple solution. Another common problem I see is very complicated entity field structures, information architecture issues. So, this is just an arbitrary example I made up with a content type, a couple of field collection fields which then have, you know, child fields underneath that. And I put a media field in there for kicks as well. And, you know, like if I've seen examples where you can have a multi-value field collection and it's got, you know, 10, 20 field collection items in it which then all have all their fields in it as well. So, I, this slows down your form submission because there's a whole lot of database interaction that happens every time something's updated. It slows down your rendering because there's just such a massive data structure to iterate through that slows down views because, you know, if you're building views that are grabbing, you know, field collection dependent fields, got all these additional joins that are happening. And yeah, so just generally slows the things down. As an example, I kind of just did a bit of a brain dump of how many inserts you would have for that example we just showed then. So, you got the first insert for your node table, the next insert for node revision, field collection item, field collection item revision, that field and the field revision table there. The fields and the field revision tables for the bottom of the chain fields. Just goes on and on. And we, you know, we also have the file entity media field going on as well. So, God forbid if you have fields attached to your file entities as well. So, it comes to about four end queries. And that was a simple example. I mean, I've seen far more complicated field collection, you know, related information architectures. And I've seen, you know, entity saves that can take 20, 30 seconds. So, if you all got a busy site and you're trying to do that, you're gonna have a bad time. I don't know, I certainly did. And just a couple of other common problems to avoid. You know, views PHP, not good. You know, it affects your views caching and it's just much better to put that stuff into handlers and plug-ins. You also got security concerns with that, but that's for another talk. Complex faceted search using the database. So, you know, Drupal's database is not really designed to do, you know, complex searching and, you know, partial string searching and things like that. So, where possible, try to offload that into solar. That's, you know, a way that we can kind of horizontally scale as well. The DB log module enabled on production. So, you know, this one, I find it really easy. I find it very difficult to debug the syslog module, you know, like in terms of, if you get an entries into the syslog, it's also getting muddled up with actual system logging entries as well. So, I used, like I have been guilty of putting DB log on production, but if you've got, you know, lots of notices and warnings in PHPR is coming through, every single one of those is doing an insert in your database. And so, not only is that then, you know, going to the disk to, you know, do an insert, you've also filling up that table, which is slowing down your database as well. So, yeah, use syslog where possible. And the other big one is node access modules, by default in Drupal 7, disable your block casing. So, rethink, do you need, do you really need to use node access modules? You know, like domain modules are a big offender, you know, you turn that on instantly, you got node access even if you don't need it. So, that's just something to think about. Any questions on the common problems? Okay, great. So, now we're going to talk about some strategies that you can use at the very beginning and the planning phases of your project to, you know, have a smoother ride. One that I really like to do on my projects is on-demand cache purging. So, if we're talking about Drupal's page cache, normally you would have what's called a periodic cache, where you can figure whether Drupal is caching pages for five minutes, 10 minutes, you know, whatever you can. Generally, the longest amount of time you can before the content's gonna become stale, and when Drupal generates a page, passes it to Varnish, and Varnish says, okay, how long's it valid for five minutes? Great, and then it serves it out for five minutes. Once that's over, it then throws it away, grabs his new fresh copy. So, on small sites, that's a fine strategy, you know, it's simple, it works most of the time. But for large sites with a huge amount of content, and you know, maybe a lot of pages that aren't changing very often, you really just, you're wasting server resources by regenerating pages that haven't changed. So, what on-demand cache purging would do in this model is user requests a page, it generates it, and it gives it like a really, really long lifetime. You know, you can potentially give it an indefinite lifetime, but you know, let's just say, make it six hours rather than five minutes. That means that that page can live in Varnish for like 10, 20, 30 times longer than it would if you were using the period of caching. So, that's a really, you know, fairly easy solution to put in place for sites where you're not really gonna have content going stale very quickly. Some considerations with this are relative dates. So, you know, if you've got a timestamp in your byline saying, posted 10 minutes ago, but you're caching pages for 12 hours, that's gonna go out of date pretty quickly. So, you wanna use specific strings. And some pages may not be suited to this kind of caching. So, you know, you might have like the homepage, for example, if you set up your rules to, you know, delete the homepage every time a node is inserted or updated on your site, I mean, you might never actually have the homepage in cache. So, it might be better to have that cached for, you know, five, 10 minutes, something along those lines. So, the way it works is that you, like the way I like to plan this sort of implementation is divide your site up into different page types. So, you know, it might be landing pages, homepage, product pages, news articles, blogs, you know, however it makes sense to divide your site up. And for each of these page types, you wanna come up with a list of events, a list of actions that would cause the content on that page to change. And when those actions take place, you wanna clear those pages from the cache. So, you can, ooh, so you can set up using rules module, you can set up those rules. You can also set it up using custom code. So, if you prefer to look at things that way, or you've got very specific events that rules doesn't support. Something in other ideas that I've seen people do to extend this is have like a spidering script to warm up your caches. So, you know, if you have to do a general cache clear, you can just hit a spider on your site and everything is already in warm for your visitors when they hit the site. Another thing you might be able to do is extend this using cache tags. So, cache tags is a utility module that tags cache entries with keywords. So, you might be able to do, you know, for example, let's just say you've got entity cache and you're caching, you know, like your article content type. And so, you can create rules that say, delete all node entity cache records that are articles rather than its default behavior of having to clear all the nodes from the entity cache. Another tool I like to use is AuthCache. So, yeah, Drupal core, page caching works great for anonymous users. But once you start having a site that's heavily trafficked by logged in users, you're gonna have issues with your server resources because, you know, they are bypassing the cache on every page load. So, this, yeah, it did mean like that my personal experience with this have went from authenticated page load times of eight, nine, 10 seconds to half a second. So, you know, that's a drastic improvement and well worth the time if you've got a heavy site for logged in users. But it does more than that. So, more than just caching the pages for logged in users, you're also able to do personalization. So, you know, very basic example, you've got Hello Nick on the sidebar of your site. You can do that using Ajax or Edge site includes. You're also able, you know, like an issue with caching pages for logged in users is the CSRF token in forms that, so you wanna use the AuthCache form module to get around that. You can also store these AuthCache entries in varnish which is another like massive improvement over storing them in Drupal's caching. And this integrates with cache expiration which we talked about in the previous slide. So, you know, like those two combined, money. So, in terms of how I would go about planning an implementation for AuthCache is to find what page types are cacheable. So, not all of them are going to be. You know, your user page might not be, your cart and checkout pages are not. And you wanna design how you're gonna segment your visitors. So, you know, generally, you know, Drupal, in the Drupal world, you just divide it down the authenticated and anonymous line. But, you know, if you've got, you know, a subscription-based site where you've got different tiers of users that have access to different content, you might wanna have base your segments on those subscription tiers. And you also need to do, you know, if you're retrofitting this, you wanna do an audit of all of the personalized content on the site that needs to be displayed to the user. And if you're planning, I would just do the minimum amount. But, yeah, you need to identify all the personalized information. And gotchas, you know, forms can be tricky. So, when you're retrofitting this, you know, you gotta test every form because Drupal adds the CSRF token to every form that gets put with the form API. And you wanna make sure that analytics, marketing tracking scripts, you know, I know that a lot of sites are using, you know, like user tracking, so they might be saying, you know, this user has visited this page and then this page, and it's using personalized information in these tracking tags. So, you just wanna make sure that you're not breaking those, or your marketing team will not be happy. For a great out-of-the-box implementation, check out Commerce Kickstart. There's a lot of documentation on that distributions project page, too. Another big thing I see is consuming a lot of data and a lot of web services in the back end, in PHP. And doing this kind of operation can be really, really resource intensive. So, you know, reading in RSS feeds, or, you know, anything of that nature. Integration with backend systems, stock sinks, that kind of thing. So, you know, any way you can think of doing it in PHP will have the same fundamental problem. And that is that you're fetching large data sets, and that hogs your IO, memory, CPU cycles, uses a lot of resources. And it results in a lot of slow, insert and update operations onto the database. So, it also has, you know, your new data, that you've just pulled in, won't be displayed on the front end until the caches expire, or you need to clear the cache to get it to bring up. So, you know, it's not even all that timely. So, the solution that I've used in a number of projects is moving this functionality to the front end. So, what I mean by that is, you know, like, for example, RSS feeds, I've used this PaaS module, and it basically just adds a little jQuery plugin that, you know, you give it an RSS feed path, and it'll fetch that and render out an RSS feed. So, rather than having, you know, like, let's just say a hypothetical example where your, each user can add their Twitter profile onto their user object, and on their user page, it, you know, displays their Twitter feed. So, you know, if you were trying to, if you had a site with 10 users, that might be fine to do with feeds. But if you had a site with 10,000 users, every time Cron runs, or it's gonna be trying to update these 10,000 Twitter feeds, and that's a huge amount of data and a huge amount of work it needs to do. So, offload that to the front end, let the users' browser handle it, and the Twitter API. Other solutions that I've used are AngularJS. So, Angular, for those who don't know, is a very powerful front end MVC framework, so built in JavaScript. The usual way that people use it is not really relevant to what we're talking about, but it is very powerful in terms of being able to use, you know, integrate with services, web services, pull feeds in, and render out data really easily on the front end. So, on a recent internet project, we had like a multitude of back end systems that we needed to integrate, and every single user of the site had customized information, you know, like based on their access levels, based on preferences that they'd set, and things like that. So, it was just gonna be an absolute nightmare to try and pull that data in to Drupal, and then display it out to the user. So, we put together a module called AngularBlocks, which is a sandbox module on this page, the URL there. And basically, this allows other modules to expose AngularJS apps as a block. So, you know, on the user's profile page, they had a whole bunch of dashboards that they needed, and each of those, each of those blocks, maybe six of them, were an Angular app, and they all pulled in information from a different system. And that was great because we were able to cache that page using AuthCache, and all of the strain was put onto the back end, you know, the other services that weren't Drupal. Another thing, load testing. Who has load testing as a part of their standard project timeline deployment process? Was that like four hands? Five, yeah. You need to be doing this. And if you're working for a sophisticated client, government, enterprise, you know, even some of the smaller people, you know, like this downtime at launch is not gonna be tolerated, and it's, you know, nor should it be. So I stressed all of you to try and integrate load testing into your development and launch process. Don't leave it to the last minute, because that's too late to fix the issues, you know, like you're gonna be trying to, you know, put the spit and polish on it, not worry that you've fundamentally made an absolute mess of the information architecture or that you don't have any caching in place. Tools that I like to use for load testing, so Jmeter, Jmeter is an application, load testing application, as you described, the way that, you know, like user scenarios. And there's a couple of cloud-based tools that allow you to run and execute these tests. So one of them is Blazemeter, another one is Blitz, and that web page test is actually more of just a simple implementation where you just give it a URL and it will automatically detect the links on the site and just run, you know, like you can say, I wanna run 200 users on this site and it will do that. Another thing that's worth mentioning are queues. So, you know, use queues when you're dealing with batch processing of large data sets, performing complex calculations, or if you need to have a sequential processing of tasks that may span more than what PHP is, you know, what your timeout or memory limit is set for. Some tools that I like to use for this are Advanced Queue, Advanced Queue Runner, and, you know, and the more less sophisticated system is Drupal Core Queues, which you can look up in the Drupal API docs in System Queue Inc. Reason for that, for using queues is that, you know, if you don't use queues, there's no guarantee that things that you are doing will be completed, you know, if your PHP hits a fatal error, there's no easy way to repeat that process, you know, like with the specific data that was passed to it. If you're using queues, everything you know is gonna be executed at least once. If it fails, you can see what it was supposed to be doing, and you can try to execute it again, and you can, you know, look at the logs, see what's happened. And System Load is stabilized, which is the important part, because these complex and long-running tasks are delayed, so then they're running in the background, rather than running when your web server is trying to generate a page for you. And Optimize Your Frontend. So, you know, things you can do in the theme layer, you know, make image sprites for all of your buttons and icons, that's just gonna minimize the number of HTTP requests. With your CSS, who is using SAS or less? Something on those lines, yeah. How many of you think about what is actually being compiled? Yeah, about half of the people are using it. So, you do need to take that into consideration, because you can do some pretty crazy things with SAS and less, that creates some extremely crazy compiled style sheets. And, you know, a lot of things you can do, consider, you know, conditionally including some CSS. So, you know, if you've got a site that has, you know, 10 sections, and each section has its own, its own color scheme and, you know, images and things like that, do you need to have all 10 sections' style sheets on every single page load, or could you just put that section's style sheets on that section? Something to think about, you know, like once again, weighing up performance and the technical debt of doing it the other way. And something that I didn't know until relatively recently is CSS rendering is a blocking process. So, if you've got really crazy style sheets doing really crazy things, and especially in older, less efficient browsers, you're gonna have problems. Asynchronous JavaScript. So, we talked about this briefly earlier, that the, you know, if you've got a really rich JavaScript heavy site, that you can have, you know, browser performance where it takes a little while for your mouse to be able to click on things or to move images, take a little while, animations don't start. What I would recommend is looking into this async.js module, which delays your JavaScript execution and, you know, it can improve at least the perception of performance on a JavaScript heavy site. And also advanced aggregation module. You know, who's used this before? Who has had a bad experience with it before? Yeah, nearly all of you. Look, it's a good module, but it tries to do a lot and you just need to be aware of what the implications of each checkbox are. Elysia Cron. This is an O-brainer. So, you know, Drupal Core's Cron will clear all of your caches on every single Cron run. And that's just, I don't know why it does that, but it does. So, get Elysia Cron in there, bypass that, you know, you don't need to clear caches every five, 10 minutes. And you can configure specific tasks to run, you know, during your low peak, your low traffic times, and you can configure, you know, like if you've got something running that needs to happen every few minutes, you can do that if you need. Entity cache, sort of a, you know, set and forget module, just turn it on. You know, if you're using, you know, commerce or bean, download, or enable their entity cache integration modules, because it's just, you know, it's a no-brainer, no risk and tons of benefits. And your floating search from your database to Apache Solar, I mean, that's a, if you can do that, absolutely do it because search is a huge performance hog. And none of it can be cached. Yes, 10 minutes. Shit. All right, we'll just swing through infrastructure then. So, just quickly talk about caching backends. So the two primary ones that I see, or two most popular ones I see in Drupal, Drupaldom, Memcached and Redis. So, you know, just to compare the two of them, Memcached, it's old, you know, I think it was some like live journal or something built it back in 2003. So it's battle tested, you know, it's widely deployed, and Aquia use it, as well as a number of other cloud hosting providers. My issue with it is it's volatile. So Memcached just stores its data in memory. So if you restart the Memcached service, if you restart your server or anything like that, you're gonna lose all of the data that's in there. And that's fine for caches, because the caches, right, I mean, you just rebuild them. But Drupal has caches such as the form cache, which is actually where it stores all of its CSRF tokens and submission information for multi-step forms and things like that. That's not volatile, but that's persistent storage data that you need to keep. So on these days recommending Redis, which is, you know, it's less mature, it's only a few years old, but it does have a one-to-one feature set with Memcached. It actually has more than one-to-one, it's very feature rich. So it benchmarks slightly better than Memcached, so it's a little bit faster, and it commits your data to disk in the background. So whilst it uses memory for its high performance reading and writing, in the background, it's putting it into the disk as well. So you've got persistence there. So you can use it for your form cache, for example. If you do use it, I would use the PHP extension to integrate with it, not the class, just because it's quite a slow class to use. Yeah, sorry, I'm just repeating myself here, but the other thing you can do is put user sessions into Redis as well. So this is a, you know, if you've got a lot of authenticated users, or you've got, you know, a commerce site where people are adding stuff into the cart and we need to remember it. If you use something like session proxy, you can then configure that to go into Redis rather than into Drupal's database. I was gonna talk about actual server topology now, so this is the simplest approach. This is what most people, you know, most small sites will be running, you know, if they even have Varnish. So it's a single server with your web server, PHP, and your database. Yeah, sometimes you'll have Varnish on there too. So, you know, if you've got this sort of site and you've got that sort of server, sorry, and your site's starting to get a lot of traffic and, you know, things starting to slow down, what do you do? You scale vertically. So that's increase the size of the instance. You can change the instance type, you know, if you're on AWS, they've got, you know, CPU optimized, memory optimized, network optimized. There's a couple of different options you can choose there, but you will eventually hit an endpoint. So eventually, adding more RAM does not help the problem. It becomes an issue of CPU. And you're gonna need a bigger box. So the next step is splitting up your box into, or splitting up the components of your stack into separate servers. So, you know, in this simple example here, we've moved the database onto its own server. And then you're able to vertically scale each of those servers independently. You know, next step, maybe put Redis on another server, or like, you know, an AWS Elastic Cache instance. But you will still eventually hit an endpoint, even if you have every single component of your stack on its own server, you're still eventually gonna reach CPU thresholds. So, the way that we overcome this is using a horizontally scalable architecture and as you can see here, we've got a load balancer behind Varnish, which is spreading out all of our load between four web servers using Apache and PHP. And then they all connect independently into our Redis, Cache backend, and our database. Now, you know, this is not the most sophisticated setup you could have. You know, for example, like the bottleneck eventually on this will become your database. You know, we can only scale our database server to a certain size before, you know, reading and writing is gonna become an issue. So, you know, the next step there might be a clustered database, you know, database cluster. And, you know, you also wanna think about your file system performance. So, all of these, all of these web servers for your Drupal files will be connecting onto an NFS server or something like that. You wanna probably, you know, eventually look at putting that into Gluster FS. And then, yes, yes. And then eventually you will attain the level of God where you're doing auto scaling. And, you know, on AWS or Rackspace, you've got a system where depending on the amount of load your servers are under, so, you know, you can set thresholds once your server hits 80% of its CPU utilization, add another server, and you can just keep doing that. And then when it hits 10%, take a server out. And so, you can, you know, if you've got a big site with a lot of traffic and, you know, an invariable demand, you can really save yourself a lot of money on your AWS bill if you're doing this sort of thing. All right, and quickly run through ways that we can debug performance and scalability issues. So, New Relic, who has used New Relic? Cool, yeah, so, New Relic is a software platform that allows you to monitor the performance of your site. So, you know, how much time is spent in the database, how much time is spent in PHP, what queries are taking a long time, what, you know, functions are taking a long time. They also have great browser and server monitoring tools as well, which I urge you to check out. Look into your MySQL slow query log. So, you know, the slow database queries are like the number one thing I see that can be improved on a lot of these slow sites. So, it's like the way that you enable that. Just these two lines that you want to put into your MySQL configuration file and restart. You want to profile your application. So, how many of you have used and installed xhproff? That's more than I was expecting, yeah. So, installing xhproff is easier than installing xdebug if you've done that. And, you know, there's modules that you can use to help, like memory profiler. But, Mark Sonoblon from AQUIA did a great set of slides that I saw. I think it was Drupalcon Sydney. And, yeah, check them out, because it was just, I mean, it was really easy to set up and he's got a great set of slides on how to use it. And your browser development tools. So, you know, Chrome or Firefox, even IE these days has, you know, tools like your JavaScript profile and a network monitor. So, you can, you know, how many requests are you making? How big are your images? How, you know, how slow is the JavaScript? Why is it slow? What's the good tools to use there? All right, so just some general tips when you're debugging these issues, you know, sometimes it's hard to see the forest for the trees, you know, like you gotta look beyond the symptoms to try and find the underlying cause. You know, why is your database running it? You know, like why is your application spending 30 or 40% of your time in the database? Maybe it's not because you've got slow queries, maybe it's because you've got a lot of inserts and how can you get around that? Maybe you can put data into MongoDB or into Redis or something like that. When you're trying to fix things, just change one thing at a time and measure it before and measure it after. Cause there's nothing worse than not knowing if the fixes you're making are making a difference and more to the point, you know, if you've got to justify your time to business people, it's, you know, you gotta show them your value, you know, like, okay, so this day that I spent and saved us 3%, you know, like a second on our page loads. And the unfortunate reality is sometimes you just gotta throw more RAM, more resources at the problem. You know, if you've got, you know, 512 meg server, you're not gonna be able to serve pages to a thousand people, you know, it's just, the reality is that you're gonna have to spend some money to make your site run better. That's it, any questions? I think we've got about a minute. No heckles? All right, yes. Yeah, definitely. So New Relic has, like, tools that allow you to, you know, identify scripts that are taking the longest time, you know, with external requests. How do you make it fair? Well, you, you, I mean, you know, like, so if you have, like, a New Relic pro account, you can log your deployments, you know, and so, you know, if they say, oh, hey, Marty, we wanna add tracking script number 20, you can go, all right, not my problem, you're the business user, I'm just the voice controlled keyboard. And you log your deployment and you can show them on New Relic, like, you know, before, you know, the JavaScript performance was, you know, average, cause you've still got 24 scripts on there. But when we put number 25 on, you know, it's significantly slowed things down. It's taken a long time to execute, cause this service sucks. But, you know, ultimately, unfortunately, developers really are just at the mercy of what the business or the client wanna do. And if they, if they think they need this extra analytics tool, then, oh well. Yeah, I mean, Muenon's great. So if anyone doesn't know, Muenon is an open source server resource monitoring tools and it keeps track of a lot of different things. And it's very customizable. So, you know, you can, instead of running on the cloud on some service, you actually run it on your server and it gives you like a nice, pretty interface with graphs and things like that. But it's actually, it's not that pretty. It's like, obviously, built by developers. But, you know, New Relic is far more interactive and gives you a lot more, like, you can drill into things. So, you know, you can see, like, okay, my database at this, you know, particular time was taken, you know, 40% of all the page loads. So, why click that? And it gives you, you know, like a whole, like the longest traces for each of the requests during that time. You can drill into that. Yeah, it is. Yeah, it's tough. I mean, and in that case, you just got a profile you're out using XHPROF, yep. Yes, it's a bit of a religious debate. It's a bit controversial chopper. Yeah, so, like, I use Apache because it's just easier, battle tested. I'm familiar with it. But, you know, Nginx has, you know, it's been proven time and time again that it has a lower memory footprint and that it is faster and it is more flexible. You can do, like, reverse proxy caching with it and, you know, put Redis behind it, things like that. So, if you, yeah, I mean, if you, if you like spending time thinking about web servers, go for it. I mean, I would, I think you would benefit from it, but that's not me. Yeah, that is a good point. So, the point there was that, so in Apache, you can just add different rules and configuration to your HD access file in, like, in your Drupal site. Whereas on Nginx, that's got to happen in your server configuration files, which, you know, some people may not have access to. There's a hand up. No, scratch your head.