 How are you? How have been your day? That doesn't sound too cheerful. Are you all tired already from lots of sessions, interesting things? I can understand that, and I realize that the last slot at a day always is kind of a different slot, but we'll try to have some fun and have a nice session, learn some little things. But there's one thing. If there's any performance pro in here, I'll spot you. And so this is a beginner session. It is beginner level. There will be some things where you'll be like, whoa, what is that? I've never seen that before, but this is by one reason, because if you've seen it now, once you need it actually, you have already seen it once, and you know what it does, and that there is something like this. So maybe some parts might be a little different, difficult to understand, but it will help you in the future, for sure. And as I said, let's have some fun. So anyone that just wants to cause trouble or doesn't like beginner sessions or fun sessions, please leave the room now. Last chance. Sure. Our topic today is high performance, Drupal, four times high performance for Drupal, and it's done step by step. We will today cover different things, but let me begin with a very, very sad story. Actually, a real post on groups.drupal.org, and it goes like this, where is the power of Drupal? I hate Drupal. Isn't that sad? So, and it always overlords the database. Sometimes the Lord reached 200, and it never reached it before. Enable core cache and the server go down the next day. I think that's a real sad story. I got you the link, so if anyone wants to verify, you can do that. This smiley is also really sad that this was happening. So I think let's do something against this. Let's get you some knowledge so that this is never happening again, and I never have to reach something like this again. In general, there is still a very much need for high performance. It is like that very simple. If you have a faster site, you earn more money. You get higher in Google. Your visitors are like, I mean, if you were a visitor and you come to some site, you want to buy something, and then you're just saying, oh, it takes two minutes already. I think I'll just leave and buy it sometime else. And sometime else usually never happens. So there is a need for high performance, and it would be good if the Drupal community as a whole would have that knowledge. And that's kind of what you want to do. So my site is so slow. What can I do? Your visitors are going to love the fast sites. They'll say, oh, this site is really slick, and it will automatically gain your advantage in comparison to your competitors until they see the session recording, obviously. But until then, you get a little edge. The other thing is, and that happens more often than you can think of, you have a nice website, and suddenly some big media comes like bignews.com was coming to you and it's saying, hey, it would be really, really great if we could feature your website. We really think it's good, and you say, well, let's do that. And then the big day comes, your website is featured. You could do millions of sales, and it's down. Because it was just not made for this amount of traffic. So the real bad here is, how do I get a blazingly fast site? I want something like this. Let's first take a look at how to do things wrong. For example, what I could do is, well, I've now tweaked my Sauerkraut settings, and I enabled some other Sauerkraut settings, and it's still not fast like xyzat.com. Why is that? I tweaked them as long as I could. And Sauerkraut in this case is APC, but we are not yet there, so I just replaced the word because it could stand for anything. Like, you're just trying to really change one thing and you tweak it, tweak it, tweak it, tweak it, tweak it, and nothing happens. The other thing is, well, I've set up 10 slave DB servers. My site is still slow. What's happening? I read on the Internet, I just need many slave DB servers. Then it will be fine. The other thing is, I've set up NGINX with advanced ag, and one is combined with NT Cash with used opt cash, but still nothing happened. So this is a person that really has a bleeding edge of sayings, and the newest technology and still it does not work. So I asked that person, have you set up Memphis? It's never like, what? Never heard of it before. So the last thing is for our high traffic day and that's kind of a true story. In a way, I've set up the static page caching for all the pages. The high traffic day can come. What could possibly go wrong? And I can tell you what went wrong. I was on the machine and I had prepared a little more than this would suffer, so I had done some tests, et cetera, and it looked all well. And it wasn't that machine, and I was watching the Lord of the machine, and suddenly there was a spike. And the Lord was giving up from two to three, to four, to five, to six, to 20, to 40. I was pressing quit to try to stop this apache so that I could do something about it. And while I was typing, I was really feeling like in a movie when the bomb is about to explode. While I was typing, it got slower and slower. The keystroke was taken, the next keystroke was taken, and then boom, server froze and was dead. We quickly switched over to a failover machine so it wasn't all lost, but it was really, really scary being on the system, and you cannot do anything. So there can always be things that can go wrong, so you always need to have an emergency case. But in general, with this high performance, we all wish that we had the magic pill. You just follow one pill and your site is fast. Wouldn't that be nice? But unfortunately, optimization is a process and not a pill. So from the examples that we've seen before, you can kind of see now in a more explanation, and here's what I need. So let me explain this a little visually. If this is your performance setup and you have it nailed down every here, it will probably hold good. But if all what you're doing is putting a nail in here and you're making it more and more and more, this is still not holding up. What is that? So this is really, if you're just tweaking one thing, it won't work. You have to work on all pillars and you have to make sure that you have a good space performance. Then this will work good. Let me give you back my presenting advice. The other thing is that you're just optimizing things, but you don't know where the pain is. Where is the pain? What is really happening here? And if you don't analyze carefully, you end up in scenario one where you're just trying to tweak something that leads to nothing. And the other thing is that you're reading some pills from groups.ruple.org and there's some nice technologies and this password and this password and you think, well, I just need to install them all and then it will work. We are again at the magic pearl here, obviously. But it's not like that. Unfortunately, it does not work like this. So do you really want to reinvent the wheel or do you want to rather stand on the shoulders of guides? Another thing is, and I've explained this already quite a lot with an example, you're featured on the bignews.com, you didn't test, your site is down, your boss will be very angry. So just summarize it again. You can optimize one part to death, just some random parts, some parts with passwords or things without testing. I think there are a lot of ways to fail. That is also complicated. Is there really nothing I could do to make my site fast? I really only want to have a fast site. Is this so much? Well, the easy answer is, hire a performance consultant. Now call 0800 Drupal Performance and enjoy blazingly fast sites. Call now in this second and you will have blazingly fast sites like me. You'll get more sales, more Google hits and everything. So that was it. Now you know that performance is really difficult. You should hire a performance consultant. Here's a number of any questions? You got me, I'm just kidding. I think having a performance consultant can in ways be the right choice. Because for example, if in two weeks you get your high traffic date, it's probably good to contract with any of the big providers that are providing services in terms of high performance and letting a professional do this. Because in two weeks you will never be able to gain all the knowledge, even with this talk, and to be properly prepared for that. But I really think we should take this knowledge into the Drupal community. And that's why I'm here. Because we want to stand on the shoulders of giants and walk the path of our ancestors. So, here's your mission. Lauding mission file. We have Drupal 7. We have several performance problems. And those are real life problems. Not really real life problems because I've simulated some things, but it's kind of all a little based on my experience with certain client sites where the most interesting things were happening. You can't imagine. So, Drup pages are feeling really slow, sluggish, and big. And they're really unhappy. This is so heavy, Lord, we feel like. And let me get Mrs. Myers-QR. She's also very unhappy because she needs a timeout. And to talk it in her words, it's like, I really need to select break. And Mr. Patch is sweating under the water. Well, I give it 100% all the time, but this is just too much. And Mr. Couch, he's really buggy. And he should be really careful about them before he's a real trouble maker. Yeah. So, to just summarize it all again, we got the Drupal pages, let's be sluggish and slow. Myers-QR, Mrs. Myers-QR, who's really unsatisfied and has so much to do. Then we got Mr. Patchy, who is sweating under the water, not just because of the heat. And Mr. Couch, the trouble maker. So, your mission, and you're all in with me here, is to investigate and fix. Let's go. First of all, we talk about server performance. Let's measure it. So, on... not the system I'll show you, I've seen these numbers. And it was really shocked, like, what is happening here? We get to that case later. But for now, we can just take a look at the top command and this is one way to measure things. You're just logging into the server. You're on top and you're just noticing how much Apache is using. In this case, you can see Apache is, like, fully working and Myers-QR is also fully working with two flags. This is something very handy. I won't give into much detail here, but you can later get it from the slides or from the session recording. This is a drush command where you can easily get the time that a page is taking. So, if you're trying to optimize something in the code, this can be really useful to kind of automatically do it and not have to reload the page. Is it working now? Reload the page. Is it working now? So, just to give you the edge. So, and now we come to the four shoulders of the guys. Let me sync my mouse. Okay. Let's go here. That's Pressflow. APC, MAMcache, and Boost and Wansh. So, how can those help me get across the site? Pressflow was really, really needed for Drupal 6, but I know that there are still people running Drupal 6 site and they will also run Drupal 6 site for some while. So, it's still useful to know if you want to have high performance with Drupal 6, you need Pressflow. Absolutely. Drupal 7, on the other hand, fortunately already has many things that Pressflow had. So, there has been really great progress in terms of high performance with Drupal 7. Nothing else is needed. So, APC, which is the Alternative PHP cache. It's highly recommended. It's very easy to install. It's a PHP extension, which means that if this is your code, again, what PHP is doing, it's taking all of these files and it's compiling them to something, and this is what is executed then. But, obviously, this takes some time, as you can see. So, what we're doing with APC is we're storing a copy of that here. So, if anyone wants to have the same files, what we're doing is we're just taking this again and we have that. And another question, obviously, is for APC, is it versatile? So, let's see. So, what we see here is kind of... We're going to the Network tab. I right-clicked and selected Select Element, Inspect Element. Now I'm reloading the page here in the Network tab, which we can now see is kind of how the page is loading. That's very, very handy for optimizing. And we can see it's a total of two seconds, this means. And it looks just like a normal Drupal site. So, what is happening here? Let's show the other example of if you have APC enabled. So, again, we open the Network tab, we load the page, select our Network tab, then after a short loading time, we can see we are now at 1.31 seconds and we're even locked in. So, APC is really gaining as much. In this example, why is this Drupal site so slow? And that's kind of a warning I want to give you here, is if you install lots of modules and you don't have APC, you have a problem. And so, let's get back to presentation. I would say, yes, APC is versatile. I installed quite a lot of modules there just to make the site slow. And this can be enough. There's also a little bug hidden here. As I said, the code is a little troublemaker here. And then we come to another optimization and that's memcache. Who of you knows what memcache is? A little. So, again, if you check one, two, perfect. Cache is very simple in a way because Drupal is caching things. You obviously have all been in the caching sessions and now everything about it. It's like this APC. We're just taking the finished page request and we're storing it. We're taking a finished block, we're storing it. We're taking a finished filter format, we're storing it. And there's one problem with databases. Databases are really good when it comes to relational data. That means you have several tables which you have to combine and then you get all of this data combined back. But a database is not good for just storing data in and getting it out on a prefix key. There are optimizations that are out of scope for the sessions that can do this also with databases and those will happen in the future but until it happens, we're going to go with memcache and memcache is really just providing one thing. You've got a variable called apple. You're storing a tan in it and if you're getting the apple out of it, you'll get the tan back. So it's really, really simple and it's very fast. So again, let's see, is memcache versed here for our little scenario? We had a page load time of 1.11 for it and we'll take a look at another video. So 1.16 seconds. Now we enable memcache. That's a normal Drupal settings file so I'm just putting the configuration in like cacheback and cachedefault class. I've downloaded the memcache module so it was quite easy once I had the demo running. And we will reload the page. Wait a moment and here we are. We're down to 996. Not bad at all. So we got some out of that. Please note that the standard Drupal caching is still off at this time so the next comparison will be a little unfair but even then I can assure you that it wouldn't be that fast. So the next technology we're going to check is boost and warnish. Boost and warnish do something different. Again, we have a cache but this time it's not inside of Drupal but it's in front of Drupal. So Drupal is kind of here for example and it's producing all of these little nice glasses. Then we are getting them out as client then obviously to manufacture a glass takes some time. So what we're doing is we're just taking a copy of the glass and we're storing it here and that's our warnish. So in memory we now have a copy of the glass and as long as the same input comes in here we get the same glass out but instead of Drupal answering this request the warnish is sitting in front and it's giving back kind of the copy of the glass. I'm sorry. Boost on the other hand is very handy because you can save files to your file system. That means for example if you have a website with thousands of pages but they are seldomly changing it can be very, very rewarding to save those files to the file system and if you for example restart the warnish cache due to some reason all of that caching is gone and also it's kind of because it's in memory memory is a limited resource it's a scarce resource it's an expensive resource it's kind of limited so it's a very good idea to just combine boost this warnish unless you're using nginx but that's not what I covered today here and then you kind of have a cache that's not so often changing and you have a cache that's always having the hot content so what visitors are currently looking at is what is there and warnish can be a little difficult to install though usually the real problem is not installing the package because on Ubuntu for example you're just doing appgap install warnish and you have it but it's more like the configuration that's difficult and for that for Drupal 6 we had a very nice boosted warnish configuration on our blog and we are currently updating it to Drupal 7 there will be coming a blog post shortly I just didn't get to the little tweaks in the end and then you can very, very easily have boosted warnish running together and you won't have to worry about things that can make using warnish complicated like we will see a little later so yeah, that sounds pretty complicated is then everything what we have here is it worth it take a look at another video so this was our memcache now we're going to enable warnish there's a certain rule to just quickly disable warnish we load the warnish server and now we load it once to get the cache to load and now we load it another time and there we go, five minutes so this is really amazing fast and warnish is also very good kind of while Apache would be sweating even if it was giving out boosted like static html files warnish is really, really good if you need highest throughput so it's a very suitable choice for a high traffic day so let's take a look at the mission update the anonymous pages say well, we are blazingly fast but we still feel a little big but we are quite happy Mr. MySQL says I've less to do now but those are syndicate users, I don't really like them Mr. Apache says I've much less to do now but again there's anonymous users, I still sweat and I really hate those anonymous UTM requests so what's happening that was actually what was happening when that one server got down the deciding question was using Google Analytics and there's two ways to set up URLs for Google Analytics one is you're having them with a question mark and then the UTM IDs the other one is you can just have a hash mark like an intro the problem was they were using question marks so we are getting a new cache entry for everything and this is just a little warnish tweak that's included in the boosted warnish configuration that will just remove everything Google Analytics related and this problem anymore as I said, blog posts will come soon so take a little look at client performance on some pages again I've seen page load size from 300 kilobytes and watch load time from 20 seconds and the reason was that it was not compressing things so again the best way to measure this is using the Google Chrome developer tab which I've already shown in previous videos so why are those pages so big and the answer is probably compression and aggregation is off so we need to compress the JavaScript and CSS and in Drupal 7 it's really really easy to set up it's really an administration menu you go to development performance there's this little tweak where you can enable those settings compress cache pages aggregate and compress aggregate and this is something you should really do before doing a go live but one caveat, you need modrevide and mod headers for this to work so continuing, there's another way that you could choose for example if you're more on Drupal 6 you could just get in mod deflate which is doing all of this for every thing so you can see that Apache is serving out and it can put quite a high load on the server so it's a good way to combine this with varnish because varnish is kind of caching those objects and if you set the times right for like two weeks or whatever and then Apache does not need to regenerate and regenerate and regenerate again so you can use those files and to set the proper caching headers that's especially needed for other varnish configurations the boosted varnish configuration is set up so you don't need this caching headers you can set this directing configuration but for example for the lullabyt configuration you need to set this caching headers and this is again done on the performance tab you cache the pages for anonymous users you have a minimum cache time of five minutes and an expiration of one hour in this case varnish would cache the pages for one hour and the client could cache some also for a certain amount of time that can be very useful because if you're saying like well I don't care if the client has like a five minute old version or one hour old version this can save a lot of bandwidth and this bandwidth can be crucial if there's many many users at the time so in Drupal 6 you would use press flow there's an option to use external caching or you can use the boosted varnish on trial and outcome let's take a look at yes it's a combination of varnish and boost let's take a look at the mission update the anonymous pages now feel fast, slick and they're just happy really happy so additional techniques you could use is you could for example use a content delivery network there's a Drupal or CDN module and this is really useful if you want to have for a user the file is very close to their location so for example if you are serving from America but those people need to have it in China in some little village or something but you know that there's kind of a distributor near there then you can push it to the CDN and it will be distributed around the world and users will get fast content so this is another optimization technique that you can apply there's also some new techniques that probably not too many have heard about yet there's AJAX and PJAX and AJAX obviously is clear in views only reloading the content that you need and PJAX is kind of the same but it's changing the URL at the top so you can use the history to navigate back and forth so for example if you had a big site for example with an image gallery you could just reload the images and that obviously gives a faster feeling because the users are not feeling well the whole page loads again etc and you can do this without having to preload all the images which especially for mobile can be quite problematic and so PJAX is definitely a module to check out and see what you can do with it there's one quick tip in terms of client performance sometimes you get this unresponsive script error on loading a page in our case this was Collabox and there were just too many links so we needed to run and we didn't want to change the Collabox code in that so the idea is very simple you're just setting it to run into the background just 100 milliseconds after the site has loaded and that's usually enough because most users are not as fast to click so and then you add the old code in the middle coming to module performance module performance you can sometimes have execute active handler things like 6 seconds in the memory usage of 100 MB, 200 MB it's important to know what's going on and one possibility is to use xhprof PHP extension for that it's well integrated via the devil module and again I'm having a little demo for that and here we take a look we'll find out why this site is so slow so what we are seeing here is I'm putting an xhprof equals one there but that's just needed because I've got some special optimization techniques in there that I'm rather advanced and usually it's enough to enable xhprof in devil and then we are taking a look at such a report and what you can see is that we have 1.6 seconds of page load times and we quickly find the caveat and that really is Drupal is scanning all files again and again and this is an old core version, this is fixed in your core versions that is just one to show you how such a little problem can add one second to each page load while the normal page load would be quite fast because it's just like 300 milliseconds 200 milliseconds can be too slow also in certain circumstances but one second is usually like that's not acceptable for certain use cases so and you can also see the memory usage more from this side so there's some comfort falls if there's a variable set on each page request your database server will have real problems if you're setting anonymous dollar sessions for saving simple data, for example for a low bandwidth flag, don't do that this will disable anonymous caching for varnish and other things instead use javascript to set receive the cookies and change the page that's working much better if you install so many modules it can be really problematic especially if you're not having APC another thing is if you want to load 5000 nodes with views you'll have like a memory usage of 300 MB that's happening with open layers there's an advanced solution which is called open layers quick query it's in my sandbox to improve the performance of modules one of the biggest secrets is to use the block caching enabling block caching and configuring proper block caching can make a real difference for authenticated users because most blocks might be really specific just either to a role or to a user or to a site or to a page so block caching can be used much more than it is normally used that's kind of an optimization for most sites that are coming in and there's something else that's very nice because there are some modules that are saying this block is not cacheable but you really want to have it cacheable and you want to have control over it and it's block cache alter just needed a patch with context I think it's applied we're not totally sure and but if you use block cache alter block directly set how the block should be cached and this is again very helpful also set up use caching it's obvious but you won't think how many sites I see where it's not done there's a nice list of performance co-patches in the groups that Drupal or CHI performance and I invite you all to join it it's a really nice group you learn lots and nice people hanging out there and we are just two less people yet come all in so I already showed the Xh4 performance video let's take a quick look at the mission update so it's only Mrs. MySQL that's a little unhappy because she just feels kind of slow sometimes in terms of database performance the question is to there can be queries that are slow like 10 seconds or so and then imagine several users doing this 10 seconds slow queries and you can imagine how the site performance is so what you want to do is you want to enable slow query log then for Drupal 6 STB tuner module really nice unfortunately no 7 port yet maybe someone wants to sponsor but you can directly use the MySQL tuner script for 7x and then there's explain queries but before I explain a little very little we get to some common tricks you can do you should use InnoDB it's the default of most current MySQL configurations but especially with Drupal 7 really recommend it be aware of the barrier you need to set no barrier equals 1 for X3 X4 file systems on your Linux account Ubuntu you can't imagine how many threads are there from developers that were saying well I update Ubuntu now I can't use MySQL locally anymore it's just getting all so slow so no barrier equals 1 is a good thread in that but be aware a little if you want to use or need to use this in production you need special hardware because this could lead to data loss and a useful guide is the Reddit handbook and I see that I'll put the badly URL in the comment another thing I highly recommend is to use the XFS file system it's a good improving file system for MySQL databases to size the partition size according to the use case so one way to fix the slow queries is you use explain for the queries that means you're using the select statement you got out of the slow query law you're putting explain in front of it and then it tells you something like this is using a temporary table this is using a temporary table this is not using indexes and if you don't know what indexes are look it up on some SQL board in that and then you add some indexes running explain again and hopefully it's faster so let's take a look at the mission update but before we get there so a little recap of the best practices um you set up the base performance then you want to have kind of your own high performance stack that you know and trust it's not too difficult as you have seen installing Monash, installing APC, etc I'm also, I promise this to someone and I'll do that, I just didn't get to it unfortunately before the session but you can expect in the comments that group.truple or high performance a little script that's kind of getting you through the setup of all of this first a little too too much for the session remember to analyze the pain points first and where's the problem? is it server based, client based, is it modules could it be the database really now where's the problem and then solve it and optimize those pain points let's take now a look at the mission update Mrs. Mya's girl is very happy Apache is really happy and the group of pages are also really really happy that means mission completed wake up now questions please yep, there's a microphone in the room it's a very basic question from the beginner my SQL the database, I have many if I delete many of the that I don't use is it helping the performance? usually unused databases if you're not using them are not kind of they're taking space on this but they're usually not important for the performance in that there's one thing to know about deleting things on the database this can be very difficult to delete things actually most production shops don't like to delete because if you're deleting on a production system and you are having some logging enabled you need to delete all of it as well and my SQL will be very busy with the deleting so usually you're just extending storage instead of deleting things usually you cite it appropriately so that you don't need to delete if it's a developer machine I mean sure delete but protection wouldn't do it also the my SQL user is all root but if I change the rule for only this database then is it also helping performance? no, it's not helping thank you further questions please if I install varnish which is cache in front of is there any user installing APC or memcache anymore? yes for authenticated users and obviously you restart varnish your timeout for varnish expires I mean you can set it like one year but sometimes this year is also over and then your pages are expiring so for this case you will still need to have APC and memcache and also usually if you if you want to use the system as an administrator which sometimes happens to add new content probably want to have it fast as well in not having to wait like 10 seconds even though the user see it very fast I reformed the question is there any use of installing APC or memcache for the sites where which varnish is okay the easy answer is yes there is because caches can expire and it's just a good baseline you just install it it usually works and that's also what you get in different configurations on all of the high performance hosting managed hosting services next question please you said that there is specific varnish configuration for say Drupal 6 I am on a server what if you had Drupal 6 on 7 is there a kind of middle ground configuration or can there be configuration that will help both sites actually the configuration works for both sites there's one little tweak to do but that's more on the Drupal site and that's what I explained in the blog post but you can use the boost varnish configuration I even have a module from way back in my sandbox that just emulating boost without doing boost when boost was not yet available so you have that on line and what you saw there as varnish configuration was also the boosted Drupal 6 varnish configuration okay so that will work you recommended using both varnish and boost together but I have been reading some articles that recommend one over the other for different context can you explain the difference a little bit more or why they would be used the people that are saying kind of that you don't need to combine boost and varnish together are kind of like saying well it's duplicated functionality it's duplicated things and in a way they are right but the problem is the argument like this they say well if I have all those files the linux kernel will have the hot files in memory anyway so it's very similar but this argument does not hold with Apache because Apache is not made for big lord if you are just serving static files it will still be much lower than varnish and it will also not hold a lord at a certain level because it's not made for that it's big and blown in a way so my approach of using boost and varnish is kind of to use the exploration features of boost and especially the disk cache which can grow very very large which are always available in combination with the hot memory cache kind of and that's the thing here for example to remove the double caching you could remove the block caching or set it lower for the disk you would get that advantage but usually it's no problem because varnish takes that memory the linux kernel is not using that for caching is there any problem in installing both boost and varnish under pressflow triple six? none at all the block post boost advantage is talking like that with pressflow questions? other questions? four minutes left now is your chance for questions little advertising for panels because there's a wonderful caching mechanism too for each pane you can select the caching time and additional dependence on the argument especially for views to repeat that in panels there's also a great caching architecture and again I recommend memcache which makes this again a little faster and in panels you have different panes and in those panes you can select how long to cache each pane and you can also do it based on the views arguments which is very helpful and that's what he was saying so further questions? caching or performance or no command session I'm hoping you'll forgive for those little things to wake you up and I hope you had fun