 So, hello everybody. Before we actually begin, just one small thing, so that's actually the first lab ever happening in a DrupalCon. And so it takes two hours, 15 minutes. We will do a break of 15 minutes. So the idea is really to first give you a presentation about performance and then actually like work together with you. The problem is they told us there will be tables. There are no tables. So we have to see the idea is really that like the second hour we build groups and we all try together to make our websites or your websites actually faster. So we tweeted at the beginning to know if anybody has a website he could show us or he actually has access to like if we have SH, SH, SH, access, thrush or whatever. Is there anybody here? Yes? Yes? No one else? Okay. Just any website you think that it's maybe or he could get faster and we basically will go in there and analyze it and check what's going on. And if you have a staging site kind of that, we can check. Yes, maybe not your live site. We will not. We don't take any responsibilities for any broken websites. At least. Better even SSH with thrush. Okay. So there are some. We can still build the groups later on. One interesting thing is who of you has never like done anything with performance? Like enable clicking the performance button or the caching that's not performance yet but who knows the button and that's it? Okay. It's not about blaming or anybody but we have to know like what is the level of knowledge. Okay. Who would say he knows more than like he actually used some modules. He installed memcache maybe even varnish. Okay. Good. So we have like and who of you would say that he actually should sit here? Come on. There is it. Okay. No, because we can actually be group, build bigger groups. So if you say later on, hey, all the stuff we showed, I use already we can. Well, yes. It's not saying we don't say that. Okay. But we can also build the fourth group because there are some a lot of people and we have more sites than than three. Okay. So there's still people walking over there in the hand, but I guess we can start. Yeah. Let's start with the presentation part. So first of all, did you have a good lunch? Great. Fantastic. That's me. I'm Fabian France, aka Fabianx from Google group, groups.drupal.org. Hi performance. Some might know me. I've did some crazy things with performance in the past and now work at tech one where I kind of belong with all the performance things. And I'm now a senior performance engineer technically there and loving it. Yes. I'm Michael or aka Schnitzel or also the Drupalcon photographer. So I'm a head technology at Maze Labs, we're an agency in Zurich and I'm doing one part of the performance in our company. So I'm Peter. That's Peter on DO. I'm doing performance stuff because I'm forced to. Clients want to have fast sides and I want what the client wants and we will help you to achieve that. I hope so. So have fun. We have a small agenda. First of all, Schnitzel already told you we will present an overview about what we think is the most important things in performance development. And that's, first of all, why things are become slow. You should know that to fix it later on. And that's the second part. Figure out what's actually slow. Really important thing. You shouldn't just throw modules at a slow side that won't help. It will make it worse. So next step will be how to fix things. And there we will have several approaches and show some modules and hopefully we can use that knowledge that we try to build here later on in the get hands dirty part. We will have one Q and A part after the presentation and after that it's just interaction. Talk whenever you want. Ask what you want to ask. Feel free to search this question in group with us. It should be really a laboratory. Let's experiment. Okay. As people are searching, there's one space here, one space here. There's one over there. So come on, fill in. Everybody should sit in here because it takes two hours. Okay. So for the first part, why things are or become slow? Now actually I would like to ask you, what do you think? What causes a Drupal website to get slow? Just throw it out. Yeah. Bad development. So bad code. Okay. Bad configuration, bad code. Yes. A lot of changes. Yes. A lot of modules. Just a lot of traffic. Yes. Big sites. Yes. Yeah. Sequential. Okay. That's yeah. Can be. Wrong cache setup. Yes. What do you mean with first party services? A third party service. So external IPA calls and stuff. Yes. Yes. Network. Yes. Images. Just a lot of content and also yes. So basically we want to point out three, but all of the mentioned ones are correct. At the end with Drupal complexity in any case. This can be a lot of media, a lot of content, a lot of modules, a lot of views, a lot of whatever. So the bigger or the more complex your site gets, the slower it will be. Just the amount of modules that Drupal has to load by a bootstrap already makes it slower. And we have to be careful that we not just add more complexity to make it fast again. So just throwing more modules in which supposedly change something or whatever if your issue is that you already have too much modules doesn't really help you. So that's really important. Then broken configuration. That's actually mentioned like misconfiguration of things. So we will see this later on a bit. But most of the time that we do something wrong or we did something in the past because the website wasn't as big and then it just got big and then we have the issue. And there is unfortunately quite some modules in Contrib which don't really care about performance or they're built and they test it out on ten note pages and it works well and if you enable it it's whatever like one millisecond smaller as lower or whatever. But then if you go in a big one, if you have a big site then suddenly it can create a lot of issues. One of them is for example the Drupal system listings call and what it does it goes through all files to find out which modules are installed. And currently there is one core bug or core issue. Let's call it issue. Which means that when a module is missing in the directory in your file directory on every bootstrap he tries to find it. So if you have a website with ten thousand hits per second he tries ten thousand times per second. Maybe the module is there, maybe the module is there, maybe the module is there. But honestly I guess it will not be suddenly there so we can cache this because the next time a user like somebody realizes this he can clear the cache so that system. So this is a lot of things that could make it really worse. And we're still trying to fix it. So does Peter, he is the maintainer of your... I'm just a reviewer at that point. So he needs some help. If you're bored any time go to this node and fix this. Help fixing it. Yeah, go to this node Eddie. It's really problematic. Just remove a module that's not need on the site but still installed and your site is blazingly slow. Why should this happen? Who does this? Well, sometimes you're lazy and then stuff happens. Good. And the next one obviously is size. Size on any kind as we had complexity. So just a lot of nodes, a lot of users or whatever will make your site slow. So these are things that you need to be aware when you build a site from the beginning. So one thing we do with our customer, we really try to find out how many nodes they will have, how many users they will have because there are a lot of things we can do correct or not correct from the beginning. So if you know what could cause issues and if you know them in the beginning you can prevent them because it's nothing easier than actually not even let problems come up then try to fix them later. I'll tell you how it is. Another nice example for complexity where every module was doing everything right in itself but together it was a disaster when I had OG menu. So for every organic group it was creating a menu and I had 5000 groups because they had been imported from legacy saying and now at my menu try to rebuild that and yeah the cache clear took 10 minutes. Okay so after we know what things could make a website slow we really need to know what is slow. So yeah and there are some more or less well known tools to figure out what's actually going on in your code and I'll mention some of them especially for PHP itself it's XJ Prof that's a really awesome tool I think it was developed by Facebook right? And it's widely adopted meanwhile and earlier you could use Xdebug. What XH Prof basically does it counts or observes every function call it records how long it takes to execute that function it records the memory consumption and yeah it brings that in the call stack so you can see what are the most expensive functions what are the most called functions and what's really important on that is if you measure there are maybe functions that are horrible slow or called quite often but you have to see if you can optimize that at all because some things you can spend weeks optimizing and it only brings you one percent or two percent I compare it it's like you can ride a dead horse but it won't bring you far so search for low hanging fruits search for things that are actually actionable where you can take action and have fast results let's show I've just recorded how to use XH Prof if you have the devil module enabled or installed you can configure that yes this one yeah login then go to configuration development performance and no devil settings and there you can enable XH Prof that will do it will add you a link to the bottom of your site so you can click on that and open the currently created report with XH Prof with the UI provided by that so we will put up all that stuff online you can look it up it's just to get an idea what XH Prof looks like I would say we could spend hours alone in how to analyze reports like that if someone wants to know how that works how to work with XH Prof and the files it generates yeah let's do that in the get herds 30 part that's how such a report look like actually and it can be quite confusing at the beginning and there's another feature that lets you compare two runs and so you can immediately see if you have made the things better or worse there's the alternative who of you know the diff feature of XH Prof have you used it before? yeah so that's probably something we could show and that's probably something that's the most useful at all from what I've gained from XH Prof profiling because looking at a run comparing it to a run is quite problematic but all you need to do and it's almost nowhere document properly is you're taking the run ID which is in your URL and you're putting one one equals the first run ID and you're putting one two equals the second run ID and then you're getting a very nice diff report so keep that in mind it's really the most useful for that we'll show you later how useful that can be one thing is that earlier it was kind of painful to install XH Prof on different systems so there is Exdebug that's more widespread I think at the moment and you can do similar things with that and if anyone is interested in that I can show you how to use it with cache grind or stuff like that oh yeah that's a really nice feature too here this graph generated by XH Prof it shows you basically the critical path of the program execution and it lets you identify slow parts quite fast but let it run again because often it's it's the database layer and there you probably can't do that much so it's just an overview a first step into getting into performance optimization so that's Drupal that's how it's executed there's also if you're interested in automated performance testing we've used it a lot for the trick initiative to profile all our patches there's XH ProfKit by me which on GitHub and which allows you to optimize the performance testing of a site given a scenario and I've now tried it out with Drupal 7 it worked out of the box surprisingly so you can use it with Drupal 7 as well and you can just have it run it will record 100 runs then give you a diff report and the only caveat is if you want to do good XH Prof you need something, you need a physical machine on virtual machines you get so much fluctuation that the results are not really meaningful but if you're on, for example, a dedicated server or you have your local desktop machine and have all your programs closed then you will get very nice results and you can kind of optimize this step like having it run 100 times you get very accurate results of that you could even put it into a continuous integration workflow well that was one side of profiling stuff that was basically program execution but luckily there's something like presentation that lets users see what you're delivering and often that's the other slow part that's actually slow and you have to profile your markup your javascripts, your CSS how you serve those things is this compressed or do you aggregate stuff like do I serve 1,000 javascript files or 1,000 style sheets and every sheet has its own request that really can slow down your side especially if you don't serve it from multiple domains so that the calls have to be serialized and not parallel there is a really nice tool that's integrated in Chrome it's called PageSpeed I think I presented just a short video about PageSpeed tab I think that's the PageSpeed tab it's part of the developer tools just open the PageSpeed tab hit analyze and it will tell you quite a lot what you could improve on the presentation side there are meanwhile other tabs like audit that also do some nicer overview of what's actionable on your side and for people who for some reason don't like Chrome there are other extensions like Weislow I think that was originally from Yahoo and it does similar things actually that's other stuff you're playing right now that's xtbacter, yeah we can show that too you want to show the network tab we can show network tab from Chrome it shows you what requests are done when, how long the requests have to wait till the stuff is delivered how long it takes to process them what is blocked in the meantime and so on so you really should just open the developer tools of Chrome and you have a really, really powerful tool at your hands to just do a fast analyze session of your page and maybe one thing we see here which is really interesting that we're actually talking a lot about this part here which is the blue one so that's Drupal doing things whatever, rendering the page, delivering to you but then after this is done then your browser goes and analyzes all the HTML and all the CSS and all JavaScript and whatever so basically all this part in here we cannot really fix with Drupal we can make it easier for the browser but this is stuff that all happens downloading CSS, downloading JavaScript etc and as you see already here the website is around 700 milliseconds and 300 milliseconds are used for PHP and the other 400 is used for displaying the stuff in the browser so here it would maybe make sense to focus first on this part than actually making this faster if it's easier, whatever depending on the things that happen down there so because I see a lot of performance stuff only talking about this and then you have 300 different JavaScript files and this takes the browser two seconds to load to interpret, to run and you could just get so much more with really looking at the whole thing because this is not what the user actually it takes, it's all together what happens on the browser yeah it's again, no your pain points if you don't know what to optimize then you're just optimizing blind and you could even make things worse because you need to really analyze what are my pain points, where is my site slow, is it the server part is it the front end part is it the database part is it even the network that's suddenly making things slow and cannot keep up with traffic etc last but not least log files like the MySQL slow query log on your development machines at least you really should enable that during development sometimes you want to have fast results that doesn't necessarily mean that are the best queries so just run it on the side of your development machine and sometimes take a look at it is there stuff in it and there is also a nice tool that Fabian introduced to me that's the PT query digest that actually helps you to analyze this huge log file it can be difficult to read so that's a really great tool if you want to have a fast overview over the low hanging fruits again it's about no where your pain points are because this tool will give you the top 10 of your slow queries and also how much it is used and you will immediately know how much of a problem is this for your site at this point after we figure out what is slow we will do this together later so you will see more of them but we want to fix slow things so we start doing this from a whole request so the first thing we are actually checking is not about Drupal at all so one thing it would actually start at the hardware and the network we are not covering this so we are talking about software which is installed about performance of hardware we need to know a lot of stuff about hardware itself and networking and so so we start here with Apache Apache is the first thing which is called if you run Apache of course you can also use any other web server at the end it's close so Apache runs and before he actually starts PHP or even Drupal he loads its own modules and default installations of Apache have quite some modules installed as we do it in Drupal we just enable everything so that the users can find out and see oh there's all this cool stuff and it's exactly the same in Apache so a lot of modules are installed which you maybe will never use so if you want to have a website a fast website maybe you should think or check which modules are really loaded do I really need them if you don't need them disable them because it's less stuff to load it will be faster at the end of the show one of them is compression at the end what it does when the HTML is generated in the JavaScript and CSS as well it compresses it so it's faster over the network there are a lot of things in the past which said compression is not faster because you need to compress it and uncompress it which takes CPU cycles but now these things have so fast CPUs and in the notebook obviously as well it really makes your site faster when the amount of data which is trimmed for it via the network and imagine here we have 2,000 or 3,000 different devices all fighting against to use a lot of wifi if the data is just smaller it will be delivered faster and mod deflate which is the module called allows therupal to tell the Apache that it should enable compression and then Apache will handle everything so basically you just need to enable it that's it and the next one is mod expires so maybe I have to explain a bit how this works if you visit the site and for example there is a jquery.js file which is loaded in therupal per default now you visit the site and you go to the next page and on this page the jquery.js is added again if mod expires is not in there the the browser will not know that it should cache this so what will happen the browser goes a second time in there and checks and downloads the whole jquery file and it will be exactly the same but nobody told him why not cache this for at least 2 days 2 weeks, 2 months it will not really change over the next time the expires allows therupal to tell via the Apache to set expiring headers to files and this will then take the browser to not load the file anymore so basically what happens, you visit the site the first time jquery.js will be loaded he sees there is an expiring header and it will be cached for this amount of time which is entered there and the second time you go there the browser will realize I already have this file and I should cache it so it is not loaded at all so you probably have only a lot of requests at the beginning and the next pages is only the html anymore and all the JavaScript files are already loaded and in Drupal it is 2 weeks by default so we are just kind of managing these kind of basic things so we are all on the same page it is 2 weeks and you probably want to treat your expire settings because there is most of the time no reason why an image could not be cached for a year or more yes, especially Drupal where everything you upload, if you upload an image again with the same name the URL will be changed so there is added an underline one so as you say you can cache the stuff actually forever so this is Apache so check your modules and check that the needed modules by Drupal are installed now we go to the next point of PHP PHP itself I guess we could do a whole conference about PHP performance but one thing we really want to mention are opcode caches so what are they doing basically what PHP is it compiles every PHP file on every run which is from a performance point of view not the fastest thing again they don't change how many times you change a PHP file on your server so why shouldn't PHP cache the stuff after it is around the first time there are a lot of things that the interpreter can cache we don't really need to know what they are doing but it can be cached and also the PHP files live usually on the disk so you have a lot of access to the disk which is called disk.io every time on every single request so what an opcode cache does it first pre-compiles the files whatever this really does but it just will be the second time will be faster in running and it keeps the files in memory so the next time that index.php and all your module files and all the stuff is called again he sees them in the memory and loads it in memory will be faster than disk.io all the time there is one small thing you can set the size of this cache and usually apc the cache size per default is 40 megabytes this is way enough for your custom PHP script but unfortunately not enough for Drupal so if you have a Drupal and if you even install commerce the cache will be 50 megabytes which is needed to imagine what happens he fills it up to 40 megabytes in one request and he realizes oh my cache is full so he throws away the old cached files to add the new ones and the next request this happens again and at the end your site will be slower so I had a lot of people I tagged them yeah just install the opcode cache and they do it and your site gets slower that's the issue that the cache is not big enough and we suggest to use 256 megabytes per site so if you have a server with a lot of websites on it you maybe need to raise this and for right now there are two opcaches mostly used one of them is apc which is for PHP 5.3 and 5.4 and there the setting is called apc.shm on the line size there is a new one in PHP 5.5 there is the send opcache which is bundled in PHP so you don't have to install anything additionally you just enable it and there is even because the send opcache is around 20% faster than apc there's a back port for PHP 5.3 and 5.4 if you're interested in using the future on an older PHP version yes and of course there are a lot of other PHP performance things but this will already make your site usually about 30-40% faster with just enabling one single module in PHP giving it enough RAM and your site is faster then next thing that also runs with Drupal is obviously the database we cover here MySQL because it's the most used database and first of all there is one thing this is the storage engine so MySQL has different storage engines the most widely used are MyISM and InnoDB with Drupal 7 it's not such an issue anymore because InnoDB is the default but if you update like a site from 6 to 7 or whatever you just need to know that InnoDB would be faster it's not the fastest engine at all but it's the one which allows you the most features the highest security or reliability because of how are they called? the transactions so you have transactions InnoDB and the biggest performance another thing we want to mention is MySQLtuna.pl MySQL has a lot of settings so if you check the .cnf ones there are a lot of settings you can do and don't ask me what they are for but there are a lot of caches and things and table caches and key caches and whatever the script MySQLtuna.pl checks your logs for an existing MySQL server so it's really important that you don't run MySQLtuna after you install the MySQL server because it needs statistics from the past because every time, for example MySQL hits a cache which is already full it locks this and MySQLtuna analyzes these logs and tells you hey, in the past you had 10 million cache misses because the cache was full so then you can adapt this cache size and MySQLtuna will directly tell you what you need to change but really important here only run this on a site already was running for at least 2 weeks I guess MySQLtuna will automatically tell you as well if you run it too early so that's really important there and also be very very sure that the scenario you're testing is the same as your production scenario if for example you're having a site and then you're running because you're testing out some things you're running some big imports that doing lots of writes to the database and some very complicated queries then your profile will be completely off as well the other thing is don't blindly optimize MySQL because if you see there's lots of temporary tables well we just increased it and then it's telling you well it needs 64GB of temporary tables and you're like what and what you want to do at this point is you would probably want to check your slow query log and you would probably also check your queries at all but why are so many temporary tables created and that's again do some monitoring with Moonin or something else get a feeling of what is happening on your site that's very important because I've seen people kind of like MySQLtuna said it to that setting and that does not always help because you could not have your scenario that you have running most of the time because some special event was happening or this kind of screwed up on the site that fixing would be much better than tuning yeah next topic we already heard the word caching quite often and that's kind of obvious because what can you do if something slow try to avoid running it again and that's what caching is about especially with the lifetime later on try to avoid as long as possible run it again and for caching it's really important that you have a strategy there is this very obvious thing like yeah let's do enabled page cache well that will help if you have very unique pages but every it will help only for that actual page to be faster and if you analyze your page and then you see oh wow I have quite often the same blocks on that pages on different pages so it makes absolutely sense to enable block caching because then you will benefit one page call will benefit from another page call because parts of that page are already rendered or prepared and you don't have to execute the whole tree again so you need to figure out what are the parts on my side that actually are the most valuable to cache what doesn't change over time do we have things like if you have locked in members locked in users do we have stuff that's not changed by that not affected by this state that's a very valuable object to cache because you can share that among every request and I think lifetimes you will cover that later on in in deep detail I personally always try to create a scenario in which I have what I call active caching that means I said basically silly high lifetimes and make sure that my systems invalidates these caches when certain actions happen because I don't like the fact that well okay it's gone do it now well that was something I could cache a year and there are modules out there that help you to do that and I will mention cache actions that allows you to use rules to react on specific events to invalidate caches and that's awesome if you have moderators that expect to see immediately what they edited on the life side you can react on note safe and immediately flush the cache and next request will rebuild the required things and now it's the latest rules version and with all these nice fields you can even create things like dependency chains if you have entity reference fields you can say okay this note is related to another note so we have to flush that too just be careful not to get lost that will really complex also be careful not to flush your entire site yeah try to avoid to flush your entire site that's always not so fun especially if you don't have fast locking or stampeding it's one keyword that can bring down your site pretty fast other modules I recommend I heard before mentioned memcache who uses memcache there are quite a lot and everyone with everyone with the official memcache module who uses something who uses version 306 or earlier from memcache php extension who uses a php memcache extension 306 or earlier you should check if you're using 304 you'll have problems with wildcard flushers if you're using 306 there's a very very crazy bug which can make all your roots disappear on the site and have had that happen to me it can also make your seem registry disappear there's a really funny bug in memcache if you request the same cache key again directly after another boom it will sacfold and then what you're getting back is an empty object to Drupal and this empty object will then by roots be treated as oh there's no roots on the site okay and that was really crazy so if you're using memcache upgrade to 308 or downgrade to 226 oh yeah that's a normal memcache not the memcache D so memcache D has performance issues yeah then I don't think it should be used to optimize your performance actually that's but now why I really asked about memcache is the memcache module in Drupal or memcache itself is very limited regarding wildcard flushes and yeah Drupal works with wildcard flushes and that makes your site slow probably I mean that's a huge impact what the memcache module actually does it starts your caching keys and tries to group them and if you do wildcard flush it tries to find the best matching group and it has to create internal tables to match that stuff and that's just horrible slow so if you have the possibility please use redis we don't show it right here on the slides but you can configure it similar to memcache as in Drupal are you cached so that all stuff is automatically cleared and it yeah it's great well yeah I figured out that when I had to fix bugs in memcache module and I fixed the bugs and we committed that but then I switched to redis actually yeah have you used memcache storage I heard about memcache storage but I never actually used it yeah he showed me how to set up a redis and it's really easy it's until you want to actually make a HA and it's a bit more difficult high availability multiple modes it gets quite difficult but to do a single server is really easy it really depends on your use case as always but did you ever do memcache cluster with fail over strategy that's not easy at work yeah another question who of you now Handler socket Handler socket two people Handler socket is something still quite new and interesting but it's stable in Percona MySQL server now and I want someone to finally try it out I didn't have the time now so I'm searching for volunteers Handler socket is awesome because they claim around 10 to 100 more performance in memcache and it's pure MySQL what Handler socket does is it gives you a key value store directly in MySQL so you can on the variable table you could just say variable set and then it was directly passed through and it would circumvent all the SQL query etc and you would probably even from a memory perspective just write to a memory row and that's it that gets later committed in that so Handler socket in Percona is really something we as a Drupal community should try out as a memcache etc. thing because we're still relying heavily on MySQL and for some things as a key value store it could even in Drupal 7 still make sense Handler socket I think everything we mentioned here we will update on the performance website so because the format of this there's just a few things on the slides and we talk quite a lot but we will make a wrap up after this session yeah that would be yes and no the point is it depends how your database is structured because database write still is an expensive operation so if you have a mostly read-only database maybe your mileage may vary and especially if you already have problems and you can scale out memcache very easily horizontally then probably Handler socket is not the thing but if you're kind of just on the MySQL database anyway then you could just use fast database caching without having to install more things like more demons etc. memcache etc. so less complexity overall Could we move the Q&A to later on I really would like to push on and get this presentation done so that we can talk individually is that okay with everyone so I try to in the next thing I would like to mention boost for everyone that's not familiar with varnish or doesn't have the control over the hosting environment boost is a module that basically stores your websites on the file system and serves it directly from there so not very much PHP execution is involved anymore than just to mention it there are pitfalls with caching like mentioned before you can get lost pretty fast in stuff like cache actions and dependencies and lifetimes and there are some other really bad things like delta caching that what I call it delta caching imagine you have a search site and you show different nodes every of these nodes uses open layers to show a map in the result what happens is if you have for example penalizer the first result will add the open layers JavaScript library and penalizer will cache that it makes a delta what's the differential of the added libraries to the stage before so the first node of the search results has the library the second one won't have it because there is no difference to the stage before and so you can create broken search results as soon as someone searches something that all search results are cached but the first one is missing the library isn't there there is a penalizer or panel plugin I promise I will release it this week and that will help you to prevent such cases the other thing to mention is form caches forms have own cache and form summits won't work if the cached forms aren't there anymore so if you cache forms shorter than the page or whatever if the lifetimes don't match forms can't be summited anymore so be careful with forms and caching so to lifetimes with Fabian now okay so now we are going a little into one of the problems with caching that he already briefly talked about is cache expiration and yes you can have for example cache actions for doing complicated things because one of the things we need to know first is when do we expire what so and of course it's pretty simple I have a node 1 I edit it well of course we know okay we need to flash so pass node 1 also the alias that's simple the other thing is what about our dependencies so we have might have for node we might have javascript CSS attached to it but then we also have entity reference and a node in a teaser could for example show the also picture but it's also picture changes we wanted to change all over the site and so in this case again we need a way to invalidate everything or we just saying well okay I'm just doing time based things and I don't care for these 5 minutes that for these 5 minutes user is getting the wrong content so there's one possibility to do it another possibility that's mainly Drupal 8 I've looked at the Drupal 7 version of cache tags and it's was too experimental for production use but on Drupal 8 we have stable hash tags and in that case we can just say well this block or this node or whatever is tagged with the user and then in the end I can just expire everything that has this user ID in some way and that's working pretty well in that the last thing is and that's more kind of a granularity thing I might have a different block based on who is using it there might be a block that's different for an admin so it might differ on the role but I could also have an arbitrary request parameter that's kind of setting a filter or whatever and I want one block to differ on that so I have some standard granularities within Drupal per role, per user, per path and that's actually done by Drupal renderer CAD parts but kind of what we need for most sites that I've seen at least is we need to have more granularity that we can extend the cache ID more to be able to say so I don't want to expire the block I need different variations of the block so adding to the cache ID that we can do that um yeah so what I'm introducing now is render cache a new kit on the block it's two weeks old it's still very very fresh but it gives you the possibility to never write any cache action because things are expired automatically I found out that most of the time the listings are pretty fast but the rendering of the entities is slow so what we had was we had a site that was built completely mis-displaced huge and was rendered by view mode and we also had some other conditions based on which view it was displayed in and then what render cache does similar to display cache which also is two weeks old is that the entity view method of the entity API is hijacked to return the cache data and obviously you could implement whatever you want what render cache does it provides a render cache node module which hijacks your node view if you're using panels you will need something else obviously because that's already hijacked it hijacks the views rowplug and especially for display suite it also hijacks the view displays render rowplug and the nice thing about render cache it's completely compatible with cache from Drupal Render and now a question because it's been around like since Drupal 7 was out around 2-3 years who knows how the cache entity was in the Drupal Render arrays works that's funny Drupal 7 has the most easy way of caching your blocks it's a very simple method in everything you could just, if you have a block you would just put a pre-render method in a cache ID based on your conditions and that's it you have that block cached in a custom way but no one knows it there's one block post from 2011 which is awesome still as you see, you don't know it now you fortunately don't have to know it because this module is doing it for you so the point is when the cache does one thing different it never expires the cache because we don't have cache tags in Drupal 7 what we are doing is something completely different we are saying the cache has never expired so you are responsible for claiming your cache what you need is memcache.redis or something else that has a least reasonably used strategy they are planned for database backend but that's future and now the nice thing is you're just creating a cache ID at the moment I'm doing a hash because I don't want mile-long hash IDs and that's based on the modified property of the entity so if any entity is modified it will be stored in the table that's pretty similar to how the Drupal 8 version of modified works funnily enough it was also committed at almost exactly the same time independently I didn't knew about it before I researched that the nice thing about the modified property is we can actually we will have a flag on my list because I talked with a lot of people yesterday who knows entity cache okay another module to add to your stack because it's kind of it almost never breaks unless you do a past condition or a condition outside of your entity within your node lord or entity lord things entity cache will almost just always works out of the box so it's a module too if you use memcache.redis or any other storage then entity cache is versed there's some benchmarks that show that even with the database cache it's 10% faster to use entity cache so just install it it's like I've never seen it break the other thing is so now we can have this modified properties cached via the entity cache and it's loaded always automatically the last thing is what about my dependencies conditions what about my granularity how can I change the hash etc and that's very easy it's a long name but there's a hook render cache entity hash alter there's also a cache info alter and the CID alter and that will easily allow you to set the cache by default nodes have the revision ID their ID and their modified property hash but if you need for example a user ID that's very easy to do so you're just setting you're getting kind of like a cache info cache info is like this hash cache property which has a cache ID granularity keys from which the CID is built and some other things so you should really check out on Drupal.org the Drupal render cache get Drupal render cache set a little that will help understanding that a little more in depth and then I'm also having a context and context can be arbitrary data but in this case is especially for example the entity type so all I'm doing is I'm putting the node UAD in I'm loading the account if the account is still valid because anti-modified works that way at the moment I'm still calling it later it could be a property with entity cache and then I can just put that in of course there could be a module and I'm working on that to also automatically include all references by default and then in your auto hook you could kind of exclude what you don't need but that would already make your block with the user in it work the user is changed the modified property change and in this auto you're kind of loading the node you're seeing all of the user modified property has changed it's done via a cache get so quite fast and with that all your blocks are automatically expired all your entities are automatically expired on your site based on what they're displaying and for a fairly complex site I've done like 30 lines of code to cache almost all entities on the site next one please yeah that are the results so there was kind of at a 4 second page floor time before and just by adding 10 lines of code I brought it down by 60% so and I don't have to deal with any cache expiration or anything don't have to deal with editors saying well I need this after one minute to show but this is okay with 10 minutes or whatever or and I can cache indefinitely until things really change and memcache will automatically throw out those things that are no longer used yeah and from the function itself you see I still have 57 milliseconds and the most at the moment is modified loading because it's still going through database but any caching would be simple and then it's like more like 6 milliseconds but still that's an enormous amount of time and a new module for you to play this the nice thing is when a cache just made it into Drupal 8 a different version than mine but because of based on hashtags and I've discussed quite a lot with cache yesterday if you want to also have hash thing and we probably have something like that but we have this hashtags based implementation it has been committed to Drupal 8 core and it's active by default so that's one thing for all the bad news that have been spending about Drupal 8 Drupal 8 will be fast and there will be render caching by default you don't have to deal with it and we're trying to do the same for the blocks as much as possible and that will be very very good and you can expect kind of that kind of speed ups after it has loaded once I'm still working to get an API to kind of be able to also in addition to the hashtags change the CID in advance if I need to to be able to do still the modified thing if I need it for some use cases where I don't want to expire the cache at all next yeah and then we're going on hope you enjoyed so after we do some caching we want to still check what we actually saw before what explained to you that all the assets which are sent to the server to the browser what we can do there better first there are two strategies one of them we talked about it already is compression but not actually compression on a gzip level it's just putting files together and the next one is aggregation so basically one really bad thing is to have a lot of requests from the browser so one strategy there is to have only one JavaScript file so you as a developer you maybe have whatever 20, 25 files but then they are aggregated and you send it as one single file and it actually contains all the JavaScript files that's called aggregation and that's already in core so you can just and all other stuff most of the time it's really confusing why this actually happens and the reason is that every page flag in Drupal and what this is is that usually when you check what a site can do and I recently had a case that we have JavaScript which has maybe some pop-ups these things the chance that somebody will actually see this is really high because it's on the front page actually and then you have some JavaScript which is maybe only for the editors so the editors maybe use the same theme but they have some help information or there we had some Qtip thingy what you can do now is as you know which JavaScript files are really important you can give it the every page flag this means you tell the aggregation this JavaScript file will be added to every single page and it will create a JavaScript file for all the one which have the every page flag enabled the Qtip in my example I didn't give the every page flag so basically what will happen is that the aggregation for this file on this page will be separate so that when the editor visits the edit page and needs the Qtip.js it loads to every page JavaScript file and an additional one and this is in the end faster to load sequentially what you actually need per page because we had issues that you add this Qtip JavaScript to every single page for every single user and 99.9% of the user will never even use this JavaScript file or these functions and the two bad things first it's data transferred and they initialize it they do some checks etc so having less JavaScript files only the ones you really need will make it faster there are two modules we want to mention one of them is magic which is really new it allows some or it came out of different themes which all implement the same things one interesting thing is that you can put the JavaScript in the footer and that's faster because usually in browsers the browser will completely stop everything else first load the JavaScript file run it and then it will continue and we've put in the JavaScript in the footer this will happen after the page actually rendered so the pictures and all the HTML would already be displayed this can create sometimes some issues but mostly it works quite well but not something you just enable you need to check your site again there is advanced aggregation which is another module and it allows you to create even more groups and a lot of different configurations with the aggregation then the pitfalls is one thing that we had in a lot of different things is that you need to check what is actually really loaded from JavaScript one example we recently had is we used facet API sliders and it gives you this really cool feature which is using a search with facet API with ranges so the user cannot only say I need a price tag with like 100 francs or dollars you can say I want to know from 50 to 200 what the module can do you can either do it with drop downs or you can do it with sliders but we didn't need the sliders we only wanted to have the functionality of the drop down the problem is the module can do both and it will start to call jQuery UI and a lot of additional stuff and that's really heavy that's why actually jQuery UI is not in jQuery in because it's really, really big but just with enabling this module every page will load jQuery UI jQuery sliders and if you see at this it's a lot of JavaScript never ever used on your website so what is important there if you have a website and you're going deep into the JavaScript look which JavaScript is really loaded and maybe think about do I really need this and if you don't need it, change it so only load the stuff you really need the last thing from my site is varnish who knows varnish who doesn't know varnish okay, good so varnish is a reverse proxy basically what happens is you let every request go into your Drupal via varnish and varnish will pass it on and when it comes back it will cache the HTML if possible and the next time the next user comes with the exact same request let's say some star page or some front page it will say hey I have cached site of this and it will not ever trigger PHP the Apache at all it will send the HTML the cached HTML directly to the browser and this is unbelievable fast it's like 18 milliseconds, 20 milliseconds it's actually of one of the main developers of FreeBSD so they know how Linux works and they know how to make stuff fast so varnish can really make sites really, really fast with Drupal there are multiple modules which support this one is the varnish module which basically disables the page cache inside of Drupal completely because you have an external one so you don't want to have an external one and an internal one which could happen that you the lifetimes are different so the external one expires but the internal one is still there then the external one is recreated based on the internal cache it just has two caches which want to do the same so with varnish module it disables and it also if you make cache clears it will send this to the varnish to ban this URL so varnish usually takes the lifetime Drupal can send set to varnish how long the lifetime should be whatever, 10 minutes, 10 years whatever you want but as we heard before when a node is changed you can tell varnish hey this has changed the next time a user comes don't send him the cached version come back to me I will give you the newest version and varnish module can do this but varnish module can only basically do the communication between the Drupal and the varnish Perch module allows another thing that varnish module needs a connection via the CLI or via some ports whatever and with some hoses that's not possible or whatever and there is another way to do this is to make a special HTTP call to varnish so instead of actually get a website you can say purge this website via a normal HTTP call and the purge module will be allowed to do this and then expire or cache actions as we heard before is basically the things you need to say to varnish okay this node has now changed please ban it or purge it whatever you call it expire or cache actions can do the both things as usual there are pitfalls with varnish first SSL if you run SSL to varnish he will completely stop or just pass it on there is no SSL implementation inside of varnish there are two things to work with this one of them is the well I've actually created a module it's called enhanced page cache and it allows you to change the CID the caching ID of a page and you can enable a mode that removes the protocol from the caching ID so you are storing HTTP and HTTPS calls to Drupal in the same caching entity so that means if you've got a HTTP request HTTPS request from a user that's stored in memory or in memcache wherever you have the cache and as soon as another user requests the page over HTTP it is not present in varnish but it will hit the page cache of Drupal itself so you can save quite a bit of storage or memory and make it even faster for both protocols and the only thing you have to take care of use protocol relative urls wherever you add an absolute URL the module tries its best by url outbound alter and file url alter to make that for that but finally you are in charge of that so basically is things it lets HTTPS run through varnish and handle Drupal it the other way is to just terminate it before varnish so what you can do with Nginx for example or whatever you want to take you can really easily build an SSL terminator which just changes the protocol so Nginx will decrypt it and send it uncrypted as HTTP to Drupal and you realize that the whole thing was called by HTTP important you need to be aware that the communication between the Nginx and the Drupal is secure because if whatever if it goes through another network or if it goes via the internet again your whole SSL doesn't make sense at all so this only is really if you can be sure that the connection between the SSL terminator and your website is secure the next thing is sessions which is also important for varnish whenever you have a session cookie varnish will also say okay now I stop working because varnish does not know anything about your site and if you have a cookie set this means based on this cookie anything can happen like showing the username of the user in a block or showing a block or showing don't show a block yes so basically if there is a session cookie varnish will stop working there are some varnish default or varnish example configurations one of them is by lolabot which I mostly use a bit adapted so if you search for varnish configuration lolabot there is one really good which has the most important things already in there and for example what they do a lot of additional javascript things like google analytics they create their own cookies and it basically explains varnish which cookies they should really search for or not so if you have a google analytics cookie your website will still look the same or the page it's not changed based on this cookie cookies will be ignored from hashing and all the stuff and the next one is parameters and back to google analytics again if you make SEO or marketing you know all these question mark utma parameters in the top and this basically tells a website which has google analytics run where it came for so if you make for example a facebook ad and a link the link will contain some utma stuff and it will tell from which facebook page it came and all the different stuff and they look different every time so if any time your customer tells you yes we will do a facebook campaign this should ring a bell because this will create a lot of requests with all the time different parameters these utma parameters and basically for varnish every request will look different you will not realize the site is exactly the same it's just these parameters which are different so there is unfortunately that's not inside of the lullaby configuration but you have it somewhere yes I have a boosted varnish configuration that was mostly used for Drupal 6 sites but it has this rack to remove utma configuration and there is also on the official varnish site in the wiki there is something for removing the utma configuration parameters another thing is that you can configure in your campaign that you want to have hash instead of for the links that's another cheap way to do it but we'll do something about it if google analytics and caching is a goal I've seen a site go down by a marketing campaign yes and maybe one thing we didn't mention it's again it's a country modules sometimes the modules don't really care about how they save stuff and sometimes they just use the session which obviously is really cool as a developer so you just need some information later on again you just save it in the session and use it then but this will cause Drupal to create a session cookie and cause varnish to not cache the stuff so it's really important to check your modules what is really there what happens there CDM's real real quick if you want your content delivered near to the user, use a CDN if you're using a CDN where you just put your domain to their IP like akamai that's it about and you just need to make sure to configure it in a way that you have some other domain where akamai can get their stuff that's called an origin server and there all the stuff is pulled from be sure to disable redirects if you plan to do something like side down redirect to something else it hit us we had to clear the one we had to clear the akamai cache and it was not nice the other thing is for akamai cache exploration cache purging you can also use the akamai module created for the white house and for most other CDN's where you just have media.mydomain.com slash images or whatever to serve your images just you want to use CDN module mostly for rewriting your URLs to go to the CDN URL and that's about it and CDNs are especially important if you want your content and your things that are cacheable delivered near to the user especially for assets so if you're doing worldwide customs then a CDN can be very helpful and pretty easy to set up and cheap too okay good so now we have a small problem the wifi still doesn't work so yes well it shouldn't be a problem but basically we planned now to do some small Q&A and split up in groups and fix your sites yeah the sites are not ritual for us right now unless somebody has it locally okay so maybe we can set this up and do it all together on one screen we could try that yeah the presentation went longer than we expected actually but first are there questions right now can you repeat it yeah I will repeat it I've read some people in line compared to this to knowledge and databases is this a viable solution is this something that one should possibly develop but a society developer should take in mind I mean is it possible for a knowledge cache solution to be faster than a Redis or Memcache so Redis versus NoSQL using NoSQL instead of Redis or even Memcache yes there are some experiments I mean there's a MongoDB module but this handlocked socket is also kind of NoSQL for MySQL to be at least it depends on kind of what you need if you need to load a lot of things where NoSQL is very beneficiary and you just have not normal queries but entity field queries so that you can do a lot of NoSQL and your architecture is built in that way then for sure try it out in that but just for caching purposes well you could probably use MongoDB as a cheap key value store for caching as well and you can query it also in efficient ways so the question was how can you actually cache a site where you have basket and you have like a product count on every site but most of the pages are always the same and I've done that the way I've used it I've used a cookie that's added via JavaScript and the only thing that was not cached was the user and cart pages so the user could register with their cart and they could look at their cart and then it was just when the cart was updated I was just putting out this JavaScript they were setting this client-side cookie and the client-side cookie I would just when the page was loaded I would update the product count via JavaScript manually worked fantastically well there is a module called off cache and there is a new version of it version 2 and there you can by block define how it should be cached so it changes the cache ability setting in Drupal and so you can say the whole page will be cached except this block and the block there are multiple ways how to load this block one of them is Achex so that actually the whole page is loaded and there's just a placeholder for this single block and then via Achex you can put it in Drupal with the cookie and then your cart if the cart is in this block will be loaded that's the Achex version there is another thing which is called varnish ESI and then the whole thing is actually done in varnish so varnish will know that the whole page will be cached and there's also a placeholder for this tag added into the varnish and then the varnish will basically do the work for you to query the back end only for this block and then you put it in the HTML and send it to the user so the difference is with the Achex it will happen after the HTML is shown so in the browser you will see a bit of flicking because it will take some time and with varnish if you're able to do this the page which is sent to the user already contains the updated stuff could you say what was called varnish ESI sorry? ESI, yes it's not only varnish and Genix can also do it but it's mostly used by varnish yeah and it was originally introduced by Akamai because you were kind of also caching your content kind of but you still need to build it up with different building blocks and then you could kind of get for example per session or per user you could get that block also cached so Akamai has the most advanced ESI implementation but it's also the most expensive unfortunately yeah but there is one pitfall if you have this multiple times on the site let's say you have we have 10 different areas where the user name is shown it will create with the normal configuration 10 calls to the Drupal which means 10 Drupal bootstraps so the AuthCache is able to do this the AuthCache module via the HX implementation so the HX will realize oh I need to load a lot of blocks so it will send all these blocks back Drupal will create all of them in one bootstraps and send it back I have not yet seen an ESI implementation of this so you just have to be worried there yeah Caches in varnish and as a tag for every different visitor we use his session ID in this way we say a different cache we serve a bubble cache like the main page and individual blocks okay yes of course question, would you recommend the patches? sorry? would you recommend not only pushed a lot of stuff into Chrome to actually check how fast your website is there is one Apache module called mod on the line page speed which they claim that you just enable it and it makes your site 30% faster and basically what they do they force to enable compression they minimize images and all the best practices they minify JavaScript so it's just a lot of things you don't have to worry I've never used it I've seen implementations of it seem like 20% faster honestly we do so much stuff already that mod page speed doesn't really help you anymore so I think it's like a lazy version of it but if you really want to know what is really happening because finding bugs there will be really hard so we just implement all the stuff that mod page speed does in your own Drupal installation so to compare it what can I still do that's really nice but I had some issues and they happened after mod page speed was enabled and then you will not or if you want to go to debug Apache it can make you slower as well sorry? it can make it slower on the website you can actually run it through it I'll tell you whether it's fast or not on a single server it's over 300 sites how will easy cache work with such a version? so you have on one server 300 sites you just need a lot of memory multi-site APC caches by file so for APC it will be one single Drupal installation because APC you might have a memory issue actually the code base is counted as per site so if you have one code base 256MB RAM it's okay if you would have send as much as in the sites folder if everything is in sites folder everything in sites folder well it doesn't matter because for APC they have no idea about Drupal it's PHP it's PHP files and as these 300 sites they share the same files of course you maybe have specific modules for specific sites if you want to know the footprint of what APC probably needs to cache you can of course hit all of your 300 sites it has one possibility and then there's apc.php there's also memcache.php and probably some other statistics tool for redis as well I don't know they use munin which can monitor it but really you want to and this apcphp it's in the docs folder in most installations just copy it over and it will show you a nice graph of how many caches you have how much the cache is full how much will be in the cache kind of like trashing like a lot of explorations out of the cache etc if you have cache the cache could be kind of like split up in different segments that's something you'd want to avoid but a very basic idea you could just on the command line to find for all PHP files count that vrpipwc-l and you have a very good idea of how many files there are yes but in your specific it's only one side I'll just comment on that on the database side if you've got it all on one database server then be careful with your mouseable query cache because having a very large mouseable query cache can be a very bad thing because essentially it locks across the whole database server so you've got 300 sites one change you can lock with the various tables and you get a bit of a problem there and I'd rather look around and APC seems to be saying PHPHP do you use orsync sometimes? yes that's probably the issue there's one thing APC needs to know when the file has changed and usually in linux there are two ways how to know that the file has changed and APC use default uses one of them and APC if you do this doesn't change it so it can happen that when you orsync between servers what we had we have like five web servers and you orsync between them that APC on the servers you synced to does not realize that the PHP file has changed yes if you clear the APC cache it should not come back actually scary ok xcash we switch to xcash ok very very similar I don't know what it is I understand there's an issue with xcash it gives you exactly the same error message but it happens if you put multiple approval incidents and you're running them through the same xcash we have that in APC because xcash has some weird difficulty differentiating between the two files because somehow it knows that they are actually the same even though we've got two versions of it what it does is because they've got individual if it's loading for all one it ends up loading like and the database cache file is the first one it was used it ends up loading in two versions it's been through it twice and that's something which they're trying to they keep saying it fits in xcash but they haven't actually and the way the only way we've got around is to run because we found it was running alive there the test instance on the same Apache instance so what we've done is we've just got a separate patch each one so we know that xcash is running on each individual one so it's possible there's also an advanced setting in APC which is real pass cache and there's some settings where you can set up real pass to be treated differently and I know that some distributions did at one time configure that wrong so that might be the issue so real pass at least should point to the there should be something findable with your error message and real pass apc because that kind of sounds like the issue what Drupal does is it does exactly like he explained it does relative includes so in APC it thinks it's the same but if you do a real pass look up then it will do a whole file system look up and see well that pass is actually different and in that case you will have a little slower performance because the Drupal will need to do the apc will need to do a file system look up first to find the right version but depending where you have the impact might not be much just measure it but it should at least stabilize your site so you never get that error again or another thing I would suggest there's still both rooms for free there so maybe let's create an apc slash xdebug redeclared issue both and we can find the stuff because we had it as well and for us it was the processing issue but yes there is like real path and things and maybe if the website now works yeah and maybe I'd like to interfere here a little bit and we see already groupings form like the apc problems group and maybe it's a good idea to have a break soon and talk more interactively while having a coffee or something so one thing one thing I want to show no it's just there are other questions than apc and there are people they've so what I can suggest is apc.php and it basically allows you to see like where the files are coming from if you're logged in you see this as well and so you see we have rather big one apc you know but you were recommending the center yes I never tried it I'm happy to try it out on one of our sites like it's a I have it locally as a drop in replacement it's okay it's absolutely awesome yes no it's free yes there are brands that are often completely wrong okay which are not the numbers I'm happy with these problem cache there's one which is provided from this one which you know but there's also php and cache admin which is far better than this one than this one php and cache admin so yes that's the basic one which was derived from apc and the one which is for php and cache admin and it's not I've used that it's quite good yeah what he said for everyone there's a php memcache admin which is much better than kind of the default port and we should all try it out the presentation page okay it's good yes we don't have what is it doing for a few seconds the html output do you have such sites that and like do you have so many requests to the same page in the same second a few seconds so instead of rendering the page for every one yes render it once per second for example but you don't have like perching or flushing it's just one which doesn't have one which has a grace value but one which also has a protection so if you have 1000 requests coming in for the same page there's only one backend request so also you could set a cache time of like 5 seconds or whatever in one is for your pages and you would still be fine kind of in that so one is to support a pseudo micro caching just support the protection layer here that there's only one backend request for every frontend request coming in we were discussing this while we were preparing this presentation I guess we all will try it out I have it installed locally but I'm a bad example I'm working with windows you know but even on windows it works as drop in replacement locally I didn't switch on our development machines that will be the next step so that we have testing environment running with MariaDB okay nice but which one Maria of the corner yes so maybe to explain yes so maybe explain for anybody not know what MariaDB is okay so basically when Oracle took over MySQL there was like a community initiative that we want or they don't want to how to say it it's just yeah well we don't like them there was actually a clause within MySQL not to sell it to Oracle but the problem was bought by Sun and then Sun was bought by Oracle so yes so basically Oracle owns now MySQL and some people don't like this so most of the main developers they said okay we fork it and it's called now MariaDB and it's a drop in replacement for MySQL and it has a lot of new cool things like a new storage engine which is a drop in replacement for EnoDB and how is it called MariaDB XRDB yeah so it's really fast and they just at the beginning we're taking basically concurrency and it is upwards I think upwards 350% faster than standard EnoDB but in terms of rights and concurrency basically MariaDB is now fully open source organized and they in it's fast so maybe you want to use it, give it a shot and they claim it's a drop in replacement so yeah next question some other metrics which are nice like you got that in later versions of MySQL as well it was mainly a granulation of slow queries smaller than one second rather than you know I would like to see queries that are slower than one second you can get two microseconds now you can do stuff like please show me at all queries in the slow lock that does not use indexes use 10 tables, sequential scans okay okay it's awesome yeah that's coffee talk okay another question so caching, rest calls and all other stuff okay well whatever so external calls to services how to cache them I could imagine two sets she's one of them just treated as a slow Drupal so basically try to cache it inside of Drupal so with varnish and whatever so to make it inside fast in the beginning or to basically think about can you use cache sets when you query the external service you can't make the service faster so there's no well actually there's one thing we're talking about that in Drupal 8 as well and the point is you have cache expiration you have cache invalidation and those should be two different time values because what you can kind of do is a very simple strategy is you can still deliver outdated content to users and then in the back end you are curing a new request to get the data new in so there's two strategies for that first of all you're setting a different invalidation time than your expiration time expiration time means during the request that this is created in you need to actually reboot the object so some users get a slow side invalidation means at this point it's invalid but at this point we are trying to recreate it so if something is requested frequently what happens is one request will have this will have this limit and we'll set a log this log and then you can either have this request just pass through and cache it again or with the updated data but if that takes a long time because external service could be slow my recommendation would be to actually put it into a queue put everything you need like the user idea or whatever you need for that request to succeed into a queue and then do the cache set there and during that time all users will still get the old content but at one time your queue will kind of succeed and the nice thing about using a queue and the queue API is that once that queue is no longer scaled what you can do is you can very easily scale out horizontally because then you can just use another queue manager that's kind of delivering your queues for you and then you can kind of split this up and you can have like just put in 20 boxes that are responsible for doing the external requests and then you can have the problem circumvented in a way that you could also have the queue and the server succeeds etc and that way your users never get a slow side so maybe you want to hire him are there questions? yeah you mentioned locking have you ever had any fun and games with Drupals built in kind of what way does that rather a lock at times what was the question for example if you Drupals recreating the variables it will basically do a lock you've got a lot of currency you get kind of stampede problems so the problem is lock stampede so I've talked about locking locking means you want only one request to succeed and the others to do we've debated especially the variable issue to dash for Drupal 7 because there's two strategies you can do we found that in most cases it's faster to go to the database to do the query and to wait for the other request that somewhere we're building the lock to do it but there was unfortunately a lot of bike shedding so that issue didn't quite get through but on the other hand locking I can also tell an interesting story if you have any old web server where PHP and E is still set to 12 for the lock and you are using very strange lock errors because then the granularity of the lock is lower than what that is and a request that's requesting a lock twice will get a lock failure it's a very strange thing just as a side note but for locking there's also a strategy to put it to memcache for example locking in memcache is very fast you can even implement a whole queue within memcache very easily by just using one key there's an increase method in memcache so you can increase it then you get a new slot everywhere and that works very well if you have a multiple writer one reader environment again that's an optimization you can do if you need it in memory queue very simple just need one reader and that's kind of reading this out and then putting those things in a queue or something that's another possibility where you can do things but locking again you can exchange it with memcache locking for example it's quite resolved that debate that was going on about how to fix it in Drupal the argument that happened about whether we should go to the database or not in Drupal 8 we have CMI and state and other things so we still have the same discussion but on a different level Redis probably also has a locking replacement for Drupal 7 so I guess before we actually tried to I guess the people which are still interested and still discussing they just stay here and we can make whatever some groups just to summarize the whole thing we did right now I guess this is the most important thing we see from a lot of night spending making sites faster measure the stuff first analyze the measuring optimize based on your analyzing and measuring and do this again so please don't just go on Drupal.org install now entity cache or even worse multiple modules at the same time because it can make it even worse and it can break your site whatever so really take performance improvements something you want to measure you want to analyze and then you optimize based on this and it will be much more happy and at the end so yes are there any specific topics you are like because I see a lot of people say did you ever had this issue or whatever I hear like APC that's maybe one thing we can do one question is like is anybody interested in reading XH prof because that's one thing I had really big issues like actually understanding what is really happening there so who is interested in XH prof okay any other things yeah you're right it's different there are the main tools that are recommended for doing it on some of them just aren't available so you have to use something else so I can see what the theory is what do I actually use so who else would be interested in Windows caching no one you know we are still here the whole week so and honestly we're really passionate about performance so I guess just connect with each other whatever go for dinner and discuss the stuff okay any other specific interest of the people which didn't raise the hand for XH prof because I guess we can only show sorry advanced varnish who else would be interested in varnish advanced okay anything else because we could do two groups that XH prof says here you maybe go out and sit on the ground and discuss the stuff to create our own both out there okay should we do this so we make XH prof in here we show some sites and outside make an advanced varnish thank you oh yes by the way as this is really new in labs even we couldn't make the presentation as we wanted because of the lovely internet we're still interested in take the survey for the presentation and just do it not yet excuse me you can explain a bit about the blocks happening 10 times yes I didn't get the bit where you said both cash what if the name is in a separate block it will call that block 10 times if you have 10 different blocks which all should load it via off-cash you make 10 requests because varnish you're not able to combine multiple ESI tags which would happen but with off-cash like the HX their E is able to combine and make only one block so if I only have one block I don't have four blocks if I have multiple blocks I don't have cash yeah try something fancy yeah hi how are you doing so you got your bearings and you're set down to be yeah I really hope you can do like a classic style and should not be just this almost impossible but I really think Andrew would love to have a resistance to become interactive and actually that enforces that and just talk about it are you sure with APC enabled even if the thing is APC hooks into phylic just try it it turns out one issue is that you have R6 starts there's the same approach exactly and then you will try afterwards if you want to know that and don't like it just run it exactly okay so can we please do the discussions outside that we can do the XH prof we only have like 25 minutes how do you have all the stock the stock the stock the stock the stock well he did you need to ask him you're talking about briefly about handle stock he did and you're talking about memcache and read it at the same time but there's actually that rotation in fairly stable that's a couple of years ago of Maraidi V and memcache memcache yeah which means if you reload the service you still have your cache correct yeah thanks so each XH prof I've just got here a normal Drupal site maybe can you close the door out there so when you want to install XH prof on your site you install the devil module there was an XH prof module but now it's in devil and after you have enabled devil there is a new checkbox you can enable in its devil and it needs two things one of them is a file system path where XH prof the directory so you download XH prof from well no internet you put it somewhere and I have it here in my I don't know there is like XH prof so that's really just the content of the XML of the zip file and you just put this in there so I need to know the URL just one second whatever I put this in there and this basically knows then where the site is XH prof HTML then is in here so this is what you need no it will search for XH prof live so it needs to know first where the PHP files are because this will be included and then it needs to know where to put the link and after you've done this it will automatically run so now when I visit the page I have at the bottom here an XH prof output link this is done by devil module and I can open it and it will show me my site and you see it's 372 mls so let's go from the top to the bottom the first one is the total time overall that it took to the websites to generate so basically this should be fairly the same as we have here so now it's crazy slow it's always like that so we had five seconds interesting so let's see the XH prof for this and you don't it's not the same so probably it was some DNS stuff or whatever so you see here it took now 331 seconds and the XH prof for it says 276 you see it here that's actually it takes 286 times of waiting that's the PHP and 44 milliseconds for receiving so it's around the same and the next one I don't know I never use this one what is this one what's the difference between this and this I guess it's IOTime well what's actually CPU processing and there is other stuff that has to be done so good so then we have memory usage overall which is interesting if you want to know like where do you need to set your memory limits and there is also peak setting which can also be interesting so that the peak is the most memory used ever because sometimes module will use memory and release it again and then here you see the inclusive so what it used overall and then the number of function calls so we see 22,000 functions or called with Drupal site important here I'm now user one so this is not what you really should use for using the performance of your website because there's a lot of stuff could happen because I'm the user one if you want to see this as locked out and you open the site as a local you see the XHPRO is not there this is because it's not running so basically what you need to do you need to set this setting I guess I just disabled the wifi here now so in here in devil there is an access developer information and permission and if you give this to anonymous users only for testing then afterwards you will have the XHPRO here as well and now we see it's actually in here really fast so you see it took like 4 microseconds which is or 4000 well 4 milliseconds let's say this way and we see this is already really fast and the reason is quite easy to find but let's check first the one which was not cashed because this was the thing so what we see now here is the top 100 functions sorted by inclusive wall time in microseconds so it does not show you all of them because as we saw there were 20,000 two function calls not of them all are different functions but it shows you the top 100 which is where you want to go it's like in the call stack and I'm not sure if this one works on my local it works so basically it's from top to bottom so the main function name will all the time be at the top and the important is now how to read the things because there are two important things one of them is the inclusive wall time and the other one is the exclusive wall time the inclusive wall time is basically the function and all its childs together the exclusive wall time is the function that it really the time it took to only run this single function so main for example just it first looks ooh it's really slow but actually it just calls everything else so that's and it itself it had only 173 microseconds so that's not the things you're interested in so we can sort so one thing you can do is run the wall time and we already see PDO statement execute this basically makes the calls to my scroll so we see at the end we had 97 calls to the database and they took all together not each 97 of them all together took 48 milliseconds then we see on serialize called 270 times we see Drupal static so now of course it will be interesting okay what type of SQL queries to run the thing is you don't see any arguments or at all so it will only tell you the parameters how many times they're called but it's interesting to basically go through because PDO statement executed if you click on it it will open this function and in this type it will show you which parent function was this called from so you go now through the call stack and if you click here we see that database statement execute actually calls these four functions and is called by query and we can go into query and now it gets big but query is called by these four functions and calls this one whatever and if we go now check the inclusive CPU there is DB query and then you see database cache get multiple so the caching actually is makes 49 calls to the database where there are 90 in total so 50% are only cache calls and I don't have any memcache or radis or anything installed here so that's for example one thing that's what he said before check what you really can optimize because if you want to optimize down here you can get rid of database well you can try maybe we'll end up but we can there is cache it's pluggable so we can change the cache so let's try another like radis or memcache or whatever you want to try so that would be one thing you could there and yeah it doesn't make sense to cache the cache again like that's not the low hanging fruit we are looking for that's actually a rather easy side that's a bit hard to find things but let's see for example the locked in why is this so fast and we can maybe compare it to the normal one and we see that in here if we check by exclusive wall time it's load user user module with 869 microseconds so the whole thing is really fast and actually if we check the call graph because this is interesting now here we see why this whole thing is so fast and this is the whole and stack what happened that's a very easy one now but we see that and it's through both of page from cash and it loads the whole thing it makes only 10 calls to where's the cash somewhere it should be that's a very good it's a bit hard to find sometimes because you can search it's a picture here cache get database cache get multiple so a lot of things will actually be called and you see there is only one call so the only thing that happens it makes a call to the cache page loads it delivers it finished so if for example I'm going here and on my local I clear the cache page table so we see here surprisingly there is the front page cached I remove this and now we do the whole thing again and now we look at it it takes longer so and you see also don't be confused by your own caching stuff you already implemented because yeah the first time it slowed so you need to know a bit what happens on your site what is installed what is not installed what happens between cold caches and warm caches you have to know which state you're actually profiling and only compare the same state only compare the same states which each other other ways you would go nuts what? then also some things sometimes it's interesting to see what happens during a node save and if you want to devil this or xh prof this you will end up in some issues so let's go here and we take whatever some node without this lovely overlay so now you see there is xh prof for this page and I click save and now I want to see the xh prof and you will see there if you go in deep there is never a node save in there any ideas why? it doesn't redirect correct so if we actually look at the node save now we use the network tab so if you remove all this and only say you want to see documents if you click save sorry we need to say all here we see there is first a post to the site this is a whole Drupal bootstrap and the post then returns with a 302 found and then it will load the node the view so if you want to see the post itself it will never be in the actual in the actual view because in here you will only see the xh prof for the viewing of this node not for the saving there are a lot of different ways one of them is actually in devil itself and you can say here that display redirection page so instead of actually showing a 302 it will create you a normal page which has a full folder and xh prof happens in the folder and disabling so I save it and now it tells me I also see the Drupal message set and I see the user is being redirected to node 13 but there is a chase prof in here so I can go here and now I see no page edit and I can form validate oh a curve request whatever why is from my module so maybe I well whatever so that's really interesting because I ended up having people telling me well no that it is really slow whatever and I go there and no it's 100 milliseconds whatever so that's one pitfall in xh prof because the hx calls sometimes don't contain anything what I usually use is that xh prof what it actually doing it creates files for each single call it creates a file in a directory and if you open just xh prof.html it shows you all existing runs so for example now I need to trigger somehow an hx call was it xh prof where you can add some parts of the url to the file no I think that was xtbug right yes what I usually use is like now I want to know this hx call if I type now something in there will an hx call happen so I go to the listing and I see all queries and I remember the last number whatever and then I would type something here now you see there was an hx call and if I go back there is another one on top I can click on it no I can't click on it that's a problem I take one which works and change the run number and then you can see it so there should sometimes if you go in the hx but it depends on the hx implementation you can also see it here in the network thingy so here in the response no no payload so you will not because yes you could send it by a drop offset message or whatever yes should we look at the slow website I don't have any slow no yes only have locals yes good question so Facebook claims they run it on their site as well and maybe we need to check the oh yeah I have another thing I want to tell you but that's good combined so if you look at devil module how it works there's some xh prof somewhere so there's a function devil xh prof enable and basically this is the magic where it happens and if we look into this no internet so you have some flags you can tell them and they say that you can there is that official one yeah well it doesn't matter you can check the flags yourself but basically this tells it should measure the CPU and it should measure as well the memory Facebook says if you don't do this so if you remove it here for example and run one you will still have an xh prof output just with less information so if you look at the normal one you have exclusive CPU time and memory use if you don't run this you don't have these and they say it's as fast as no as without so what we used is that you have a random numbering here for example and you only run this every 1000 requests or something because there is even an xh prof but there is a module xh prof graph and it shows you the graph about the speed and the function calls and all the stuff so you can run it on the production side I would just suggest to not run it via the devil module because having devil module installed on the production side is not it's from performance point of view not such a good thing xh prof generate the files for anonymous users but not generate the link development the link is posted let me find it no it's not the right one so xh prof so basically you say each extra of data disable and then you make a new object then you say save run and then the link is saved in here they just build it so here they basically the id is returned so you can save the id so after you stop xh prof you get an id and then you can save it wherever you would like devil module just posts this in the footer for you but you don't have to do this one thing that is interesting you see here all the time the whole site and one interesting thing are these renders with the ads and also call user function array so this can be really confusing because so basically if we steps through one so we have main okay this calls menu exit active handler good page deliver html page render and now it's interesting okay which render because render is called multiple times on the site then you maybe have theme and so you have no idea now at which part the theme function was called so sometimes it's interesting like I did a lot of stuff in block so I just when actually blocks were called I wanted to know what happens in the whole block system and everything else I wanted to get rid of so what this usually works but in Drupal a lot of things are called by a call user function and for xh prof that's the same function all the time so what I do I don't know if there's a better way to do it but I just copy out this in the enable so basically if you do this you don't have to do this at the beginning of the site so if you want to know this somewhere later you just copy out this in your module at the beginning of a function the shutdown as well at the end of the function and you get the whole xh prof only for this function and you can work in there and all call user function stuff will be part of this whole system so that's sometimes really interesting yes any more questions for xh prof? okay any specific xh prof questions? show a div Fabian was talking pretty fast yeah Fabian was talking how does it work? you know the run one and run two simple as that the source I need to keep yes I think what did you compare actually well two runs yeah but you disabled for some runs some statistics you know remember no I didn't aha you mean I compare now naturex call with another one okay let's run just two of them so we have one and you have another one so it's run one yeah and that's mostly what's used in car development if someone requests a profiling that's what you're doing with the xh prof kit a lot of tests and then you compare that to a proof that you're right with your statement about performance and that's interesting so you saw I just clicked the page twice and one of them was actually one percent faster so it's really important that just like if you're really nitty gritty small adaptations the stuff depends on so much things that happens on the computer like you see I have all whatever tabs open and things so if you really want to go deep your two versions even disable everything else or try on a bare metal server which only runs this or just run it multiple times and maybe we can plug here Fabian's xh prof kit how was this if the internet ah look so that's the line side and the xh prof kit automatically runs your stuff multiple times like 100 times I guess probably can configure it so if you want to see okay is it really faster you just run the stuff multiple times you take the average time and then you compare the average times so that's important to know as well a more scientific approach then yeah give it a shot and at the end that's my local computer and it's completely different to whatever you have on your production side so sometimes it's really interesting to run xh prof on your production side to if you want to really really see what happens there the function calls shouldn't be different but the timing while all the stuff yes exactly yes and sometimes it's faster on your local depends on whatever like you don't have network connections and all the things hardware drivers like the hard disks if they have like barriers set and stuff like that that can have a huge impact okay so back to the we still have a question one question left you mean hacking core so you mean actually like changing not hacking core but changing how it is already in core based on whatever yes you mean on a performance point of view most of the time it's usability that we change it I have to say that node edit is not something I usually make fast because it's not what all the time is used so normal editors you can tell them well it's a bit slower what we saw is especially if you have a lot well we have one website where actually edits a node and there we did a lot of things for caching of references so entity reference we have a site which has like ten thousand terms if you want to show this in a select drop down first you have a problem on the browser side because browsers will die from this and another thing is loading because you couldn't end up in ten thousand entity loads so of course you can either make entity cache or you actually cache this form element so yes we did parts of this but then it's not specifically between core and not core sometimes we need to do this in contrib as well it's just changing yes but basically it's a form alter and it's a renderer with cache the things that nobody knows but exist you can really easily cache this then okay they're finished so we should finish as well okay thanks that's it as I said we all three are really passionate about performance so if you ever have a question hit us we are around the whole week hope to see you at the extended sprints though yes if you want to help Drupal 8 faster join us on Friday for sprinting just Friday are you leaving Friday? are you serious? there's a whole week I know I'm here what can you do? I don't know he has duct tape because his car was robbed and so they most of the problem is that you just have a lot of problems he has lots of duct tape and he what do you need? he needs duct tape you need duct tape? I know that so most of the time it's a problem that you want to have I don't know I don't forget in Drupal 7 I have a beach cache it's like 3 or 4 hours beach cache