 All right, so I have a brief presentation about the Moodle universal cash Which it's gonna go in my pocket Which has been around since middle 2.0. I think or 2.1 and Has changed quite a bit The performance profile of Moodle and has had a very important impact in How we use and experience and configure Moodle in at the end of the day? so I'll go very quickly over a little bit of history the first question is why do we need a special cash for For Moodle and one of the key things to be aware of is that when we use PHP every Page that we serve every URL that we serve starts from scratch The PHP engine does not remember anything from the prior request. So we have to Say which files to load which files to compile potentially and Out of Kind of at the outset of serving that page it has no configuration in memory we overcome that by Having a good pre-compiler a caching pre-compiler which now is standard in PHP But for many years we did not that have we did not have that as a standard in PHP So people would use e-accelerator or APC or different forms of off-cache now Almost everybody's using PHP 5.5 which includes off-cache and that's all these things Out of the box. However, it's important to remember that that's not that the mental model or the The general model of PHP step every request starts from a blank slate and that's different from other languages Sometimes used in in web development Such as Java or or Python or Pearl where you can load up a server load up certain amount of configuration and variables and then serve pages we we don't get a chance to preload that and Because Moodle has much of its configuration in in in a database That puts us in a situation where we have to be fetching a lot of information from the database This makes Moodle very configurable, but it makes a bit more work. So the cache is very important And being efficient about how we retrieve things from the from the database is very important during during the one point X During the history of one predicts, which was fairly fairly long We spent a lot of work working on caches, but it wasn't very successful. It was successful But it didn't it didn't have them the the desired impact We did also do a lot of work on improving a little bit the code flow and how we run is sequel queries and that had a fantastically positive impact It was also a lot of work because different from a cache. This had to be On a per area of code. It was a lot more work intensive. So we did all that work a lot of we are very generous we the large group of people applied a lot of work there and and So over the one point X era Moodle developed improved its scalability Basically by improving code flow and improving sequel code now when we went into the two-point X era that wasn't enough and at the same time we had I guess we had learned enough about dealing with caching code that the Moodle universal cache was written and It has a much better API and much better implementation. It's a much more useful cache however, something interesting happened we had we built the Moodle universal cache and Almost no code made use of it because to use the Moodle universal cache You actually the code has to change to use it and very little code in early two-point takes Versions used it. It is from two seven onwards that we have most of the Moodle code perhaps all the Moodle Moodle code base using intensely the the Moodle universal cache and So now we are in another stage in another era of of the performance profile for Moodle So still when we use we use the Moodle cache because it's enabled by default But it's very very hard to configure if you want to stray from the default So one of the questions that I've had for a long time and I'm supposed to know about performance I've been doing this for a little while and And I've been daunted by the Moodle universal cache configuration One of the things that I found that I had to sit down and study all the configuration. I was Sounds ridiculous, but I was I was quite Confused about it at the beginning and although all the terms because I'm familiar with it because I was I didn't implement it But I was part of the only confiliate early discussions when we were talking about the late in the 1.9 Cycle talking about the design of the Moodle universal cache I should know about it, but Took quite a bit of reading and sussing through the code to understand what the hell is going on but it is it is really complicated and It's hard to build up a mental model of what things each type of cache will do and how it'll exactly work out and The documentation is very good about About what each item will do and then it says well go out and test it right and That's fine, but it's fairly hard to test and it's very hard to test reliable It's very hard to measure and to know what are the right measurements So I sit out to do a bit of work for that And to look to learn and to figure out how I should configure the Moodle cache and what What I could say, you know what what I could document and communicate about it and This is a little bit distorted the story of that of that work So the stack used for this experiment that has lasted for the last couple of months not hasn't been my only my only The only hasn't been the only work on my disk, but it's been pretty intensive Has been I have worked both on hardware and virtual machines I have worked when I have worked on hardware. I have worked with hybrid drives and SSDs All the work is based on Linux all the work is based on rel 6 or rel 7 and Also a little bit of testing on on recent versions of Fedora to understand what's happening with New York code all the work has been done using Apache and PHP and two modes mod PHP and Using fast CGI, which is very similar if anybody here is using nginx the performance profile of using a patchy with Mods mod fast CGI It's very similar to using nginx It's been based on Maria dv55 and Recent versions of PHP To further to for more detail about about what the focus was we we were working on on relatively simple configurations looking at how a straightforward one machine configuration or a small cluster running in a virtualized environment will perform in different with different With different configurations of Of them of the mood universal cache. We wanted to keep the configuration simple so any configuration advantage that Requires to complex of a sit-up gets minus points obviously because it's a big hassle to set up And if it's brittle or if it's complex to maintain That all adds to the work we we have to do and we want to keep it streamlined We want to keep it simple easy to upgrade easy to manage Clear easy, you know clear and and straightforward to monitor. That's that's a key That's a key goal and in general we wanted a better understanding of how Moodle performs under pressure and So what did we use for measuring for to take our measurements we worked? We worked our way through several large model installations, so we took an example data set from a large customer of remote learner and we Picked up some reference pages pages that were popular and notoriously heavy So essentially we picked up the moodle home page and a few other key course pages right that's that those are traditionally the pages with the most content and the content coming from the most diverse array of Moodle plug-ins, right because the Moodle course page will require different things from different different blocks and different different Moodle plug-ins, so Those are the pages we use we switched off some of the authentication controls and we use the patching sorry we Don't switch up the authentication But we basically provided a cookie an already authenticated cookie to a patchy bench and we run the system through a patchy bench performance now some people will use Jmeter are you folks here familiar with Jmeter? So with Jmeter and similar tools You take a tour through your web application and it captures all the traffic and then you can replay it That's an alternative approach. It's more laborious and and Quite a bit less practical if you want to change what things you're testing and it's also very prone to giving Giving different results and different And sometimes somewhat somewhat muddled or or false results So we use a patchy bench, which is very simple for a quiz this page or requests that page And we complemented that with visiting the pages in real time while we're load testing to see what an end user what what the actual perceived perceived User experiences does it load quickly in my web browser? Or does it load half a page and get stuck and then load another half of the page or does it load the page? But not any of the associated resources It's a it's a combined approach that allowed us to move faster and test much faster than if we we had used a more in-depth test of each run which we did many many runs and When we did want to look at a particular run We did go very deep but over a particular runner and over a particular type of request So while we are load testing and throwing a lot of traffic at the server we would run One query With the php profiler on so we would get a very clear path a very clear record of where time was spent in execution and What we found was extremely interesting so The middle universal cache is very well used so moodle today issues very few database queries and that that's very good news so that that's excellent news and On the other hand we see a huge number of queries to the cache itself, which it's good but it's a tricky aspect of good because What we find is that the the flip side of the cache performance is that we see that any change in latency to on the on the With regards to accessing the cache has a big impact on the overall times so We basically focused on local cache because we found that the shared cache didn't make a large difference and Working on the local cache we worked we tested MIM cache and mongo but found that the latency the added latency Made the performance quite a bit worse We essentially found that the note the standard file cache which is what is enabled out of the box It's fairly good, and it's what you should be running, and that's what you are running anyway and We did find that after a little bit of work and a little bit of patching There's an in-memory cache Which is based on the old a PC which is a PC you which can be loaded in a modern php by five That's even faster than Local files probably by a factor of 10 now Here's the thing we can make The cache go much much faster. However It does not change the overall page load time. It does not change the time to serve pages we're hitting the cache a lot and The per request time is 10 times better if we use a PC you Because it's in memory, and it doesn't incur the cost of calling the kernel to retrieve a file The the kernel is very fast because it retrieves a file that is essentially in memory But by skipping the kernel we can get each request much faster. However, it did not matter It moved it it only improved overall results very little less than 5% and So we went to the PHP profiler to find out why and the reason why is that we spend something like 20 to 30 percent of our CPU time We are having a CPU bound when we are under when when a web server is under a lot of traffic And we spend more than 20 percent of our time running muck code so so the improvements are real the overall the overall improvements are real and But the you know when we are developing we all and we're working on performance We work on one bottleneck and so this bottleneck is resolved and now well now there's the other bottleneck right and that's that's in a sense That's always how it works. And so we have resolved in a sense muck has been very successful about Resolving the bottleneck in terms of database access and it has in turn well has expanded what we can do in a single single machine installation or even in a cluster sit up, but it has moved the ball to to muck itself so From an admin point of view from a mule admin point of view It's the the my conclusions are simple. You can just use local files the standard that the default is a good default You don't have to stress about it. You don't have to worry about it use use the file store. It's okay However for the development team the the results are a little bit of an interesting of an interesting challenge because We spend what seems to at first blush a lot of time just deciding which Which cash store are we going to query? And this time is significant right I Did work through the profiles and it wasn't clear It's not like there is one particular function that is a hot spot There there wasn't a particular place where we were spending all the time is prayed all across the muck So the question is can we devise a strategy to simplify and to streamline the mule universal cash? Perhaps might be even worth to lose some capability or to lose some option In order to cut this down significantly because one of the key things when you're developing a cache is that it's going to be used a lot and and so the code that is supposed to Make your application faster that code itself has to be very efficient So my I was already talking With Dan here one of my projects for the hack face on Friday will be to look at what we can do on that front The other aspect that I think it's interesting is that Is perhaps of secondary order because the defaults are very good But I think it's worthwhile to look at whether we can simplify the configuration Now with these changes with with with the changing of performance profile that comes from from the improvements in Moodle in recent versions of Moodle 2.7 2.8 2.9 One of the things that changes is that we used to be heavily Database bound the that the database used to be the thing that held held Moodle back in terms of performance and now that has shifted and now we are very very strongly CPU bound and in that context one of the Extremely strange findings is that reducing the number of a patchy processes or or of PHP processes Improves the response times Right and that is a very counterintuitive Conclusion, but it has the the ideal number of PHP processes. It's a multiplier Usually two times the number of CPUs dedicated the number of cores dedicated. That's a very that's that's that's a Radical change from the type of configuration we did for a patchy in years past Anyway, has that been geeky enough? Is that being technical enough? Okay Anyway We do have five minutes. I hope that I So currently it seems like we're set up perfectly then because our web server has 24 cores and And so yes, but we're looking at examining Amazon cloud etc in other ideas So and we also just use one web server so that one's itself to using the local cache the file cache. How do you have multiple web servers and Local local file cache for the for the muck Is it do you? How does that work out so? Okay, so once one of the things is that You have so the local cache will store things that are safe to store locally and So it will be distinct from from the the cache that has to be shared the Once you have a cluster yes for the shared Cache you do have to use memcache or mongo. I I think in that space what dominates or in our testing what dominates there really is the TCP latency So you need to work on on on your network. You need to make sure that it's as fast as you can And you probably want to run mongo As as a cache rather than as a store. So there's the option to tell mongo To basically give you asset like warranties. You don't want that right you want to say go fast and lose It doesn't matter if you lose your data. So that's what you can do to Make sure that the rights are as fast as you can but the limitation on reads is going to be your latency So I do have a few ideas to improve To to reduce the impact of latency because we shoot lots and lots of queries I'm gonna work on that at the hack fist and that should Improve the performance the plan that I have should improve the performance of stores that are shared such as mongo and and And meme cache and the plan that I have should apply to all of them Shouldn't be like just a fix for mongo or just a fix for meme cache. We'll see how it works So you're saying about using Apache bench rather than JV to but Presumably just doing very primitive get requests And no posts and no dealing with session, you know session keys and all that kind of stuff I mean with Jamie to you would do you could do more complex Performance tests. Yes. Yes. So it is admittedly a narrow view My intent was squarely to force a lot of work on the middle cache. So Yes sessions are a concern and in a sense what I've done is I have Work my base to remove them as a factor right from this testing reasonably like if you put your sessions, so if your sessions Your sessions can run fairly well on any face and can run fairly well on on meme cache or mongo, but You only do one pitch although the session is sizable. You it's it's just one pitch and perhaps one right per per request, right Whereas we're issuing maybe couple hundred reads and from from the from the move cash and Let's say I did my best to remove that as a factor from the variations I tested, right? I'm Yes, there's there's also work to be done there Now to kind of follow-up from a question or comment over here We did a bunch of work on clustering memcache for latency reasons and we added that in core in two Seven I think maybe two eight So basically you get zero latency on reads, but at the sacrifice of latency on writes And I'm curious if you'd looked at that feature said any I'm guessing by the face the answer is no We can talk after this. Yeah, I'm interested in that. I'm really interested in that. Yeah So you get even if you get zero zero latency and reads even if the store is It's in a different system. No, it's the way that it works is that We call clustered memcache and it's you have a memcache instance on each front end and The the the store configuration is such that when you write you write to everybody But when you read you read from the local host version, right? So the rights are very expensive. I understand So on stores like strings Right where it's very rarely updated, but very heavily read you get a huge perform Well huge in a sense performance benefit out of that Because you're never updating but you get near zero latency on the read because it's just on the local machine So that's something I haven't seen it. Yeah, I'm keen. I'm keen on catching up about that I'm pretty new to the caching with a moodle idea But how how do expirations usually work is always on a schedule or there are certain actions that cause caches to expire or How is that coordinated by the universal cache? The if okay, if I remember right the expiration on the file cache So the API allows you to say the time to live which marks an expiration different Cache back hints will use that differently I Think memcache handles the expiration internally So you just it just gets handled by memcache for those backing stores that don't have native support for it We store the expiration inside the payload and check it Maybe maybe we do find a payload, but we read it and say well actually it stale so when we won't You know that the driver for it for that Backing store will not we'll say I didn't find anything because it found something, but it was old In terms of the file store We did find that It did not effectively it I'm not sure if it didn't happen often enough, but we found that we needed to prune Expired entries more aggressively than moodle did so we ended up We ended up running a few extra cron jobs to clean up File stores, but but there is an input there is a TTL and it is and it is enforced But you could end up You could end up with Entries that exist in the in the cash store That are expired, you know the code will read them and And not really reported back to the calling to the calling Code the code that uses the cash. However, they can Add to the burden of carrying the cash, right? So sometimes is for some backing stores you might need to run a cron job to clean it up to prune old entries All right, I think that's all the time we have today. Thank you Martin. Thank you