 Thank you all for coming here. I hope you all had a great lunch, and you've enjoyed DrupalCon very much so far. If that's the case, just raise your hand. There will be some time for questions at the end of the session, so I would appreciate if you were asking questions at the end. This will be a lot of stuff. There's been gotten a lot of thought into what we can do in core about automated performance tracking. The attempt rules I will show, but overall it's just a lot. And it's also advanced levels, so please, if there's something a lot too much going on or something, then please just ask after the session or during the questions. When you have a question after the session, please use the microphone so that it's recorded properly. So let's start. First of all, let's talk a little about the motivation. Why does Drupal need this? Why is automated performance tracking in any way useful for Drupal? I mean, we had like Drupal 5, Drupal 6, Drupal 7. We didn't need it. Why do we now need it in 8? The point is the landscape has changed. Work performance was pretty important already years ago. It is very, very, very more important now. It is a very, very important factor in the two days web square. And there's even like this for mobile, you need to deliver all the page in just under one second. Else users are just going to skip your site, going somewhere else. And if you are having like an e-commerce site or something, then you will lose real customers. So performance is no longer something where it's like, yeah, I'm writing this little website. And then in the end, if there's a performance problem, I'll just put in like caching system, activate the cache, and then it's all good. But size has gotten too much more complicated. So there's authenticated users. There's user interaction. There are different ways of, especially in the shopping sphere, to engage users to buy products. There's real-time data that needs to be collected. All of that takes a lot of time. And while sites get more complex and more complicated and such overall also slower, there are certain penalties for slower pages like there's a Google penalty. And there are definitely studies that show in compared e-shops. In the faster shops, the users are buying more and more. So performance is something you should be planning in from the start of a project, not as an afterthought. So what's the common state of performance tracking in Drupal? We have a wonderful, fantastic performance gate. There's just one little problem. It's not enforced at all. Normally every commit with a Drupal that wants to go in, every patch needs to go through all of the gates. There's an accessibility guide, user experience guide, gate. And there's also a gate for, I forgot. Anyway, performance is also one of the gates that's very important. But the problem is at the moment we cannot measure the performance. So we cannot really enforce it. We can say, well, if something is output, like it's on the seam layer, then this probably touches the critical path of a page. So the performance is probably impacted by that. But in all of my performance work, what I've seen is it's not the obvious changes that are creating the big performance problems. If there's one line of little change that's done somewhere, that in the end makes your side 50% slower. And it's those things that you not even notice. It's like adding a decorator somewhere, and suddenly everything is slower. But because especially in the Drupal core, we have not yet built big sites with it, we cannot know what impact little changes have. Because we're just seeing, well, it's like a Bartik or base seam in the core. And well, it's fast enough. It's OK. But we cannot really see what is the real impact. How would it impact once this was live, once we are building real client sites with it? And that's a problem. So the performance gate exists, but it's not really used and enforced. The current state of performance tracking in core was kind of before it was like, hey, how long does a test suite take? If you have patches that go in and afterwards the test port runs for like three hours, which happened in the early development of Drupal 8, then it was like, probably we have a problem, especially with the CMI. There was a big issue where the config entities were not cached yet. And then it was like, well, the test ports are now so much slower, it may be we have a problem here. But there was kind of one of the only indications that were overall seeing that. Then it's like, okay, some developers choose to do it. They do like a quick, oh, I just do a quick AB minus N to just check if their patchy benchmark is around the same or is around the same. Okay, let's get the patch in, let's continue work. And overall, only for certain obvious issues, like our nice trick initiative, they made us profile every patch for the trick initiative, while other things in core like Blink Profiler or something have been profiled just, like the link manager have been profiled just very little. Because well, that's just in the back end somewhere, et cetera, it's not front end facing. So with that, we have probably missed all the real and hidden performance regressions. Because overall, what we've seen in Drupal and everyone was seeing Drupal 8 has gotten slower and slower and slower. But no one can tell when it happened, why it happened and how it happened. And we had just gotten to a point where suddenly it was not just baby steps, but because we have this nice little thing called Drupal 7 as a comparison, you were like saying, well, Drupal 7 was that, Drupal 8 is that. What has happened? But because over the whole of those things, you were just having like a little performance regression, little performance regression, little performance regression. With Drupal 8, you are not actually seeing that it gotten slower. Just in a comparison to Drupal 7, that gets very apparent. But then we're like saying, well, we now have symphony, we have all that container stuff, we have all the other things. So that's not really a surprise. Fortunately, there's also some nice work going on in the contract space. Symphony has a great tool, which is a symphony debug toolbar. And there's a nice module, which is called the web profiler. Which allows to at least see the query time, which database is used and the number of DB queries per page. Pretty simple. It also creates nice little reports, where you can see things like caches or the PHP config. The web profiler, while doing a lot, is at the moment also more like an introspection tool of how many routes do you have to find? What is going on on the side? It's useful, but it's, to my knowledge, at least not yet used by core developers as much. And also it's just, again, it's local. So I can say, well, I profile this with web profiler. It was like only 200 milliseconds slower or whatever, or 20 milliseconds slower. But do I say the truth? Maybe, maybe not. Probably, yes. But it would be nice to be able to see, well, where was it 20 milliseconds slower? What happened? What did you see that this was not problematic? And especially, what did you test? Because if I'm changing the aggregator and I'm testing the front page, there's obviously no difference. The other thing is the current performance tracking is really only the Drupal core. So client sites can be very different. And especially at the end of life of Drupal 7, we are seeing a lot of client sites that are having such huge complexity because they're trying to use new Drupal 8 paradigms. But Drupal 7 is not suitable for that because there's some code that's just not gonna work with that. For example, we found that the Drupal fine seam functions took just 300 milliseconds or 200 milliseconds on core. But on the client side, it could be as much as 3.5 seconds. And there was, in this case, due to some other circumstances, a real problem. But there was something, it was very simple to optimize, but there was something you would never find if you were just concentrating on core. So that's also by, as a performance engineer, I say it's very important to go out into the field, optimize reassights, and not just do it as an academic exercise, but because if you've ever seen a client site with 300 modules around 16 beam blocks and it's all loaded together and then there's fields attached to that, et cetera, then you're like, okay, this is a different problem than what core might have thought about when it was kind of developed. So that's what I'm coming on a little later. There needs to be some ways to get more real of that. Overall, there's some interesting questions to answer that cannot be really be answered easily. For example, is a Drupal 8 slower than Drupal 7 at the back end? How much has it gotten worse? For web pages, how much does a render cache help? How much does it, for uncached pages, what about the services? For example, there was one project where I was thinking even about Drupal 8, but because the bootstrap time was so much more than Symphony or Drupal 7, it was decided to not do it because at the moment the web services of Drupal 7 was so much faster that it was not worth it switching for all the benefits Drupal 8 was giving. What about the frontend performance? There we are faster because we're not loading any JavaScript by default. Then, how many database requests do we make? How many did we make in Drupal 7? How many are we making now? Another interesting question, how many bytes are transferred on average? Are we like sending 10 megabytes of data, compressed data, or is it more like, oh, some K, whatever? So overall, all of that leads to that. We need to automate that process. It needs to be possible, both for your client-sites and Drupal Core to automate that process to get an understanding of how much has my site gotten slower, does my site confirm, chooses standards that are set by the client and also for the Drupal Core, how does it work? How can we make it that it's automatically and that it's not such a pain? Because I've written a little tool called xhpoughkit and Mark Sonnebaum has written a tool called xhpoughlib. They allow to do several aggregated xhpough runs automatically and then in the end, xhpoughkit is giving you the minimum while xhpoughlib is giving you an aggregate of all the runs and then you're getting even a much better idea of what the performance is like. We measured every single patch for the trick scene conversions and almost every seamer knows how to use xhpoughkit from the trick initiative, but most core developers don't use it. Which makes me a little sad because it's like the tool is out there, I'm happy to help or implement in that and for the trick we've gotten really good results in that and we showed that kind of everyone can use it because if seamers can use it back and developers should be able to use it as well, very simple and what turned out that when we're using this xhpoughkit to profile the patches that we're doing for the trick conversions, we've built a lot of other performance problems in core that were not found because kind of no one was doing that extensive profiling then. Those who had to do it because the front end was always like there's needs benchmarks. But overall, even with xhpoughkit, it was still very painful and hard work. I remember one Drupal con in Portland where we're just profiling patches. Another big problem of the whole optimization is you have to manually set up a scenario. So one of the standard scenarios we're using in Drupal is like 15 nodes with 20 comments or a page with 300 comments and setting up that manually takes a lot of time because the scenario needs to match what actually has changed in the code. As I said, if you're a touching aggregator, it doesn't really help to profile the front page. So what we really would need in core is a common case of scenarios and these cases should not just be like arbitrary, like well, we have this module that profiles that page, but it should be based on real use cases, like user stories. As an administrator, I want my modules page to not longer take than one second or whatever. Or as an administrator, I want to find content in 300 milliseconds so that I'm not interrupted in my normal workflow. Those are goals, those are performance goals and user stories that then could be set up as scenarios and that then could be even enforced. And the nice thing is if Drupal core had that, then you would be able to use the same scenarios or it might be with a little changes on your client's side. Especially for the seem to trick conversions, it was sometimes very, very difficult to find the page to measure and to be able to do that properly. So overall, the question is to ask how can we make it simple? Could it be as simple as running the test suite or could we even combine it with test suite? And I've put some work in that, there will show later. But before answering that, let me go a little over what is automated performance tracking and what would we need to track? How can the performance be tracked and what types of performance tracking are there? Overall, there's a front end, there's a back end, there's a database and there's a remote services overhead. And then there's also this little thing called scalability. Scalability will be important once you are featured somewhere because then your one webpad is not enough, you will need like an army of some and you will need to know how to scale them properly, how to load test them properly so that you know how much traffic do I need to be able to ensure that my site performs with that many users. So it's all about accountability and planning properly. But overall, what we are seeing here is not linear but it's more like a tree structure. We have the front end, we have the back end, then it's talked to DDB. If the database is slow, then the front end will also be slow. If there's a remote service call that's happening like some pages like to queries a Facebook graph doing the page load, that's not a good idea. Put it into a queue, making in the background, cache it properly and not do it during the run time. Try to avoid expensive operations that you cannot control. If that remote service goes down and you have not set a proper time out and the default is 30 seconds for a Drupal HTTP request, your site will be down. None for the front end, there's also the page assets like JavaScript, CSS, images, how many images do I have? How many images do I load by default? How do I aggregate my JavaScript? There's a lot of different types here that are there and overall you can also classify all of that in time. IO, like database, IO, input, output, where CPS doing nothing but the database is doing a lot and memory. And memory is more important for scalability, for example. Now there's a thing of what to track. So there's the time, the speed and performance, the raw. You could, for example, say, painting the page that takes 10 milliseconds, but the block controller who's online that takes 560 milliseconds, or database request takes 300 milliseconds. That's already useful data. But it's even more useful in tools like Nublick or others are providing you is to give you the average. So executing the JavaScript takes on average 50 milliseconds. It can be faster, it can be slower. And just going via the average, you can then maybe find outliers that are your real pain points and slow pages. The web user page takes on average 780 milliseconds, or the memcache request take 1.50 milliseconds on average. Why is that important? Because we decided where the memcache was like responding in 14 milliseconds, or what's that, 40 milliseconds or not a problem. But if you have like a hundred requests, that's already 1.4 seconds. And so it makes a difference. But if you were just using it the whole data, you would probably see some fast memcache request, some slower, but with the average, it gets pretty clear that you are having a problem. In this case, the memcache server was in a different network. And putting it into the same network solved the problem easily. Then there's quantitative data you can use. So the number of bytes that are transferred, or the number of function codes. Quantitative has one big advantage. It is not dependent on external factors. Because if you are having like speed tracking, you are so dependent on CPU that it's not doing something. If you are having iOS, so dependent on the network, that is not doing something. But with quantitative data, you can be sure it's raw data that never changes and that you can reproduce always. And there's also the other quantitative data like the number of cache hits you are having and the number of file system accesses. That's something that Drupal Core has worked very hard on. But control modules have not. And that might be also the fault of having no automated performance tracking tools that are telling you like, hey, it's not a good idea to use require ones because that actually does make a stat call to see if the file accesses. And if you put your code base on NFS, you can expect bot-strap times of 500 milliseconds to 1.2 seconds. Just had that in a site. So it's very, very important to ensure that you're like tracking what you have included and then using just a simple include is kind of the most performant way to do that. What Core is been trying to do so because even normally your disk is not busy, assume there's some backup joke or something going on, suddenly something that's always performing, always fast is suddenly slow at midnight. But by taking care of and measuring well, for example, this S trace during the test suite, well, there's so many stat calls we are doing, we should have zero of them or we should have as less as possible of them. You can ensure that you're not going into that trap. Another thing is quality and quality is very difficult to define. For example, the quality of the aggregates, quality of aggregates means how often does the user need to get a new aggregate if he's visiting your most important pages. And it's difficult to define because you are trading speed in terms of network retrieval for the number of bytes you are having, like the size of the aggregate, versus how many aggregates you have to reload because they're not in the cache, versus other factors. Overall, you have like the front end per meters, and that's we'll just go real quickly on. So we have the time to first buy to kindly the most important thing. If you, for example, I've seen my render cache session yesterday, we showed off a big pipe saying where the time to first buy was kind of trying to send the request as soon as possible and then just pushing more data afterwards, which is important because just after this, vaping time is over, the browser can start to do something. Then there's a parsing of the HTML, CSS, and JavaScript. The JavaScript is executed in different stages. There's a document-ready event where you can do things, there's a window.law event. So for example, again, and that's really just something you normally do not think about in development, but which is very important is if you're having something that must not be there when the user is actually coming to the page because it's like, hey, behind the tab or something, you could as well use a window.law event to add that actual functionality. Any sign will be loaded faster. So not everything needs to be in behaviors and not everything needs to be in the document-ready event. There's possibilities to do things after the page has completely loaded already, which again makes it faster. Then there's the first load time with no assets cached. Then you have the cache hits versus the cache misses, the quality of segregation, as we said, the number of bytes transferred. If you want to look at that in more defaults, slides will be up. So then there's perceived performance of the page like the Facebook big pipe. Usually Facebook would have a load time of 3.5 seconds, but how many would go to Facebook if you like Facebook anymore? Because it would be like, oh, I don't really want to wait that long for just to watch what my neighbor did yesterday. So due to big pipes, they're just having like, after 200 milliseconds, you're seeing something and then your stream appears and then some other information appears and then in the end the ads appear. You can even try that out if you go to Facebook and you put like question mark, big pipe equals cell flash. Then you can see how slow Facebook would be normally if it was like a standard dropper site currently. How can we measure the front end? How can we, now in 2014, that was not possible before, but browsers have much improved on that. It's no longer just taking a look at click, how long does it take? I perceive it's a little faster. No, we can actually time that. Most Chrome has a network inspector. There's a great profiling tool in Safari Chrome Firefox where you can actually see now it's painting, now it's executing the JavaScript, now it's loading some images, now it's waiting for something. But there's also the possibility to record all of that and that is the HTTP archive 1.2 files and that's what's also used by the famous web page test org where you can even download that HAR file and then take a look exactly of when was loaded what. There's also the browser timing API. For example, for web page test org, you can send more information like that. So that are possibilities to test things but what you really need are realistic scenarios. I've tried to do a little web page test org with pages that were dumped out of the test suite. They were woefully fast. They were using stock, almost no CSS, no markup and almost no content. That is for the front end, not really useful because what we are having is complex pages with complex widths, with complex functionality and to determine the aggregation quality it's not enough to say hey, I've got the front page, I've got four aggregates, really good. But if you hit like your form or five most important pages after another like a user's coming to the page, you want to sell them something, they're going to the cart, they need to load for other aggregates, they're buying something and another four aggregates, that's not gonna scale, that's not nice user experience and that is something that's very hard to measure unless you've set up a realistic scenario, that's a scenario part again. On the back end we have the parameters of the page execution time, page size, the bytes transferred, the number of PHP function calls, file system accesses as already explained. Then in Drupal 8 we have surrender cache so it would be freaking nice if we knew how good is my render cache performing, how much cache invalidation do I have on my side, how many cache hits do I have and how long does it take to rebuild important caches like the menu router, does it take like 10 seconds, does it take 20 seconds, or is there a problem and it takes three minutes, which was also on the side we recently had. How can we measure that? The back end, there's as I said, the xhpough kit or the xhpough flip, the xhpough kit allows you to upload your runs online, just check it out, github.com slash lines at slash xhpough kit, Apache branch mark and xdebug. Then there's the symphony debug console with a web profiler as already shown, there's the strace, and while the web profiler is already nicely using the symphony functionality to give some information also already about caching, there's internal introspection for the quality of the caches, for what core is doing, this needs to live in core and not in contrap, because in core it's where we need to be accountable for and core is not accountable to performance at the moment. Here's just some sample output of how that would look from xhpough kit, this is something that you can then paste into a issue to kill just like a baseline comparison, that means it runs it 100 times and then it compares if there was any fluctuation on the system at that point, so it compares it with itself and then it compares it with another branch and then you see like there was almost no difference here from the patch it got like three milliseconds slower, which is obviously not so big of a problem. And at the bottom you have like a little link that you could click and the nice thing about that is any one of you could profile core, every one of you could take a patch that has a needs benchmark tag and they would just use the xhpough kit, you need to talk with someone, hey, does it patch change? And they could, you could take a look and then you would install Drupal and then you would apply the patch, clear caches and run the xhpough kit and then you can abort it and the nice thing is then you don't have to like trust that what you are seeing is yeah, it's 20 milliseconds slower, why is it slower? But you can click on the link and you can see the xhpough output automatically as all the data you can draw back and you can see, ah, that's why it's slower. And that was very helpful because the trick contributors were kind of like doing the benchmarking and it was slower and they were saying, okay, that's okay because it only happens like one time per page and that's okay and then was kind of saying to the core committers, well, this is slower but it's okay because it only happens under that circumstances. The amount of the database, obviously, average request response time, the number of total queries, but what's nice to have there is we need it's code by DB operation so 1% in that query, 10,000 in that queries, number of slow queries, number of queries without indexes. Then the query cache quality, DB cache quality, for example, if your query cache is used across all sites and you have several sites hosted on the same thing, then what we had was a query cache contention because it was using cache tables and database, normally wouldn't do that but the client insisted and because they were all kind of writing to the same cache database tables, the query cache was totally blown out of the water and the test suite that took normally a run of two hours was running four hours just because of the query cache contention log and changing some parameters fixes. Again, the database can be measured with a symphony debug console, should be showing you all the queries, currently it's broken, so I'm not doing a demo. Then if you get a slow query log, there's something really cool you can do to get some idea. For example, what we usually do is we do it monthly, which is kind of automated performance tracking. So we're taking from production the slow query log, then we run the PT query digest command on it from Pacona and that's giving you a very nice overview of all that you can do, of all the queries, which are the most slowest queries which are hit most often, which have the most problematic, no indexes queries, et cetera. And if that's empty, great. But don't forget to lower the threshold because two seconds by default is a little high. So just put them to 300 milliseconds, then you will see something. Another possibility is to use a MySQL tune, which again can say you, well, the query cache is like only 30% used, so might be a problem, or it's 90% used and there's a nice hit ratio, great. So most things like MySQL, et cetera, have this kind of introspection internally, but in quality could also kind of collect this introspection data and report it because we know how long each query takes, we could aggregate it, we could average it, we could report on all of that. And I don't think the overall rate would be so much more if done in an efficient way. For example, there's a possibility to use a little memcache queue or something like that. They're just writing data to memory, which is obviously not that slow. The last thing is remote services like memcache redis or a remote API call. You could have the average request response time, number of total requests, the number of operations, like request time, for example, if you have 40,000 memcache sets, that's an absolute number that might be okay, but if you just have 2,000 gets, that might lead to a problem in your code or 8% part operations for your web service. But how could we measure those remote services, the HHPOP that works some? But overall, there are no tools except things like Nubelig, which gives it for you on the map, but for the pure open source only solutions, there are no tools. You currently don't know, so what we could do, we could use a guzzled driver, we could add some introspection into there and also report while this page is doing like this Facebook graph API saying, oh, it's getting a Twitter feed, et cetera, on the live data and that would make me as a performance engineer very happy if I was loading a page and then in the debug console it was having like a big red cross and was saying like, hey, this page make live HDP request, that's not a good thing to do, but also make the development much easier. The last thing is scalability. As I said, it depends very heavily on the infrastructure, so I'm just assuming a standard large scale Drupal structure here, SSL termination servers, Varnish cache servers, a number of APIs, one MySQL master, sorry, one MySQL primary thing, and MySQL, Waclick cars, that's a new language, and KMM cache ready servers, LL Apache Solar servers, and then one of the things that's most important for the scalability here, less automated performance tracking, but still important is how much memory we need for PHP and that's the max clients is equal to server memory to the PHP memory, for example, 300 MB with eight gigabit RAM, you have like 20 to 24 clients. So if a client comes to you and says like, I need to support 100 clients and you have to say to him, well, with that memory requirement for that site, we need at least four webpads, five is better. And that's a nice good Apache buddy, they will automatically calculate that for you. How could you measure that scalability? Now we're back to the optimization part. You could do a JMETALO testing, you could do your testing based on scenarios you've created by clicking full site with HAR files, or you could set up BHAT things. If you have your BHAT set up correctly, then it will give you huge opportunities to also do performance testing, because what BHAT should be testing are your business use case, your most important business use cases, like your user coming to the site, putting it into the cart, and then obviously buying the product. And if that's slow, you'll know in advance. And now when the customer tells you I've lost a thousand customers and it's all gone haywire. But the customer tells you I've got 1,000 new customers because it's so fast. Overall, scalability is freaking hard to measure. It's so diverse based on the scenario or website you are having. If you have a static page surfed by a vanish, you can have 1,000 pages per second. But if an authenticated user, it might be more like three to four pages per second if it's a heavy load site. For core, there's probably not much we can do, because core does not match most client sites for which scalability matters, which are complex enterprise sites. So this is just not feasible. There are certain problems of the performance, tracking and core. First of all, you need a way to install Drupal reliable. That sounds like an easy task, but it is not. Especially in the, it might be simpler now, but especially in the beginning of the Drupal 8, Dress Titan store was broken every time. And you could use, of course, the test installer, but that has other problems. So you need a way to reliable and store Drupal. And then you need a way to reliable, create users, nodes, terms, files. And again, Drupal generate was broken for many versions of Drupal 8. So it was kind of not a tool which you could do in Contra, but you either need to do something with core or use something else. One more problem if you're measuring speed with HH ProfKit, even with 100 runs and with HH ProfLake, even with averaging over that, if your machine is kind of creating a DVD movie while doing the one profiling and not doing it while you're doing the other profiling, you've got a problem. It is not possible with virtual machines to do accurate performance tracking. I've tried it. By the highest EC2 instance that I could for just a few minutes or an hour or something and tried to run the HH ProfKit and I was getting such high fluctuations that was unusable. So I decided in the end to just purchase a dedicated server and then I was able to do reliable benchmarks on that. And you need a high number of runs. So not like simply test me that's trying to cram as many sites on that one server as possible. You can only run once on one dedicated server. Yes, there's now also dedicated servers on demand which might make this more possible but there's a much higher investment in speed measuring. Now that's also the problem of statistics. You have the minimum versus the average. The minimum might be way off even though in our tests it has shown when we compared the HH ProfKit which is giving you the minimum run versus the averages, calculated by HH ProfLIP, there was kind of always the same at matching and expectations. It could mislead you. Of all to make sense, you would need six data points per HH ProfRUN like the minimum, the maximum, the average, the median, the 95%er and the 5%er to really be sure that there are no outliers that there's nothing wrong with the test. So even just profiling not just not even automated profiling is already very hard in that. And then that's how to report the data. Currently HH ProfKit only uploads the minimum run. Just didn't get to that yet. But in reality what it should do is it should call the HH ProfLIP and calculate the six data points and additionally upload that. And with that you would have a nice information. For the front end we could, for example, save the HAAR files during the test suite. I've got a proof of concept of that in my sandbox. There's a visual regression testing branch. There's possible to do the full test suite, run the test suite, have it output HTML files. But as I said, the results, especially for the front end, were not that impressive. All right, core introspection. Contrib is good, but we need this in core. What can core do? One thing is, let's use the test suite. There's so much data generated during a test run we can use it. We can, for example, collect a unique request to repair test run based on the hash of the request that we are sending inside the test run. We can collect the timing, the HH Prof run, the HTML output and the assets. But we need to, and then we also, once we have said core introspection data or we have a web profile in core, we need to collect all the data that we got in there. The problem there is, what is a unique request? You, a test run might make several Drupal gets calls and then it's also difficult to compare things. So there's like a deterministic random generator I've written and a monolithic clock service to the test run has always the same request data so we can actually compare things properly. And now this test run is similar to this test run even though the code has changed. We could, after test runs, collect the slow query look after the test suite. Clear it, collect it, report it, mail it to someone. Anyone wants a slow query look? Then we have, for the front end, we collect the HR data after the test suite but take the time to first buy it based on the record data which would already give us much more information on that. And for the back end, we could compare the XHPOP average as per request theory. Behead is a great saver here. There's a lot of talking about integrated testing, et cetera. And that's already saying Behead is a possibility because as I said, if you're doing Behead write and it tests your business cases, Behead already represents a real user using a real browser using your website. We could install a query liable and we could pretty much, even with JavaScript, kit users, articles, notes, et cetera. And because we are not kind of measuring how long it takes to take a note or not need to measure that as much as viewing a note, the speed difference that Behead is a little more slow would be no problem in that case. Behead can also collect HIR data while it runs. TechOne has a prototype of that in our Behead branch where we have tried with Funtime.js to actually run the Behead and collect the HIR files and afterwards sending them to web page test org for seeing how fast, slow as the site has gotten. And Behead can also be used for the load testing. So if you set up your Behead to do less internal things but really just put the user des in the scenarios, then you can run a lot of that parallel on the server and can use it for the load testing. For example, as a user, I want to buy a product and then you are putting it into a cart and testing that scenario. And that's not even that difficult to set up if you are already set up for automated testing. And there's a lot of talking core also of we need Behead testing or we need MIG testing where we can drive the core via JavaScript to also JavaScript data introspection. As I said introspection and core, we have the data. Let's use it. We know the number of query in a request. We know the average service request time. We know the cache quality and the profiler. There's a lot of that already. But the next thing is we need to actually use the data and not only track it for one request but collect it on averages. Have something not like New Relic but Drupalic that's really getting this data and importing nice graphs out of it. Let's call the framework that Contra can even use more to get some more useful data out of the test suite. Put them in the stick random. Make time calls into Drupal time. Collect statistics. Yeah, Performance Collector Service Symphony has that. So that's great. We just need to ensure we can go the next step and report that as the average min max. Then we need the cache quality for hits misses. The profiler already has the hits and misses but there's no kind of saying like over time what is my cache hit ratio because having just in the web profiler saying just one hit is not very useful especially for things like caches you need to know it over time. Collect the data by request. Report via Symphony to your back to where that's done. Wow, but we need it in core. That's no performance gate. Okay, there's something other cool we can do. We could create the profiler purely in PHP. There's no need for an extension like XHBAR for something. It's probably not that performant but it works. We could have like a web browser. So then you go back to the profile my page. GoDaddy has done that with WordPress and then you've got now exactly how things are and not only down to the level of services or what we know in Symphony but really down to the level of what we have called find this string 1,000 times that's a little too much. And well, might have been talked to on. Yeah, the request again and profile. There is something new that was just released and or not yet released but that's in private beta right now. I've had the chance to get a demo by Fabian Mordentier yesterday. I got informed of that before the talk and I talked with him and it's free for open source projects. It already has that nice button in the Chrome toolbar where you can like have be on your side and then you click the button and then you profile it. It has an API, it collects the data over time. So we could think even though it's a larger community discussion because it's after all still a proprietary service even if it's free as in beer for automated performance tracking at least maybe until we have something built for core but a SenseAirlabs Profiler looks very, very promising over all and it's giving you all that while you were looking at XhBuff, it was giving you way too much data and SenseAirlabs Profiler is giving you just the data you need. Symphony Strikes again or SenseAirlabs in this case. How can we get started? Collect the data. Let's get the Profiler in core. We have a performance gate. This is collecting the data. This should be in core. Then we need to report the data and we need to report the data over time. We need to have a Drupalic. We need to empower Contrib more to extend into the introspection services and all of that Symphony already provides but there might be some other things where Contrib is not yet getting the data but that we will only know once we start collecting even more data. So there's so much data. There's so many possibility. So let's make the performance gate enforceable at least for Drupal 9. Thank you. That's okay. Lots of these things that we indeed love to see in core. So great talk. Thanks for spreading the word so to speak. I have got three remarks and questions. The first is develop generate. That was indeed broken very often in Drupal 8 core and was very frustrating. And we've got some good news regarding that because as of three or four weeks ago, every single field type in Drupal core now implements a, I don't remember the exact name of the method but something that allows you to generate a random value for that field type so that makes it so that we can finally give an entity, figure out which fields are in it and then generate random value so that we can create values to well generate like in core. So the API for that is now in place. Wonderful. So that's the first. The second is the BHA testing. You mentioned that that is real user testing. I don't think it's real user testing because there is no actual user there. You're still controlling the latency. And I think the critical part is that you are controlling, not controlling the latency and you're not controlling the CPU environment that the user is running in because real users are burning DVDs, for example. So it's very useful. It's very valuable but real user monitoring I would consider actual collecting of data with actual browsers of people in the wild, so to speak. But it's a tremendously useful thing to still have but I would call it synthetic monitoring. Makes sense. And then the third, if I remember. All right, so you were talking about introspection in core, moving web profiler in core, gathering the data so that we can actually enforce the performance gate. I would love to enforce the performance gate, by the way. We are making progress on that to some extent. I'm working with, I'm not sure if Luca and Luisa Luca from web profiler project is here but we are working together on making, on getting the essentials of web profiler in core, the data collectors. And we are working on, or hopefully that will happen. I'm not sure if it's going to be accepted into core but things are looking well. We currently have the standard and minimal install profiles in Triple 8 but we are working on adding a profiling install profile that comes with a community agreed consensus built set of scenarios. So for example, a view with 50 nodes and each with five comments. I don't know what it would look like exactly but some scenarios so that we can have those scenarios easily recreatable and every single commit of Triple 8. Therefore, we can compare apples to apples instead of apples to oranges. And that comes then with those data collectors built in and the coolest part. We would have a, probably also the most contentious part. We would have a, just like we have test coverage for the standard install profile, we would have test coverage for this profiling install profile. And for these various scenarios, for example, a page with 50 nodes or whatever it would be in several scenarios, of course, we would track how many cache gets there would be, how many state gets, how many cache misses, how many cache hits, how many things are render cached and so on and so on. So that goes to some extent towards fixing that and creating more awareness. But of course, it is not complete. What we would need is XH prof profiling as well and met much more data. But it is a step forward. So hopefully we can work together on that one. I'm very glad. Yeah, thanks a lot. Just one little remark that we had is still useful because what we actually and exercise we should be doing in core is we should be trying to set up from Drupal 6 to Drupal 8 like every two weeks all commits and see how performance has developed over time because that would be kind of the most useful graph for all of that. And then kind of setting up the scenario with BH means it works also with a Drupal core version to down half that profile yet. But yes, yes, yes, yes. Those scenarios sound great. Wonderful, thanks. Just a question about the feasibility of the testing with scenarios, 50 nodes with 300 comments so each sounds very good. Could we up that number to about 100,000 nodes to actually simulate real use of Drupal in an enterprise situation? Well, usually you wouldn't have a page with 100,000 nodes. No, no, but in the database, having 100,000 nodes in a database can still have a significant impact. That's another point that you bring up for scalability, et cetera. You're totally right that 50 nodes in a database is nice but it does totally not represent a site with 300 modules, one million nodes in the database which have two fields each. One of those fields is field collection. This field collection has another 10 fields. Another field collection which has another 10 fields and yeah, that's a real scenario. And you only have the nodes that have certain value and something, yeah. Yeah, that more goes into the direction of having like a profile, like a big data profile which is a very useful suggestion, absolutely. Further questions, remarks? So then, what did you say? Please evaluate the session, give me feedback. It was a lot of stuff today. I hope you still learned something, enjoyed it. Well, we've both fancy performance. You can always join the group of Drupal Archive Performance Group. You can hang out in IOC with me and others in hash Drupal performance or Twitter about it with hash Drupal Perth. Have fun and have a great DrupalCon.