 All right, let's go ahead and get started. Yeah. So everyone, thanks for coming. My name is Ezra Gilda's game, and this session is Understanding XH Prof, pinpointing why your site is slow and how to fix it. So how many people have heard of XH Prof already? That makes sense, because you're in this room. And it's in big letters on both sides of the room, too. But how many people have used XH Prof before? And how many people feel like they used it, but they weren't really sure what they were looking at, but it seemed like there was definitely a lot of numbers. Yeah, a few hands. You're correct, there are a lot of numbers. So hopefully with the session we'll help you understand XH Prof even better. And how many people have never used it before, but really are excited to use it? Yes. Very exciting. Great. So what problem are we trying to solve? The problem we're trying to solve is slow websites or web applications. So of course, when your site takes a long time to load, unhappy users are gonna leave. Also, if your server can't sustain a lot of concurrent people using your application, then it's going to crash. Seems like we're right next to the kitchen in this room. So if you're hearing some of that clanking, I apologize. Yeah, so every second counts. This is a pretty well-documented phenomenon on the web, but user abandonment tends to increase as time for your page load increases. So you wanna make sure that page load time is low. So what does XH Prof tell us about? It tells us about page execution time. So that means time spent by the server generating the page that's sent to the web browser or for serving to a mobile app, then it's the API endpoint. XH Prof can also measure membrane and CPU usage. It does not measure the amount of clanking happening directly behind the podium. All right, so there are a few things that XH Prof doesn't measure that are also relevant to performance that we should just identify and understand that we're not gonna discuss. So one is time to first bite. That's when the user makes a request and then they're waiting for the server to start responding and sending data. That certainly is a big factor of an user's perception of page execution time or page load time, but it's not necessarily something we see from XH Prof. We don't see page render time, so that's the time that the browser spends, actually rendering the page based on HTML and CSS and other assets that are sent back. So you could ship those assets out super fast, but if the page isn't assembled in an efficient way, it can still take a long time to generate in the client browser. If you wanna look at render time, there are some great tools for that. Chrome Dev tools and Y Slow are both really good tools. And so there's a lot of factors, including those that I just mentioned, that are part of the user's overall perception of performance, which can sometimes be different than the way that the site or the server is actually performing. And of course, the user's perception of performance is really important. Wow. I don't know if that's getting captured on the recording, but there's quite a cacophony of sound happening in this room. So designing optimistically, this is something that I'm not gonna talk about in detail in this presentation, but I highly recommend that everybody go out and check this presentation out by Mike Kreiger of Instagram. And it's a few different, really simple concepts on how they design to make the application appear to perform more quickly, even though technically it's not necessarily sending data any more quickly than another app. All right, so our goal with XH Prophet to reduce page execution time, improve the user experience, and also improve the concurrency and efficient that the server can serve with the given set of server resources. We also wanna avoid speculation. So this is a big problem when it comes to performance issues on a site. I know that I certainly, when I've heard folks talk about looking at a site, oftentimes you'll hear just what ends up being speculation. And this kind of makes sense, right? Like when we think about why there might be a problem, we tend to think about past things that we've seen. And sometimes the past things that we've seen will be related to what we're dealing with now, but not that's not always the case. So you'll hear a lot of maybe it's this, maybe it's that. And the issue with speculation is that then you can spend a bunch of time on an investigation that is fruitless. It doesn't turn up results that are useful. You might also even spend time working on remediations that you think are going to improve performance and then you find out actually that those remediations were not effective. You'll also hear really vague problem descriptions and this is kind of my number one pet peeve. Things like views is slow. XHPROF really lets us dig in and get much more granular about what we're seeing. We wanna avoid those kinds of misconceptions. So like views are bad, I'm going. And panels are bad, right? Be smart, I'm going, is kind of the message here. So tools like views and panels empower you to do inefficient things. They're very powerful tools. If you wanna put 60 inefficient listings in a single page, views and panels are great tools to do that. But it's not a negative aspect of views and panels that that's possible. It's just side effect of having a powerful tool. On the flip side, views and panels both provide a range of really powerful configurable caching that can actually help you improve performance on your site and make it easier to do that than many other tools. So the views query and render cache, panels paying caches, and there's some add-on modules like panels hash cache. All of those are really useful tools that can really help you improve the performance of your site if you know how to use them wisely. So an example, kind of a problem statement that ideally we'll be able to produce after working with XH Prof is something that's more specific, right? So whereas maybe before we said that some part of the site was slow or some feature is slow, we really wanna get to the level of detail like this statement here, right? Entity loads are slow because when we load entities we load field X, which also loads Y data, which spends Z time in the database. And on top of that, XH A loads a thousand entities of type X. So now we have a really specific root cause and a little later in this presentation we'll actually look at a few specific examples of issues that were identified with XH Prof and the remediations to those issues. All right, so just to recap, we're gonna use XH Prof to pinpoint the root cause of performance problems and develop a surgical remediation plan. So we can go in and fix the exact problem that we're facing. All right, so how many, so we did a show of hands before, but it sounds like many people have seen this UI before. This is like, this is about half of the XH Prof UI. This is the left side, which is the best side. I mean, there's just really useful information on that side. I'm not sure why that was so funny. I think I'm really finding the easily distracted by sound. Anyway, so here are the things that we're gonna see here that are important to see. The number of function calls. For example, in this example, you can see the function PDO statement execute was called 366 times within a request. The wall time, which is basically the, we'll talk about wall time actually in a second, but we have this idea of inclusive and exclusive, which we'll define. We've got memory usage and CPU time. So for inclusive and exclusive, this is a really important idea. And I think that for me, once I understood this idea and how to see this in the UI, that was like a turning point in my understanding of XH Prof. So exclusive time tells us about the time spent in this function only, and inclusive time tells you about the time spent in a current function and all the child functions that are called after that function. So CPU time and wall time, super simple. CPU time, anybody wanna guess? Time spent by the CPU, shocking. Wall time is time spent by the CPU plus time spent with disk IO. So if CPU time and wall time don't match and there's a discrepancy there, there's a good chance that you're waiting for the disk. You're waiting to write a file or read a file somewhere. So what are the kinds of things that we're going to find with XH Prof? And many of these are not specific to Drupal, but many of them are also Drupal specific kinds of things. And as you're profiling in Drupal, it's useful to be able to recognize some kind of Drupalism things. For example, an entity load so that it's more easy for you to recognize in the future when you're profiling on a Drupal site. So we might find a function that's called many times unnecessarily. And really often, not always, but often, a really easy remediation there is to add a static cache. You might find slow queries, right? And so one approach to understanding those further is to run the page again with the develop query log and use the built in explain feature. We can actually run a MySQL explain on a query and get information about why that query takes a long time. You might find many fast queries, but there are so many of them they stack up and then you deal with resource contention in the database. This is a really interesting thing and this is a little bit more of a Drupalism because of the way that Drupal builds database queries. But what I've put on the screenshot is output from develop query log on the top and then XHPROF output on the bottom. And so you can see that there's this database query that has so many clauses, right? These are node access clauses in this query. So many of them that I couldn't fit them in a single screenshot. However, the query still executes in 85 milliseconds. So is this query a problem or not? Well, it turns out that the actually, the performance impact of this query has tends to be on actually assembling the query rather than executing that. And what that means is that often in Drupal, we put queries together in an object-oriented way. So, or just through assembly rather than writing out a direct database query. So the node access system, for example, will go request all the node access parameters that affect a given query and they'll concatenate them into a string and then eventually that gets concatenated into a string and actually the process of assembling that query can take longer in terms of execution time than running it. So that's a handy thing to know and that's another example of something that you wouldn't see just by looking at the slow query log. So we might find excessive entity loads or we might find excessive calls to memcache set and get what does excessive mean, right? Well, it depends, right? You need to understand the specific application that you're working on to understand what excessive means and what an appropriate remediation is gonna be. So for example, if there's data that you definitely need within a request then your strategy should be make it easier to compute that data. If there's data that you don't need within a request, no reason to optimize the process of getting that data. Like, you know, this is another case where if you just wanna optimize database queries, you can spend a whole great long time optimizing database queries. But if it turns out that there was another path where you didn't even have to make that query in the first place, why spend time optimizing it, right? We wanna conserve developer time and make the most efficient remediations we can. You might find page blocking calls. This is such a common one. So if you have a site that makes a request to a third-party API and it does that within the page request cycle, let's say, you know, you need to get information from a third-party API to display information about a user, just as an example. Then you're gonna spend time and the user's gonna have to sit there and wait. And that's what you can see in this, in this little screenshot from an XHPROF run. You can see that there's this SOAP client do request and it's taking basically almost half a second, 467 milliseconds to make that request. Meanwhile, that's the time the user's sitting there waiting for the page to execute. If we can queue that and have it run somewhere outside the page request, then it's going to be more performant. Just so you know that there's a little bit of counting going on of you people in the session. So if you're wondering why someone's pointing at you, that is probably why. Great, so we might find excessive calls to Watchdog, which is the Drupal system log. Notices can slow down your site. So if you have a lot of PHP code that you really mean to go back and fix one day, but it's not causing any critical errors, it's just causing notices, those notices in the process of writing them to the database when they occur can actually build up and incrementally add to your page execution time. So fix those notices. It's another thing we might find. Views and panels render time. So we talked about this a little bit earlier, but when you're rendering a view, it may be that the process of rendering the view is actually more expensive than anything else. So I think oftentimes we hear that views gets a bad rap for assembling inefficient database queries, which is sometimes true, but very often the time that you spend with views and incur performance penalty actually comes from rendering the view, theming the results rather than running the query. We may also find a snail writing a turtle because somebody thought that would be a good thing to add to a website. If you find this, you want to just take it out. All right, so how do we get started with XH Prof? I'm not gonna redo pages and pages of documentation because that would not be a smart use of our time, but there's a few different ways in general if you want to install XH Prof and you're running brew on Mac, you can do brew install XH Prof. There's a really handy Drupal XH Prof module, which is a very easy way and convenient way to do XH Prof profiling in your site. There's also this handy single file tool, which is great if you wanna share an XH Prof run file and have someone else look at the results of that you found in your own XH Prof analysis. So we wanna avoid having bad data in our XH Prof runs, so there's a few things that you're gonna wanna do before you get started to make sure that the data you're looking at is useful. So first of all, if you have a menu rebuild and a registry rebuild within Drupal, within a page request that you're profiling, the page is going to be slow and there's gonna be a lot of extra processing going on in that page that is not going to be useful to you because it's definitely not common to have a registry rebuild happen within an end user's page request cycle. In general, you wanna leave the develop query log off since that can also add some incremental overhead. If you wanna come back and turn it on to an analyze a query, that's great, but in general I recommend turning it off. Another one is XD bug, definitely turn this off, make sure it's completely off. This really can add a non-trivial amount of time to your wall time. And I've certainly made a mistake of not leaving it on and then spent time looking at results and then only to find out that I didn't disable a XD bug in my environment and it's definitely a bummer when that happens. Another thing that I think is sometimes easy to overlook is you wanna test as an end user the same level of permissions that you're optimizing for. So if you log in as user one or another administrator on your site, that user's gonna be bypassing a lot of the permission and access checking control that will happen behind the scenes. And that's stuff that you wanna see because that's likely stuff that's contributing to page execution time and stuff you wanna optimize. You also wanna know which caches are in place. So one thing that can really throw off your investigation is if you keep profiling the site, you could look at the same page and get widely different results each time. And sometimes that can happen when you're hitting various different caches and you're not sure which ones you're hitting. So there's a handy script again by Mark Sonnebaum to disable the views and panels or rather the views caches on your site. There's also the views cache audit tool in Drush that's useful but this is a handy little script and that'll disable all your views caches and that's a great way to just make sure you're getting more consistent results so that you can hit the same page again and not have to worry about whether you're seeing a cache or uncached page. Same for panels and pain caching and then there's other kinds of caches. Scrooge McDuck. All right, so let's do a really quick demo before we talk about some more XH Prof scenarios. Okay, so I have a local installation of Drupal Commons here and I've done some stuff to it to make it slower and I've also enabled the XH Prof module. I've granted the access to Vell information permission to regular and authenticated users so that you don't have to be an administrator to see this information and that's nice because the XH Prof module when it's enabled and set to profile every request has this handy little XH Prof output URL on the bottom and so all you have to do is click that URL at the bottom of the page and you'll see the run information for that page. So let's refresh this page and see if it's fast. Spoiler, it takes a few seconds, right? That's not ideal. So this is the first example. I'm actually gonna just copy this link and open it in a different browser window so that I get the administrative theme. So over here I'm logged in as the administrator but it's the same site. I could have just clicked that link and viewed it as that end user. So this is our first XH Prof report that we're looking at, right? And on top you can see the total wall time, total CPU time, they're pretty similar. And we're seeing the top 100 functions. So one thing we could do just to take a look around here is we could start by looking at inclusive wall time. And this is a handy way if you wanna just look kind of top down and see is there any point in the request where there's a significant drop off in time, right? And so you can see pretty quickly as we scroll down, there's this big drop off where we have 1.7 seconds down to 0.9 seconds, right? And that's right after we run the comments posts in it function. So that seems like a suspect. We could also sort by exclusive wall time. So let's go ahead and do that. And if we sort by exclusive wall time, we can now see that there's one call to comments posts in it. And that one call takes 1.7 seconds. So we know that this is something we wanna take a look at, right? So comments posts in it. And if we click that name of that function, we can see the call path that resulted in this function. It was called by user funk array. If we click that as the parent function, basically what we're seeing is a Drupal hook implementation and invocation. So module invoke all, all modules that invoke hook init are being run, which is taking us to the function comments posts init. So let's take a look at comments posts init. This is an example, but you can see that comments posts init does some really inefficient stuff. And the comment is add magic to sauce. I don't know what that means. I'm not sure who this developer was, but we probably wanna remove them and their code from the project. So if we go ahead and let's just do a before and after, right? So if we go back to this non-administrative user, haven't made any changes yet, we can go ahead and refresh. It's still taking, you know, it still takes a little bit of time. So let's now go ahead and remove this problematic code. And I'll actually just clear the cache since we're, actually no, I'm not gonna clear the cache. I'm just gonna comment out the code here. So we're gonna return. And so now this code is not being run and we can refresh the page. And it does load much, it seems to load much faster, but let's confirm that. So I'll open this new xh prof run and we can compare it to the old one. So now we see that the inclusive wall time was one second, whereas before it was 1.7 seconds. Let's see, somewhere around there, right? Before it was 2.7 seconds. So we saved about 1.7 seconds of wall time. And we can say, by looking here, right, comments, posts, and net, we can search and just see that that function is not even showing up in this list of functions. So we don't have to speculate, we can say comments, posts, and net, we saw right here that it was responsible for, too far back, right here that it was responsible for 1.7 seconds of inclusive wall time, or better yet, exclusive wall time. And now when we removed it, we lost 1.7 seconds from the page request, the page execution, and we see that that function is no longer called. So we can pretty definitively say that this function was the problem. So this is an example where it's unlikely that you'll have something, a block of code that's quite that bad, but in fact, you actually will see different issues where there is something that is very clear that it's a performance problem, but it wasn't clear to that developer when they wrote that bit of code. And we saved a whole bunch of time guessing, we didn't have to speculate, we went right to the problem. So obviously that's a simple example, and now a little later I wanna walk through some real world examples of things that we found by the XH Prof and what the remediations were for those issues. But first I wanna talk about how to deal with when the site is slow, right? Often you'll get a request that, oh my goodness, the site is slow, what do we do? And there's a lot of urgency around that request. Performance is very important and understandably folks get a little panic to when performance is not good on their site. In general, we wanna measure performance, we wanna understand what exactly is going on on the site in a more specific way. And so one thing that I like to do is to develop a plan around what are our goals for performance on the site that's very specific. And the reason for that is that it allows us to identify when we've reached success and because we've defined what success looks like in advance. So there's no, because otherwise you can tune until the cows come home and you won't know if you're done. So what I like to have is success as defined by execution time on a specific page logged in a specific user under specific conditions. So we can say, user Steve will log into this page under this condition and it will take one second or less to execute the page. That's how we know when we've reached our performance goal. And that's a very specific goal. Two people can look at the page execution time and agree either it's one second or below or it's not, right? There's no gray area there. And it also, by quantifying things in this way we can actually quantify our improvements that we've made on a page by page basis. So at the end of, you know, we can do before and after for all of those conditions and say, look, for users with this role on this page we consistently improve page execution time by, for example, two seconds or 50% or whatever it was for that page. And that's a really appreciable thing for you and ideally for the stakeholders that you're working for. Sometimes when you'll have more than one person working on a site and recording observations whether it's in the same environment or a different environment it can be really handy to have a structured way to write down your observations. And so I like to just do a simple Google spreadsheet where I record the date, the environment, the user that was logged in, the path that we're looking at, the execution time that we saw. If you have a URL to the XHPROF run file so you can see specific results that's always handy. The person who observed that and then the notes. And the reason for that is just if you're looking at, even if it's just three different pages if you're looking at the page tweaking some settings, trying to disable caches you're gonna load that page many times and it's likely that you will forget the exact set of conditions that you used to profile that page and what the results were. And so it can be really handy to just have this kind of as a scratch pad either just for yourself or share it across a team of people working on a site to help improve performance. All right, so some case examples. And these are things that we found mostly things that we found working on the common scene or other things that we found as related to that. So the first example, does anybody want to take a guess of what we're seeing here? There's one thing that at least to me really stands out. We've got, I'll give you a start, right? We've got the function ctools, token, content type, content types. It's called 84 times and it takes 2.8 seconds of inclusive wall time from that function on down. So does anything stand out to anybody about what we're seeing here? Maybe something that seems a little unusual? Yeah. Yeah, so you're spending over a second localizing strings, right? That's the t function in Drupal is the function that we use to localize strings in the UI and make sure that they're translatable. So that seems like a long time and in fact, it is a long time. One second out of 2.8 seconds is definitely non-trivial for this code path. And also the number of calls, right? We're calling t over 128,000 times. That is too many times to be calling t. You probably don't have 128,000 strings on your UI and if you do, you could consider a more simple UI. Just a little bit of advice. It's not a UX session, but I'll put that out there. So this was a real world example. That's the Drupal node if you want to look at the details. But this was actually a very simple issue to fix, right? We just needed to add a static cache to ctools, right? Adding that static cache, we could see the results of that, right? We eliminated 128,000 calls to t in this particular environment. We reduced page execution time by two seconds, which was, I think, two seconds out of almost a three second page load. We significantly reduced the memory footprint. And the fix was really simple, right? Now, this is another example I would never have speculated. Oh, we're probably calling t a lot of times and need to add a static cache. But with XH Prof, it's very easy to dig in, see that, you know, it sticks out almost like a sort of thumb once you're used to looking at that data and come up with a really surgical remediation. That's actually very easy. Any questions about that specific example before we move on? Static caches, all right. Here's another static cache example and I'll just talk through this one. Basically, we have the Drupal node access grant system, which is how we determine whether a given user has access to perform a given action on a specific piece of content. And the list of grants that they have, which is like, you can see this but not that, basically. It never changes within a page request, yet we do compute it multiple times within a page request, so in Drupal Core. So this is another example where a simple static cache really improves performance. And it's also another example of a system where I think, you know, the node access system definitely balances a lot of factors to produce. It makes it possible for you to configure access control in a complex system. So in some ways, it's not as performant as a system that maybe you wrote completely from scratch. That's always kind of the trade-off that we deal with with a assembled system like Drupal. But there's still things that we can do to improve performance there. So rather than rewriting the whole node access system, for example, or just tossing it out, we were able to make a surgical improvement. I think Eric Web worked on this issue. Yeah, thanks Eric. Benefiting all Drupal Core users. Another issue, let's open up the Drupal.org URL. So avoid unnecessary entity loads in group reference fields. This is another one where basically we identified a code path in organic groups that under some very extreme conditions that happened on almost no sites, but did happen under the site that we were looking at. If you have a node reference field or an entity reference field that references a large number of groups, like 7,000 of them, that can be not performant. And the reason for that, it turned out, was simply that we were loading all the reference entities anytime we loaded the containing entity. So in this case, we had users, users reference groups, users could be part of 7,000 groups. Anytime we, let's say you're logged in and you wanna look at a list of seven users, for each of those seven users, we load the user account and all 7,000 of that users groups. That was not performant, right? And we could have added more servers, we could have, there's lots of things we might have done, but in fact, it was relatively easy to write a patch to just not load those entities. So that was a pretty useful remediation. So this is a more detailed example of the problem statement that I generalized earlier. So in a site with 43,000 users, where each user is a group and can reference other groups, the user is trusted context, we experienced page execution times of 15 to 20 seconds, apparently resulting from large numbers of entity loads and then I went on to describe the code path that made that happen. And then the fact that we observed that when we changed the code path, we went from 15 to 20 seconds to one to two seconds of page execution time. And this is a patch that was committed to organic groups. All right, and so the last one is optimizing node access query building. So this is another one where, I'm part of that little blackout delay where we switch between context here. This is another one where, great. So this is another one where we were able to optimize performance of the node access system and the way that we assemble grants, right? And so it's a little bit, you could say that this is an edge case just in that it only happens on sites with outrageous numbers of, outrageous amounts of content, but it's still important for Drupal to scale to that number of sites. And so I'm trying to find the ultimate patch that we committed here. So another great Eric web patch. And as you can see, it's a relatively simple change to the way that we assemble node access grants. We define the new API function. And in the scheme of things, like it's not a super long patch, but this patch had a really significant performance improvement. And it was another case where we took a system that folks, I think it's easy to say, oh, that just doesn't perform well and we made it perform better by specifically looking at the challenges to scaling that system. So those are the case examples I wanted to show. I'm happy to take questions if folks have questions about XHPROF or performance or other things. We've also got some super smart people in audience that may or may not troll. Hi, yeah. Is there a reason that you choose to use the XHPROF module instead of the XHPROF support and develop? Yeah, so the question was, is there a reason that I like to use the XHPROF module instead of the XHPROF support that comes with develop? Really, I just like that the XHPROF module puts that handy little link in the footer and it's super easy to just download the module and turn it on and configure it. I think they both work great. It's just kind of a matter of personal preference as far as I can tell. Develop puts a link at the bottom? Lousa. All right. So we have maintainers of develop in the room. This is getting hard to repeat. Do you want to speak into the mic? All right, so yeah, and I can summarize. I think everything you're getting at most. Uh. That's Mark Sonamow folks. So the develop support and develop, what it does is it handles the enabling and disabling. It does give you the link at the bottom, but then it requires you to set up the original UI. So when you go to the Facebook project for XHPROF, there's the XHPROF PHP extension and there's a small little PHP app. And that is like the original UI and it works fine, but it's just another thing to have to set up. It's very fast. I actually generally use that myself because it's very fast. But the reason I wrote the XHPROF module was if you're going to use it in production, I mean, it has a few nice options for like setting it to, like setting a sampling interval. Like you can set it to run on every like 300 requests so you can get some good data in production. It also lets you have that UI available and you can act with Drupal access control so you don't have to like set up a new VHOS or whatever. But if you're local and you already know how to use the old Facebook UI, it doesn't really matter. You can use it however. Most is going to come up to the mic. Is he going to talk about XH GUI? I don't think he is. We'll see. Yeah, so yet another GUI for this stuff. XHPROF.io is the name of the project and the URL of the project. So you set that up on your web server and it analyzes run information. So that's kind of the front end and the way you get runs to be recorded is you go to PHP.ini and there's two lines that you have to put in there that are in the project. I think Ezra's scrolling through it now. It's prepend and upend in PHP.ini and then you get your run information. So the advantage is if you include this stuff within, if Drupal is the thing that is starting profiling, then you're going to miss the beginning of your request because it takes a while for Drupal to start to instruct XHPROF to start recording. So you get more stuff in your run. And the second thing is that this is a nice GUI that works for things that are outside of Drupal and the Facebook GUI is really dated. Mark's is better than that. This might even be better than the XHPROF module, in my opinion. Oh, Eric Duran uses a different one. Maybe what we should do is just collect a list of all the different XHPROF UI's that you can use. I can, can I just augment what Mesh just said? So we'll collect a list and link it from session. Sorry, go ahead. I'll just augment what Mesh just said. If we're talking about doing a lot more symphony stuff, we'll be doing a lot more stuff in Drupal 8 where we may be not using Drupal. So having something that has a GUI that's not relying on Drupal might be a good idea. But what I actually wanted to ask was about visualization tools. Sure. Is there any, are there any ways to visualize the data the way I've used like GraphViz with extabug dumps and stuff? I think it depends on what you're trying to accomplish. I know that. Visualization, you know, just immediately jump at the problem. Immediately, yeah. I know that we've used R to do box plots and things like that, using data. I think it sort of depends on the kind of visualizations you wanna do. Mark, do you wanna say any more about that? The original Facebook UI. Do you wanna say it into the mic? So the original Facebook UI, if you have GraphViz installed, there's a little link at the top to see a, I don't know what that plot is called, but where it puts everything in boxes. Yeah, the call graph. Some people find it useful. I've honestly never found it useful because I can just sort and see that big number. Is that just experience? You quickly learned. Maybe it is experience, but, because every once in a while, I'm just like, I re-think about it, I'm just like, maybe I do need to look at this big thing of circles and I look at it and it's like, nope, this sold me exactly what I could see in the sorted graph. Yeah, right, cool. Hi there. Ezra. You had mentioned disabling extabug before you run this and so I'm curious, does that mean disabling it in the INI or just not turning on a book, Mark, that if you use that to disable it and then listen for it in your IDE? Yeah, you want to make sure, like if you go to the PHP info status page, which Drupal exposes an admin report status, you want to make sure there's nothing about extabug in there. Okay. I'm curious, how do you track your changes over time? Like, is it the way you showed where you just keep opening tabs and making fixes or do you actually? Personally, I like to save the run file. If there's a salient part of the run file that is useful to me, that might take a screenshot of it and then I put it in that spreadsheet format that I just personally like to use and that way I could see, oh, this page with this user under this set of conditions on Tuesday was significantly improved from the previous Friday after we did X, Y, and Z. I think it's just a matter of personal preference, how you want to document that. I know there's tools like New Relic, for example, which will take aggregate data and present that as a graph over time. And that's another nice way to track it. I just like, personally, my preference is to just have somewhere where I record the specific page execution time for the same set of conditions in different instances. Yeah. So if you're using cloud hosting, for example, like Pantheon or whatever that comes with a whole layer of caching that's in front of anything that you're doing in your code and you talk about how you would, if you're developing locally and testing locally and then you're hosting on a very different platform and you're not using a paid service to do your profiling, how do you take things that you see on your live site and then translate them and how you test them locally or on a different environment? Yeah, I think that environmental variables like that can always be tricky. For example, if you have memcache running on your production server but not on your local server, you might see different results as a result of that. I think it sort of depends on each case, but there's a few ways to deal with that. One is to run a virtualized version of your server. Another is you can have, you know, there's tools like Megalodon or just Chef recipes that allow you to recreate the same set of hosting stack locally. I'm not sure specifically if Pantheon offers that. But there's, you know, there's tools like that. I think that some hosting providers let you run XH Prof in production. That's something that I know Acquia has as a beta. It's not like officially there, but it's something that you can... You have it enabled on all servers. It's enabled on all servers, but somehow it's in. So yeah, if you can run it in production, that's great. But in general, it does depend based on those factors. Is there a better answer to that than the one I gave? I feel like that's the truth. It's not a super satisfying answer, sadly. The only other thing I would say to that is that we can easily, I think we'll easily overestimate the need for that. There are environmental differences. Yet I can say like, I mean, as a performance engineer, I do like 99% of my profiling on my MacBook Air. And I can usually tell like, it doesn't matter if it's the same speed or if it's slower or faster, it only cares that you're looking at relative performance. So I can see like, if I made a thousand queries and that was quick on my MacBook, I know that a thousand queries is more queries than I need to be making. And so I just look at more of like the number and what's happening in like what Ezra was talking about earlier where you have to like reason about your performance profile and think about like, is this, is what I'm seeing here, what I expect to be happening? And if it is, then it's usually fine, but usually you can find outliers like that without having to deal with the environmental differences. Yeah. Mark also hosts 99% of client sites on his MacBook Air. Yeah, Mark has the cloud in his pocket. All right, any other questions? Don't be shy. Yeah, please step up to the mic. Which requires traversing the conference center for folks at home. I was just specifically curious what the overhead for XH Prof is in relative to running standard environments. I think it's small. It's some small incremental amount of overhead. I don't know what the percentage is specifically, but it's not so significant that, put it this way, if you're working on a performance issue, it's probably significantly greater than the incremental overhead that XH Prof is going to add. Yeah, it is large enough that you don't leave it on forever for every request in production. That's why, so Mark mentioned sampling, the sampling rate earlier, but the sampling rate means is it's just, you can run XH Prof on one out of every 500 requests or every 300 requests and that way you get data that's from production, but you're not impacting every single request on your site. Excuse me? Yeah, the CPU flag adds the most overhead. So you can turn that off. And I think that the XH Prof module does that for you now automatically, doesn't it? It's an option. Great. I was just largely going to agree with you. Yes, as long as you aren't profiling a page, the impact of XH Prof is essentially zero. So turn it on so you have it when you need it. That's all. Jeff. Hi, Ezra. Hi. Sorry, I don't know your name, but I'm gonna reference your question about production profiling. And this is a talk about XH Prof, but tools like New Relic I think are more valuable for production monitoring and profiling because what you're looking for is aggregate data in those types of environments. Like I can profile locally to get an idea of what's happening specifically on the thing I'm looking at, but over time I might wanna see things that surface that I may not be aware of. One of the best examples I have of that is when we were working on a client site, it's a really big community with forums and lots of comments and the text filter to process the comments was slow, but not so slow we would have noticed it on any individual profile, right? But in aggregate with New Relic, we saw over time that this one method was running a lot and we wouldn't have just stumbled into it and we wouldn't have known that in aggregate, it was one of the most expensive things happening on the site without something that aggregates a lot of profile data together so that we can see what happens over time with it. Yep, thanks Jeff. All right, any other questions? So thanks everybody for coming. Please do submit session feedback to DrupalCon and feel free to submit it to me. This is definitely, it's like a complex technical topic, right? But hopefully based on your feedback, I can get better at explaining it. So thanks for coming, take care.