 All right guys as you as you see we've got 52 slides here and 45 minutes to give them so we're gonna get started Thanks. Thanks very much for coming to this talk It's Hope hope that you're in the right place. The talk is using black fire to profile Drupal loading time So this is about Drupal back-end performance We're Alex Durgachev and Dave Vasilevsky from from evolving web Somebody somebody gave me some marketing advice and said I should include an about the speakers slide. So here it is There we are and so about evolving web we're we're small and scrappy Drupal development shop up in Montreal We've been doing this for for a number of years since Drupal 6 just came out and was in exactly the same phase as Drupal 8 is now And we've been very involved in in Drupal cons and local events and trying to contribute what we can We we we do content migration Multilingual responsive all the usual stuff and we try to do You know larger custom development projects as well. So black fire comes in handy for that We've also got a great training program and some of the customers we work with is Canadian federal government lots of University sites and and just generally large organizations What we'll talk about here today is Just what is profiling? What does that mean? How do you do it? We're gonna do a demo of the tool that we love to use black fire. Oh, hi George and because you're not here for a marketing talk about a tool you're here for Learning how to actually use it with Drupal will will share what we what we learned while applying it to some Drupal projects So so the first section is just motivating You probably already know that loading time is very important. That's why you're here Let's see. So there's there's the user experience angle I mean people like me are incredibly impatient if a page is taking too long to load I just hit back right away and go to the next Google result Obviously, there's a concurrency and scalability angle So if you were if your page is loading slowly and you get a huge influx of visitors Which is your goal right when you're making a web website You don't want it to crash and of course the first thing you're gonna think is throw more hardware at it And that's a smart move in the air from cheap hardware and cloud and all that But that has a limit and also the moment you start like going from one server handling all the requests to multiple servers or multiple app heads or a DB server or more Replication that that right away adds a lot more complexity and makes your life a lot more difficult So you don't want to do that until you have to and it's better to say lean and mean There's there's definitely financial implications on both the previous points I don't need to tell you that but an interesting thing is a is If you probably don't don't know this but Google's main innovation at the very start wasn't just coming up with a better search algorithm this page rank thing that we all heard about what it was is their version of page rank was Perform it They were able to do things that other guys just couldn't do because the web was too big to calculate this by having a sampling distribution of like, you know They looked at which sites is linking here and there to each other And they were able to do it in a non-deterministic way with with random sampling and they were able to do it faster And that's for that reason that in the first in the first five six years of using Google They would have put down this search result rendered in 22 milliseconds on every single page I mean that was Google's in that's how they got big. So hopefully There's a lesson in there for us So first I want to I want to re-emphasize what profiling like backhand profiling doesn't measure It doesn't measure the overall browser rendering time How long it takes to download the HTML the CSS render all that run the JavaScript fetch all the other images and assets? That's that's not something that that we're doing here. What we're doing is is Kind of looking at your back end and looking at exactly what's making it slow So I came up with a snappy slide profiling is like an x-ray for your for your code performance and it really looks at what is going on in the Each function call which one of them is taking longer and And figure out how do I fix it? So what we're going to see is profiling gives you Time spent overall in the whole page and per function call It shows you how much resources is being used for each function call including how much of the CPU how much of the wall time how much RAM Overall how many DB queries are used also the the network usage and and very importantly disk usage as well These are all the things that are going to make your your site take time to render And this is what you're trying to figure out what is going on and try to get that internal view and a profiler like like black fire is a tool that hooks into the PHP engine and Instruments of function each function call now instrumentation roughly means You adding manually at the start of each function call microtime and save that and then at the end of each functional microtime save that subtract the difference and that's your sort of inclusive time and then For every nested function call it measures that to and calculates all that and and a profiler gives you a big graph of that to analyze So why is this important for Drupal? Well Drupal is not the lightest web system out there in a real-world context Drupal sites have a bunch of contrived modules have probably a bunch of custom code that some of you wrote and Realistically It's gonna be a hundred or a couple hundred queries. It's it might be 30 modules at the lightest end to 200 modules on the heavier end or maybe even more so Drupal sites in the real world tend to be kind of sluggish and This is why I too like black fire is particularly important keep in mind that core Drupal core has a Lot of eyeballs on it. So it has a lot of great optimizations already So there's less low-hanging fruit in Drupal core. Maybe now with Drupal 8 just having come out There's some but contrived modules even five years after Drupal 7 was out contrived modules have so many problems that no One's ever run a profile tool on them crazy, right? But it's true so And probably the same goes for your custom code base I mean if you if you don't put that out there and people aren't using it Unless you use the profiler on it then that no one's done it just a poll of the audience who here regularly Or at least once has run a profiler on their custom code Pretty good. All right great. So that's about two-thirds of the people in this room So and who was who does this on a continuous basis who does it like, you know every release? Okay, a lot less hands here and Yeah, so so this is in part because it's a bit of a hassle in part because it's takes time in part because the existing tool Isn't so user-friendly. So you do it when you have to so Hopefully that's beginning to change and and you'll come away from this presentation being motivated to say hey This is easier and more fun than I thought. Let me do it more Great Another thing to remember is a lot of the performance problems with the Drupal are often Handled by sticking varnish in front of it, which is great because varnish is a super efficient cache but it's but it clearly works in a in a in a Like a section of your use cases I mean most visitors will receive a fast page load except the ones that don't or when you have a High-traffic site and somebody clicks edit and a note and click save and all of a sudden your home page isn't loading anymore because all the caches just got expired There's there's the problems with cash stampede if you have search and you let especially if you have a search-driven UI that that's gonna all throw off the varnish cash and and so and that Goes without saying that it only works for anonymous users not for you anyone who's logged in and has a cookie and generally On a personal level as a developer working on a slow side is demoralizing I mean at the end of the day no matter how cool the functionality you're building or how nice the design looks if it just takes A while to load every single time you're just gonna be miserable at least that's me So I think this is my favorite slide here. It's profiling Summary like what we did is we we started using this black fire profiling tool recently and Both for a few of our existing projects that we were doing now And we even went back on a few recent projects and and ran it to see if there's any dramatic Low-hanging fruit that we could report and sure enough there were so take a look for from a good university There's there's a listing of all the Programs and courses that students have to take to graduate That's a very lean site already that was loading in three four hundred milliseconds max per page But it turns out in one hour we we found a very easy Vestage of Drupal 6 that we never upgraded or changed it in Drupal 7 because it kept working But it was a performance problem and saved 13% on every page load for a mysterious client Who shall rename remain nameless we've got a very dramatic speed up on their home page for anonymous users And that was not 300 milliseconds. That was over a second on average. So and that's been dogging them for years For for our own site in Drupal 8 take a look. We had a very dramatic Core improvement. So we we were able to shave off almost 20 percent. Sorry like around 20 percent of a page load It's a light site It took a lot of work because it was a Drupal core Improvement and we're still trying to figure out how to make it so that it works not just for our site But for other sites and how to contribute that back But this this particular performance problem is dogging every Drupal 8 site out there and we'll talk about the later on and And and finally for Linux foundation last year we did a big internet project for device certifications kind of the new Internet of Things standard and And that was us and that was pretty fast as well But we did have one view that was slower than than most of the other views and we couldn't figure out why and using this tool We were able to to shave add a cache and shave a lot of time off Basically node revisions are not really cached. There's no like Proper like views node loading and entity loading is cached unless there's revisions involved So we worked around that So hopefully you're you're motivated that was the the motivation section The the next section is is our profile profiling methodology the same marketing consultant in the back of my head So we need more acronyms. So we came up with our methodology called mafia. All right, let's see what that means So M is for measure Figure out exactly what you care about to measure use a benchmark tool like chrome dev tools We'll show you like how long each page takes keep that open As you browse your site as you browse other sites and as you browse the web to connect your intuition What's a fast site and a slow site to the actual number that it's giving for back-end loading time? That's that's a trick that's going to take you probably a week or two to do on a passive basis And then you'll have an intuition for what's slow and what's fast Also, keep in mind that some some pages are cached some pages are not sometimes there's you're logged in sometimes You're not exactly what URL you're looking at whether there's a redirect involved So there's subtle things that you're going to be paying attention to and benchmarking on an outside in kind of you Also, keep in mind that when you're comparing things you can vary a given page of a site site to site serve environment Like maybe it's slow on dev, but fast on prod. Maybe it's fast on dev slow on prod What else is running on that server at the same time? Also another good benchmarking high-level trick is just disabling some modules if you're like trying to figure out what's going on that's Good to vary things up like that and and and of course for measuring You're gonna use a profiler to actually figure out what part of your code is slow Is it the database queries that we're running? Is it some web request? Is it some inefficient calculation that I should be caching and I'm doing an every request unnecessarily? so Let's see how to the next step in mafia is a for analyze so once you're looking at a at a Profiler output, which is a call graph and some other metadata you're gonna you're gonna look at it and say what is being slow What are the the low hanging fruits? There's always some I mean if you haven't run a profiler before in a code base Chances are there are things that you're gonna be able to spot within a pretty short amount of time And so look for things that are easy to fix and then look for things that are just clear bottlenecks So focus on the the big wins in our experience the things we found in that category are Slow calculations that should be cashed like if you're rendering a whole menu on every single page And then you're not caching that for some reason That's a problem. So Drupal core does this but what if you have a contrived modular that renders a mega menu? That's gonna be an issue that probably dogged some of those sites slow SQL statements That's that's a big one Not not too much in Drupal actually I think Drupal has enough complexity that it's usually in the application layer But once you've optimized that a way then you're gonna have a dot bottleneck in the database web requests like if you have microservices Sometimes we see unnecessary entity loads or too many of them and lots of things like that that we could just go on another subtle thing that you're gonna Need to be aware of is that sometimes there is no obvious bottleneck But this page is still clearly slow or the site is still clearly slow And you try to figure out why and oftentimes there's a server environment. That's a play and so you have to the profiling Who won't show you this aside from the fact that there's no bottleneck That's what I'll show you but you might have like swapping if you're out of RAM On our dev environments, we use vagrant and at some point we're using virtual box shared folders And so our dev sites were just every function call was slow because it was trying to like load the load the code base off of a bad Lee implemented file system Are there processes running on the same server? Maybe you didn't install op-cache. So keep that in mind, too Okay, the next step in mafia is f for fix so make a change And see if it helps so it's very important to do a comparison, especially in Drupal, which is very complex You don't know the whole code base. You don't know exactly if you if you Optimized something and hook in it. Who knows that? Maybe during page rendering time you got rid of a slow query But during page rendering we need the results of that query anyway And that would have been statically cached and now it's not so it's just gonna run that query again So you think you just saved a hundred milliseconds, but really you just moved it from here to there and So that's that's an important thing to consider Next step of mafia is I for iterate which means keep measuring again You've made a fix you've confirmed that this helps now. Don't stop because there's gonna be more More things to look at more things to fix Of course at some point you're gonna have to know when to stop what is fast enough and that goes back to user expectations you know performance analysis of like your your infrastructure and and that intuition that I was just telling you you got to build up like you have to learn what the numbers mean for Each site and how that translates into what your business goals are Another thing to to keep in mind is please Try to log your runs try to keep detailed notes depending on the tool Like if you use something like black fire that makes it easy for you to keep a historical lot record of all your profiles If you use XH prof that wasn't particularly organized There was no way of labeling them and so after a day of using XH prof a few years back I remember just saying what the hell do I do with all these so log log your runs and and keep in mind What what you're comparing? And the next thing in mafia is I actually couldn't come up with an a I don't know if anyone here in the room Can can do better so so we I even used to thought the Zora's comm for no nothing Okay, yeah All right, so That was the the theory and I just want to circle back and talk about the the tools so for for front-end which is that next session somewhere over there you've got We always just start using Chrome developer tools But there's a whole suite of tools that help you figure that out So go go learn that and and in my opinion, that's just as important as the back end It's it's it's yin and yang of optimization of your page loading time So don't don't forget, you know, even if you're a back-end developer, you need to be aware about front-end performance Once you actually want a benchmark a given page usually looking at it once isn't enough You got to get a bunch of runs So we we often use tools like Apache bench or J meter which is much more complicated Apache bench just says hit this URL this Many times with this level of concurrency J meter It lets you script. It's a Java program that lets you record and script and replay logs of real visitors So that's a great tool as well to get an intuition for how how your infrastructure can handle the load and how long it takes There's a there's a new category of tools out there called APM application performance monitoring the most prominent tool is new relic and and that's kind of a mix between server monitoring and Getting a picture inside your actual application. It knows about SQL queries It it can can give you charts of in production of Exactly what you Drupal site is trying to do This is great for figuring out why the hell your server just crashed and we highly recommend you use it on the other hand It's not always that helpful for optimizing the specific bottlenecks So that's where the next phase of tools come in which is a PHP profiler The most prominent one that everyone's been using is XH prof. I bet most of the people in this room who profiled use XH prof Yeah, yeah, so we've been using that for years and it's it's pretty good. It was made by Facebook But it's kind of showing its age. So the one we're talking about today's black fire, which is Kind of a huge improvement on it All right, so what is what is black fire? It's it was created by Fabian who was actually in the room or at least he was a second ago Fabian All right, and Fabian's also the creator of symphony and twig It was started as a fork of XH prof so Eventually it became its own beast and and most of it or all of it was rewritten, but From a user perspective, what does it give you over XH prof? It gives you a much simpler installation. So you actually have less excuse not to use it all the time much better docs a Far nicer call graph that you can actually zoom in on easily And a better just user experience like all these annoyances that you had looking at stage prof Like for example the logging of the runs like and not be able to compare things and label things That's that's a little tweak that makes a big difference Another important thing is it supports comparisons. It's actively maintained I'm as far as I'm aware they they don't have a version of XH prof that works with PHP 7 black fire Or he's got one and these guys just are building the product and making it a fun experience to use and It's it's not an open-source tool, but like good hub It's the base level is free what XH prof gave you gave you that's free and a lot more And then at the same time at an enterprise level there, you know if you're using this all the time for a large team That's when you want to go premium So the first the first case study we'd like to do is for McGill University and my colleague Dave will lead you through it Thanks Hey So now that you know about block farm, I'm gonna show you how to actually use it first of all How many people here have actually used block fire before? All right great, it'll be well the guys in the red shirts So great you're gonna learn something so we're gonna talk about the McGill University Website that lists all the courses and programs that students have to take and this is a really important site to To a profile because this is this every Every fall when schools about to start about the day before the deadline for registering all the students suddenly realize Oh my god. I have to fix this. I have to register for my courses and if the site was slow would just go down That would be pretty bad Also, it's a search-driven UI in many cases. So we can't just enable caching. It has to run. Well even uncached So I'll show you a page here Here's one of this we have this running in our dev environment and It's a reasonably okay webpage and I'll show you how to do a profile with it So we have our black fire Chrome extension installed And all we have to do is hit profile and there will be a live profile So please the gods of presentations be on my side And so it does a bunch of things profiles the page and Then it sits around. Oh, there we go. And now we have a profile to view So I click on it and they go to the black bar site and so this is hosted on black fire servers And it's available for you to share with other people in your team, which is Yeah, I'll get there, which is another nice feature of black fire And so this is a pretty complex interface for the first time you see it It's it's much more complex in some ways an xh prof, but it's also got a lot more functionality So I'll give you a brief tour of like how all these things work Along the top we have all the different metrics for the global run of your programs This says that your program took 225 milliseconds of total time This one is network and disk time. This is CPU time. This is memory This one's for Network and HTTP requests that are done by your program. So this is great to get an immediate overview of what happens What's your what's your website doing? And you can say is 225 milliseconds too slow, then I really have to look at it if it's not too slow, then you can be happy and you can Go on another page in The center of the page here, which I can zoom in on you have a call graph And so what this does is so when your program start when a Drupal program start it starts running index dot PHP And then index calls some functions it calls menu execute active handler for example But it calls other things too We can see it calls Drupal bootstrap over here And so this provides a graph of every function that's important in your program everyone that takes a reasonable amount of time And what other functions it calls is really great because it lets you both zoom in on things Let's you see what exactly did this function call But it also lets you get get an overall view of what is the structure of your program look like you can see in many Cases just by looking at it once you get the hang of it that oh, yeah There's a lot of recursive calls happening here And that's a really good way to learn your way around a code base that you haven't seen before You can also see that for each each little box here each function It shows you how what percentage of the time it took and how many times it was called So this one is called takes 50% of the time and it's called once But there's more detailed information on each function on the left hand side here Which is a list of all the functions for each one It says how many times it was called and how much time it took both exclusive and inclusive time So here you can see how much time each function the functions ordered by how much time each one took and Then you can also zoom in on them You can see exactly how much time this one to this function took nodes paid view node page view But an important thing here is that it shows you what that one called Here as a list of different things and that these boxes here are different from the ones here Because this one shows you how much time node show took only the times that it was called by no page view Well the box here for node show wherever it is which show you how many times how much time it took when it was called From anywhere in your program. So it's a useful differentiation to know about Okay, so now that you know what the interface looks like let's see what's going on here and what's making this page slow You can see in the in the call graph here that some things aren't read And those were the what they call the hot pass the things about your program that are particularly slow And so we'll look at one of those we can see here that one of the things that's taking a while is theme the theme function That's really normal in Drupal. That's okay But then we can see that there's a bunch of things that theme calls And one of them is here is a pre-process page for the theme It's pretty unusual for a pre-process page to take 13% of your total load time. That's not good So we'll look at this one a little more we can follow it down through other things that it calls It calls Drupal get form eventually it calls some stuff for the search box and finally we get down to here Which is a function in a custom class called load academic faculty nodes and we can see this function was called twice But it does 36 calls to node load That's usually a bad sign You shouldn't be calling node load that many times and that explains why this function is taking too much time So we can look at the code for this function now now that we've been prompted by black fire So we'll look at the code and we'll see what this function does is indeed it goes through a list of faculties And for each one it calls node load That's that's a bad idea in Drupal if this code comes from Drupal 6 where there wasn't any other option But now in Drupal 7 we have node load multiple that lets you load multiple nodes all at the same time and do One query to get all the fields or to get each field of all the different nodes at once So we can really easily fix this we can just change our code so it goes through the faculties Collects the nids instead of loading each one and then loads all the nodes all at once with node load multiple and This is a lot faster when we did this Oh This is an anonymous view with cash and turned off so that we could profile it No, I'm sorry. So you need to install a black fire plug-in and give it your credentials Yeah on your server, so you can't just do it on every single page anywhere if only yeah, that would be pretty cool Guys, we have a feature request So, yeah, so you can see now that this function now takes a really small amount of time It takes only four milliseconds, which is almost nothing So we saved about 25 milliseconds this fix took under an hour the first time we did it and had a measurable effect on our site So we think this illustrates how easy it is to get started using black bar and why you should start And now I will continue Great great that yes I actually had all the components listed out and and my colleague Dave forced me to take it out So there's a you got to install a PHP and extend a PHP extension That's super lightweight super minimal called a probe it's written in C and all that does is is it instruments each function and sends that data over an HTTP socket to the black fire agent, which is a an actual little application running listening in a port Compiling all that and then sending that over to the black fire servers Yeah, so for for everybody else the question was is there a partnership with with Acquia Yeah, yeah for the hosting yeah I I'll just answer to to to be faster. They I know I talked to Mike Maris in Acquia He's aware of black fire and and he said that They're they've got that in their backlog right now and he said I want to I want to hear cut more customers Asking for this integration and then I'll bump that up on my agenda. So so please you know what to do All right, and so you install black fire one of the great selling points of this I'm not I'm not selling it But one of the great selling points for me was it's easy to install They give you a really slick page that like you copy and paste four or five lines like this They it's offered for Mac Ubuntu red hat even windows. I don't know how they did it, but didn't try So the even like if you're logged in you just have to log in to get up with get up to sign up It takes a second they give you the server ID and the server token So you just copy and paste all that from one page and you've got that installed on your on your site We run it in Docker that works just as well or you can run it directly in on your Mac So that shouldn't hold you back from trying it Now I want to cover some some more interesting features. We already talked about the need to have baseline baselines and comparison from before and after and Although we had to cut this from our demo black fire makes chrome extension makes it really easy to To say hey this profile I'm about to make Label it like this and make it a reference that I'm going to compare with later And then after you run that second profile and then you say compare it with this previous reference It'll generate a call graph of diffs every single Function that appears in this new call graph of diffs It shows you that how much was saved or or lost in terms of performance improvements And that's that's amazing and this is a key feature Another another really handy feature is they have a command line app command line tool called black fire client that That you can use for Ajax requests for things with cookies for post requests for for testing web services and And we're going to show that later But one of one of the specific use cases for that is to is to profile drush So what you got to do is you got around? Blackfire run that's the client and then you type in whatever PHP command You were going to type command line in this case drush launcher CC all in case you're wondering we say drush launcher instead of drush because drush the overall executable forks and That'll fool the if you fork that'll fool the profiler So we got to use the inner tool drush launcher Another key feature is that these profiles because they're uploaded to this to the service You can share them with your colleagues with your friends if you write a blog post like we did about it You can share it with the world So this is really going to make it easier to have a conversation about how to profile and what's slow on your site So that's that's a really key win Another mini case study we had to shorten this for you guys Sorry was for this client X and you know, it's it's always like this the clients who have the most to gain from profiling Are the ones who don't want to talk about profiling? So they're remain remain remaining anonymous So the issue was their home page is sometimes taking five six hundred milliseconds Sometimes a minute to render for anonymous users with varnish in front of it and that's that's not right Sorry a second, excuse me. I think if you take in the front-end resources, it was a minute, but that's a different conversation So first thing we did is we open Chrome's network tab and we looked they were like ha look at that More than half the time for the back-end rendering at least on my laptop was Was spent doing a 302 redirect from the home page to slash again for some sort of a language default setting I don't know what kind of complex logic they're doing but it shouldn't take 357 milliseconds to figure out that the browser language is English and Let's look at that. We don't know anything about the code base So let's use block fire to you to sort of give us this x-ray of this unknown code base for a consulting client It wasn't it wasn't something that we wrote So we go into Chrome we right click copy as curl. You've all done that. Oh Yeah, this is a redirect request the Chrome extension will get fooled right because when you When you actually go to the home page it'll redirect to slash again And then the URL that you can activate black fire on is already slashed again. And so your black fire Thing won't actually catch the first half of that loading time. So that's a little gotcha So right now we open that Inspector the network tab of Chrome dev tools we copy as curl. Who's done this here? Copy is curl. Yeah, this this is a handy feature of you debug with it all the time What is going to look like is everything after that black fire thing? So like curl URL all the HTTP headers including the cookie like everything if you want to be logged in you're logged in and then we pre-face black fire with that and then that tells the black fire command line tool to add extra HTTP headers to tell the server Hey, this request needs to be profiled and instrumented and uploaded to blackfire.io It does its magic. It's a command line thing and it actually shows you a little summary Then it it gives you a URL for the graph We don't have time to look at the call graph, but right away we noticed In the hot path was this lovely thing a theme hook TQ pre-process page that calls Drupal go-to after doing a little bit of language logic Okay, does anybody see something wrong with this picture? You probably shouldn't be doing a A simple redirect inside of a theme pre-process hook by then you've rendered the whole page and And you've gone through everything you want to move that up as soon as possible in this case It was is quite trivial to move it to hook in it and and just save I don't know like 320 out of those 350 milliseconds So that was an easy win on this particular client There was like four other easy wins with caching of mega menus and all that that we also did So there was a there's a lot a lot to optimize We got a much better performance and of course we have to compare the benchmark and like I said it was from 350 to 74 So and we didn't know anything about the code base at the time So having a tool like Blackfire to show us a picture is very helpful I've got a few more tips for you One of the nice things about Blackfire is that when you do a profile of a page It'll actually load that page ten times and average the result. That's called aggregation Sometimes that's great because it controls for caching and stuff, you know Like otherwise you'll have really crazy variability one time you're looking at a out of profile And it has this many milliseconds and other times it's that many milliseconds And you don't actually know what you're looking for what you're comparing So having this average makes you a lot more sane compared to what we had to do with xh prof Other times it will confuse you because you need to profile the uncached behavior or or just that one specific request Like if you know what you're doing so sometimes you got to use that checkbox that they have in the Chrome extension That says when kicking off this profile disable aggregation, and that's what it means. It says do it only once This will also explain why sometimes it says a function was called 2.5 times It's whatever it's because it was averaging sometimes it was called to sometimes it was called three Another thing to remind you is that Blackfire doesn't keep function arguments So every time the theme function is called it measures it and pulls it all together as for that little box for Theme whether it was like theme table or or theme page all of those calls are pulled into one If you need to help you debug and figure out what's going on if you need to separate that out Blackfire does have an extra feature that lets you sort of separate one function call to have separate notes for that But that's in pro Okay, another tip is x debug first We we think that profiling is is a great first pass to figure out a code base But after you do that you're gonna have to read the code and if it's complex enough You're gonna have to step through it in in a debugger. We use x debug and PHP storm So that's that's a great combination Unfortunately x debug completely skews profiling So you've got to turn that off every time you run Blackfire and turn it back on when you want to use it and turn it back off So I think they're working on optimizing that but as far as we know you got to restart Apache a lot There's also a profiling overhead to consider when you're looking at the numbers from your benchmark tool And you're looking at the numbers from your inside of Blackfire There's typically Blackfire is gonna be a bit slower because it's instrumenting everything We find it sometimes between 20 milliseconds or 20 percent and a hundred percent of the request So it is substantial enough for you to keep in mind it varies But we despite this variability we did still find that it's consistent enough to not be such a huge problem Just keep it in mind, but it's consistent It skews everything kind of equally because every function call so it's it's it's not so bad another important consideration for using a profiler to optimize is the fundamental computer science trade-off of of Data structure versus running time or in this case memory usage versus running time if you if you say hey This mega menu is loading too slowly We're we got to cash it and we're gonna because we're using it in three places and through the request So we're gonna had a static cash. Well, you just added a Meg or whatever half a meg to your memory usage for that request and and if you if you took a intro to computer science algorithms course They will always tell you about like in place algorithms are slower data structures are faster But they have a higher memory requirement, and there's there's a fundamental trade-off here in profiling as well Another key thing that we discovered that we never saw in any blog poster docs Is the fact that the blackfire SDK which is a like a set of PHP classes that you can manually include in Drupal or your own PHP app has an enable probe and a disable probe method That's a start profiling from this point on an end profiling here And what that'll allow you to do is sort of like have a micro profile of just that code section to deal with that pooling of different costs to theme and the like if you want to debug it It's it's a dangerous tool because it doesn't show you the whole picture So you can use it to figure out what what's going on in a particular section of the code But make sure to run the profile again after you've made your fix on the whole site to make sure that the difference is actually helpful And I think the most interesting case study that we have for you today is for our own site and in d8 and Dave will cover that Okay, it's on All right, so we upgraded our site to d8 We talked about it at a couple of Drupal camps and so you can go see a presentation that we gave about that It's on the internet and we really like d8. Actually. There's a lot of cool features. We love views. We love ck editor We love twig. We love all these features, but there's a little problem. It's slower than d7 and that makes kitty sad So here's a page on our site the view of blog posts that was slower than we'd like It's actually really fast when we cash it because Drupal 8 has built-in page cash and dynamic page cash and all these great things So when it's cashed It loads in like 40 milliseconds and we're happy But every time we edit a node that invalidates Drupal 8's cash tags Including the node list one so the view has to do all its queries all over again It's the render things all over again and it takes a while it takes a few hundred milliseconds Which is way longer than we want and longer than a tuck in d7 So we did some profiling To try and figure out what was going on and we encountered a problem here that As Alex mentioned black Friday's aggregation it'll hit your site ten times But if you're trying to look at uncashed behavior specifically This is a problem because the first time it hits your site It's uncashed and after that everyone's cashed and so you can't focus in on the uncashed behavior You could just disable aggregation But we don't want to do that in this case because it'll make our results inconsistent will profile once and we'll get one number We'll profile a second time We'll get a different number so we want to do is to keep aggregation on but find a way to focus on uncashed behavior So we found a cute trick to do that Symphony and now Drupal 8 has this concept of an event subscriber that can listen for events that happen on your site And one of the events that can listen for is when I first see a request it should do something So we added an event subscriber that listens for requests and as soon as a request comes into the site It pretends that it just edited a node by invalidating some cash tags and you can see what the code looks like here It's pretty simple and so what this does is every time we load we go to our site We pretend that we just edited this node and we get the uncashed behavior don't commit this It'll make your site terrible, but it's really nice for profiling So we ran a profile and we found out that the reason why our site is slow at least one of the reasons was actually in Core this time. It wasn't our own stuff There's a function called get visible blocks per region that figures out which blocks should be visible on your site And where they should go and this is a really important function because it doesn't just run Once in a while it runs on every single page even if big when you're using big pipe It's still to figure out what block should be where? So so yeah, it runs all the time and it's really slow on our site. It was taking a hundred and seventeen milliseconds Which is a bit over a fifth of the total site loading time So that doesn't make us very happy and it sort of makes sense that it takes so long It sort of makes sense that it takes so long because we have so many blocks on our site We have like a hundred and thirty blocks, but that's pretty normal for jupy late there's a ton of things that are blocks and I don't think it should be unusual to have that kind of that that Quantity and it definitely shouldn't slow your site down this much So when we looked at this function that does get visible blocks per region function We found out what it was doing was loading every single block for your theme Getting them all from your config and from the database Which already takes a lot of time because it's loading all these blocks just to check if it should show them or not and Then for each one of us to check whether it should be visible using a bunch of different conditions Like what is the page path look like what kind of note are you viewing? What user is visiting the site now and it's a pretty complex set of conditions that involves a bunch of different Metadata and it takes a long time to compute so This kind of sounded familiar to us because we have the whole node access system in Drupal If you're loading if you're looking at one note in Drupal It'll load the node and do an access check But if you're want to look at a whole list of nodes that would be way too slow to load every one just to find out if it Can be seen so instead it uses it stores a bunch of information the database and uses a single database query to figure out What should be visible so we built something that does that for blocks We called it block access records and it does a single DB query to figure out what should be visible you can see we did a comparison profile here and It saved 80 milliseconds on our site for every single uncached request and it has enough stuff built in that it should work on Most other Drupal sites out there So we hope you'll help us try to get something like this in correlator because we think would be a great improvement for Drupal If you remember from that earlier Earlier slide about how long things took that was the two-day optimization exercise So that's that's clearly a non-trivial module and it'll be non-trivial to integrate it with core But I was talking to Vim the other day Vim Lears and he was quite interested in that So maybe that'll be Dave's claim to fame and one day all right, and We want to quickly cover the fact that while Blackfyre is an amazing Profiling tool. I think yesterday I was talking to Fabio and he said it's more it's much more than that or his vision for The tool is it's really about application performance management because he wants it to be something that you run on every commit You generate the profile you integrate that as part of your deployment and an ongoing process, so this is where Blackfyre premium really comes in it sets up a notion of environments with team members who can share profiles amongst themselves It'll have more generous data retention policies you you're you're encouraged to set up assertions and testing scenarios with different pages You've got CI integration. It's got slack notifications It's it's got custom metrics that you can define for those assertions and it even has a lot of built-in PHP recommendations saying you're not doing this Common optimization do it like such as install the twig see extension I don't know how many specific ones they have for Drupal for these Recommendations that would be nice to have more but the one that always runs in our profiles that we see is You have more than 10 SQL queries and that's not optimal Yeah It's appropriate for Drupal so it'll make the tool better and make all of our lives easier great And recently they announced an on-premise version if you work for a huge enterprise that will never think of Sharing these profiles with a different organization. That's that's an option as well They've got a black fire booth and they're they're here to both answer your questions that I cannot answer and and also to help you Install the tool and try it today. So please take advantage of them while they're here That same marketing advisor that I've got told me I need more cost to action Everywhere and on the websites and presentation. So here are the cost to action for this presentation first Please fill out the feedback form so that we get invited back to speak at Drupal con if you liked it If you didn't like this presentation pretend Yeah, you didn't see the slide There's there's code spits going on tomorrow. That's your guys are back-end PHP developers and we need your help Dave will be doing it and migrate stuff and there's a lot of stuff going on tomorrow So please please come to the convention center and participate Follow me and Dave on Twitter if you're trying to learn how to use that there's a black fire booth with with actual PHP superstars and We should we should also mention here's a link to the block access records module that you guys check out We wrote some blog posts about black fire here for both Drupal 7 and Drupal 8 We got another case study that we didn't even talk about and and we have a Talk that we did at mid-camp about upgrading to D8 for our site. Maybe that's interesting for some of you Suzanne my colleague has as a great training program that she runs So if you guys need Drupal training Or your team probably not for for people who are here But for your teammates who are both for content authoring theming intro to module development We've got that I'd like to invite you all warmly to Drupal North next month in Montreal Montreal is Beautiful in June. So as you see it's it's all purple and pink. So please come and it's gonna be a great regional summit and Actually just one more thing you shouldn't have left that guy who just left The folks are from block fire We're very excited that that we were giving this talk and they meant and they reached out to us And they said hey We're gonna be at your talk. We're gonna answer all the more advanced questions. So Fabian is here. Nicola was one of the key developers for black fires here and and they even Generously offered to give out this book 24 days of black fire that actually I picked up in Drupal con Asia And I got so excited about it that I Was inspired to give this talk. So basically everything I I Covered I've stole from this book. So you might as well might as well get your copy here So that's that's our presentation and and at this point we'll take questions and and thanks a lot and And I think Fabian will take most of the questions Anybody no in fact I Yeah, so the question is is there a conflict with with New Relic and black fire And running them both in the same server and not only did I get to know but earlier? I learned that there's an integration. So if you're if you if you have New Relic installed I think they'll they'll integrate with black fire as well. So that's that's a great combination Any other questions? Go ahead. I Don't believe XH prof has a heat. Okay. So does black fire have a heat map? I know XH prof does as far as I know black fire doesn't I'll have to I'll have to punt to them Why that's that wasn't a priority. I couldn't hear that. Sorry. Yeah, it's not really a question But it's a recommendation. So I happen to trace you they're similar to New Relic They have heat map functionality great. So if I you mentioned that black fire integrates with New Relic So if there's a future integration for trace you Okay, awesome. Good. Good. Thank you. All right. Yeah, I use that Netta. Okay It's fantastic because it also has trace debug and includes all the Drupal hooks as well. Oh, super. We'll try it Thanks a lot Anybody else we got a just a few more minutes All right. Well in that case, thanks so much. We'll be around and more Yeah, cheers and and more importantly go talk to the guys in the in the red shirts get your book and ask them the more difficult questions Thanks