 It's time, and I think that we can start. Hi, everyone. My name is Giannis. I come from TechOne Consulting. And this will be the talk about performance audits and the top problems that we see when we are doing them. If you want to follow along on your own device, this is the link to the slides. I will give you a few seconds to scan the QR code. But I also show the same slide in a few minutes for people that will come in later, potentially. So as I already said, my name is Giannis. I'm also known as slash-rsm on d.o. Senior Engineer at TechOne Consulting. I used to lead the DAID9 media initiative. I also used to work at examiner.com, which was the biggest Rupa website at the time. It was, I believe, the first production website on D7, went live even before D7 beta was released. I think that it went live on alpha. And we had a lot of content and a lot of traffic. So yeah, we were fighting with these kind of problems. And yeah, performance was always something that I felt really dearly about and was very interested in. And that's also how I got to TechOne. We are, we work with many technologies, but we are mostly known for our work in Drupal. We are second all-time contributor. We have largest concentration of core contributors of any organization. We also donate a lot to Drupal Association, among other things. We have a full-time infrastructure expert donating to them that helps them run Drupal.org and all the tooling that we have and all the infra-related stuff that they have there. And yeah, I would also like to say at this point, if you are profiting from Drupal, consider financially supporting it, either by funding the DA, sponsoring the conference, or funding a module maintainer. Only if we all invest in Drupal, it will continue to thrive. And like probably most of the companies in this space, we're almost always hiring. We work with clients from all different areas, from enterprise, government, nonprofits, education. So a lot of interesting work and a lot of interesting clients, which brings us to performance audits, which is one type of a project that we would do. Tag1 has been perceived as a performance expert in Drupal space for many years. And a lot of times when people will experience performance problems, they will come to us for help. That usually happens way too late and on a very tight schedule. One of the recent performance audits that I had was for a project that was planned to go live in a week. That obviously didn't happen, but they went live in two weeks. So we can do a lot of things, but not everything. And these projects usually start with a review of the site. We would check the code or just poke around initially. It's very useful if the client tells us which areas of the site are the most problematic, and then obviously we would go and look into those areas first. And we try to identify problems. And some problems are easier to fix, are lower hanging fruit, and some problems are just because of the wrong architecture or something like that, and that is obviously much harder to fix. Especially for the ones that are on the tight road map, we try to focus on the lowest hanging fruit and try to push it out of the door and then provide recommendations what to do afterwards. And depending on the client's needs, after reviewing the site, we would only provide recommendations, or we can also work with the client to provide fixes or to guide the client's team to do it themselves, whatever it's needed. Here's the QR code for the slides again, if anybody came in later so you can still get it. Yeah, and these areas that I will talk about are roughly organized based on how easy they are to fix. It's obviously debatable, and it can be different from side to side, but it's roughly in the order how we would also approach fixing them usually. And the first one is the incorrect use of cache metadata. What I mean by that, Drupal has this awesome render caching system where it will cache the markup that is generated, and when it's cached, it will be delivered super quickly. But our sites are complex and we have authenticated users, we have anonymous users, we, some parts of the site are displayed differently to different users at different times, depending on many different conditions. And this is where cache metadata in Drupal comes into play. And it basically, for those who are not familiar with it, it has three concepts. First is max age, which defines for how long something will be cached, and by default, it's indefinite unless you change it yourself usually. Then we have so-called cache contexts which define how something that is cached will vary. If it's the same for every user on the site, then you don't have many cache contexts, but it could change depending on permissions, depending on cookies, depending on IPs, ranges of IPs. There are many possibilities. And then there are cache tags. Cache tags define on which events something will be invalidated. So let's say you have a block that displays a node. That block should obviously invalidate and rebuild when that node is updated, because it probably depends on the content of that node. So these are cache tags. And usually when we are using incorrect cache metadata, we have a weird box, something like, oh, I have a block that should display something, but it's displaying something else for me in some weird situations, not always, and it potentially only happens on production, because they are where you have live traffic. So, and it's usually quite hard to debug. But when there is something going on that is weird and it's in some weird situations, displaying wrong content, there is a good indication that you have a problem with the cache metadata. And one really easy thing to verify that is to disable cache, render cache locally. And if the problem goes away, then it's probably cache. And then you need to go into that piece of content that is mid-behaving and review the cache metadata that is attached to it. So, how do we do it? First thing, we need to understand how caching works. Without that, it's really not that complicated, but I still see that a lot of people struggle with it. I'm not sure how we fix that. There are good resources outside. Maybe we should even improve them or provide additional ones, but that's the first step. And it basically comes to what I explained earlier. These three concepts that you need to understand and basically have everything you need. And then we have some debugging outputs that we can use. Initially, Drupal only had option to enable cacheability headers. And you would see when you enable that, you enable it in the services file and don't enable it on production. This is for development use. But when you do that, you will get headers that will show you what is the max age of the page, what are the cache tags that the page uses, and what are the contexts. This is great, but it's also limited because this is aggregated information for the entire page. But a lot of the times when you are debugging something that is related to this problem, you have to deal with just a part of the page, just a single block or a single format or something like that. And as of Drupal 9.5, there is something in Drupal that allows us to see that as well. If you enable that configuration option in the services file, you will get this HTML comments above every block that gets cached separately. And now you can review it, which cache metadata is on it. Even it shows you how long the rendering took. So you can already review the performance part of it. When it's uncached and if it's rendering for two seconds, maybe there's something to optimize. And you will also see if you have a cache hit, if you have a cache miss. So usually my approach would be disable cache, review the metadata, review the performance of this piece of the page. And then when I'm satisfied with that, we can enable caching again and see if we, if the whatever we're working on is actually getting cached or not. If you wanna learn more about render caching and how to deal with that, Jody from Renaissance and I are having a talk tomorrow in the same room and you're kindly invited to join us. Next one that I see really often is disabling cache to solve bugs. This is related to the problem we were talking before. But instead of going and debugging and figuring out what's going on and what kind of cache metadata will give you the behavior that you want and expect, the cache is just simply disabled. And this is done, and I'm not even sure if I should tell you that, is done by putting a max age zero on the render array that you're working with. Or if you're using views where you have the configuration for caching, you switch to non-cache plugin and it does the same basically. And the solution here is just never do it. Cause I've really like, I've never seen a valid reason to do it. Instead of, oh, I don't know what's going on and I don't have time to figure it out or I don't know how to figure it out. Let's just do it like this and it works and users are happy and go on with your life. But it ruins cacheability of your entire site. And then you will have, depending on how much traffic you have, you will have much bigger problems down the road. And if you already have a lot of traffic, you will notice it pretty quickly. The even bigger problem is if your site grows and this isn't the problem and then you potentially even develop a habit of solving your bugs like this and you accumulate more and more of this, so-called solutions, and then your site grows and you get more and more traffic and then it becomes a problem. And then it's even hard to fix because if you're doing it for years, then it can be everywhere and it's really hard to find everything because then it's not just about enabling the cache again, then it's about, because you've done it for some reason. And then at that point, when things go wrong, you have to first find each instance of this and then you have to figure out why it was used and fix it properly as you should done in the first place. So it can be really time consuming if this goes on for a long time. So please don't do it. You will thank yourself later. Next, sorry, yeah, I'm pretty sure all of us are. The next one that a lot of people are not aware of is the entity list cache tag. And this is a cache tag, so let's repeat, tags are metadata that defines when something is invalidated. And entity list cache tag is a tag that will be invalidated when any entity of this entity type is saved, either created and saved or updated. Like when anything in this entity type changes, this is being invalidated. And by default, each entity type gets it. Even if you create your own custom entity types, you will still have this cache tag being invalidated when you save them. It's in the entity API. And it's always called like machine name of the entity type underscore list. And then, it's a great idea, right? When you have something that it's affected by a lot of pieces of the content, it makes sense to have something like that. And then views come into the play. Views have to be really general. And views don't know when you are building a view that displays nodes, it doesn't know if you are displaying every node on the site or just some subset, like a single content type that might be just a small fraction of all the nodes. And by default, views will put this list cache tags to every view that you build. Which means that by default, if you just use it out of the box, every time when you save a node, all views that are displaying nodes will be invalidated across the site. And then, nobody ever built a site and used the nodes for everything, right? And then this happens. Because especially if you are updating these nodes frequently, if you are importing something that runs at Cron, that runs often, and it imports something as nodes, it will invalidate every view on your page. And when you have a lot of traffic, things go kaboom. There is a module that fixes that. And I'm pretty sure this module could be in core, I guess. But core or views out of the box is not smart enough to figure out what to put in there to be smarter. You basically have to tell it because you know what kind of content you have and how it behaves. And this module, called views custom cache tag, provides a new plugin for views caching. And it will remove the least tag, the one that views adds by default, and let you define other cache tags that you want to invalidate on. And very easy, yes? I'm not sure. Could be. We can check later. Thank you for pointing it out. Benji was talking about this. It's probably node underscore list colon. But I'm not sure. I'm really not sure. It could be like that too. We'll check. So Drupal will provide you this content type specific cache tags out of the box. But it can't use them by default because it doesn't know what you need, what you want. And with this module, you can tell it. So you will remove the list cache tag and then you can use this more specific ones. That's the easiest way to get the improvement. But then you can also do it custom. You can create custom cache tags and use those and optimize even more. One example that I had with a client is they had a block that displayed birthdays of the call. It was an internet for the company to communicate and whatnot. So they had a block where they displayed colleagues that have birthday this week or something like that. And now this block was being invalidated every time when a user was saved. And then on top of that, they also used a single sign-on system. And single sign-on systems often sync fields from the single sign-on into Drupal, which happens on every login, which means that on every login, you were saving your user entity, which means that this block was being invalidated every time when somebody logged in, which was all the time. And here we just added this little custom snippet that checked if the birthday date changed. And only then we triggered that custom cache tag. And now from invalidating it all the time, we went to, no, not very often. Now we are coming to things that are more related to views. And first thing is using a lot of joins in views, which happens very easily and very quickly because it's related to how Drupal builds its data module. When you add fields to entities, each field will be in a separate table. And that's great and flexible. But then when you're building views that are only displaying these fields, it's not even that big of a deal. But when you want to create conditions, filters on those fields, you have to put these conditions on the tables that are joined into. And queries with joins can't use indexes efficiently for most part and then the especially big problem are left joins because the data set, the database needs to work with explodes. Let's imagine if you have an inner join and if you have a lot of entities that have an empty field, then only the entities, the rows, that have the value for that field will be joined in and the data set the database needs to work with won't grow as much. While if you use left joins, all the entities will get this joint tables in and the value of the field where there is no value will be null. And then if you do this few times, it explodes in size and then database needs to work with that data set, which makes it slow. And there are few things that you can do here. You can use custom entity types if that fits well because when you're using custom entity types, you usually know which fields you will need and you can make them a base field of the entity type. And base fields will be in a single table unless they are multi-value then they get their own table. But if they're a single value, they will be in one table and you can index that much more efficiently. It's easier to work with. And I know that writing custom stuff in Drupal is sometimes not the most popular thing, but it's not a sin seriously. Sometimes you can achieve things in very simple ways. And you still get all the bells and whistles. You still get added forms and formatters and views integration, all those things are still there. So it's not that bad. It's also not forbidden to not use views. Like you can create a custom block for something and it could be way more efficient. It could be easier to make it perform well. It really depends from use case to use case. Because sometimes when you have really complex things, views become really hard to work with. And sometimes it just makes sense to do a custom block and print the data out of it. It's also not a sin. And where that is not possible, we often use the so-called in subquery trick. Which looks roughly like that. On the left side is how views would usually do things if you have a condition on a field that is in a separate table. It will join it and then it will put a where clause on a column in the table. What usually can work quite well is to separate it out into a subquery. And now in the right example, we have one query that is inside in and we get entity IDs out of it. And now on the main table, we are doing a work clause on the primary key, which is always indexed. And this can be way faster. Especially if you then make sure that the subquery is covered with an index as well, you can go for a complex query from taking seconds to execute it to taking milliseconds. It's a neat trick. It means that usually you have to either alter the view or what I prefer to do is to create a custom plugin for the condition. Because then the main reason why I prefer that is that it's nicely isolated and it's clearly visible in the views UI that you are using something custom. Because then when you open the view, you see the name of the filter can be something that is not standard. And you're like, okay, something special is going on here and then you can take the label and search the code base and you will find the plugin. That's the reason why I prefer that over altering views and also other things in Drupal for that matter. The next one that is kind of related but it's a separate problem is views overuse. As I said before, views are great, but especially if you have complex things, they can get quite unperformant very quickly. Sometimes you can see people using views as formatters and this immediately adds a join to it, which makes it work worse. For formatters, we have field formatters and we can use them in Layout Builder and whatnot. So we should really use that. And as mentioned in the first section, views don't have the most efficient cache metadata because it cannot have it because it needs to work for a general use case and it doesn't know the specifics of your special case. So unless you go in and optimize the cache metadata, which sometimes could be hard, you can be more efficient if you go with a custom block or a custom format or something like that. Another thing that happens very often and it's a really cool trick, it's one of the lowest hanging fruit, is the count query. Views will do a count query over the query that you built by building the view if it needs it. And it needs it when you are displaying the total number of items that are in the view and it needs it if you are using the full pager because it needs to know how many pages there are. And if you build a query that takes two seconds, this immediately means that it will run twice, which means that you are now at four seconds. So if you're just able to get rid of the count query by using mini pager, for example, or not displaying the total number of items if you can afford to do that, you've just cut the render time of the view almost in half because it only affects the database part, not the rendering part, but it's a huge win if you can do it. So a lot of the times we would do that first and then go and optimize the view further just to get this initial gain to keep things moving. Yeah, as I said, don't be afraid of custom things. Sometimes it's more efficient and, surprisingly, sometimes it's even simpler to do. But most important thing here is that you have to measure how long something needs to render. In views configuration, there is an option to display the query when you are building the view. Do that, use that. It will also display some performance information, like how long it took the query to execute, how long it took for the view to render after that. And if you work on realistic data sets, or at least something that is close to realistic data sets, you will catch these problems sooner. And a lot of the times approaches, oh, we'll do this before we go live. No, you won't have time at that point. You have to do it as you go. And the best approach is to build it into the team culture, to make it part of the peer review process and have it as one of the check boxes on the checklist. Have we measured how long it takes with 100,000 nodes in the database? How it performs under realistic conditions? Then another views-related one is overcomplex views queries. And again, it's similar to that one. Here, especially when we are used to sidebuilding, a lot of the times the goal is to make the view configuration simple in the UI, which doesn't always translate into it, performs well. So, same recommendations. Measure it, have the query displayed as you build the view. Sometimes you change something that seems insignificant and it completely turns the query around. And if you don't have it displayed, you won't notice. And then when you have the query, use the explain function in the database to let the database tell you how it approaches the query. And learn to understand this information, because it tells you a lot. Even if you don't have the realistic dataset yet, based on the output of the explain debug command from the database, you can already figure out if it will perform well or not. But remember, looks good in the UI doesn't necessarily mean it will perform well when in use. Then, the next one that is very common, but it's especially after the fact, really hard to solve, is overuse of Contrib. Drupal is great, like there is a module for that. We all love it for this. Me too, don't get me wrong. But when you have a website that has like 300 or 400 or 500 Contrib modules installed on it, that just adds complexity. Even if there is nothing inherently wrong with all those modules, just all that code that needs to be loaded in Iran and all the configuration and everything, it adds up. It makes things heavy. So think about it. Do you really need a module or you can do the same thing, achieve the same result with a really simple code snippet that is custom and does exactly what you want it to do instead of installing this huge module? And also, keep in mind that Contrib is not perfect. Even Core is not, right? But Contrib is way less scrutinized than Core. So, you know, I even say when you're deciding to use a Contrib. module, do a review of it, treat it as it was a custom module. First time, especially if the module is big, it will take some time, but you will get better understanding of what is going on in it. But then when you will be dealing with updates, changes will be probably minimal for the most part. So subsequent reviews won't be that time consuming. And then, even in Contrib, like sometimes there are things that you would not expect them to be. Like in context module, which is one of the fairly popular modules, there is a bug that disables auto place holdering, which basically in turn disables benefits that Big Pipe provides for you. And the issue is six years old, there is a patch that fixes the problem, but it's still uncommitted. So if you want the context to really work well with Big Pipe, you have to patch it. And there are probably numerous examples like that in the whole Contrib space. So don't just blindly trust them, all those modules. Review them and trust, don't trust verify. And then this one is usually the hardest to solve. It's the data model issues. And we've touched on this before. A lot of the times, if you're using nodes to build everything, it's not the best approach. And then on top of that, if you're using two complex field types, like paragraphs, it can also add a lot of complexity and slow things down. Paragraphs are great if you're trying to use them to use components approach to content building. That's fine. But I've seen an example where Drupal's site that we are auditing was a data store. And it was only exposing things over API. And paragraphs were used in that data model. And when I asked the client why they did that, because paragraphs gave us this great filter that we could use in views when we were building the views to feed the API, which I totally understand. But no, it made things slower because paragraph is in another table. It's another entity that you need to load, so it just contributed a lot to make things slower. And you need to think about things like this when you're planning the architecture. As I said, if you're building some content where you want components, paragraphs could be a really good answer to that. But if you are building like a data store for AI researchers to feed their algorithms with, probably isn't the best solution. And again, don't be afraid to create custom entity types for a use case like that. When we were exposing things over API, a custom entity type for that would be an awesome fit. But the problem is when you've done it, when you already invested into it, that basically means rebuilding the entire site, which is not always possible because it's a huge time and financial commitment. So this one sometimes can be fixed after the fact, but most of the time we have to think about it when we are in the planning phase. And also don't blindly trust the data model that Drupal builds for you. Maybe do a prototype and check the database how the things work in there and then think about what it will mean when you will start pulling data out of the database how this will affect you. So maybe you have to tweak something. It's not all doom and gloom. I am coming from this perspective when we're dealing with enterprise sites. We have to be pragmatic. If you're building a brochure site for a local sports club or something like that that will have limited amount of traffic, you can probably get away with not doing anything of that and doing everything wrong, quote, unquote. Because you will never have these huge amounts of traffic and huge amounts of data. But you can still think about it and make your users happy because the site will be more responsive. But if you are building an enterprise site you absolutely have to think about those things. Because if you don't, fixing these problems later will be more complicated and more expensive. And especially tricky is if you have something that is not enterprise on an enterprise level yet but it could become later. And if you already accumulated all these technical depths then, yeah, it will be really hard to fix. So be pragmatic. And, yeah, to conclude, don't guess or trust but measure how long things take. When you are doing that, when you are debugging performance only change one thing at a time because if you change everything and then measure again and you see an improvement, it's fine but you don't know which change caused the improvement. So do it step by step, one by one. As we said already measure on realistic data sets build it into the team's culture make it part of the peer review process have somebody in the team that or ideally everybody thinks about the performance aspect of it and ask questions in peer reviews. Don't be afraid of custom code sometimes it's way simpler than you might think and we're in such an industry we should never stop learning. There are always new things coming out and we have to master our tools and this is also one of the areas where this completely applies. If you don't know how the caching metadata works and render cache works then it's really hard to debug problems that come from that. And be pragmatic if you have a small site don't bother. But if you have a huge site or an enterprise site think about this for sure. And speaking about tools this is a promotion for another project that we are working on at Tag1. We are developing a load testing framework but it's called Goose and it's written in Rust and besides Rupal we also love Rust because it's really nice to work with and performs really well. So if you... maybe this is one of the things that you can do you can create a simple load test and run it as part of your build pipeline and to measure if things improved or performing worse. And there is also a library of helper functions that we also developed which is called GooseX and as part of GooseX we already have an example for load testing Umami. So if you want to see how to use Goose with Rupal that Umami example in GooseX would be a really good place to start. Another thing that we are currently working on and we are very excited about is the automated performance testing for Rupal Core. We teamed up with Google and we invested quite a lot of money to do that. We started by improving Rupal's Core Web Vital metrics which was mostly done by enabling lazy loading of images and things like that and then when we completed that project we decided to start working on the automated testing and the goal is to have support in Rupal's test stack for performance tests and to have it run on Rupal's Association's infrastructure for all Core commits. And when we have it ready it could be even used for Contrib or for your own websites. For Contrib the limitation is the infrastructure cost so if we will be able to find funding then in theory even Contributed Modules could run their own performance tests on every commit or at least before a release or something like that. If you want to learn more about that initiative we have a talk just after this break where we will be talking more about Core Web Vitals and what we did in that area and more about the automated performance testing. And that's all what I had and now I will be happy to take some questions. Yes? Yeah, so I have to repeat the question for the recording. The question was how would the result of the automated performance test display in the issue queue? And yes, I think you would see the metric and probably some visual indication if it performed way worse than the baseline. Something like that. And yeah, it would totally like when it's finished it will totally run on DA infrastructure so it's for the community. Sorry? Yeah, I guess it is. We can talk later. So the question was whether there's a way to fundraise that. Yes, sure, we can talk later or if you're interested in funding you can come to the talk later and we can talk there or you can get in touch with your association like somebody will point you in the right direction. So the question was since we don't want to stop using views what's the process to make sure that views perform from the beginning not waiting till the last week before the launch? And the answer to that is when you're building a view make, display the query understand the query and ideally use develop generate to generate as many entities as you plan on having on the site and if you do that there is a good chance that if a view will be really bad you will notice it when you will be doing the preview especially because you also have the performance stats there in the preview so if you have a realistic data set which you can generate with develop generate then you will already have the impression even if you go with a smaller data set you will feel like half of what you're planning to have or something like that and things start to grow like a view takes half a second a second that's already too slow so realistic data set and have the data and the query displayed that's basically what you need to do sorry the question is if you only care about this when you are dealing with authenticated users and you don't care about this when you have anonymous users because you have page cache or varnish and to some extent yes the problem is less problematic but at the end of the day you will have to you will have to build those pages at some point for varnish to cache it and if it's really bad and if you have a lot of traffic you will notice it of course the problem won't be as big but you will notice it and especially if you are not using correct cache metadata and you are clearing usually when a cache tag clears you would also clear a tag on varnish or Nakamai or whatever you're using and then if you are invalidating all the time then you will have to rebuild it every time and if it takes a long time to rebuild it you will notice it so it's less of a problem but it's still a problem thank you very much have a nice break