 We'll go ahead and get started. Once again, you are in the room for planning for performance, and we're going to talk about that today. So I'm glad you came here. If that's not what you're here for, you might want to go somewhere else. Or stay anyway. Or stay. Please. All right. So planning for performance, if you're interested, we also have a slide at the end where we note our usernames and stuff like that on Drupal.org. But just to kick off, I'm Jeff, Jeff Beeman. I work at Aquia in professional services. I'm a technical team lead there, and I focus on big projects and leading architecture on those projects. I've been using Drupal for a long time, since right before 4.7 came out. And then I'll let Eric introduce himself. So I'm Eric Webb. I've been in Aquia about four and a half years. These days, I focus exclusively on performance, scalability, infrastructure, that whole side of the equation. My whole background is actually more on the system side than necessarily the Drupal side. So I kind of come at it from a different perspective from the system's people and really trying to bridge that gap. So our presentation today builds off a presentation we both did last year called Making Sense of the Numbers. We're not doing the same presentation. But if you're interested in that material, you can go back to the site from last year and check it out. That session discussed strategies for building effective performance tests and making sense of the resulting data. And it's philosophically similar to what we're talking about today in terms of a sort of almost religious reliance on data and goals over intuition. So there are quite a few things we aren't going to talk about today. But some of these things are really important to performance. But we're not going to discuss things like CDNs or cache purging or profiling. However, a lot of these topics are really important. And if you were around yesterday and you're interested in performance, you may have catched Ezra Gildes game. He works at Acquia. He presented on XH Prof, which is a profiling tool. And if you don't know what profiling is or you want to learn more about it, you should check out his presentation, because that's a great place to start and it's really valuable information. All right. So lessons we will cover here. You want to? Sure. I think the biggest thing, this actually is a good lead-in from the XH Prof session yesterday, is even though you have the data, you don't want to just jump into data. So a lot of what we're talking about here is really set yourself up for success. Don't just look at what your preconceived notion of how things should work or how things should be improved, because leading into that intuition oftentimes leads you astray, because you're taking maybe a personal bias or personal experience. And our goal here is really to kind of break through that and look at things a little bit more systematically. Another piece, and this is sort of the contrary to talking about XH Prof or all the other great tools that are out there, is the tools only take care of what you actually know about. So if you think about request profiling, if you're not request profiling command line operations, or some part of the application, or you're not doing it in dev, maybe you're not catching things. So the thing about tools is they're all reactive. You can't, tools are not going to reach out and grab more data than you ask them to grab for you. So that's why this, I think a lot of sessions and a lot of information really tries to take a tools first approach, but it often leads gaps. And part of what we'll talk about today is to really cover that full spectrum, there's a lot of tools that are really needed to do it right. So just kind of will illustrate what all those tools are as kind of a segue for you guys to look at what's out there and get some more information. But the key is the tools are a discussion point, not the entire discussion, which I think gets lost very often. Another I think interesting piece is from a tuning perspective, tuning implies that there's something wrong that you're trying to improve. Just by definition, that's what it means. So if you look at your entire performance strategy as tuning, you're basically admitting that you're gonna create some problems and you have to build in some time to undo those problems that were created. So it's sort of admitting defeat in some way when you're just immediately saying tuning is our answer. So I think this is probably the least important slide of the entire session for everyone, but this is how I think every performance talk starts out. So obviously, these are very statistically accurate reasons for Drupal to have bad performance, and 3 AM is a very good reason for that. Interns, the first Drupal project. I think everyone's first Drupal project, let's be honest, was probably horrible. Myself, I definitely included. And there's always someone's first Drupal project. So when you're just relying on tuning or relying on that reactive basis, again, you're gonna create the problems and just assume you're gonna have to take the time later to catch up. So, and the slide is kind of a joke, but it's also serious in that reading other people's reports of their top performance problems or sort of collecting this idea in your head of what the most common performance problems are with Drupal isn't a good strategy. It leads you to rely on intuition. Like, assuming that views is slow or panels is slow, those are assumptions you make without data and a strategy behind them. So yeah, it's a joke, but it's also kind of serious. So one thing that I think is really important to note here, and this is coming from kind of a personal background of both of ours, is when we're talking about performance and we're talking about these kind of lessons, we're really looking at it from a services perspective. So meaning, work for an agency or a freelancer or something where you're building a site and then at some point handing it off. I think a lot of, especially if you're really into the DevOps side or the infrastructure side, a lot of the big difference of those talks is they're building systems that will host many different applications over many years. And tooling and creating all of that structure is very valuable. If you're trying to churn through projects, build, hand off constantly, you gotta think that setup time to do all of that background information is really gonna add up really quickly. So our perspective is a little bit different than those types of talks. So I think at Drupalcon, a lot of people come from that agency sort of services type perspective. So just before we get too far, it's just important to say that's what we're thinking of. So when you're planning, oftentimes you don't get a lot of chances later on. Once you've delivered that, you're putting your name on something that you're giving to someone else and saying, this is ready to go. So it's really important to kind of look at it from that perspective and realize that not every lesson learned is the same for every way that you're actually managing or delivering a project. So one thing, going into more of the tool discussion, this is kind of the first segue into why the tools really sort of catch you off guard is when you're going into that delivery, you have deadlines, if you're maintaining a product, there's deadlines, but you have a little bit more room over time, when you launch the site at a point in time, you don't get a second chance necessarily. That's the worst time to have a false start and have to find your performance problems after. So that's why when you're in that tuning perspective, again, you're assuming right before launch, you're gonna magically fix all these different problems that you ran into. The data that you are gathering, which is very valuable, not all data is or should be actionable. So we'll touch on this a little bit later that caching isn't always actually the right answer, and we'll give some examples around that. But it's important to have the data to inform your decisions, but it shouldn't always dictate that a decision has to come. And one of the biggest things that I think we run into a lot on really large projects is tuning is really about making small optimizations in a lot of different places. If the architecture itself has structural issues, that's not tuning, that's rebuilding parts of the site, and that gets really expensive. So maybe even more than anything else we're talking about here, that's kind of the lesson, right? Bad architecture is very, very expensive to fix. If at the end of the day you just have some poor performing code, you're forgetting to cache some things, that's just natural and that happens. But if you're not thinking about performance from the start, things will sneak up on you, and that architecture is not something you can magically recover from. So with that sort of background, one of the things that we want to make really clear is this session is not going to be about anything like sexy, we're not talking about any like sexy technologies or anything like that. This is going to be really sort of straight laced, buttoned up planning that you can do from the very beginning. It's not about tools, it's not about really even Drupal in a lot of cases. So let's talk about a few use cases. Do you want to run through these? Sure. So just like whether you're from a services background or more of a product or internal background, you look at projects differently. The type of site you're building also from day one changes how you should approach performance. So if you have just a really static site that's for sure way are fairly simple, most of the performance benefits you're ever going to get are page caching and very simple things that solve a lot of problems very quickly. As you get a little more personalized, something like groups at Drupal.org, you see a lot of customization on a per group basis. Something that if there's 400 groups, a lot of times there's 400 copies of content listings and different things that are really something you can predict how often they change or how many there could be. And then all the way fully into what's fully dynamic. So something that's personalized to you to a specific point in time and there's no real way to plan and sort of mask any kind of performance problems. So as you're in the planning phase, fundamentally understanding what type of site you're building really changes how often you want to look at performance early on in the project versus saying tuning might be okay in very certain circumstances. So I want to look at a use case. So delta.com is where I spend most of my time on the website or on the plane. So this just happens to always be up. So I run this a lot. One thing that's important is from a marketer's perspective or from the non-technical side of your organization. Their vested interest is to get people into the site, know what they're doing, use the site, and get the most value out of it. From a performance standpoint, that's also extremely beneficial. So the example we have here is we're looking at, this is an anonymous screen essentially. So the two most important things are either log in to see what you're looking for, or I'm brand new, I need a book of flight, right? It's a very targeted experience. There's not a lot of extra content on the page. You have to optimize for no reason. It's exactly what someone wants to do. As someone logs in, it now changes to what are the two most important things to that person. Now that we know who they are, it's about just a quick status of who they are and also what is their current travel. So if I'm logged in, probably more often than not, I'm either checking in for a flight or looking at upcoming flights. I do that a lot more often than I book a new flight, right? So you really need to think about, if I showed this and booking a flight and all this other information, not only is kind of the UX diminished, but the performance gets more and more complicated as you're adding more and more of these personalized elements. So this is, I think, the common ground piece between performance and marketing, that if you wait until the end to sort of have these conversations, I think both people are actually at a disadvantage. So it's important to have these conversations up front, and UX can actually bind a lot of the non-technical and technical folks into something that's really best for everyone. All right, so let's talk about defining goals, because in the use case of Delta, for example, if you are a performance engineer on that project, you need to be able to work with the marketing department and work with the UX team to define goals for performance and define your strategy for approaching the implementations. So context is critical when you're defining a goal for your site's performance. Not all pages on a site have the same performance expectations. Your expectations for performance might change based on user type. I think most commonly, people are familiar with unauthenticated versus authenticated traffic, right? Anonymous users have a different performance profile and different performance goals than we have for authenticated users. And our page types might define what types of performance we expect. Like your homepage, you might want to be lightning fast, whereas it might be okay if a sub-page that lists all of your upcoming flights on delta.com. It might be okay if it's a little bit slower. Or you might have API calls that have some performance expectation. So let's talk about some examples. We're gonna show good and bad examples of performance goals. So first, a bad one. The homepage should load in three seconds or less. I mean, that's bad because it's pretty slow. But it's also bad because there's not context and there isn't sort of measurable goals with it, right? So the homepage should be delivered to anonymous users within 500 milliseconds and fully render within three seconds is a much better goal because we can measure, we can tie it to a role, like a user persona on the site. And we're also defining our expectations for data delivery versus page render time. And we aren't gonna get into that whole sort of topic here today, but understanding the difference between getting data to the user and then getting that data rendered, those are two different things. Delivering HTML and rendering HTML plus CSS plus JavaScript plus images, those are different concepts. Another bad example is any single page should never take longer than two seconds to fully load, right? And this is something we do, it's tempting to do for yourself. Your clients will ask you to do a lot. It's unacceptable if the site ever takes longer than two seconds to load and fully render, right? And in these cases, you never wanna say never. Never is a really bad word when you're defining goals because you aren't accounting for all of the various things that might cause a page to load slowly sometimes. So time to render can be dependent on the client network, browser performance, and you should always assume that there will be performance outliers. Things will happen that make page delivery slow sometimes. So this table here is an example of a good way to build performance goals. If performance goals are about quantifying the performance of a site, the goals should be well defined as well. So, and the spreadsheet, these slides will be online after the talk. So don't feel like you have to take notes on this entire table structure here. But when you define goals like this, don't just use them for launching your website. Use them actively during development. Like when you get to the end of a sprint, you should be checking performance. In the middle of a sprint, you should be checking performance. So this is an example of a spreadsheet. Do you wanna talk about maximums? Yes. So just to add one thing to what you just said also. I think the trap here is also comes in with the whole tuning concept. If you're looking at this just for launch, again, you're assuming you're gonna play catch up at the very end. But if you're really gonna be agile about development and make sure that as you build, you know when something's really done, you really need something like this up front, not just a checkpoint before launch. Maximum is an important concept though because just like on the last slide, as there are unknowns, never is almost impossible to actually enforce or impossible to really realistically get. So maximum usually does not mean never above this. Maximum usually means 99% of requests, 99.5% of requests. Some sort of number that is kind of an effective maximum, maybe is a better way to think about it. If there's one in 100 or one in 200 requests that just happens to be bad, then that just kind of comes out in the wash over the course of however many hundreds of thousands or millions of visitors you get. So maximum is not a statistical maximum, it's an effective maximum. All right, so caching isn't just about turning on caching. Caching isn't just about going to Drupal's performance configuration page, checking a couple of boxes and hitting safe. A good cash strategy isn't a binary decision. It's about what we're gonna talk about over the next few slides here is four different criteria that define cash effectiveness and can help you make decisions about how to cash things on your site. So we're even gonna talk about instances where caching doesn't make any sense. Or it'd be a bad idea to try and cash something. So there's actually, I think this is really the crux of this whole discussion. Is understanding when caching is appropriate, understanding from a functional perspective how that impacts performance. And this is the planning part of planning for performance. So as we go through and discuss each of these topics, these are things you wanna look at as you are planning to build something, not after you've built it. So cash effectiveness can be, sorry, the way presentation notes work here is like crazy. We're using Google Docs and I keep losing my place. So cash effectiveness is about cardinality, granularity, cash expiration, and rebuild cost. And now we'll go on to the next section. I'm not gonna define any of that. No. So these are all weird terms for the most part. And we'll give you some examples of each one here. It's also important to mention these are our terms. So they could be completely wrong, but it's how we talk about it. So it's just the terms we've decided on and some of it's related to if you're big database guys, you can sort of understand some of these terms in the same context. So we're sort of reusing terms from other places in the stack effectively. So let's start with granularity. Granularity is the functional difference for each cash item. And we got three examples up here. The first one on the left is a user profile block. User profile blocks are going to be different for every user, right? In the middle we have articles, like a block that you might put on news articles to say, here's some information about this article. It's author, what section it's in, it's rating. That block is going to be different for every article on your site. Popular content in a group. That's a block that's gonna change based on the group you're in, right? You wanna talk about cardinality? Sure, so this is definitely one of those borrowed terms. This is sort of the quantifying the cost of caching, essentially. So if you're caching something per group and your site has more groups than users, for instance, that's more expensive than caching per user. Because you gotta know how many groups there are versus how many users there are. If it's global, the cardinality is one, right? If it's per user, that cardinality will actually change over time, and you actually need to account for that. If it's different by the day, if it's different by the location someone's visiting from, maybe that's per country of their site visitors, versus the little micro geolocation of someone, which is essentially unlimited. And so the example here is a block with tools for users, like a sort of quick links menu or something like that. And the one on the left is for authenticated users. The one on the right is for administrators, and they have different sets of links based on their role. So in this case, the cardinality would be the number of variations equals the number of roles on your site. So the putting cardinality and granulary together is basically the equation for figuring out what that total cost is to rebuild any individual cash item. So cash expiration is the point at what triggers the cash being expired. So in this example, the popular content block on the left. If we build it, if we build that block and we cash it, at some point, we want to rebuild it. And we have to define when we rebuild it. So in the popular content block, we want to rebuild that block every 15 minutes. But in the block that's about latest content, we want it to be updated every time a new post is added. So that expiration isn't just about time, and it's not just about action, it can be either one. So, and I think this is a big piece that, I think this is probably the piece people think about most already. But when we talk now about the time to rebuild a cash item, that's where these kind of first three concepts really come together. Is if there's a lot of variations to it, or the amount of effort to actually expire something, or the frequency that it's expired, that all constantly raises the cost of what that cash item is, and how much it's really going to take to solve the problem. So the cost can be sort of quantified as time to rebuild, like time to, the amount of time it takes to rebuild a cash item. But you can also think about it as resources. So it may not take a long time to run the algorithm to figure out what popular content is, but it might be a very performance, like it might be a very CPU intensive algorithm, or it might require you to run a lot of database queries, that kind of thing. That would mean that the popular content blocks rebuild cost is high. It's an expensive block to rebuild. Whereas the latest content block is just show me the latest articles on the site. It's a very simple thing to ask Drupal to do. So the rebuild cost for that block is low. And within to maybe give a few Drupal specific examples here, if you think of expiring one block or one view, then it expires and you rebuild it. It's very, the ramifications of clearing that cash is very simple. On the other hand, if you're clearing the menu cash, for instance, to add one link, that one link is cheap. But because of how the cash is constructed and understanding how that fits in, the cost to rebuild it is much more than the change that you kicked off. So what you're doing doesn't actually necessarily tell you what that cash rebuild is. It's actually more about the structure of what is coming out of that. So this is about creating a cash strategy. This isn't about just technologies and tools. It's about looking at your site in terms of strategic decisions about what is acceptable and unacceptable for frequency that something gets rebuilt, or how much it costs to rebuild it, and things like that. So you can't just build your site, like get all the way to the end of building your site and say, oh, yeah, we should probably cash some of this stuff. It might help. And a lot of that is because not every architecture supports the same performance plan. Earlier, Eric showed the slides that were DrupalCon Austin, groups.drupal.org, and Twitter as three examples of sites. Those three sites cannot have the same performance plan. If you're building a site that is entirely anonymous traffic and it's sort of like a brochure site, you're not going to rebuild caches very often. And the cost of rebuilding the caches may not be very expensive. Or you only have one variation because everybody visiting the site is anonymous. And none of this is mutually exclusive to using caching. I think a lot of it, and myself, I'm even, I come victim to this too, is I maintain a module called Drush Cash Audit that lists out panels, views, blocks, all of that to tell you what is and isn't cached. And I think I mentioned the phrase earlier, grab the data and then choose to optimize. And I think that's a point even here is it's easy to fall into a giant listing that says, these 15 things aren't cached. That may not actually be wrong, but it's easy in the grand scheme when 80 plus percent of them should be cached that it's easier to say, well, I'll cache everything. And it actually has some kind of long-ranging impacts potentially. Right. Yeah, so it's not just a technical choice. So you don't download Drush Cash Audit, run it, and say, okay, everything in this list needs to be taken off the list because it's not cached. Right. And like the Delta example showed, these aren't decisions that are just technical decisions. These aren't decisions that just developers should be making. Stakeholders should be making these decisions. Stakeholders will tell you how important it is to have certain information be refreshed on the site. And that's not a decision you should be making as a technical person. You should be informing that decision by providing guidance about how expensive something might be to rebuild, for example. But the people who drove the decision to put that thing on the site in the first place need to be a part of that discussion. And talking to stakeholders 10 months before launch is probably a cordial conversation you can work through. Talking to them 10 hours before launch, figuring this out, probably won't quite go as well. So this is our obligatory, blank is dead, long live blank, all the other lovely catchphrase you see in every single presentation. None of this is about Drupal. Whether you're building with Drupal or some kind of homemade application or WordPress or anything else, the concept of caching strategy and how that affects users is the same. So I think this is one place where if you're new to Drupal or you're new to web applications in general or you're non-technical, this is all the same conversation, no matter what your experience level is. The conversation I had about this kind of stuff three years ago shouldn't be different than it is now. How people use a site has not fundamentally changed. The type of site may have, but it's not about if we use views here, we can do this. If we use panels here, we can do this. That's not the functional requirement. That's not the initial need of the site. So let's walk through an example. What we're going to do is show you a community site front page. If you're familiar with Drupal Commons, these wireframes are sort of inspired by Commons. But we didn't build a Drupal Commons site to show this stuff. Because we wanted to illustrate the fact that these are decisions you make before you even get into building your site. So this is our community site. This is the unauthenticated home page. And sorry, it's hard to see my notes. So the blocks on this page are a featured articles block. And this block gets populated with articles that have a flag set to say this is a featured article. The next block below that is recent content. Then we have a login block and a banner ad. And we have recently created groups and popular content. So this is probably a pretty common thing you would see on a community site. And while performance is important for anonymous users, it's hard to talk about all the crunchy, nasty stuff that comes along with performance if we're just talking about anonymous traffic. Because an answer might be just cash the whole page for anonymous users. And so we're going to talk more about authenticated traffic because it gets into the meat of what we're discussing here. So this is the authenticated version of our page, which has just a couple differences. We replaced the login block and the banner ad with a My Profile and My Groups instead of Recent Groups. And then we changed recent content to recent content in My Groups. So you can see already we're differentiating the homepage a little bit. And if you're thinking about the terms we talked about before, cardinality, granularity, we're already affecting our architecture a little bit here, right? So we'll go through each block and sort of talk about each block in terms of the terminology that we laid out before. So this is the featured articles block. It is a global block. So this block is the same no matter where you are on the site, no matter who you are on the site. We have one version of this block. So we say it has a cardinality of one because it never changes. It's never different. The granularity is global and it expires. So we need to rebuild this block whenever new content is created that's flagged as going to the homepage or the front page. And based on a small assumption, we're just going to assume this block is fairly low cost to build because we know that all we have to do is ask Drupal for the most recently created featured blocks or featured content. So I think this is a good case of first getting into now that we're looking into granularity and cardinality. We can start to understand when caching is good and when caching may not actually be useful. So for example, for this block, there's an effectively unlimited number of versions of this because it's going to change on a per user basis. It's recent content in their individual groups. So if your usage pattern for people is that they typically only come to the front page occasionally, but they go to other pages very frequently. You're going to build this cache it and by the time you're going to show that version again, it's already outdated. So what you're effectively doing is taking the performance of whatever you're building, adding to it the cost of actually saving it to the cache and then deleting it before you ever show it again. So this is a very important example where caching just isn't always the right answer. It's an easy answer, but you'll actually save for very large pieces of content. The time to cache that is non-trivial and the fact that you're never going to retrieve it again is actually a performance issue, not in a performance enhancement. So the user profile block is a block that again has unlimited or effectively unlimited potential revisions, right? Potential versions. Its granularity is per user. So every user's profile block is different. And it should refresh based on the data you see there. I don't know if you can read it very well, but we've got things like the number of posts this person has created, how many points they have and whether they have any notifications. So this block needs to be refreshed whenever a user does something that would cause that data to change. So if they create a post, if they do something that gets them points on the site, or if they get a notification, and we don't want to sidetrack too much, but like notifications, we might say those might happen really frequently. So let's not even make them a part of the block cache. Maybe we rely on an external service for that. So the rebuild cost becomes not low and it's not high, but there is value in caching it because we're only looking at the homepage, but we might have a bunch of places on the site where we show user profile blocks. And it might be valuable for us to cache this thing. So we're just going to go through a few more examples at this point. I think it's hopefully fairly clear where the thought process is and where we're trying to evaluate this information upfront from a functional perspective. So some more examples that are per user, there's a very low cost to rebuild it, but in this case, it almost never expires because it's not frequent that someone leaves or joins a group. So another case of it's unlimited, but here it's a good thing, whereas in other unlimited cases, it's not good. So that's why all four of these are important factors that will determine what your cache strategy should be. So just a few more examples, very high costs that are global with one variant. If the time to rebuild that is high, it makes even more sense to cache. It's a very obvious choice, and this is sort of the prototypical Drupal-type example. So the problem is that caching isn't always a super simple discussion because it gets a little worse than the topics that we've talked about. So on the featured articles block, what puts content in that block? We talked about it being a field on a node that says that this is a front page article, right? But you might have a complex workflow behind that. You might have like an approval process that has to happen. So it may not be as simple as just saying if a piece of content has this flag, put it in the block. So again, like your use case might alter this particular instance. In the user profile block, again, we talked about notifications. Maybe we'd yank that out so that we can more effectively cache the profile block. And then maybe reputation points come from all over the place and reputation gets updated all the time, and maybe it makes sense to yank that out as well. And then the last one that gets kind of hairy is like, what do you do if you have permissions affecting certain things? So the popular content block might be a really good example where you might have a really popular piece of content in a private group that a user doesn't have access to. So what do you do? And this is why you have to have these discussions early because a stakeholder might say early on, that's OK. Like if it's private content, we don't put it in the popular content block. And then that's like a simple discussion. But if they say, no, it's absolutely vital that popular content from private groups show up for all users, then your cache strategy for this block is going to change. So I think, as we're shown here using something like Balsamic, put this stuff into your wireframes. That's where you're actually doing the work. That's where you're having the discussions. This doesn't belong in architecture document and something that the stakeholders will never read. This needs to go in call-outs in those upfront wireframes, mock-ups, whatever that first process looks like. So this is always my favorite part of presentations is there's always kind of the snarky section. I figured I would take that myself. Again, this is from a services perspective. So there's opinion here. The last section of what your cache strategy is, that's just important for any project. But how you put that into place is totally up to you. This is, I think, a really important thing to note is when you're creating new tech, it's not solving brand new problems. It's just solving problems in your current state. So don't think about everything in terms of Drupal because Drupal hasn't solved these problems because nothing has. They're the same problems there always will be. It's just some of Drupal chooses to solve it in one way or another. The problems are always still there. So from a tooling perspective, I mentioned earlier, if you really want to rely on the tools to get the data you need, you need a lot of tools to fill in all those gaps. So from top to bottom, from the very front end of how people are using the site, all the way back to those raw server logs to make sure you're not missing anything. So the setup time, the cost of doing this gets very expensive if you're just going to rely on that kind of data. Assumptions is a really key one. Just stay away from them. It's pretty simple. There's four nevers on one slide in all caps. Hopefully that's the obvious part here. A lot of these are driven by our experiences with supporting clients of ours at Acquia. So these are sort of like, don't do what we did. Kinds of advice, right? Or don't do it twice. Yeah. So do you want to talk about it? I know we're running a little short. So I think not relying on assuming things are going to work the way you want them to, don't assume all environments are the same. Verify they're all the same. Don't assume that your caching layer is always ready. When you're developing a site, you're constantly working through it, but you have to account for, let's say a varnish server gets restarted at some point, or MIM cache gets restarted at some point. If you haven't accounted for that or know that that could happen, then your caching strategy could actually blow up, not help you in those circumstances. You never know what's kind of hiding in modules, or what's hiding in Drupal Core for your use case. If you're really working on large, high-performance sites, most of Drupal does not really account for that in necessarily very intelligent ways. So a few examples here. If it's a performance problem, it could actually be Drupal. It may not be you. It could be Drupal Core. It could be views. It could be some big module out there that you kind of assume, well, everyone else uses it, so it must not run into these problems. So just a few examples here of node access query building. Ezrax actually mentioned this in his presentation yesterday. If you have lots and lots of organic groups, the time to build that query actually dwarfs the time to run the query, but only in use case we have lots and lots of groups. Also, when you're using node access, you can't use block caching. Block caching is a very efficient cache because it's just raw HTML, but let's say you're using workbench moderation. So you're using node access for a handful of content authors, but you can't use block caching for the other 99% of your site's visitors. So that's another case where node access is just a specific example here, but more generally it's your use case may not fit the use cases that your module or Drupal necessarily supports out of the box. Node access grants, they're the same throughout a request, and on a recent project we realize they're actually recalculated every time it's run for every single node access query. Most sites don't use node grants, so they never run into it. We were, and it was really killing the site. It's a core issue, but most people don't run into it, so it hasn't been solved. And then contrary to a good example is the mail system module, which you look at and say it's a very simple module. It's used by 60,000 plus sites. What could go wrong? It actually has this fun little feature where in hook and knit, it loops through every class that's available and sees if mail system could use it. So if you have lots and lots of classes, it's really expensive. If you test the module by itself, it's perfectly fine. Another example. And that makes Drupal very sad. Thanks to Chicks for this one. So the general philosophy of the whole talk here is about catching performance problems early and not even like catching them while you're developing, but catching them before you even develop them. Before you create a problem, you can try to plan and prevent it. And some of the ways you can do this is by making sure that you have tools available to everybody on their local environments. For example, XHPROF, all your developers should be using things like develop query logs, things like that. And you want to hold all of your developers accountable for performance in the code that they write. And I skipped the first one because the first item and the last item kind of go hand in hand. Don't try and measure performance without good data. And you can get data, but good data matters. So if you're building a site that eventually will have, when you go into production, you're going to migrate in 100,000 nodes and a million comments, you should be developing and testing against a site that has 100,000 nodes and a million comments. Your performance data isn't going to matter if you're testing against 100 nodes and 100 comments because your site profile is different if you're not accounting for the differences in content. Similarly, and Ezra touched on this a lot in his XHPROF talk, don't rely on intuition when you're trying to figure out why a particular page is slow. Data matters and profiling can help you find good data. So just to sum up, planning for performance means setting good goals and defining solid strategies. Establishing a cash strategy involves the components that we talked about, looking at cardinality, granularity, cash expiration, and rebuild cost. And you should be defining your cash strategy early in the project. And finally, your data matters. Data matters a lot and good data matters. Maybe I should have put good data matters. Bad data is worse than no data at all. It's like writing a test that doesn't test the right thing. You look at a pass or a fail and say, well, it passed, so it must work. But if it's not checking the correct thing, you're going to be chasing the wrong failures or you're not going to be chasing the correct things to begin with. So we went a little long and we've only got a few minutes left, but if anybody has questions, we're happy to take them. Just hop up to the mic if you want to ask a question. And here's our Twitter handle, so we're happy to continue whatever conversation afterwards. So feel free to reach out to any of us. And we have to make the obligatory, we're hiring phrase. We're required to do that. We had a question about implementation of the cash strategy. You know, you presented some ways of expiring a cash on some very specific criteria. So how do you go about actually implementing expiring it, like when new content shows up? Do you have a hook every time content is added or something? Yeah. So I think that's sort of a second part of this type of conversation. But there's a lot of modules out there to do it for you. Like views content cash is a good example. Or panels hash cash or views argument cash. There's a lot of things out there that provide that for you. So I think once you've defined what that strategy is, then you're more informed of what you're looking for after the fact. So I think that's a good example of if you have a specific need or something like that, we can talk after Twitter or something like that, but the options are out there once you've defined what it is you're even looking for. I've found that sometimes when folks first learn about caching and caching strategies, especially staff, they say, well, let's just kind of cash everything and let nothing get stellar in a few minutes. But of course, that's not necessarily reasonable. Do you have any suggestions for opening up a discussion on cash strategy, especially with non-technical staff? So I would actually say usually the non-technical staff don't want to cash anything. Usually they look at it as, well, if it's stale, that is the end of the world officially. But then the technical people say, well, if we cash it, we don't have to be called at 2 a.m. and have to deal with this. So I think a lot of it isn't necessarily about what the caching looks like. It's about someone functionally saying that block is different for every user. And then you can make the determination of what that takes to be cached or not be cached. Whether something stale or not is something you can solve with a good implementation. But if you don't know what the difference is, then you can't even know whether to cash it or not. So I know that kind of circumvents the question a little. But I think it's just the caching is a fear of stale versus fresh content. It's not a fear of does this expire in 10 minutes or is this removed when some other action takes place. So as long as you can avoid having stale content, I think that's where you sort of break the conversation of what do we functionally want here? And then how do we actually implement that? And then on the other side, you can make an internal decision of how do we make it not stale. One of the things you focused on was sort of thinking about caching early on when you're still doing wireframing type stuff. I mean as part of that, almost that thinking about caching would even dictate some of the design or UX stuff even. So Jeff mentioned like notifications is a great example. Yeah, and the previous question is sort of related to this where if you're having that conversation really early on, you may realize that the homepage has way too much crap on it. It usually does. Like every site we have ever seen has way too much crap on the homepage, right? And having cash strategy conversations early on can help people be okay with cutting the cruft and redefining what their strategy is for the homepage and really picking out the most important things. The other thing is a good cash strategy can reduce your overall cost to rebuild caches. Like if you're efficiently doing it, you have more room to do some expensive things. Like the popular content block could be driven by a crazy algorithm that uses Google Analytics data and comments and votes and radioactivity and all kinds of weird stuff. And that might be okay if you can reduce the overhead of building out other caches. One other thing we didn't talk about is prewarming caches. So like a popular content block, you may decide you never want to build it when a user views the page. You want it to be being built like on Chrome or something like that, right? So a user never experiences the cost of that rebuild. Hello guys. You guys talked a lot about good data, having good data. But sometimes when you have a new project, you don't have that good data. And most of us developers we use develop generate. Is that a good strategy or you guys have other suggestions? Yeah, so in general, generally speaking for simple use cases, I think develop generate is great. We've actually started experimenting with methodology for building out better representations of content. We work on a lot of sites that have very complex architectures where you might have, like Commons is a really great example. Commons has groups and nodes are related to groups and you kind of, when you are testing a site, you want a representative, you don't want to just say like generate 100,000 nodes of like whatever you decide. Because it's not representative of what your site is actually going to be like at the end. So we've been experimenting with, there's a library, it's not a Drupal specific library called Faker. And it's really good for generating like more realistic fake content. And then we're trying to figure out the best strategy for turning that into Drupal content. So far we've used like Drush commands to populate a queue that creates a bunch of content. Alex Ward sitting right there actually had some really good ideas around using Migrate to do it. But the idea would be that you're generating fake content, but it's more representative of what your site is actually going to look like at the end. And it also allows you to not think that you're reading a Roman newspaper. You touched a little bit on perhaps doing some caching on Cron or other types of asynchronous kind of operations for the caching. Are there some general strategies that you have about when that might be appropriate, when kind of considering the four different aspects of things to consider when thinking about caching? So if I could take that, I think a lot of it is, I think what the definition of stale is, is a good piece there. Because by doing it in the background, you're assuming that someone will get something that's somewhat stale. So that being stale should not be affecting things like latest content. Where latest content is latest, it's whatever just happened. But if it's something like popular, where popular doesn't change on a second, or the metrics for it change second by second. But effectively, things won't move around that quickly. So I think that's a better case for being generating the background. Because the concept of stale doesn't really apply in that case. Or generally, it's anything that would be better suited for Cron. Things that are external web service calls. Things that you cannot guarantee some predictable performance for that particular operation. Anything you would want to do on Cron anyways. If that's infiltrated certain blocks on the page, it's the same concept. You want it to be in the background because you don't want page views to go from 500 milliseconds. And then something is possibly getting worse. And now you start saying three seconds because you have some external dependency that's causing some trouble. So I think a lot of it's how confident are you in the fact that rebuilding that will be consistent. And when you're talking about data, things like standard deviation help with that a lot. If any one page can deviate by 50 plus percent of the actual load time of that page, that tells you that you can't accurately predict how the site will perform. Because things could be stacking up on each other. Or it could be some dependency that you're not thinking of that's actually creating more and more problems. We should probably call that good and clear out for the next session. But thanks everybody for attending.