 My name is Erin and I work at Aquia and I have worked there like three and a half years in support and then the last quarter I've been a technical account manager and I spend a lot of my time thinking about caching and caching issues and how to identify and resolve like some really weird problems. So um first I'm just gonna like do like a quick review of some of the differences between Drupal 7 and Drupal 8 and then um with regard to caching and then I'm gonna get into two specific types of problems and my goal with this talk is to show you the method I've developed for trying to troubleshoot these issues so that you can then do it yourself. Should be awesome. And I'm so excited to see so many people. How many how many folks to your hands are still like the Drupal 7 world with caching problems? Okay. And then how many folks are like becoming fluent with the Drupal 8 world of caching? Yeah. Okay. That's cool. That's a good mix. And I like to give it one more minute because I don't know about you but I am always late getting to sessions so except for this one. So just to kind of cover it briefly. Drupal 7 cached an entire page and saved that whole page to the render cache. So if you change something in a block on a page it wouldn't necessarily update the page especially if you add several caching layers with you know really good time to live on them. So what you had to do was do a whole cache clear to regenerate your front page and that's very resource intensive on the back end. Drupal 8 thread because you can cache a lot of the page but still have like areas like that I have here in the red that are you know dynamic areas for logged in users. So you can update it just for that particular user or you can have like a block I put on the top here see your school and you need to get out an emergency alert. You can just use cached tags and clear out the block at the top of your home page and be able to have that everywhere without having to expire your whole cache which is rad. But it does sometimes result in some issues where things are lingering on the page or things are not showing up the way you expect so that can be a little trickier to troubleshoot in Drupal 8 now that you can cache all the things. Just like best practices if you use cached tags with your views it's much easier to propagate that out to a CDN content delivery network. So if you're using Aquamire you're using cloud flair to help deliver some of that static content or the dynamic content faster at the edge layer use cached tags. Enable page caching dynamic page cache is awesome because then you can cache this three quarters of the page. Big pipe if you haven't seen Wim's big pipe talk from a little while ago you should. It's good. I repeat some of that here but definitely definitely set this to as long as you can and you can use Dresch to set your max age even for that like with Drupal 8 and using the purge module to selectively invalidate cache when content's updated you can set that time to live out to a year and then when your content editor is updating something on their site and they hit save then if you're using the purge module it'll just update that right away which is a wrap. They're happy. You're happy. Your caching is forever so people are accessing information quickly. It's a wrap. And then I have another talk that I gave specifically on PHP caches. There's opcode cache and there's user cache and there's some fiddly PHP specific caches. That's a different talk but check that out too. So let's get into the interesting problems because they're kind of fun. We had a client UC Davis and they had this persistent issue on their alumni profile page and their news page so this was like dynamic page generated through views where some of the profiles would show up properly. They would be styled and you would see the article tag around them and some of them would show up completely unstyled just text and that would have field tagging around it and they couldn't figure out why and we couldn't figure out why for a long time. This is an interesting problem because it only occurred occasionally so we'd have to try an intervention and then wait for whatever mystery occurrence was making all the styling go away. It was persistent and it was even though this was a university site it was a large multi-site so a lot of different sites running off the same code base. The other sites on the code base work fine so with something specific to this site we're like okay what could it be? So one of the things that we use a lot to learn about what your caching layers are and anybody can do it and this is just on the aqueous site is use curl and you can take a look at which items are a cache hit. So there's a Drupal cache hit and then which items are in Drupal and which items are like we use varnish so this would be a varnish cache hit or it could be for this site uses cloud flare it could be a cloud flare cache hit. It can be this X cache hit can be whatever it can be updated by whatever caching layer is closest to you and in some cases that's varnish and in some cases if you have a WAF or you have a CDN that's the one immediately close to you. Both should respect the max age and then if you are using cloud flare they have a cf-ray ID which you can use to contact cloud flare and get more information about what's in the header and what's in the cache. For us this is pretty juicy because we're like okay so if it's a Drupal cache miss we're not hitting the render cache this is a Drupal cache hit there is a cached item it kind of tells you which layers of your cache you're seeing data from. Sometimes you need to get information you know with Drupal 8 you can cache authenticated sessions right so sometimes you need to be able to use curl on an authenticated session and in which case this is like a really handy method for getting around that. You can grab the session ID and attach it to your curl and be able to see where the cache layers are and this is just kind of an example but when I was talking about before here's your Drupal 8 site on whatever stack that looks like. Your Drupal 8 can cache to the database of memcache or redis just depending on where you're hosted. We use varnish at aquia on load balancers so when we strongly advise you to cache as much in varnish as possible because it pushes the content out quickly and then that can be picked up by a CDN like cloud flare or we also encourage you to use web application firewall. It's nice to have that ready in advance in case you need it. So this is like what you could be dealing with. Now when you're trying to figure out something that's stale content or something content that's not rendering as expected you're using your curl trying to figure out okay so is it picking it up from this layer or this layer or is it all the way is it a Drupal cache hit is it all the way in the Drupal in the database or memcache. So what I advise people to do is to simplify and a quick way of doing that without actually making any architectural changes at all is just use a cache busting string and as you can see it's well the cache busting string will miss these outer cache layers get right to the Drupal site. It's handy. Another way of doing like trying to figure out where the problem is in a caching layer and one that we used to actually troubleshoot this problem is to use Dresch DC. You can still Dresch clear cache in Drupal 8 and if you do it kind of systematically and selectively it can help you reveal like where exactly the caching problem is is it in the theme registry. Usually it's going to be right here in render and that's what happened in this case. We've cleared UCC to selectively clear render cache and the unstyled content immediately was updated and was styled and we're like okay that tells us that it's Drupal and it's in the render table. So just to kind of restate or say in a different way if your normal caching layers look like this and you're having trouble figuring out where the problem is occurring that you're troubleshooting simplify get it right down to either load balancer or if you can work locally and just check on Drupal 8 a copy of the code in the database in the files. You can also temporarily disable memcache or redis if you're using that. If you're finding that Dresch clear cache render if you're flushing just the renderer and you're just caching to the database that tells you that you can search USQL and search the database render table and see what you can see see if you can detect where that faulting data is coming in. For this specific one because it had to do with styling we also enabled twig debugging which was super handy for helping us understand what the formatting was. Obviously after you've done troubleshooting please remember to re-enable all your cache layers. We've had some sites where they did this really great troubleshooting, discovered a problem, fixed it and then forgot to put things back when they're like why is my site done during a huge load event so please remember to check that back. So going back to my example here what we did is we simplified we take it just down to the Drupal database we turn off memcache we turn off external cache just for the site we went to the article and then we said okay let's look and see if we can find where this event occurred to have this unstyled content and we use this select query wanting to select things but we use this query to help us figure out where which teaser was faulty so we went through and we found that this one that was the faulty one like when we're showing up in the cache as unstyled content was using fields for the theme and this one that was correct was using the node teaser for the theme like hmm what we found then we had a hunch okay because sometimes you get there you find the evidence and then you're like how can this be so we took a look at the view settings for both of these and we found that this was using the regular article was styling the default teaser and the RSS field was also using the default teaser we're like oh okay so this is the same so when you went to the RSS feed or when anybody accessed the RSS feed for that page it saved it in the cache render table looking like this unstyled it's styled with fields and then when we cleared that out it went back to normal and when we accessed that as a news article first it went back to normal and that's because for that default teaser view it isn't aware of cache contexts and that's a known dribble bug that's been there for a little while in fact I was checking it right before the talk I'm like did they fix it they fixed a lot of these issues but this particular one is still not fixed so what I recommend is that you create a display mode if you're going to build like a styled list and use it on your sites good practices to build a display mode for this view or for any custom views and not use the default teaser just so it's funny to me because everybody uses the default but the the people working on this are thinking oh the default teaser is an example that people can use for styling when in reality what do you do when you're setting up a site or a lot of people just accept the default so like hey this must work and then they apply the theme styling to the defaults not realizing that the default is unaware of the cache context so that's my big like this was neat so you can solve it in two ways you can create a custom display mode I think that's the easiest you don't want to end up with a bazillion of display modes for views but it is kind of nice any mommy has an example of there's a few of them that are styled out or you could just change this right you can use change the format that you're using from the default so there's two ways to avoid that in the future so this is just kind of like a thing that we run into all the time at aquia that we'd love to like share is just double check it's like this be a by default but absolutely check your views this is in the edit thing and just make sure that when you're caching you're using tag-based caching so it can be picked up by your cdn layers and even if you don't have a cdn it will catch tag-based caching works really well so this is problem number two and I'm this is kind of like an example when this used to happen before dribble 8.5 this used to happen a lot with translated sites there was a bug in the reader redirects for multilingual sites that got resolved in 85 that would cause the same problem but this is kind of the symptom the symptom is I've been working on this site I have this block I know it should be there on the site usually this is the one you get with screenshots and a bunch of arrows and it's kind of the example I have here like where did my block go it should be showing up in some contexts and not in others this is where I'm gonna do a little like switch because this is not actually the site that I'm talking about but but this problem of where did my block go or my content is unexpectedly missing actually crops up a lot so there's a kind of similar to the kind of troubleshooting we did before there's a pretty straightforward method of troubleshooting these two first is you know make sure that you've double checked all the basic stuff like it should be showing up in the block settings and that your visibility is set so we ran into a lot of that where the block disappears but it's just really simple triple eight settings then start making notes is there a difference in behavior for authenticated users users who blogged into your site or anonymous users because you can hit different caching layers right you need to kind of double check is it is the block disappearing on certain pages for authenticated users or maybe authenticated users are always seeing the block and just anonymous users can't see it just clearing crash does drush cr fix it or can you be more specific with that does drush cc and clearing render cache fix it is there a custom module uh that you know custom modules are awesome ways of adding functionality to your site but every once in a while you can you know make an assumption with your programming that causes your blade to work unexpectedly and in this one like swill alert we're absolutely talking about a custom module and then also use curl get an idea what are the caching layers that are between you your point of looking at it in the drill ball so in this site our customer made a custom module sales tool that was just disappear for site visitors authenticated visitors would visit one page that has the block and it's there they go to a different page related sense of content they come and it's there you go to the third page that isn't supposed to have the block and the block disappears then they go back to the first page that has supposed to have the block and the block is still missing for the rest of their session that was the sales tool block is made missing and that is affecting all the anonymous users too in this case it was nice because we were able to download the code and database to a local environment and when caching was disabled the block always behaved as it was expected which is fine but disabling caching is really not a solution in production really not a solution in production so we had to figure out this problem and resolve it so what we did to kind of track track track this down is work with a local copy and and then search cash render and take a look and see what was happening so when we accessed the page that has the block and the block's appearing we have this cool like cash redirect line and cash render when we visited then in the same session visited a page that where the block's not supposed to appear we found that the cash render the redirect line had disappeared and it was replaced by this like markup string that was the zero value like really that's pretty strange it should have something it shouldn't just have a placeholder of zero so we tried like a few other ways of testing clearing cash in between and we found that this behavior was consistent like we really needed the cash redirect line to be there to be able to see the block so then we looked through the code right and we rolled out double eight and we found this piece of custom this custom module that is has this logic to it hide the block if no links are to be displayed like wait a second that's where this markup with the zero value is coming from so my character put together some example code of how to fix it so this is just kind of example code of how you would potentially fix it and then when we changed this code in this particular custom module then when we access the page again we were able to see this cash redirect was still there as it was supposed to be and the block started appearing unreliably so that's kind of like a really quick kind of overview of hey you know double check your modules if the block is coming from a custom module just double check the logic or disable the module temporarily and see if you know the problem goes away you can't do it actually because on this example because this module is building the block you're trying to troubleshoot but then you know change your coding strategy and that might resolve the thing so when I submitted the talk there were still outstanding multilingual issues that would make cat cached multilingual content just sort of disappear but good news is this part of the stock is really short now because a lot of those have been resolved there have been some great work on that so it's awesome um there are a couple that I am aware of this browser language detection isn't cash aware is one that's currently active um and then this one I thought I'd mention uh just because it was so baffling for people people shouldn't be using jubilee point five anymore hopefully they're using a newer version uh but if they are using an earlier version than that and they're finding that translated content is just sort of disappearing it might be this bug um and really that was like a redirect that was returning a 403 and so but it was baffling because suddenly translated content would just disappear and then because I work at aqua this is the same we have it's either cash or cash so in your production and you have you know 10 billion of your closest 10,000 of your closest friends all visiting your site all that same time you can use your caching layers or you can end up paying more money in hardware that's the way it goes we prefer people use caching layers and this is just kind of an example of um one of my current clients they're using a decoupled application and they're using cloudflare and akamai they're actually using it in two places here and then in front of dribble eight for their api also but um this can be a little complicated to try to troubleshoot but it's kind of i'm finding that this kind of setup is kind of more and more standard so i thought i'd share it it's good so thank you very much this is awesome having a packed room please ask questions we'll have to share so so would you recommend installing something like views custom uh cash tags because what i'm finding is that uh when debugging cash tags all views that use content as the base have node list right so of course adding content basically anytime you add content update delete all of those views are now cleared so would that be the recommended version you would you would suggest or is there a different way you would approach that i usually end up advising people to use purge um with whatever helper uh module works with purge that works with your hosting you know like there's an aquea purge that works with the podge that purge module that specifically talks to disobeying requests to the our varnish layers but uh there are other purge kind of boltons that will send a band request which is a lot less resource intensive and it will still update all the views though so taking the like varnish out of the equation and just sticking strictly with drupal 8 um so i understand purge but my guess my question is really i taking that out would you recommend something else to take away the node list tag oh i mean you could try it see what works for your particular application i haven't noticed it being like so resource intensive that it's showed up on my um like i think that that even though you're updating everything it is kind of like a minimal tax still on performance okay hello hi there so based on this case study drupal comes with uh page cache dynamic page cache and big pipe so do you guys have all three of those turned on or just turn on all the caching things turn on all three of those yeah check out wim's talk about big pipe it's really good and it explains how how they all work together that was listed in the slide right yeah and then my second question is um you have cloud flare then you have varnish and obviously you have authenticated users in this case study so i'm assuming that the the cloud flare and the varnish was not caching those authenticated users and you were relying on big pipe actually they will if you have a dynamic page cache then parts of the page will be cached in varnish and picked up by cloud flare too parts of the page yeah because um the way that drupal builds the page in drupal 8 it's not trying to render or cache the entire page right it kind of builds it up so that you can have cached content for authenticated users especially if you're using dynamic page cache or big pipe right that will be cached in varnish and cloud flare right then it's only dynamically generating like the custom right so you're saying a page would be partially cached in varnish because varnish is doing page cache right right so you have a partial page cache yeah it's kind of interesting to me it's kind of neat it works pretty well too how that would work yeah because if it's the same page for a different say student then it must have a different variation of that page because it's only one page but it's just customized for that right student to be in varnish how would varnish so if you're using big pipe too it'll load all that cache content and then we'll load the dynamic areas right i understand the big part of the part that i'm not seeing is how is varnish handling if it's just the page yeah it's more more easier to see when you're looking at the live site and you're use the curl and it'll show you which caching layers are picking out okay thanks this might be a stupid question but uh we have memcache installed you know on our aqua sites and i'm wondering if internal page cache and dynamic page cache and memcache you know are those three separate components or yes memcache would actually in you know like we don't have to use internal page cache anymore because we have memcache now you they work differently so memcache what it does is it caches request that you would normally you would otherwise be sending to the database right it's kind of works the same way so instead of being like a cache whereas the page cache is usually in here the layers before you get to drill well i see okay thank you you're welcome i like that question was good thank you and um just a reminder we have stickers so please take some one more question yeah um you talked a little bit about custom blocks and having dynamic content in the page and needing to set cache tags for that we are running into that problem where do you set the cache tags is that something that you're doing in the configuration and purge cache tags are usually enabled by default so um in drupal 8 and then also i showed you the setting in views um so if you look at the header if you curl and include the dash h for the header information it'll usually list out the cache tags that's it's using looking at the list will help you better understand exactly what has the cache tag associated and what doesn't because we're only seeing it on a on custom block i mean everything else we have we're using purge we're using marnish it's it's only with the custom blocks that this seems to be coming up okay yeah so um that's where you start to kind of get fiddly like into the code and try to see whether um that's actually engaged you know happening before you engage like the dribble cache context or the caching system or maybe you're like missing a little piece of logic that allows it to do that we know what's happening in dribble you're like we just don't know how to fix it how to fix it yeah um so the other thing would be to look at a module that's caching really well on your site and compare how it works with your custom module to see um but i would i would love to know more about your problem so follow up with me later all right hey so i have two questions i guess the first one is what benefit do you get out of having multiple cds set up on the same i guess project um so not it depends on what you're using your cdn for um we have some customers that are using their cdn to help deliver the site quicker to the edge right um if your site's hosted in the us and some of your consumers are in china it makes sense to use the cdn to push that to china or you know uh we have some customers in singapore who have um you know different places in the the globe where they need to get the images and the content too so you can use that um and the other purpose of a cdn can sometimes be to help keep your site up um and perceived as up when you're doing work on the back end yeah i just know that you would but this is purges for for both of those cdn yeah no it gets complicated you're only doing it usually if you need to and usually this is just my slide it has a couple of examples but um usually you stick to one solution or another okay and then my other question is so i know akamai specifically has limits on how many cash tags they can process per hour so in terms of like for your customer facing views and that kind of stuff i understand using cash tags is that's kind of a must right yeah but for sort of your admin UI stuff would your recommendation be to not use cash tags um if you're like kind of pushing up on that limit yeah you could you know disable some of them if you're getting close to that limit yeah if you're hitting the point where you're hitting that limit too um you can also use something like purge to um the other thing that purge does is it compresses the cash tags so it's not so much you know if you it's a shorter thing so if you have a character limit we definitely filter out so we have configured only enabled so there's no reason to be purging config but it'll still send those cash tags we filter those out yeah and we we we're trying what we can to reduce it but it's also just not knowing exactly what what is getting purged but you know we would have 500 cash tags per request right that would go up and you know that's too many yeah try to filter and get it down or use um modules that are a little less it's hard because then you're you're doing a different kind of optimization instead of a caching optimization it's tricky I have a question for uh web services caching uh so we use aquea content hub to publish content meaning that every time when we publish content or update content um enumerates all the data and sends to the content hub right uh it typically is fine when we first publish the content it enumerates uh correctly to the content hub but when we update the content more often than not that we see that content is actually cached oh which means that you know even the content we see uh being updated on the title on the side we can see the change but we can't really see that change reflected in the content hub got it so what's your recommendation for that kind of caching so on the content you mean like your master site where you're sure yeah so any size that you publish the content to content right I would put in um a support ticket on that uh just because so I I worked with the aqua team and they said uh one of the