 So the first version of this talk was given at BuzzFeed headquarters, so I have an alternate title flag So a few disclaimers first So, you know, I try to touch on a lot of things during this talk, but it's by no means comprehensive There's literally, you know, a hundred different ways you can add performance to your website so I tried to focus on some of the main ones and some of the main areas and So if I'm missing something, you know, don't hold it against me So I guess we'll get started So performance, I think first it's useful to separate it out into two different kinds of performance and two different ways to deal with performance on your site front-end and back-end and they're very very different in how you deal with them and and The implications thereof and they require like significantly different approaches So I guess first let's talk about front-end performance When I hear people talk about performance a lot they talk a lot about like oh, I shaved 50 milliseconds off my you know endpoint or my my view or whatever and really if you're doing website performance and you really should be starting with front-end performance because 80 to 90 percent of the user response time on a typical web application will be on the front-end not on the back-end So saving 50 milliseconds on your view is not gonna make a difference saving one second on Optimizing your JavaScript loading procedures is gonna make a difference. So Steve Souters He's a work that Yahoo and Google at basically as a performance evangelist and he's written great books about Performance applications for O'Reilly that I would highly recommend but let's start with the front-end so front-end work so The attributes of front-end performance work They can be based generally universally applied So what does that mean it means that when you make a fix or improve something? It's gonna affect almost every view or every every page of your site They often require systems and tooling changes. So it's not something It's a lot of it is like dealing with oh, I need to override collect static to do something I need to compile something and put it in a certain place so that it works So there's a lot of systems tooling if you guys have op teams For bigger companies a lot of times your office people have to get involved with some of this stuff and Another attribute is it there often clear system independent best practices So this is not often the case with back-end performance work a lot of front-end performance work There is a best way to do it. There is the known best way to do it so I Guess I'm gonna start with going through a few best practices and then talking about how you would do it in front and performance So the first one is caching static assets forever. So Why would you want to do this you'd want to do this because you serve your site up? You know, you're doing 10 15 20 requests a second Your JavaScript or CSS your images aren't really changing. So you don't want the browser to constantly Be re-getting those resources every time someone refreshes the page or a new user, you know They go to a different page that has the same JavaScript so the ideal way to do this is to cache this Assets CSS file JavaScript file image file forever and only change it when The image itself or CSS or JavaScript changes. It's called basically. It's called fat far futures caching is what a lot of people call this So how do you do this? How you do this what before it used to be fairly difficult now You know ever since several versions ago Django has the static static files as a standard part of Django all you really have to do is Set your static file storage to be cache static file storage or if you have your own storage use the cache files mix in suddenly if you look at He opened up like firebug or web developer tools and you and you see what happened your images or your CSS You'll notice that they're renamed like cat.jpeg became cat.123.jpeg and that's Something that basically Django does for you and it changes the name of the file every time the asset changes Therefore things are cached as long as humanly possible So the second best practice bundle bundle minify compressed static assets, so First of all, what does this mean? So bundling means you have 11 JavaScript files on a page You take you take that you reduce that to one or two Minifying means you take all extra white space and long long variable names anything you can out of that JavaScript file Compressing simply means to basically gzipping it So this is like typically this is done and by one packet so in rails this comes like standard and in Django You need to basically a static asset manager to do this the reason you'd want to do this is Reducing the number of requests is a strict benefit for making your applications fast if you ever look at How applications are loaded like if you if you fire up Firebug and you look at the network panel and you look at how they're loading you'll notice There's this like stair step pattern to how Assets are loaded it'll load like eight at a time It'll stop it'll wait for one of those eight to finish and it'll get another one and and that's because Browsers basically have a limit per domain of how many simultaneous requests that allow on that domain This is a little bit better in the more modern browsers, but if you ever have to deal with i8 or god forbid i7 They have you know for for assets for connections to a particular domain at one time So you'll notice if you have 14 JavaScript files, it'll be loading for it'll stop It won't do anything until one of those is finished downloading it'll get another one It'll get another one. It'll get another one So what you're trying to do is you're trying to reduce the number of requests as low as possible And static asset managers do that So to use to do this kind of management basically you got to use a External package two really good ones Django pipeline actively developed the guy You know he takes a lot of pull requests and is always adding stuff and from the flask world web assets Which also has a Django web assets extension to it also great at this stuff So bonus points the other other thing definitely pipeline. I'm not sure about web assets support is in Your CSS files if you're referencing images and those images are under 32k it'll take those images It'll turn them into base 64 and it'll embed it right in the CSS file. So I don't know how many you guys like maybe like seven eight nine years ago the best practice for reducing number of requests for like Websites that might have like 50 little icons was was having your designer create sprites And then using those sprites and doing these like CSS offsets to get to the next image and next image and next image So with data URIs, you don't really need to do that you can you can have 50 images each with a little icon you can reference them in your in your CSS file and The the static asset manager will find that replace it with a base 64 equivalent and you won't have to deal with dealing with managing sprites Okay, another best practice Serving static files via CDN. So CDN is going to provide less less latency to your user If any like bigger shops like you guys do performance monitoring or testing and a lot of the performance monitoring tools Will show you like the results as if your site was loaded in Brazil and as if it was loaded in Russia And as if you know because I'll they'll keep servers in those areas and they'll tell you what the latency is So you won't really notice it maybe locally because you're in New York and your servers are in Virginia But in Brazil they might be getting a completely different experience of your website. So that's why it pays to use a CDN So using CDN's in Django is Fairly simple. I mean there's a bunch of ways to do it one good way we like Use a extension called Django storages which provides custom storage classes we happen to use the s3 one we put we Store all our static assets in s3. We point our CDN to s3 and we're done Okay, another best practice So serving more stuff is static assets. So this is a kind of a general one, but I think it's one that is Really applicable. So a lot of I think I'm pretty sure a lot of you guys Now you're using angular using backbone using ember You're doing less of your templating in Htm on the server side. You're doing a lot more your templating on the client side So as as a lot of these applications turn into like single-page applications where the server side basically just returns JSON and the The rest of stuff happens on the client side. There's an opportunity to store more and more things of static assets As you store them a static assets, they take load off your servers. You can use the CDN You can cash them forever. There's all these advantages So front-end templates like your angular templates your amber templates are perfect candidates for using a static for serving a static assets What that requires is a little bit of tooling. I mean it depends on exactly what JavaScript framework you're using and how your stuff is set up But it can definitely be done. We do it with angular and it works brilliantly other thing you can look at is Are there things that are there pieces of data that only change in between releases like us personally? We have all these like fairly complex menus. They're really just like big pieces of JSON We were serving them as as you know as part of like a payload and on the server side And it was like getting stuck on the page and the page would load it By a JavaScript what we realized is we could actually just serve those as static assets have angular grab those static assets And it added it took off several hundred milliseconds from a lot of our key endpoints Okay, so let's let's go to back-end performance work Well, you'll notice so it's front-end performance work it it it's a lot easier to go through because there are some global Things you can just do and you can say this is the right way to do it You should do it this way back-end performance is a lot more nuanced often it can be done on a only on a case-to-case basis, so Every site is going to be different the views are going to be different your approach to making things faster is going to be different a Lot of times they require co-chains. It's not it's not just systems and tooling and using a different static file storage You're gonna have to get in there understand what that slow view is doing and deal with it Like I said, it's very site specific situation specific So there's not a lot of global stuff But okay, there are some global stuff you can do just to do is you can check off your list. You can use cash sessions By default or I think by default Django has DB sessions turned on so every every session Makes a few hits the DB you can switch that to either a pure cash session or a cash DB session And it'll take a few a few hits off your database. You can use the cash template loader So a cup I don't know four or five Django's ago The templates basically every time you'd make a request the template would get recompiled they fix that And the solution was to use the cash template loader, which is basically just a Django setting use it you use in production It works. It's strictly faster Maybe it's slightly controversial, but I don't know if it is so if you're starting a brand-new project or you have a Existing project that is making like heavyweight use of Django templates. You might consider switching to ginger to it's generally considered to be faster and you know, I would only do it on a new project or someone where it's really performance critical because it is a pain if you're actually switching on and You have a fairly big site But you would you should see some performance benefits from switching to ginger to or starting a project on ginger to instead of using the the standard Django Templating engine, okay, so now I'm kind of the real stuff. So So how do you start? Performance optimizing on the back end. I mean first it should start with a disclaimer You don't want to do this on every view. You don't want to do it everywhere You don't want to like go and say oh, I'm just gonna make my site faster There's a waste of time because you'd be optimizing stuff that like five users a day touch And unlike unlike front-end performance where it adds some complexity, but it's like a hard limit to how much complexity it adds Back end performance. You can go crazy You can take every view and up obfuscate it to get you know, 50 milliseconds out of it So you really want to pick your spots and pick the key points that are performance sensitive that hundreds of users thousands of users touch every single day and work on that because typically performance Optimization on the back end is gonna add code complexity One programmer is gonna do it and the next programmer is gonna come and not understand what why this is being done It's not often obvious. You're doing a lot of times doing non-obvious things that that only makes sense if you're looking at it from a Performance perspective, so you really want to pick your battles but Let's say you did find a problem view a view that was slow that that was it critical to your application How do you start? You don't want to just start like randomly like oh, this is you know sequel curry looks slow Let me go optimize it. Oh, why are we using this data structure? Let me go fix it If you do that, you're basically flying blind So what you really want to do anytime you do back end performance work you start with a profile a profile basically tells you How long is this view taking the return? What are the elements of it? You know, what are the pieces that you should be focusing on? There's if you just Google Django Python profiler middleware you'll see like 50 of them There's various good ones. I linked to one that we like. I'm sure there's some that are just as good if not better So what does the profile look like you're not really supposed to be able to read everything there? But that's what a profile looks like how it typically works is is you install your profile middleware and you give it a Hey custom like Get parameter say and profile equals one and when the profile middleware sees that instead of returning your normal view Your normal JSON it returns something that looks like that So what you see is typically you'll see the functions that are called how many times that are called How how much time each call takes how much time the total calls take? This profile profile middleware specifically it also gives you modules and how much how much time is spent in each particular module So this gives you a map of okay I have this view that's slow. What part of is it? So is it my function? Is it Django? Is it sequel? Is it just Python? I'm doing is it this library that I didn't know about that I'm using where where do I look first? So looking at that typically what you'll see is you'll see like like let's say that's a sequel problem You'll see 90 95 percent of times spent in Django slash DB slash like my sequel up high So that gives you a good indication. Okay. It's a sequel problem If it's not a sequel problem, you'll see a lot of like various Python functions that are taking up a lot of time so let's So, yeah things to look for Tons of times spent in sequel it's one place you can really pretty pretty easily improve functions that recall called way too many times you'll see Basically every time we've done a profile on a like a performance that's a thing that seems like it's running slower than it should We've often been embarrassed almost ashamed of like what we've seen like oh my god Why is this function getting called 10,000 times? You know, it should be getting called 50 times There must be something wrong So that's the kind of stuff you look for things you don't expect and it's very site specific You really have to know your application a while to know what looks odd to you and what looks normal But that's the kind of stuff you're looking for either sequel or like oh this function is being called way too many times Or it's taking way longer than I would have expected given what I know it's doing So let's start with sequel. What if what is the problem with sequel you look at your profile 90% of it's in sequel So what do you do? You start with there's great tools out there to help you Kind of diagnose and solve those problems Django debug toolbar, which I'm sure almost everyone here knows is Is a great tool you install it you turn on the sequel specific plug-in every every front end view You see you'll get a sidebar. You open the sidebar. You'll see every sequel query. You'll see how much time each sequel query takes So that's very helpful to begin with if where that kind of falls down is if you have Back-end only views basically things that return JSON So Django debug toolbar won't really attach themselves to that because it actually looks for like an HTML to attach to So in that case there's another tool called Django dev server which basically replaces the standard Django server and You can turn on different modules and the different modules one of one of the modules basically as the view is As the server is doing work it'll output every sequel Courier it's doing and it'll output How long each query it's taking it'll also it'll also like mark when the same query is being done 50 times It'll say like 50 duplicates so so on the back end that's a really good place to start so typically sequel problems they fall into two one of two big categories one the sequel query You're doing is taking too long To there you're doing too many sequel queries So you know each of those have different ways to solve so There's basically if you if you look at every Django release it seems like every Django release they Release another way to get around a sequel issue So I think since the days of 1.0 select related has been around and then each release There's been they've been adding different ways to like get your sequel to do what you want it to do So I'm gonna go through a few of them This list is by no means Comprehensive, but it'll touch a lot of the major things that we've seen that can can help with sequel So select related select related it solves a really common problem in Django. You have a foreign key you have a car and and you have passengers and passengers refer to the car they're in And you're iterating through passengers and you try to like print out the car They were in and each query of a passenger will also do a query to get the car So you'll see like if you're printing a hundred hundred passengers You're making a hundred queries to get the car So you something you might expect takes five, you know, here's this view. It should be five queries You know, it's doing a hundred five queries Actually, if you go to Django admin Django admin is really bad at this if you if you if you ever open up Django debug toolbar to a Django admin where you reference another model inside the list for you You'll see a hundred queries if you're ever wondering why your Django admin is slow. That's that's usually the reason So the way to solve this select related select related basically you you you basically tell Django Why are getting the passenger also do a join internally in sequel and get the car So we don't have to get it a hundred times each each time we're iterating So values values list so a lot of times The Python object the model that Django returns to you when you do a or I'm query is is just Kind of getting in the way of you just getting the data What you eventually really do is just turn it into a dick then return as json anyway So it turns out there's like a lot of overhead with with creating this Django object that is your car or as your passenger It does a lot of stuff to make that happen You can basically if you know that You know you're not doing anything with the object you're not calling any methods All you're really going to do is you're going to just take that and put into Python dick anyway and return it Values values lists are two ways to do that. They basically return a list or they return a dick with with the fields you wanted And you can basically bypass a lot of that Python overhead DB index equals true. I mean it's like, you know a 40 year old See, you know ever since relational databases existed. There's been indices basically and a lot of times You're you're querying by a particular thing that doesn't have an index on it and then ends up being a lot slower than you would expect So DB index is one way around that So another page of sequel tricks you can use so prefetch related prefetch related solves a very particular problem, so Select related dimension previously it doesn't work in particular. It doesn't work with many to many relationships It's it's because internally select related is done with a with a sequel join And it doesn't really map well to making that happen in one sequel query By using joins so what prefetch related does it does the basically the same thing it? It allows you to like, you know, let's say in our case a passenger could have been in multiple cars So so there's a many to many relationship. It makes that Work without doing a hundred queries and instead of doing the kind of the matching and avoiding a hundred queries In sequel what it does is actually does it in Django and in Python really It gets the two tables you want it does the matching for you and it allows you to avoid a lot of extra queries prefetch related even if You don't use in many to many queries that it does have certain advantages like we've seen If you're doing select related on four or five different tables internally in sequel That's like a five-way join which is like Typically a really bad big no-no So sometimes prefetch related is as faster because it's Python basically does a better job than than sequel would only so only is So if you if you open up Django debug toolbar, and you do your typical or M stuff, you'll see that every time Django gets a model or or a You know series of models is actually getting every single field from from sequel It's not really a problem and until it is right So you have you have five or six fields on a you know a hundred row Table you'll never really see it as a problem. You have 10,000 rows and each of those rows have 80 columns And you happen to only need four columns in this view You're just returning the name and the URL Let's say and there's 76 other columns then it becomes a big problem So only solves this problem basically what only does is it says Instead of getting all these 80 columns out of this table get the four that I know I need and I'll just work with those it kind of happens with a little bit of magic. So if you happen to Access a field that wasn't in your only it'll it'll actually query and get the field so it won't fail And this could be a good thing in a sense that your stuff won't break Why I have this uses caution it can also be a really bad thing So if you're using only and you're getting these five fields you need Another program comes along and says oh, I need this six field and they access it without without every other thinking about it We've had situations where suddenly this this this view is you know 3x slower and all the what happened Oh something that was like five queries has turned into 500 because now each of each and every one of those iterations is going and Getting the object again because it's missing a field that wasn't in the only so you want to be sure When you use only that you are very very careful with it You know use it with caution comment it well, you know, just be careful. It's useful, but you know, it can also be dangerous The first kind of the in some ways the opposite of only rather than saying hey, these are the fields I want you're saying these are fields. I don't want similar similar thing. It's a performance thing, you know, I You know one common good thing you can do is you know you have 80 fields and and You don't want to get too specific about about exactly the fields you want But you have these 10 big text fields that you know, you don't want you know You're never going to need in the context of this view Use defer you take those out and you save you save that time that it takes to get that data so bulk bulk create so there's like now bulk functions bulk updates and bulk creates in in In Django, so a lot, you know, most of your performance problems. Maybe are you know read base? You know, we're a like a financial data company. We insert a lot of data We get it and in big blocks and bulk create has been great for basically speeding up the inserts So what if all you know, those are some tools I think there's some other ones too, but if none of that works you can always fall back on raw Django basically gives you Access into hey, I need to do this raw query And just get out of my way. So, you know raw is kind of a last Last resort thing you can do but it can be helpful sometimes So Denormalization I mean that's a classic has nothing to do with Python and Django. It's a classic way to deal with Performance problems as they relate to SQL, you know a typical case You have a blog and the blog has comments and you want to print the number of comments So that can be really slow if you have to do if you have to do the query every time So instead every time a new comment is written you update this thing called num comments Which keeps a running count of the comments of the blog. So it's a very typical Old performance trick, but it's one that's still I think very widely useful and I just want to throw it out there, but it is you know, it is something to consider If you're doing SQL, do you really need to be doing SQL often times? Yes, sometimes. No I'm just using the sequels of basically a key value store. I'm just using the sequel to search on Well, you know, it turns out a lot of these a lot of these You know specific domains Are these days handled a lot better by alternate databases like Redis for your sets and lists and sorting, you know Memcache for a key value storing, you know, you know, whatever the case may be, but it's something to consider when SQL is seems to be failing you so the other side of Performance optimization on the back end it's Python so You know You look at your profile and SQL is five or ten percent of the total total runtime So it's not a sequel issue really, but you're seeing this Python stuff happen So, you know, what do you look for? How do you how do you deal with it? So, you know classic computer science you look for bad algorithms first, right? You look for some things that are you know and squared that don't need to be You know people, you know, we've seen how we have had programmers of my company You do something you don't even think about it when it's you know, there's ten objects in the in the database You know a year later. There's a hundred thousand objects in this bad algorithm. You did suddenly shows itself as oh wow That's n squared now. That's a big problem So about bad algorithms is something you look for you know Doing extra work in loops, you know, I've seen that a lot like people People checking things inside the loop that don't ever change because of anything that happens in the loop Like that's something we I've seen a lot You know things The other one like, you know, you have this function. It's like you don't you don't ever perceive it as slow It does what it does efficiently it returns, you know the result you're expecting But it's just slow enough that if you ran run it 10,000 times inside a loop suddenly it's a big problem so really basically Performance on the Python or non-SQL side I summon as up as people doing bad stuff inside loops like almost nothing is a problem until you do it a thousand or 10,000 times Then it's a big problem So that's what you look for you look for your slow function or and then you try to Find that place where oh wow like why are we doing this or why is this take you know a hundredth of a second except We do it a thousand times So fixing this stuff So I want to give a warning like sometimes you just you have to get weird like Sometimes the stuff like doesn't make sense like what you're doing No sane person would do it until you have this performance problem. You just solve it and I wanted to give two examples of of Things that we've seen in our code So this is one we saw so we saw a decimal point one. It was used as a constant and a function and Turns out decimal point one isn't like a normal constant, right? It's a it's a function It's an object creation really and you know at least until Python 3 3 3 You know the decimal is implemented purely in Python. So it's kind of slow So we had a function that had decimal point one and it was using it as a ran rounding exponent It's like something pretty common if you if you guys are familiar with the the decimal module It wasn't a problem until we had this thing that was returning 10,000 rows of data It was it was rounding each one So it was being run 10,000 or 20,000 times and suddenly this thing was taking a quarter of a second This constant like this thing that was doing nothing, right? So the solution for us was to make a really dumb constant at the top of our file called decimal point one That was evaluated a decimal point one and use it It worked it immediately suddenly your profile like oh there goes a quarter of a second off your profile So this exact problem, and I guess we're gonna do a pull request pretty soon so we do a lot of stuff with decimal this exact problem is in Actually the core Django code if you look at how a decimal thing gets saved One of the things it does is use this constant decimal point one for its rounding It's exact thing and it was it's just recently showed up in our profiles when we tried to save like 10,000 Rows using bulk create with decimal so it's something you just never think about until you you know You do your profile. You're like oh, what's this? So example number two like Reverse so reverse is great like you know don't repeat yourself just you know Reference the URL by name give it give it its arguments. It'll return to the the real URL Reverse reverse is kind of slow You know it's fine if you do it once or twice But we had a we had a situation where we were returning an Excel table with with like 5,000 rows It had the symbol and it had the URL you could get to the symbol It's just an output. We need it so we were running reverse 50,000 times so on on the exact mind you the exact same URL endpoint not different one. It's the exact same one, so it looked exactly like this and And doing it with real data it was taking a lot Longer so instead of what we did is we ran it once with a placeholder argument And then we use string replace to replace that placeholder argument with the real data Kind of crazy you would never want to do it except when you're trying to make something fast and you need to do it, right? so that's kind of two like weird points and I Guess I show these weird points not necessarily to say that you're gonna have these same problems Well if you work with decimal you will probably actually have this exact same problem, but really to show that it's it's it's very Application-specific and sometimes you're just gonna have to do strange things to get things to go fast It's not obvious and it's not something worth even doing until you know that something is a problem because No, same person would do this kind of stuff, right? You wouldn't do it as just like standard from day one you write this kind of code because you think it's the right thing to do We do this thing like in very very choice places that we know we have to do it because otherwise It's like it sounds it's like ridiculous So what if You don't you've done all that stuff your your Python is as fast as it can be you've done all the weird stuff Your sequel is as optimal as you can be And you're still it's not fast enough So what's the the classic solution to performance problems when you've done everything else? So cash right so cash in and cash some more anything you can cash cash it So that you know catching is tricky. Well, there's a saying that says cashing is easy and validation is hard so Yeah, you can catch anything but obviously users are expecting a certain have a certain expectation of how up-to-date your data is going to be So for you the challenge is okay Let me isolate This thing that I want to cash that I know is slow Let me think about how often this thing needs to be you know changes or how often it needs to be up-to-date Is it something that someone won't care about? Even if it's you know if the exact number is six hours old or is it something that's so important that You know it needs to be you know within a minute Up-to-date so once you've isolated these things and then you you cash them. So how do you cash them? So to start with Vango Vango provide some tools There's a view cash, which is a Python decorator you can put on Any view and it'll it'll Read the arguments of the view and it'll cash as a key based on those arguments in that particular view It can be helpful. There's a template fragment cash. So you can it's basically a Just like a template function you put around any any piece of You know templating and basically anything that happens inside of that is cash. So remember that in Django Couries are lazy. So if the query a lot of times won't be Evaluated until the template is like looping over the thing that the courier turns so it's going to cash all that stuff So that can be useful But we've we've personally found a lot of the like more useful Levels of caching aren't don't come necessarily standard with Django For us some of the most useful things has been isolating a function whether it be a function a Method inside an instance method inside a class or a class method inside a class and saying hey I want to catch this So we have interfaces into stocks and the stocks have you know data series they return So a really convenient thing for us to do is have the stock it has a class method or instance method And it returns this data based on arguments and it's a very convenient place for us to say hey cash this for an hour So for that you need to basically a function level cash Something that'll look at that wrap around any function and cash So a really popular caching package is called Django cash use hills which does function level caching and it does a lot of other stuff We we actually wrote one called Django cash helper Which only does function type caching and nothing else because we wanted we didn't want to deal with like other things the package did We just wanted this hundred lines of code that can cash a package So if you if you Google Django cash helper There's very short documentation and you can just use it And you basically put a decorator around any method instance method class method whatever and based on the name of that method The where it is like what module it's in and the arguments It'll create a key and I'll catch it for how long however long you tell it to So and then lastly There's a query caching so query caching is something I think at least I notice get popular in Django Maybe three or so three or four years ago where there's all these packages released That we're trying to solve the correct query caching problem And basically how they worked is they they attached either automatically or either implicitly or explicitly You would tell it you would you would tell particular or M calls to say to cash for a certain amount of time So two of these were one was called Django cash machine And I think it originated out of Mozilla Django cash ops I think it was just kind of a guy like a consultant And they tried to solve kind of two different things one was just caching queries The second one they tried to solve was automatic invalidation so knowing When this query should be out of date and then stop and then clearing the cache When they know it was going to be out of date So I mean this is a kind of a big topic, but I wanted to put it out there So these things do exist and they are used I think Mozilla still uses Django cash machine machine although I could be wrong But you know, they're complex and there's a lot of caveats So, you know invalidation is hard and knowing when to automatically invalidate a query is hard You can still use these tools to like explicitly invalidate, but implicitly There it comes with certain caveats, but it can be kind of extremely useful And I think yeah a few minutes early, but that's that's the end of my talk