 So I'm just an introduction, as was already done. I'm a software engineer at Venmo. My name is Chris Adams. My Twitter GitHub is AdamC64. Interestingly, I'm not the Chris Adams who's also a developer, Python developer, Django developer, and Django contributor who works at the Library of Congress, ACDHA. So that's not me. And neither of us are the gentlemen Chris Adams, a 90s era professional wrestler who's here for disambiguation's sake. There he is. That's neither of us, just in case you're wondering. Yeah, it's kind of interesting having a very common name and seeing the collisions that can possibly happen. So we're here to talk about Django. Django is great. We all love using Django, I'm assuming. And Django, but Django is really a set of tools. And those tools work together in interesting ways and helpful ways, but at the end of the day, what we're using when we're using Django, when we're writing Django code, is a set of tools that work a certain way. So tools are great, but tools can be used in good or bad ways. This, in one sense, is very obvious. But when you're actually getting into things, it might not be so obvious to you when you're using them, especially for the first time. So the Django ORM is one of the tools in that Django provides to us as an object relational model, easy wrapper over SQL queries, which used to be much more tedious to be writing. So one thing to remember is that whenever we're using a tool, especially a new tool, is we should manage our own expectations for tools. So many people approach a new tool with a broad set of expectations as to what they think the tool will do for them. However, this may have little correlation with what the project actually has implemented. So this is just kind of a long way of saying that tools can be misused, especially if you're new to them. So as amazing as it would be, if they did, unicorns don't exist. Tools are not really perfect. They never can be perfect in this imperfect world. So we have to really understand them and not deceive ourselves, especially when encountering new tools, that this tool is the one panacea that's gonna really get everything. It's gonna understand me, it's gonna know me. It's really gonna, you know, I'm gonna be understood by this tool that I'm using. Actually, that's not true. It's your responsibility to understand the tool and to use it correctly. So Django ORM, among its other things you could say about it, is an abstraction layer. It's not only is a tool, but it's an abstraction layer. Abstraction layers are great because they take us away from messy details, but also risky for the same reason. They take us away from messy details. Same reason, okay? So we don't have to do many things that we used to have to do before because we have the ORM, but we might forget or not understand that the ORM's doing things that we don't expect it to do. So don't forget you're far from the ground when using the ORM. You have the ORM API itself. You have Django's implementation. You have Python, you know, maybe you misunderstand some way about how dictionaries or keyword arguments work. You have lots of layers when you're using the ORM, but then you have the SQL itself and you have your specific database backend, which might have its own quirks, might manage indexes in a particular way, might do things slightly differently, load things from, return results from cache that you're not expecting, things, you know, and that depends on your configuration. So you have a lot of layers here. So one thing that I really like about Django is the query set, the programming interface that query sets implicitly give us. And explicitly give us. I mean, we can use the query, we can define query sets, we can do many things with them, we can tell them we want certain things to happen when we define query sets. And so query sets have two features that's worth keeping in mind when you're thinking about them. One is that query sets are lazy. And the other is query sets are immutable. So this is a review for people who might know this already. So what do these terms mean? Lazy means that a query set doesn't evaluate until it needs to. So many people, you know, don't get, you know, really don't realize this. And when you do realize it, it's a great aha moment because you're realizing what's going on behind the hood. When you define a query set, you know, model.objects.atall, what you're doing is you're defining an instruction, which is deferred. It doesn't get executed until it needs to. And there's certain cases when it does execute. And many cases where it doesn't execute. Query sets are also immutable, which means that if you chain them together, you say, you know, I have a query set and then I say dot filter, what's returned from that operation is a brand new query set. However, that new query set, it as much as possible inherits all the features and the properties that you wanted of your old query sets. You can chain filters together. You return a query set, apply conditional, have another filter that only happens in the case of that conditional. You can do all sorts of things. And every time the query sets are returning and you're getting a new one. So you don't have to worry about the references to the old query set. You can store one, you know, return one to another, send one to another function. So query sets are always new. And so, you know, the quick examples here, you know, each of these operations is, returns a new query set and does not actually hit the database. So worth keeping in mind that what's not happening and it's the same level as what is happening. These, however, are situations where query sets are evaluated. There are a few of them. They're in the Django documentation. These are just a sample of three. You know, there are other ones also. Query sets are evaluated when you make a list out of them, when you use an index or range operator and when you iterate over them. So, but the interesting thing to think about here is that you can define a query set very, very early in your view function. It's only at the for loop, for example, in the third case here, that the query set is actually evaluated. The SQL is sent to the database server. So anyway, things to keep in mind when dealing with query sets. Okay. So I want to highlight some patterns which are problematic. And oftentimes people don't even know that the pattern is problematic because they're really doing things that look very straightforward. I'm writing my view function. Everything looks great. I mean, you know, everything looks very straightforward. It looks like I'm using the query sets the way I'm expected to. But actually, when I understand what's going on, certain things, it's easy to miss. So let me kind of point these out here. So we have a sample app, which is a blog app, kind of boilerplate. And I've taken away like various other attributes to these models. These are just the relations between the models. And one of the features of Django model objects is that when I, well, I'll just go through the examples that we could save. So we have basically blog post and a comment on a post. Submitter, which is a foreign key to the user model post which has foreign key back to the blog. So you're posting on a blog. And, you know, likeers, many of the people who like that post, okay, and comments on the post which are done by a particular user. And, you know, the foreign key back to the post that they're the comment on. Pretty basic model. Pretty straightforward. So here's a view which can be used to just generate a list of the blogs that are on the site. Maybe this is a management view or something like that. Looks really simple. Here's the template that renders that view. Looks really simple. We have a reference to the blog detail page. We put the blog name there. And we say submitted by the person who submitted it. Very simple. This seems like just home run, like really easy thing to write, the push it to production, get this thing working. And it generates something like this, okay? As would be expected. All right, so. And now I just wanna talk about the detail view for a little bit. The detail view is a way of seeing a particular blog. And I, you know, get the blog or 404 if the blog doesn't exist. I request the posts that are for that blog. And then I render that the blog detail HTML. This is a little bit more complicated because there's a lot of different features that maybe my users wanna see. They wanna see who likes the blog. They wanna see the comments that are on the blog. And so this is a little bit more complex. So and it generates something like this. Okay, you know, I just used some sample, lower mitzvim generator. So, you know, we have comments, we have people who liked it. Okay, so you, this looks a little bit more complex in terms of how this is generated. So now as general developers were left thinking, okay, I've done my job, right? I've gotten something out. I've completed the work, right? It works, right? But really at the end of the day, as I said, the Jango ARM is a tool and the tool is functioning a certain way. One of the main interfaces that we have towards our data is through the SQL query language. And Jango is generating, the ARM is generating SQL. So how do we find out what SQL is being generated to in a particular view? And the problem is if you can't measure it, you never know if there are problems, right? You know, how do you know? There's information lacking. And we can theorize based on what we think it's doing, right, this magical, I know what the ARM's doing. Like, you know, it's just super easy. You know, I'm using .all, so it's doing my query there, right? No, it's not. There's lots of false assumptions we can make. So really in general, and this goes not just for Jango, it goes for most tools and most technology, you need to measure and you need to get an actual printout or a display and transparency into how the tool's functioning and what it's doing. And as precise as possible. The more information you can get here, the more power you have as a developer. And I think everyone would agree with me here. So how do we get this information? You know, we can't be naive with our tools. How do we get this information? Well, one really, really cool project is the Jango Debug Toolbar. If you're not using it, I highly recommend you use it or have some way of getting information that's equivalent to the kind of things that this provides out of the box. So I'm not gonna go over how to install it. They have good documentation. You can do that. It's relatively straightforward. So let's use the Jango Debug Toolbar to take a look at our pages. So first we'll take a look at the blog list page. So here's our blog list. And notice that one of the cool things at the Jango Debug Toolbar is it gives us a panel which is, it basically injects HTML JavaScript into the page. And also the data about the execution of the page. And it gives us these tabs that we can click on. So we need information about the timing that it took to generate the HTML, details about the settings, the HTTP headers, the request object, okay. These are, I speak for a long time about each of them. I'm gonna focus for this talk on the SQL. And notice that there's 52 queries being generated and being executed to actually generate this page. Which seems like a lot for just a list, right? So we did the thing, right? We used the OM. It looked really straightforward. We used it. And what happened here is we actually, we didn't realize it. We generated quite a lot of queries. And this is a bad thing because there's a network round trip that has to happen. The server has to send the query and also receive the response. And the database server has to process each individual query. So there's a lot of extra overhead for having all these things happening. And, sorry, if we click on the SQL button, the page actually, this general devote tool actually gives us a list of all the queries that happen. Which is kind of cool when you think about it. Here it is. It gives us a timeline. I'd go off a little bit more of the things just for the sake of displaying, but you should definitely check it out. So, you know, now our next question is great. We have the queries here. Where are they being generated? And actually you can click on the plus and you can get that information. The plus to the left side will give us the full query and also the actual line of code where the query is generated. And I should say here, this is not a query set that's generating it, that's already happened. In this particular case, these individual select statements are being executed through the foreign key, log dot submitter. So Django, the query sets are lazy and also the model objects are lazy. The foreign key relations aren't gonna be accessed, aren't gonna be loaded until they need to be or if they need to be. So that's why it saves in terms of an optimization. Maybe you never reference this submitter object. And so you don't need to have information about the off user model object. So here it is. Blog dot submitter is being referenced that's generating these many, many, many individual queries and the more objects you have, the more queries you have. So this is actually a very bad situation and there should be a way to do this much easier. So this is where select related comes in. Select related works by creating a SQL join and including the fields of the related object in the individual select statement for the model itself. So the off user is gonna be joined with the blog objects in this case. So the SQL is gonna return back multiple times as many off users as there are. And you think, well, is this bad? Well, it's actually much more efficient because you're batching it, you're doing it in one shot and so there's only one network back and forth to get that information. So it's actually much better, much fewer queries. So it gets a related object in the same database query and however, to avoid much larger results that result from many relationships, so like a many to many field as opposed to a foreign key or maybe a reverse foreign key where you're getting the opposite side of the relation, select related won't handle those cases. It'll just handle, think of it in terms of foreign keys and one to one fields, it's helpful and when you're gonna be iterating over them in a loop. So here we use select related. We tell it we wanna select the submitter of the blogs and you can have commas and have multiple fields which it will join together and get all that information. So now we do it and notice two queries and there they are, okay? And notice the inner join on the query. So we've just completely reduced the number of queries and even if we have much, much more users and much, much more blogs, this isn't gonna increase just gonna be two queries. So this is really great optimization and a great tool, especially when you're using loops. So all right, the blog detail page now is a little bit more difficult. So now notice we start with 44 queries and if we take a look at the queries in this case, it's a lot less straightforward, right? You don't just have a list. And what I usually do is I start from the top, open my code and I actually go down and see where these things are being invoked and I just try to be intelligent, try to identify patterns, patterns are good. The same thing's happening over and over again. Might be a group of things happening over and over again. And in this case, I'm just gonna reveal it for you here. The patterns that you identify if you take a look at these things are that there is a submitter far and keeping access on the comment. Okay, so now we're talking about comments that are being generated and who submitted that comment. And there's also a query on the likers attribute, which is if you remember is a many to many relation as people who like the post. So we have comment.submitter and post likers. So prefetch related is an optimization that's used for this case. And instead of using an inner join, what Django is gonna do is gonna prefetch those relations, those many to many relations. And then it's gonna do the operation we expect of our model and it's gonna join them in Python. And that's actually more efficient than having database do it for you. So there's a reason why select related doesn't do this by default. Prefetch related is gonna prefetch those many many relations because it could just result in a massive result set. So it's much much better to use it this way. So you remember prefetch related is for many relation that has many related members. It does a separate look up for each relationship. Does a joining in Python. And this allows it to prefetch many to many and many to one objects in addition to the foreign key and one to one that is in select related. That select related helps you use, helps you manage. And it also supports generic relation, generic foreign key. I don't know what version of Django that is. That it does support that. I would check that. But you can use it for those kinds of things. So we put it in here. You can use double underscore syntax. In this case, this is the most efficient set of arguments that will help us in our particular example. There might be more that I overlooked. I didn't spend that much time trying to find the optimal, optimal, but this simple optimization, this one line of code is gonna save us. So if you see it's gonna prefetch through the comments table and get the submitter, it's also gonna have the comments information, which is gonna be helpful for us. It's gonna implicitly do that. And it's gonna select the likers. And one line and we're done. Well, we only have seven queries now. I'm sure you can get this even smaller. But one line, just amazing amount of work that you can do. If you notice, prefetch certain things before others. So it's just really, really helpful pattern to think about. And to bring into your own toolkit as you're working to make your views better. So in summary, the query set API method, select related and prefetch related, automate some of these best practices to avoid extra queries in views and in templates. And you select related for one to many or one to one relations, and prefetch related for many to many or many to one relations, including the reverse foreign key relations. So, yeah, it's basically it. So thanks, this is pretty straightforward. I hope this will allow you to start using these tools more effectively. I have a code on GitHub, so please clone it. And yeah, thank you very much.