 Talk came out, came about because I was trying to, because I was trying to solve a problem. I was creating an API and part of what the API had to do was to basically, the API had to provide a whole bunch of information, right? So maybe on the order of like half a million records, there was another consumer that had to consume about half a million records, right? So this is kind of like where the talk came from. Because I ran into some performance issues, right? So one of the things that I use is I use SQLize. It's an ORM. I mean, are you guys familiar with SQLize and ORMs in general? This is for Node.js is backend, right? Okay, so ORM is an object relational method. And basically what it does is it helps you to basically access relational databases. All right, so usually if you want to page through a result set from an SQL database, most of the time what people like to do is they like to use the offset method, right? So basically they set a limit and then they set an offset. So the thing about it is for small result sets, actually that's okay, most of the time it works fine. Okay, but if your results set gets larger and larger and larger, what happens is for the first page, they find the first page and then after they iterate through the first, let's say 10 records. Let's say if you have 10 records per page. Then for the next page, they have to iterate through the first 10 records and then the next 10 records and so on and so forth. So every page that you query further and further down the road takes longer and longer to come back, right? Yeah, so basically that's it. Okay, so this is kind of like what happens if you use the offset method versus the seek method. As you can see, the more pages you have in your result set, the longer it takes for the offset method to actually return your results. So the thing about the seek method is it's all basically about using indexes in the database, right? So it's very important. So the idea here is to use the index so that you can get directly to the page that you want and it's usually demarcated by some kind of, some kind of, let's say unique ID, right? Eventually the idea here is that you want to preserve the sort order of it so that you can, I can go to this part and then after that, I can go to this part and then I can go to this part, right? The thing about it is because of this, you have to modify the where clause in your SQL, all right? So let's say for example, you have a where clause with some criteria, you have to add on more criteria that uses that particular index that you are using to actually segregate your data, right? To basically maintain the sort order of your data so that you can page to the exact point that you want, okay? So some, usually we, let's say for example, you have a list of, I don't know, posts, right? And most of the time for the list of posts, you would like to have, you would probably order them by date and time, right? Of course some posts may actually have the same date, let's say if it's post by day, for example. So to maintain the sort order, not only would you use an index that is based on like, for example, the post date and time, but also for example, the primary key. So eventually, also most of the time, you use the primary key plus something else for the sort order, okay? So the thing about the seed method is for the very first query that you use to enter into the result set, basically it's just as per normal. But the only thing that you do do is that, let's say for example, you want to return 10 results, I would add, you would add on one more, right? So that you know there's another page. Okay, the thing about the seed method is that you can sequentially get lots of pages, but it's very difficult for you to say I want page five and then I want to go to page 10 and something like that. So you go one, two, three, four, five and throughout. So the reason why we add on one more is so that when we actually take one more record than we need to actually indicate whether there's actually more data that whether there's another page, whether we can page, whether there's another page for us to extract, okay? So as I mentioned before, the last record is used to determine if there's the next page and then this is how we construct the subsequent queries, right? So if you look at, let's say invoice 11, they find there's an invoice date there and then there's an ID, that's the primary key. So we have to construct the query like this. So for example, you have your where clause, which you have your general search criteria, okay? And then you have to add on this portion here, which kind of helps you seek to that middle of the result set. So as you can see, what happens is we select the invoice date of invoice 11, right? And then after that we have this thing here, okay? That basically says that select don't select anything, don't select anything before that ID, right? So it's to make sure that you don't pick up any records that have the same invoice date as invoice 11, all right? Okay, it's kind of technical, okay? So, okay, so back to SQLize. So what I've been talking about here is basically just all SQL, all right? So as I mentioned before, SQLize is a popular object relational mapper that, you know, people like, I mean, node.js people like me like to use, right? And this is how the code would be like, right? If, for example, we're using SQLize. So for example, we would have the, you know, we would find all and then you'd have your where clause. And then of course, this is how you would code it up using the various operands in SQLize to actually seek into the result set that you would require. Okay, so I did some benchmarks, right? Between the offset method and the seek method, right? And so I generated like 100,000 randomly generated records and I kind of took them 100 records at a time on a little VM that I had on my machine, right? And so basically this is the kind of the results that I had. So I didn't expect it to be that much better but it actually pretty much is, you know? It's almost on the order of 60 times like. So one of the limitations of using this method, right? Is that because you have to add things to the where clause, if you want to change the sort order, you basically have to change your where clause, that part of the where clause here, right? That allows you to seek into the data. Okay, and as I mentioned before, you can't actually jump to an arbitrary page. So if you start at page one, you've got to go to page two and page three and page four, right? You can't go from page one to page seven or something like that. But it's probably okay if let's say, for example, you are implementing like a mobile app, right? And then, you know, somebody just scrolling through their posts and all that. So, you know, nobody jumps, right? They, you just have this infinite scroll and then, you know, the mobile app just keeps on querying the API. You can backwards but you have to have, you have to construct two queries. You have to basically say, I want to go forward and then you have to construct a query that goes backward. This is kind of complicated but it's possible. So if you, let's say, for example, you wanted to have like a, you want to go forward and backward, it's possible. Okay, so basically I was doing some research on the internet and that's how I found it, right? This, so this guy, he's a database guy. So he kind of like, you know, kind of, you know, gave me some inspiration on how to do this. And I have a gist implementation for the SIG method using SQLise. You guys, I mean, this is available on the Talk.js GitHub so you can take a look. Alrighty, so I've come to the end of it. So I just want to, any questions? Sure. The thing that's making the queries, is this from the front end or the back end? This is from the, this is from the back end. So what? Sorry, is there a requirement on the back end to be d-stateless? Excuse me? Is there a requirement of the back end to be d-stateless? Is there a requirement? Yes, yes, yes. So you can't do like an initial query that just gets the list of all the IDs and then scrolls through that with a window. I mean, you, okay, so let's say for example you want to get a list of all IDs but then you have to get like, you have to, you still have to, you still have to suck that all down. So that's quite significant. It all depends on how much information there is there, how many IDs you have. So like I said, this one has like 500,000. You know, the result set that I have is about 500,000 and we need to page through. So even if we suck all of that down at once, I think it's a bit much. I was referring to just the IDs. Oh yeah, just the IDs. I mean, even then, would you really want to have all the IDs there? Because what happens is once you do a query, you get all the IDs. Let's say for example, somebody's adding something to the end of the result set, right? Then you'd only have those IDs, right? So let's say you get like 3,000 IDs, somebody's adding on, somebody's inserting a few more records into it, then you'd still, you need to find a way to actually get those IDs, right? Those IDs that people have inserted inside. Yep. The time, thank you. The time you saved like performance files, how much time did it cost to develop them for us? And especially this is a simple query, but once you start having like filters, how complex can it get? Is it like really something you recommend to everyone or only for certain use cases? I think that if you require, let's say if you need to, if you are going to do a query, right? And then you're only gonna look, maybe the first hundred or first thousand records. So every time you do a search, let's say you're doing a search and you're doing a query, first hundred records, then you don't need to do this, right? But let's say for example, you want to feed information, right? I need to query something and I know the result set is gonna be quite large, then you would wanna do something like this. But as Brandon mentioned, I mean, the requirement was for it to be stateless. Oh, anyway, thanks for all the questions. Any other questions? Have you considered using different satellites that might support the queries in a different fashion, for example, like a time series, rather less? What happens is, yes, I mean, there are always different ways that you can use other kinds of databases, right? So for example, I mean, let's say for example, for search, so if you can use normal SQL queries to give you search or you can throw everything into a elastic search, for example, right? But, you know, this elastic search is kind of big, it needs a machine or that kind of thing. So I think we didn't want to complicate the stack anymore than we had to by having like different databases because you said to mel everything together anyway. And so that's why we use this method. Any other questions? Oh, one more. Yeah, I think for this problem is actually sort of like synchronized, we use the sign and count form function. Okay, so the find and count all function is literally your offset. If you have the find and count all, that uses offset method. Yeah, and then if you declared your complicated indexes or multiple columns, then you would guarantee that it would sink to the right place immediately and your code could be very simple and present the usual way in a sense. You don't have to lock around and in the complicated conditions, right? Because using that function, you have the entire column for the result set and you also get the records that you need. So you just compare the sum of the return results and the amount of skip versus the number of records that they actually match for filter, you would know whether or not you need to continue to make the same mistake. That means if you offset x and y, but you know that your filter condition returns that, if x and y exceeds that, sorry, if x exceeds x plus y, then you know that you have to continue to patent it. Yeah, I mean you, okay, so the thing is this, right? If you have a very large result set, if you do a find and count all, if you do a count all, the count itself takes quite a while. All right, the count itself probably takes, I mean, depending on how large the result set is, like I said, if the result sets are relatively small, it's not a big deal. But the result sets are relatively large, right? Okay, then it takes a while. As you've mentioned, the find and count all, the find and count all, basically that's what I've used. I used the find and count all to do the offset method and that is what has returned me those kinds of results. So I guess what I'm saying is that if you have this particular use case, then this is one way to solve that problem. I mean, like I said, there's actually many ways to solve issues. I don't think there's always one way to skin a cat, right? But I'm just saying that if you have, if you are wedded to, let's say SQL database, you have to put your, you know, for legacy reasons or whatever, then this is one way to solve the issue. I think like if you just have the right competent index, but in the right order, that matches the query, I think even the time could be really quick, yeah? But you can take this offline. Yeah, sure. Thanks. Oh, yes? That's a quick one actually. Is this, how is it different with cursor? Oh, it's nothing to do with cursor because as Brandon has mentioned, it's a stateless thing, right? So the thing is you could, actually if you wanted to, like you said, you can use a cursor. One of the reasons why we may not want to use a cursor is that when you do the cursor query, you tie up, you can tie up all the records. I mean, they may lock various records or, you know, so if you have a lot of results to actually pass, then what would happen is some other function or some other process may want to write to a record which you have locked up with your cursor and then also you get a database lock, right? What happens when more data comes in and you need to insert rows in Whitby? Assuming, I think it works well when you keep adding rows to the end of the table, but what happens if you want to insert a Whitby 4 and 5 and export a 5? Do you keep renaissance the idea over and over again or? Okay, so for example, usually we try to make use of, for the remaining the sort order, we try to make use of IDs that don't change, right? So let's say, for example, if, so that's why the primary key is usually an ID that we use to help maintain sort order in addition to whatever other sort criteria we would like to have, right? But I mean, I get your point. So what happens is, let's say for example, we are sorting on some date. What happens if somebody adds on a date that's in the past that we've already gone past? Well, then it's in the past, then it's kind of, then you won't pick it up until you, unless you go through the entire process again. All right, thank you guys.