 Thanks for coming and thanks for having me. I'm really happy that this talk happens in the morning because I'm fresh. I Will be doing life coding. So this is gonna be difficult both for me and for us. I Will make it I will try to make it as easy to follow as possible, but please pay attention and If anyone has problems seeing the letters, please let me know This is the terminal Is the font big enough? All right All right, so let's dive in My name is Michael. Hi. Nice to meet you. I'm going to be talking about caching in microservices and first of all a quick Intro into what we will try to do today This is the setup I prepared. It's a fairly standard setup with a microservice in an API gateway at the front So the browser will send the HTTP request to the API service This is the API gateway that is exposed to the outside world and then Internally this this service has a client HTTP clients that passes the request To the underlying service and this service has a database in it. It's postgres You're running on top of SQL alchemy and then this said that to the service will Return the response and the API service will pass that response back to the browser and the thing we are trying to do Is to implement caching in such a way that the browser doesn't need to know about that so Assuming that queries to the database are expensive. We would like to avoid them as much as possible Without requiring anything to be implemented on the browser side and The way we're gonna do that is using the Itaq mechanism. Oh, sorry about the colors. I don't know what happened here So typically what happens what I just described the browser sends a request For a to-do list with some kind of ID To the API gateway then this is past Well, first of all, the API gateway needs to check if you have access to that list So if you are a collaborator because people might use the same list work in a group and This requires querying the database and then API service will ask to the service about the list itself and this also requires the querying the database for the list itself or the all the entries and so on So the way itaq caching works is that during the first call When API service issues to get request to to the service On the to the service side We are supposed to compute the hash of the response after producing it and send it back to the API service This is done via HTTP header code itaq and it's a basically just a string But it should uniquely match the response and API service is going to store that and then pass the body back to the browser But the second time the browser asked for the same thing The browser behaves exactly in the same way and the API service will first check if it has any hash stored already And if it does it's going to send a get request to the service But this time it's going to include an if not much header with this string This allows API service to verify that the cache is fresh and the underlying document did not change in the meantime To the service is supposed to verify that and if the cache is okay and fine The document wasn't modified in the meantime then to the service is going to reply with 304 not modified status and not include the body And then API service will fetch the body from its own cache and then return it back to the browser so that's the Overall view and we're going to try to implement that live So let's get to work What I have here is This is the to-do service. So we're going to start with that. This is the backend service that has the database It's done in the fast API. This is the service and here somewhere here is a request Like the view that returns our to-do lists and I figured the nice way to Capture the response and cup compute an attack for that is to implement the middleware But in order there's one important thing in order for it to capture the whole response We need to make it a pure ASGI middleware not a fast API one Because the response might be streamed So when your service starts sending back the body it might do it in chunks and We need to retrieve all the chunks before computing the whole hash for the whole body So in ASGI parlance a middleware is just an asynchronous function But this function needs a state so I'm going to wrap it in a class Because I'm lazy I'm going to use our data classes everywhere where I come so we need to think we need to Know the application and we need to know the The cache storage itself and the cache storage itself is also going to be a data class For now, it's going to be empty for simplicity So now when you have those you can install the cache in the in the backend service. So let's do this Memory cache and cache middleware and now here we're going to instantiate the cache Just like that and then install it Middleware this is going to be cache middleware and we're going to pass our Cache as the argument And then whenever a request comes it will go through that middleware and the way it happens is that ASGI server Will call Will call this middleware basically So we're going to include an under call for that And the arguments here are going to be scope Whatever that is and Two coables one for receiving data and one for sending data back to the client We don't really need to know the details about about what the scope exactly is but we know one thing the scope Contains type because in ASGI between the application and the server. There are a couple of things happening And we are interested on the HTTP traffic. So we're going to check for that first So if type is not HTTP We're going to just call the application passing the arguments and return Now if if the scope is of HTTP type, then we know that we can create a request out of it request object from the scope and then having that We're going to pass Everything as is to the application But this time we're going to overwrite the send method so that every chunk of the body goes through our function And for that we're going to create another function called Cache send and for that we need Access to the cache we need access to the request itself and the original send And this is going to look like that first it needs the cache it needs to request itself and It needs the send function and Then the callable is also synchronous and it gets something called the message That is this is message is like a part of the body or part of the response that it is supposed to go to the browser To the client the HTTP client Now those messages we are in HTTP scope because of the type we checked earlier So we here we're going to receive two types of messages. It's either be a response start or response body so first of all There is a state machine internally So we are supposed to get this response start first and then body a couple of times if the response is chunked So if the message type is HTTP response start then we're going to just store it and obviously we need a field for that and it's going to be a message or none starting with empty and And that's it So we are going to accumulate that but not send it to the client yet and And then if it's not HTTP response start then we know it's an HTTP response body So we can start accumulating the chunks Obviously we need something for that as well and this time it's going to be a list of messages It's going to be empty at the beginning and after all of that Right and that the last chunk contains a field called more body set to false. So we're going to check for that. So if message Says that there is more body coming Then we're going to just return and we're going to keep accumulating So eventually the application will send us a chunk that has Sorry if more body is true Eventually the application will send us a last chunk that has more body set to false And at this point we know that we have the whole response Stored in memory. So if since we do that we can well now we can produce it to the client And before Sending it back to the client we can modify it a bit for example to include this eta header But first of all, let's send it back to the client. So it's it's transparent So when we did that everything should be transparent The middle where I think is is already installed here. So we're going to try it and it doesn't work Of course Let's see what we did wrong Body yes, thank you life coding. Thank you audience Yes, and this is the response everything is transparent nothing happens But we what we can do now is we can compute the Well, first of all reassemble the whole response from the chunks So this is going to be that and then a computing intact for that and we're going to base 64 encoded immediately On the body. This is the digest So this is our attack and we can now Say that our response start headers We're going to add that header the e-tag header the one we need to the response and Also, we're going to store whatever we have just produced in our cache and we're going to store What we need for that we need We need the request we need headers of the response we don't need the body here So let's let's let's imagine we have this and we're going to do that in the meantime Later on and any duck So the the the property for the response headers is going to be trivial Because we have the response start we spawn start contains all the headers we need so we're going to just Wrap it in a nicer way To convert that from row AS GI data structures into something that you can use later on And we're going to store it. We don't have the store method yet. So let's implement that So this is the store store takes self takes the request takes response headers and takes any tag and First of all, we need the cache key So let's create that from the request quest and then Store that in our cache as an entry and The entry is going to contain it ag and response headers So we don't have the Key method yet. So the cache key is going to be the method a tuple containing a method and URL for now This is not going to be sufficient and you will see that later on But let's start with this and see what happens And the entry is an e-tag. That's the string string and response headers Headass Whatever happens here We store stop complaining, please And we set the header so everything should work Let's see if it does it doesn't self-response Headers not row Memory cache has no attribute cache Obviously as I forgot to add it This is going to be just a dictionary of tuple to cache entry And it's going to start empty by default and now we have something now Supposedly our backend service now Computes the e-tags and to return them back to the client now. We don't see those because we don't blog anything So let's move to the client and see if the clients really sees that This is the client and the HTTP client that is already prepared for some work We have a session Like HTTP client session that is overridden and It customizes the cache response and cache request. So we're gonna try Adding our code here to see if the e-tag is there now. We're gonna start with Just calling the parent start Now when the response is initialized and IO HTTP does everything it needs We should have half you should have access to all the headers that have been passed By the service to us So let's see if the attack is there and if it's there, we're gonna just look it and see Let's see if that helps It is there. Yay Right. So we are the client. We have just received an e-tag So we need now our own cache storage in order to store the Response including the body this time So because we are lazy, we're gonna copy everything The cache is gonna behave Pretty much the same way With a couple of differences. First of all, the response is gonna be the whole response not just the headers Now here we need to change the class because in IO HTTP, it's called request info. It does the same thing Response now we need a response and Key is gonna be Constructed in the same way. This is important like the client in the service need to have the logic that computes the key in the same way So let's keep that and now we need some bookkeeping in order to Add everything like like half this cache accessible So let's do that We're gonna do the same for the request to make our lives easier later session needs the same thing and We're gonna do partial applications so that all Responses request objects created for the session point to the same cache and The way this this session is implemented in on the client side is that it creates a new Client session for every request that comes in from the from the browser and it's also a middleware So we're gonna add the cache to that in the same way that we use as we did before and here past the cache right so Whenever we install the session middleware, we need to pass the Memory cache to that and it's gonna be propagated down to every response and request object And in order for that to happen We need also to instantiate the cache on the client side this time And this is the session middleware that we need to augment with the caching So back to the client We know that the e-tag is there so since we know that we can now access the cache and store The thing we need and the thing we need to pass here is Request info fortunately, we have that because in a IOH tp all response objects contain also a pointer to the Request that caused the response We're gonna store self and they took itself. So let's try it Yeah, nothing broke. So this is supposedly happening So now we have stored the the cache on we have stored the E-tag we received from the server on the client side and now in order to do the second call Whenever we issue a request we need to check if there is something in the cache and Instead of adding this e-tag header this time we need to add if not much on the client side So let's do that if there is an entry Now we need to get method. So we're gonna implement that in a second then We're gonna update the headers and we're gonna Send the e-tag we have in our cache in order for the server to verify that and the get method is gonna be trivial requesting for the returns either a cache entry or none and We need the key constructed in the same manner and self cache get We're gonna just return that Nothing really interesting So now whenever we issue a request everything should work But this time on the server side, we should be receiving if not much header. So let's check if this happens We're gonna check that in the middle where as well right here because right here We have created the request and supposedly this request contains if not much header. So let's see if it's there And if it's there, we're gonna just log it for now to see if it if it's if it's really there So the first time around it was not there Because it this was the first call but the second call indeed contains that and it contains the same It act that we have just generated in the first call So this is good and we're gonna check that information now So the client says that I have a resource and the hash of that resource is this and that so please check it If it's the same and or the resource changed in the meantime So we're gonna check it So if the client says that it has some some hush, we're gonna see if there is a matching entry in our cache and That Whatever the client send is the same as what we have on the server side For that we need the get method. So we're gonna just copy that because lazy is good It's needs request and it works the same way and now We know that whatever we have is the same as what the client has so we're gonna simulate All right. Well, we're gonna just respond with 300 for not a modified and we're gonna do that directly by just sending the messages in ASGI So the first message as we know is HTTP response start It needs to contain the status That's gonna be 304 and it needs headers and Hedders we're gonna take from the entry But we're going to encode them in ASGI as row headers and Then we're gonna send Another message that this time is HTTP response body and we know that there is no more no more body And we're gonna return So this means that the second call should now return a 304 with empty body But only if if not match matches Whatever we have in the cache already. So let's see if it's true This is the first call and this is the second call and nothing happened No, why because it should let's log it first So of course get request if not matches entry attack Let's see, let's restore the Log should be working. So the if is not there It doesn't work this part worked as I remember It does So for some reason entry is not there Entry is there Alright, I must have encoding e tag in cash entry should be a string and it's a it's bytes This is because I forgot to decode it here And now it should it should work Restore everything here and now try The first call and the second call now This happens It is correct and everything is alright, but the client now says that the response is invalid. Why? Because the response now contains no body and the client expects that to be some kind of Jason because the client Doesn't retrieve the body from the cash yet. So we need to do that So this is the response and now we need to check that if self Status is 304 and We have an entry This means that we now need to Pretend that we received the same body that we have stored in the cache previously So let's let's do this Just by cheating And to response status we need to do the same thing for the reason This is the string describing the status and now the body and otherwise We need to check the data. There's one thing that we also need to check here. I forgot about that We need to check that we cash only to hundreds Because otherwise everything is gonna break when we try to send posts or patch requests And we probably also need to check that the method Was to get and so start of these 200 So now now let's try again. This is the first call This is the second call and now if you look closely you can see that during the first request We did query the database and the in the second call. We did not this is exactly what we wanted and Supposedly our work is done here. Well, no, why why because Right here when whatever we issue a request we identify ourselves with this header so we are asking for lists owned or Accessed by this particular user so If we say that we are a different user Our cache is gonna work badly. It's gonna leak data and it's gonna return the list That is the same because we cashed by URL and method We don't cash by any additional headers. This is wrong So let's fix that we have nine minutes so we can fix that Right It works or it may work in HTTP by abusing or using a header called vari there is a header in HTTP that says that The content of the response the body or the format depends on some headers sent by the client and This is exactly what happens here. The response depends on the Header X user header sent by the client and this is even visible Right here because we query the database and Not only based on URL, but on the value of one of the headers. So let's do this Let's set this header To X user But this is not enough. We need to now include that information in the cache key that we use So let's go back to the cache and see what happens The cache middleware and everything here stays the same Because we do to the cache itself and that the mechanism we send the whole request so we have access to both request headers and response headers So what we know that this should include The headers here and we're gonna add them to the key We're gonna add them as a series of tuples. The first part of the tuples is gonna be name of the header and the second Second part the second item is gonna be the value of that header in the request And now we need to include that everywhere Whenever we access the key, this is gonna be a bit of a problem That's we're gonna solve the problem as well now Whenever we store something We know that the very very headers comes from response and this is gonna be This might be empty And we're gonna split that by the semicolon because we might have multiple and we're gonna Filter out empty Empty headers just in case Now this is the store method that was easy now the get method is more complicated because now we don't have access to the response so we need to Get these very headers from somewhere Fortunately, we can just memorize them This is a dictionary And we're gonna memorize them by URL assuming that the API is sane and the same endpoint returns the same variable all the time This is a big assumption, but we control both sides both the API client the API gateway and the underlying service So we know that we can be sane and here. We're gonna just retrieve it quest You were all it might not exist. That's gonna be an empty list by default Right So now we incorporated the vari header into whatever cache We do and now it is supposed to work So whenever we get the list for user full Once and twice this is cached and there is no query to the database But we do that for the data user. It's gonna return an empty list and we're gonna ask again For the for the another user. It's not gonna ask the database. This is good Let's see what happens here. Oh When we ask For one user and then for the second user Like switching between them it always hits the database. Why? Well, this is because we only implemented these vary headers on the server side And I said before that our client needs to construct the key in the same way if it doesn't It's not gonna work as well So the only thing that is left is adding the same mechanism to on the client side So let's do that and hopefully we'll be done And this is gonna be really really easy We just need to copy the code headers list of strings The same thing needs to happen with the default ticked, sorry right here Now the store needs to be modified in the same way But this time we get headers from the response object that's the only difference and Then in get We do the same thing Very headers and we need to pass that To our key function, right now. Let's see what happens now We ask for the list of full it is cached very good We ask for the list of bar it is cached very good And now we ask for full and it is cached So everything works Everything works until we change the Object itself and this thing I will post on discord for this talk because I'm running out of time Caching validation is gonna be not so complicated as you think And I'm gonna send the link to the wrapper when this is implemented But for now, let's let's stop here Because I imagine this was Enough for everyone certainly for me right, so Unfortunately, I will not be doing a Q&A here, but I will do any Q&A in the hallway because I really need coffee now So feel free to grab me immediately after the talk if you want or on discord or whenever during the conference Thank you