 Hi everybody thank you very much for joining we are super excited to have this unique experience talking at the DEF CONF so my name is Sean Paz I'm a solution architect at Red Hat IGC and with me is Orfridman I'll let him introduce himself Or. Hi I am Or, I'm part of the RGW team that's it. Thank you very much. So we are here to discuss with you about how you could leverage NGNX in order to use it as a CDN layer for a Sephardous getaway. So moving through the agenda for the next 25 minutes we'll start off by talking a little bit about CEPH and what is CEPH in general and we'll dive deeper into CEPH's architecture in order to understand how we can use the CDN cache layer in order to gain more performance in our cluster. We'll talk about what are the reasons that we need to use a CDN in the first place and we'll also show you how we have integrated those two components in order to get much more much more performance from our SF cluster. We'll show you some performance statistics so you could see the the difference between using this feature and without using it and we'll also have a demo that you can visualize it in your own eyes and you can see the how the performance differentiate. We'll end up with having a short Q&A session so it's important for me to say there are no stupid questions feel free to ask any question that you have and we'll be happy to answer it at the end of this session. So we'll start off by talking about CEPH. So what is CEPH? CEPH is basically a distributed software defined storage solution which is unified and by unified I mean that CEPH supports all the relevant storage protocols that are in the market at the moment. So by using CEPH you could have block file and object storage all located in the same cluster and we see CEPH being used in various use cases mainly in modern workloads such as cloud infrastructure, data analytics, major repositories, backup and restore, etc. So diving a little deeper into CEPH's architecture we see that CEPH is basically a software defined storage solution which means that each component in CEPH is actually a piece of software. All the software pieces are eventually talking and interacting with each other in the cluster and this is what gives us the main clusterized experience when managing the CEPH cluster. Looking at the components we see that we have two major components which are first of all the mounds which are responsible for the authentication and the cluster management and those these are responsible for storing the actual data that is being saved to the cluster. There are some additional components but they are less relevant for this specific presentation and it's important to say that the fact is that CEPH is an SGS solution so basically each one of those components is represented mostly as system D services or in the latest versions of CEPH containers as well. Looking at the data flow we see that we have three different storage protocols which are Redos Gateway for the S3 service, RBD for the block storage and CEPHFS for the file storage. So each one of the objects that is being read into each one of those storage protocols is being divided into a different set of pools. So for example if I'm a user and I'm trying to interact with the Redos Gateway S3 service I will end up with using a certain set of pools and each object that is reading to a pool is actually getting its location calculated so the calculation is being made at the client level so objects are reading to the cluster for example to a specific pool and they get a unique hush that will eventually define their location in the cluster. The result of this calculation is actually the PG which is the placement group that the PG helps us distribute the objects in the cluster so each object is being divided into a different PG or we have a group of objects that are relevant to a specific PGID and this is how we distribute the objects between those and with the PG's we also control the data protection strategies that are now divides into two protection strategies one is replica where we save the object multiple times in the cluster and the second one is the register coding when we chunk the object and we distribute it among all these in the cluster. One thing that is important to mention though it's the crush map the crush map actually holds the cluster state and the topology so basically when one of the one of Ceph's components fails whether it's a disk a server a rack or even a data center the crush map is getting automatically notified so this is how Ceph knows how to replicate the data into another location in order to maintain our wanted recovery. So why do we need CDN anyway? So CDN just for record is our way to cache objects and fetch them into an external component which is not relying on the Ceph cluster and the main reason why we should use CDN is first of all because it offloads the intense workloads from the cluster so basically we saw in the previous slide that two users for example that are trying to reach the same object from the same bucket are eventually ending up using the same set of disks because this object has its own location so as the the number of user grow we will end up saturating the disks so having the CDN can help with that. Another reason is because it decreases the latency dramatically think of it when we take a look at our cluster performance eventually it's being directly affected by the weakest part in the chain so the weakest part at the moment is the disk layer so right now when we take the objects and we fetch them into an external component we actually are allowed to retrieve them from RAM which decreases the latency dramatically. When decreasing the latency I mean we all know that as the latency decreases the number of operations that we could let's say interact with the cluster is a lot higher so we could gain more performance when getting the CDN deployed in front of the Redis Gateway. We see a lot of use cases where customers are deploying their applications to the edge or the far edge and eventually their storage systems are being located in the central cloud so there is no logic in having your application closer to the end customer and eventually the storage is located in the in the central cloud so in this way you could take the CDN and deploy it closer to your application so the end customer experience will be better and the response time will be lower. Last but not least you could also have you could also use a read ahead mechanism where for example a generic use case you need to stream a video one gigabyte size so you will end up with fetching and reading a different set of bytes each time when streaming the video so this is how the cache layer could help you with the read ahead mechanism so it will prefetch all the certain sets of bytes that you need to read in the future so you will eventually end up with having more sustainable and persistent latency and response time so in this point I'll be moving the control to or or the stage is yours buddy. Thank you so we chose NGNX as a CDN in front of the RGW the S3 Gateway and we chose it because of NGNX as a couple of advantages advantages on other solutions like varnish for example and NGNX is really easy to change NGNX and using and customize it and using for example UWA to change things and it's quite common course CDN providers like Cloudflare if I if I remember right and so we chose NGNX so how it works this solution so one client is sending a get request to the NGNX the NGNX will send the request with another header and we'll let the RGW to authorize the the request the the main issue with with CDN is that when you are caching the object you will get eventually a public access object and you don't always want to create the the object without the ACLs and sometimes you want to just you cache it and not letting and and not letting anyone to access it after it so NGNX after it will get the 200 or 404 from the RGW or 403 it it will then send the request without authentication and will store the the object the the body and on a configured the directory so after it if another client will send the request the NGNX will send just the authentication API request and will just send from the from the local directory the the body so we are getting authenticated CDN for S3 this is something special there because we don't have this solution and somewhere else I mean I think it's quite easy to just cache HTTP requests but here we are providing authentication mechanism that and we are not delegating permissions to the NGNX NGNX just many in the middle and it almost can't and change the request or not handling the authentication itself so this is the advantage here and the second thing that Sean talked about is that and there is a readout mechanism it's something that there I don't think there is a readout mechanism in other CDNs and NGNX has a nice feature called the smart cache and which means that if someone is wants only I don't know a zero byte to one byte so the typically the CDN will cache only the first byte and that's two bytes sorry and that's it and for NGNX there is a mechanism to override the range adder and request all objects without partial content and after it if someone will send I don't know byte to three it will get it from a local cache and NGNX will not need to send the request again to to rgw and to request the the body again so it will only send the authentication request and next slide please Sean next oh thanks and so as we talked about there are two new APIs for this feature that we have added to rgw the authentication API and the cache API and the authentication API is for the authentication obviously and if I mean both APIs we don't really need them we use them only when we want to I mean if we want to cache and to create a public access objects eventually on the NGNX we can just change a little bit the NGNX configuration and we will get without the authentication API we will get much faster response time because we will just send the the body from the NGNX without asking the rgw about the permissions and the second thing is the cache API so the cache API is a little bit complicated and the thing is that most of the applications are signing the range adder as part of the AWS v4 signature so the thing with the cache API is that we are wrapping the adder's requests before sending it to the rgw we are using a spatial a user with spatial capabilities we are giving the NGNX the ability to wrap the all requests that the user have sent to the rgw and and this way we can change the range adder so even if because the NGNX needs to change the range adder to get the all object if you want to use read add it needs to just send the request without and without the range adder so it's a problematic thing because the the rgw wants to get the range adder too and it wants so it will return a signature does not match so this way the spatial user is wrapping the all request and the spatial user itself can override eventually the range adder so if the the range adder is signed with zero to two and the NGNX will send without range adder or with the range adder with different the range it will allow the rgw will allow the request this way and so this is the the second thing we we've added with this feature because we saw for example video streamings and and so on that they are using range requests and we wanted that the NGNX will send only one request and then just bring it to the client from page cache or from disk and this is the thing and we can also not using the cache API and just uh using the cache on top of range adders so if the user will request zero to one and then one to two it will need to then NGNX will cache those requests separately and next slide please Sean thank you and so in this slide we can see that we are getting almost twice the performance and then rgw alone without the NGNX CDN and the second thing is that we will get more if the client is remotely and the CDN is remotely if we have any a bandwidth limit between the client and the rgw and so we are seeing something else that on 16k we will get a better performance on the rgw than the NGNX the thing is here is that the NGNX is sending two requests right or even one request but is still in the middle so on the yellow line is still the NGNX still sending the authentication API authentication call so it will limit eventually the the number of requests for small objects I mean the the overhead of requesting the authentication is quite the same but for larger objects and we don't see that it will impact I mean it will impact to use the NGNX for bigger sizes and for 10 megabyte object size the limit was probably because of a bandwidth limit in the lab and as we thought we got a little bit slower request when the object is not cached because we are sending in the first request we are sending two requests and then we are sending just the authentication request and then GNX is probably much faster because it doesn't have all the layers just storing a single file or not single files just files on the local directory and we could use NVMe for caching the cached objects and so I want to talk a little bit about the demo we are going to present the demo is using OpenResty which is a kind of NGNX wrapped with Lua scripting for the cache API we are using Lua scripting to wrap the and the user headers and we are using for the cache API tool NGNX authentication and AWS signer module and we will see in the demo that the performance when we are dealing with the edge location the CDN has a lot of impact on the requests you will see that it's quite the same RGW and then GNX without cache and then you will see that with cache it will get much faster response time so we can go to the demo just let me know that you see the screen right so we are compiling the OpenResty and as you can see we are using AWS hot module and auto request and HTTP slice it's another feature of the NGNX and the cache API that the NGNX can not just read the whole body NGNX can read the slices that we declared for example give me half megabyte each range so it will just send in parallel half megabyte requests to the back end to the RGW I think it's important to mention that here we have two splitted screens so the right screen is the Rattlesgate way itself and in the left screen we are actually deploying the NGNX cache layer a guide how to deploy the NGNX with the RGW and so on is in the documentation and we share it with the slides we replace the NGNX with the OpenResty we create a deal for NGNX to save the cached objects basically this is configurable as in the NGNX corner we could decide which directory will be used for the cache for caching the objects and eventually we rely on the VFX cache in the NGNX server itself to retrieve the objects from RAM we could also store those objects and cache them in in RAM itself so here we are changing the cache user we are using the access scan secret key we are going to create on the RGW for the cache user and we change the list of RGWs to only one RGW but we can use as much as we can as much as we want sorry and for the for the NGNX configuration so now we create it with AMZ cache which means that we can wrap the request currently is the request to the RGW 5 megabyte file and so 50 megabyte file it's important to mention I think that in the first test where we are reaching to the Redis gateway itself remotely the 50 megabytes file is being retrieved in 29 seconds and as you see as we go on the object is being retrieved from cache so there is a lot of difference between the first test and the second test and even all the tests that are following so I think it's time for Q&A feel free to ask any question that you have Sean we have three questions here from your Q&A panel so the first one is NGNX is kind of load balancer is it helps it does it help the performance by load balance caching it could help but the the limit there is I think the the capacity of the single NGNX NGNX has some load balancing by sharding the URI so you can eventually shard the caching across NGNX servers so you will get if you have for example a free NGNX every server with one terabyte NVME you will eventually get free terabyte of cache of different objects and another thing I want to say about it is that this cache is of course eventually consistent it means that you will get all the objects than what you have I mean if you are updating frequently the the underlying layer and not just writing and reading you will have to or just decreasing the in the NGNX configuration the time for caching I mean oh and the NGNX will send a request just to check the heat tag of the object or you will not be able to use it if you want to get the latest object or if you don't want to use a smaller update in the NGNX so Alex is it okay for you is the answer is is okay Alex feel free to let us know in the chat here regarding the same okay so I'll answer the second question of Alex and the cache layer is on NGNX and the I mean the data is in the self cluster and NGNX is only saving the the requests for a limited time and eventually NGNX will send just the authentication API the authentication requests and not the all request I mean if the object is a one gigabyte it will send just the authentication request and the RGW will not try to fetch the data only the metadata of the object to check for the ACLs the NGNX knows where to send the request in the configuration we're just configuring the back end servers and the question of Devrim currently NGNX has QoS but no we don't have a real bandwidth by a RGW user on top of NGNX is it is this your question Devrim one thing to add with your permission is that basically we should decide what is the proper configuration I mean according to our organization and our S3 service as Orr mentioned before the cache TTL is configurable in the NGNX conf so basically we should we should keep in our minds that there are objects that are being frequently updated so we need to flush the cache more often and not for example have this configuration saved for one day because if the objects are being changed four times a day so we won't get the latest version of our object because objects are reputable so this is from that perspective thanks thanks Sean so last question here by David Duncan and then we can move towards the end and and if there are any further questions we can move to discord so David mentions that so this is interesting do you see an ease of opportunity to use other types of services or caching layers now for CDN for flexibility I mean he means not just for NGNX so currently there is another approach using cache layer inside RGW so it's another thing and yes we are not currently investigating in other solutions for external CDN and the thing is that if the for example varnish or any other solution of HTTP caching can implement the sending of the authentication API and maybe using glua or something else to wrap the request after it it will be suitable to I mean it's not limited to NGNX the solution is not limited to NGNX just the the first implementation is of NGNX the spatial finger is the implementation of the authentication API and cache API so we can cache objects and we we are still using a authentication there and so this is the spatial thing to just cache HTTP response is quite easy but to cache HTTP response of S3 and authenticate it it's something we didn't see somewhere else and this is it David do you did you did we answer your question okay yep so David David says thanks all thanks Sean and thanks a lot Sean and all for this amazing talk so if if anyone wants to have any further discussions you can join session room for and discord for reaching out to our speakers here thanks again guys and thank you guys bye bye guys thank you very much feel free to contact us if you have any quick any additional thank you very much