 presentation, they may trigger some event in which case are a certain burst of users coming into our company. Why it makes a big trouble to our company? Because if the burst of end user is beautiful, we can just open the machine in advice. But the burst is unpredictable. And this we cannot always open a lot of machine, or otherwise the server calls will make the boss kill us. And this auto scaling is not fast enough. You may wonder that I can open a lot of other sort of easy kill machine in a second. But the scaling in other ways is very hard. You can open a lot of machine, but you have to fill in the database with data. It means expressly with MongoDB. Expressly, our MongoDB is supported by some third party. It's quite hard to change the support content. And this our only solution. If there is a certain burst of incoming user and it's much beyond the server capacity, we can only say sorry to pass to some of user to protect our system, except for total outage. You may ask, if there is a lot of user using Engine X, using Engine X for ways limiting is enough. Why you have to make so a lot of work? I can only tell you that our company, although it's major business, it's a streaming platform, but there is a lot of mini games in our apps. User can pay a mini game for 30 minutes. You can imagine that if you start a game at certain time, you cannot connect to our server because of ways limiting. You will be very, very angry because you can't complete your game, especially such a game as store. And it's during ways limiting, instead of performing from client to random user, we allow existing user to continue their game while blocking the new user, translate to technical requirement, when we use the access token to distinguish the user. When we found that the total number of active using access token is more than X, the server will enter system filtering mode. In system filtering mode, we will perform a check-in for all requests. If that access token has been used in the previous 30 minutes, we consider this access token is used by existing user and we allow it. With the access token, it's not used in previously 30 minutes. We simply block it and return all 29 to many requests. Here is our visual approach. It's definitely fair, but I want to show you as an example of how to use threatless incorrectly. First, to know if an access token is used in previously 30 minutes, a very, very simple solution is we use that access token as a key and store it in threatless and set the TTL 30 minutes. If the system filtering mode, we just check if that access token has corresponding key and exists in threatless and resorted home. But this approach has a quick problem is we are unable to tell what is the number of active access token in our server. And this another problem is the key value in threatless has much higher memory footprint than hash set. And this we use hash set instead of key value to store such key web access token in threatless. You can use this example. You can see that in every 30 minutes, we create a hash set. If there is a new access token coming into our system, we put the access token into the hash set we are currently using, for example, data. The access token takes its last usage time as the timestamp. And if the system suddenly jumps from normal mode to system footprint mode, when we receive an access token, we just need to check the current hash set and the pbsf hash set and see if this hash set has usage within the previous 30 minutes. It's simple, but it takes it. Why is fail? System filtering is why we want to perform system filtering. We perform system filtering is to protect our internal server controller against the burst of traffic. But in the re-seal approach, every request during system filtering, he needs to ask for the threatless if such access token has appeared, has used in pbsf 30 minutes. The, I repeat, it's why we want to perform system filtering because the burst is totally unpredictable. If we allow all the traffic to go into our back end, our back end will crash and result in outage. And this, we need some mechanism to protect our internal back end component, for example, threatless elastic search, the MongoDB base, my signal, against such traffic. And this, why re-seal is a fail-up? It's a fail-design because it needs to access the threatless for every request during the system footprint mode. And it will result in a crash in the threatless. And another, which problem is? Because the threatless in the threatless cluster environment, although you can have a lot of load in the threatless cluster, but one threat is only stored in one threatless loop. It means that during the system footprint, all of such traffic will jump into one threatless cluster loop, and it will result in a crash spot. Such, although in the threatless cluster, others will be normal, but in that threatless cluster, in the loop that's stored in the headset, it will be over-programmed with a lot of traffic, and case outage. And this is the reason that we are currently using. First, I repeat our goal. First, threatless is one of our key components in the back end. It must be protected against such threat of traffic. It means that no matter the 2DS, our threatless must be protected from such threat of traffic. And we need a mechanism to understand if our SS2 can have been used with PPS 30 million, without SS of threatless. And this, we need to use local memory to store such SS2 traffic. But everyone knows, in application service, the local memory is not unlimited. And this, we need some constant local memory mechanism. Same mission impossible. My boss always won. My boss always won ship, server fee, but it seems impossible. But it's possible with home filter data structure. I want to introduce what is home filter data structure. It has constant memory usage. If you create the home filter data structure with your PDP data memory usage, and this data structure, you can add a string into that data structure. And then you can test if that string has existed in that data structure. However, you know there is constant memory usage. It means that there must be a trade-off. Constant memory usage means that there is a time with positive. I will discuss more about it later. And one of the key important features is the home filter supports approximate time. Although it's not an acceptable algorithm, but it can tell you the approximate number of items within that data structure. And finally, it supports merging. The merging of two home filter data structures is very, very fast. Why home filter is so helpful? First, the false positive is controlled in a low wage. It's not important because, for example, if your server capacity can support 10,000 users, supporting 10,000 and extra 100,000, it's not a matter to our server. And one key point is the server capacity is a PDP fighter number. And in home filter, the false positive number is helped by our server. The memory footprint we stuck in the beginning. It means that we can give large in large memory calculated by our server side capacity. And this, we can always keep the false positive within a suitable number in our system. And here is the R1 simplified architecture. You can see that there is a lot of application server. Previously, I have mentioned that home filter is stored in local memory of every application server. And these days are helpful. If user A go into the server one, we need a mechanism to synchronize that SS token to all of the application server. Otherwise, if we just go into the server one and in the next 10 minutes, we go into the server x. It will be false positive because it is not synchronized. And this, what is the role of Reddes here? Reddes is a place of synchronized storage. In every second, the application server will try to perform a sync operation. In the sync operation, it just get master copy in the Reddes and try to merge the master copy to the local copy. And then after merging, it will put back the merged copy back into the Reddes. And this, if our SS token is added in one machine, after two seconds, it will be populated to all of our application servers. Then ask, why we use Reddes instead of other components? One key point is almost every back end, they use Reddes as hash. It means that if we use Reddes as a centralized storage, we have no external back end component. No back end component means that no special approval from our boss. And one more important is no extra approval from our SLE team. I should not disclose too much. One important point is previously, I do not tell you what is exactly stored in the Reddes. But you can see that in every second, every application server will try to synchronize its local copy with the Reddes master. Then there is a race function. In order to avoid the race condition, automatic lock is used to protect against race condition. This, instead of just using a key value to store the own filter content, we use a hash set instead. In the hash set, the race is the content of the boom filter. And the time, the TS, the time-same-film is used for automatic lock. It means that every application server will try to grab the master copy to merge with the local copy and back-sync to the Reddes. If the time-same is failed, then it will grab the master copy again to retry the operation. Because we need to use the automatic lock hang, automatic lock hang, one of the key points is check and set. So we need to do a script to write such operation. But it's very simple to do. In our application server, there is thousands of incoming requests in every second. Every second, you try to read and write to the boom filter local memory. This, we need to use the read-write lock for the current concurrency control in the application server. The read-write lock allows us to perform multi-fractional tests because test is a read operation and single-fractional test operation. To further speed up the performance, the update of the boom filter is deferred. Why? Because in every request, the request do not need to wait for the boom filter to be completed before it returns back to the client. And our application server is writing in Go language. In Go language, there is a data structure called for the channel for all new SS token. The request will just put the SS token into that channel. And then there is a background fact to slowly get to slowly get from the channel and add back to the boom filter structure. I can see that some of you is currently performing a video on the boom filter data structure. And you may wonder that there is F operation in boom filter, but there is no x5 operation and no td operation in boom filter. And how can we resolve it? Instead of recreating another boom filter and publishing it as paper, we do not have enough time. And this will perform a still but direct disclosure. I think everyone here used it the next window as your server. OK. I think everyone knows what is log rotation. Log rotation means that, for example, Apache will write a lot of logs. And every day, we delete the oldest log file and create a new one for current day. And it's called log rotation. The log rotation approach can also apply on the boom filter. And this, instead, in our local application server, instead of this boom filter object, we create 30 boom filter objects. And in every second, we remove the oldest boom filter object and we create a new one. And this, and this, and this, and this. If our boom, if our boom, if our boom, if our access token has not been used in 30 minutes, it will finally be removed in such a way of boom filter. And one more key point is the access token is added to the latest boom filter. And this, not all the boom filter needs to be synchronized with RedDesk. Only the currently using one needs to be synchronized with the RedDesk. Here is our V1 architecture in detail. You can see that in RedDesk, there is a lot of boom filter. And only the last one, the current one, the using one, is synchronized with multiple application server. The V1 approach, some foot on the table. And it fell during our peak hours. Why? Our application server has, you know, will scale up and scale down according to our number of requests. And this during the peak hour, we open a lot of talk, we open a lot of talk content for application server. And this, the application server, all of the application server, we try to synchronize with RedDesk. And one key point about the optimization law, is that the synchronized contract increases exponentially with the application server number. And this, once the application server number increases, and finally, a lot of things will fail because it grab from the RedDesk and the rest as master copy is updated by other application server and resulted in fail. And this, it needs to grab again. And finally, because of such a contract, application server will retry, retry, and retry, and resulted in abnormal, high bandwidth usage. And in v1.1.1, we just update slightly. On the automatic log algorithm, we perform a give up because we do not have enough time to rewrite everything. It means that the application server will still try to grab the master copy from RedDesk and merge with local copy. But if found the contract, it means the master copy, when it tries to upload it back to the server, if the master copy is modified by other application server, it will give up and stop the upload. Is it truly meta? No, because the thing is done in every second. Even one or two iterations will fail. But the local copy will eventually sometime upload back into the RedDesk master copy. So it doesn't really matter. It doesn't matter. If your system, your app will request you to access the RedDesk for a similar thing, it's a horribly bad idea. Your RedDesk will almost die in CDOS. Similarly, before the age of 3WT, the access token is random strength. It means that a random string access token, how we can hope is a rated access token, we will check it with either RedDesk or MySQL to check if that random string is issued by our server. However, it is a weakness to be exploited by CDOS. Because in CDOS, I do not need to. If I do not use the 3WT, I use random string for access token. The acceptor, he can generate a lot of requests and do not need to lock in the system. And every request will try to access the RedDesk to know if it is a access token generated by our system. It is not good. And when key pointers, RedDesk has normally the performance bound is membrane or network IO. However, be careful of the hotspot. If there is a hotspot, some of the nodes in your RedDesk cluster will have 110 usage and case break trouble. And one key point is many web-developed parts, they will say that, oh, we can use some library to write some beautiful website. Oh, this, we do not need to pay attention to university cost and do not need to pay attention to algorithm or data structure. I can show you that. No, absolutely no. BUNFIL data structure is caused by 10 years ago by university professor. If I do not pay attention to that university cost, I will unable to fulfill my forced requirement. More fought. In the realm of high QPS, consistency is sometimes sacrificed for performance. You can see that in our room through the solution, there is some force possible. However, it doesn't matter. And if your system is woven and woven, you may need to consider some trade-off between consistency and performance. And one key point is efficient use of local membrane is the key of high performance. I cannot disclose our character algorithm in our application. However, I can tell you that a lot of requests is fulfilled by local membrane in our application server. Because if we do not use the local code, the data used by a lot of users will become a hotspot in the rest of the test start and resulted in performance deviation or even crash in the rest of the test tunnel. There's a lot of time. And this, I want to show you some extra things in our company. In our company, our F-lang page uses a Nitterbot. The Nitterbot will show the users the top 2,000 users score. And the score usually there is only the increase. The only decrease is we found some user is cheated within a time or we roll back their score. However, it's very, very abnormal case. V1 solution is convenient and will be used by a lot of application developers. The Nitterbot invades a data structure for sort of set. The V1 solution, we use the sort of set and store the user ID as the key and the score as the score in the sort of set. And this, to get the top 2,000 user, we can perform the get by range operation in that sort of set. It's very convenient and is even our whole. And finally, our sort of go crash again, because there is hotspot enough in our Redis cluster. Why? Why? Not just because of re-operation. Re-operation can be reduced by use of local memory. But one key point is big. After user have finished a game, they need to update their new score in that sort of set. It means that sort of set will become a single calling of all user activities. And it resulted in serious hotspot. In virtual algorithm, in high performance, we need to sacrifice something. Instead of storing all items to all users, we only store the top 2,000 items only. Why is it important? Because the update or get operation is adopted in all operations. Stolenness items in a sort of set will obviously improve the performance. This set is still a hotspot problem. But because of every update and every cost less time done, then the system performance has improved a lot. And the drop-back of virtual compatibility if we found there is a user treatment and that user is the top 2,000 user. The whole list need to be rebuilt from our main database. Because if there is a user treatment, there is no longer 2,000 records in that sort of set. And of course, we need to fill in the data from our main database, which is very costly. But it really happened. And then to truly solve the hotspot problem if you want to save it. Instead of storing the data as one key value component, we store the data into a data shape. And every data shape has only the top 2,000. The data shape is saved by user ID. We perform an MB5 hash on the user ID and then modified H. We use the remaining lambda to determine which shape it should be put into. And each shape has only the top 4,000 user. In every request, in the read request, we will simply get the data from all the H shapes and then perform a short time in the application server. Although you can think it's a stupid algorithm, but we migrate the single hotspot problem into a smaller hotspot problem. And finally, I cannot disclose the number, but after we save it into a big enough number, it will resolve the problem. And instead of using sort of set, we use hash set. Because update and hash set is big O1, the update and sort of set is big O log n. And this is the performance of update in Chris Masterson. And after we get the result sorted, sort of leaderboard, it will put the result into energy. Where does key value for catcher? Here is some home exercise, if for you. Of course, everyone here, I think you know what is MQ and Ruby MQ, but in some case, we do not need to perform a for-tolerant key operation. Can we use the redness to perform some task key operation with task priority? I repeat, sort of set should be our work in all course because it involves log n operation. There's n of my presentation. I know my English is poor and I'm too nervous. Hopefully, I wish you can get something from my presentation. And I will put the slide in my key hub, I can't. You can search Triton Hold in GitHub. And if you have any questions, want to ask me, you can search Triton Hold in Facebook. And my profile is this one. This one is not a work, it's a handout. That's all. Thank you.