 Hi, my name is Siddharth, I am SVS on the Internet, I work for Mbipe.com, we are a edtech company based in Bombay. Can I just say like, I know you guys know how would you have it here, but you guys have it really really nice in Bangalore, in Bombay if you say Redis Minicon, you are not going to get anybody over there, it's really nice, you are very lucky. So counting things very fast on the Internet, in other words, how to build your own real-time stat server. On the Internet is interesting because there is net scale, web scale, so we are talking about doing this for lots and lots of people at the same time. So before we go any further, I would like to give you my definition of real-time stats. There are a lot of stats that are counted as real-time, web analytics being the most common use case of them, but I am more interested in going one step beyond, which is where real-time stats actually change the user experience. So when you are collecting page views, you are not actually changing the user experience based on the data, at least not in real-time. So if you have a blog site like Upworthy, you probably don't need like this version of real-time stats, someone comes and offers your real-time stat solution, you can write an article about it, you tell them to go away and you can write an article about it. But if you have multiplayer gaming or adaptive algorithms like today's favorite example, QuizUp, you can learn this one weird trick to improve user engagement. The trick is real-time stats and the weird bit is doing it in Redis. Redis is a bit weird if you are coming from a traditional SQL background. It forces you to think about your data in a different way. You are no longer going to be able to use those standard relational approaches that you used earlier. Let me show you our use case for a real-time stat server. We make a practice application so kids can prepare for entrance exams. And as you can see, we support like a number of exams, there are three or four or five subjects in each exam. Each exam has these, each subject has these units. Students have chapters and internally chapters have topics and topics have questions in them. And we want to make the experience as nice for the user as possible. So for example, if a person is having trouble with a particular concept, we want to be able to adapt what he sees next based on his performance in real-time. The other thing is this is a very, very competitive space. Kids are, it's all about being in the top 10 percentile, top 30 percentile being the top rate, et cetera, et cetera, et cetera. So we have a, we have a thesis that we can increase user engagement A by making it more adaptive and B by gamifying it, giving them leaderboards and other things to be desirous about. So why would you use Redis for something like this? Redis is an in-memory key value store with some very cool features and we'll go through each of them. Why do you want an in-memory store? Frankly, if you want to do it for thousands or millions of people at the same time, you are better off not trashing your disk every time you update a counter and RAM is to disk what an F16 Hornet is to a garden slug is just orders of magnitude faster. And this is a recent statement made by someone more famous than me, but I forgot his name. Redis is the new disk, disk is the new tape, and tape is the new office decoration. So why do we want a key value pair? Why can't we use SQL for this? Any ideas? Why can't we use SQL to run a real-time state server? Yes, but why not? Yeah, basically, you know, you can't, so for example, we have over 7000 topics in our database. We can have an arbitrarily large number of people looking at statistics pertaining to those topics. You can have a prop who's got like his report card open for every student with like last six exam stats and or whatever last six day stats and each of them are updating life. You can't run those number of SQL queries in any sort of reasonable useful time frame. And if your re-computation takes longer than your refresh interval, basically you will end up burning your server down. There's not much to do there. Caching won't help you because if you're a really, really busy site, your caches are going to get busted all the time. So if you want to reliably show fresh data, you're not going to get much value out of caching either. So what we do is we use Redis and we embrace the entire Redis philosophy and we do denormalization. Now, I didn't go to computer school. So this may not be the exact correct technical word for it, but I think it gets the point across. You instead of creating a report on instead of creating the report by saying select star group by and so and so when it's required, you create it when it's written to. So the one thing about real time status is you don't get so much ad hoc variability anymore. But in return for that you get these every time an event happens, you take the delta. So for example, you have a data structure which for this user 250 to is made 101 attempts and he makes one more attempt all you have to do is like increment the counters over there and now this this report is reading this report is free and always will be you can read this for thousands of clients simultaneously all day long and you won't even be using up any CPU or IO, but denormalization sounds weird, right? It's like not normal. I mean, how am I supposed to deal with something that's not normal and it is a problem. So so it's a problem, but if you embrace it, you can you there are strategies to to deal with it. Some of the problems you'll find is keeping data in sync, right? Everything is fine and someone goes to Rails console and deletes a user attempt. Now the stuff that's cached in your report or denormalized in your report is out of sync with the actual if you run an SQL query against it, it won't be right. So how do you keep data in sync? Strategies differ based on what the situation is, but you could say something like don't use Rails console or run health checks every now and then and sync data recovering from outages is the same thing. Here you want a strong guarantee that every event is processed once and only once. If you have an outage, can you like really just send the last 10 minutes data or however long the outage was again and be guaranteed that things don't get counted twice and you don't get any ad hoc reporting, which is fine though because normally your real time use case is quite limited subset of all your data and you can actually do without without ad hoc reports. So it's almost like you go down the street, you're thinking about a real time stat server and the devil comes to you and says I'll make you a deal. I can give you your real time stat server, but you're going to have to take the pain of denormalization. So at some point someone is going to write a database which just does all of this for you, but that point unfortunately is still in the future. Luckily for us though, most stats that you look at if you like really break down the reporting problem, most stats are either counters or gauges. Anyone here not familiar with these terms? A gauge is this is the value now. So your current memory usage is a gauge, whereas number of page views is a counter, it increments each time. Most of them can be counter the gauges and if you have sets and arrays, you basically covered 99% of the use case of whatever it is you're going to need in your report. So it's an in-memory database key value store with some really cool features and frankly any amount of time you spend reading the Redis docs is time really well spent. There are some amazing features hidden in there and literally like it's sometimes you meet your soul brother and you say wow, you made exactly the database that I wanted, but a lot of the cool stuff is hidden deep inside the documentation. So yeah, do read the documentation. If you're building a real-time stat server, do not leave home without this isolation. You don't want concurrent access to the same data leading to bad counters. Everyone knows this, right? You want to increment something. You read it from the database, increment it by one, write it back again, but in the meantime someone else has read the old value, incremented it by one and so on. So you don't want that. So you want each write to the database to be isolated, Redis does great at isolation. You want it to be atomic. Each individual Redis call is atomic. However, if you put a few of them in a Lua script, they are not technically atomic anymore. If one of them is syntactically wrong, the ones before it will run and then it will crash on the wrong one and the rest of them won't get processed. So there is a small corner case around atomicity in this thing and controversial topic, PubSub. There is no way you're going to run a real-time stat server without your database notifying you when stuff has changed. Polling this database is no fun at all. It's really expensive and it's 2014. So we can hopefully have more databases with PubSub in the future. Redis PubSub works fine. I agree it's not durable. But if all you want to do is, hey, this has changed, push this out to over a web socket, it works fine for that use case. So this is what we did. We have a client, which is the app, the one that the user is facing. He creates, there's an event, the event's delta based on the reports that you want to update is calculated and sent over whatever transport you want, HTTP or AMQP. Someone picks it up off the queue and sends it to Redis. In Redis we have a lovely little Lua script. Lua is a very fine language. It's a little weird first time you come to it because they've used all the wrong symbols for everything. But apart from that, it's really nice. And you can maintain a list of watched reports like clients that are actively interested in changes on these reports. And Redis will send a notification every time one of these is updated. Another service listens to these notifications and basically sends this out over Faye or Pushar or whatever your favored web socket protocol is. No, Lua is actually like it's a very twisted little thing. It's meant to be embedded inside other programs. Yeah, it runs on, it runs on. It's embedded inside Redis. Yeah, it has access to all of Redis and it runs inside that process. So it gets all the isolation and atomicity that Redis provides. We'll come to this after the demo. Yeah, a demo, okay. Can everyone see this? So I have used my phenomenal design skills to put together a little demo for you guys. We'll reload this page. So it's really simple. Choose one of the two and you get a random number and choose another one of the buttons and it'll update the score fingers crossed there you go. So what it's doing is essentially it's saying that so it's updated all of these all of these. So just give me one second. It's updated. So you have one report for the Node A. You have one report for the Node A1 and you have one report for SVS's relationship with Node A1 and with A and with A1. And every time I do this, it goes and updates all these reports. It's doing this in real time and sending me the updated report back. And I can hopefully it's performant enough. There are a few interesting things that are going on here and I'll come to them one by one. Yeah. How come? How did you? All right. So what it's basically doing is there's a little Sinatra app running behind. It gets the whatever post it says the level is A1. The username is SVS and score is 71 like basically this is what it gets. And so I sent something to my Redis to my stat server which breaks it out like this that these are the slugs that I'm interested in to update. These are the keys in the key value portion of it. And what I want to do is I want you to increment attempts by one. I want you to increment the score by 81. Then I want you to whatever you get at the end apply this expression to it. I want something called average which is I score divided by I attempts. And so I was working on a ranking feature. The ranking feature hasn't really finished yet. So I won't be showing you the ranking feature. But basically you just say update this with this and that's all that you need to say once you do that you can get to the interesting bit. It's about 83 lines of Lua. Can you guys read that? Yeah. This is Emax. Let me just subline text it. Yeah. How much time do I have? So the main takeaway is not it's so interesting what the Lua script is doing. The main takeaway is don't be afraid of Lua. Lua can do a lot of phenomenal things. You can make more complex logic. Put more complex logic in your Redis database than Redis allows you to do this thing. And the main reason you use the Redis you use Lua is to say I want all these six things to happen at once. I don't want anyone else coming and messing with the data while any of these things are going on. Yeah. A Lua script will lock like any other. So all of them are atomic like they happen only that thing is happening in Redis at the same time. Yes. So executing a Lua script is also atomic operation which means that you have to be very careful that your Lua script doesn't hang or basically that because then everyone else gets blocked behind it. So what we do is very simple. You remember that I underscore like we say to we convert them into like Redis primitives. So Lua is very strange. They don't have a string split. You have to implement your own string split when you want to split a string. The other thing Lua very interesting. So they have this concept of global and local variables. So Lua doesn't allow you to access the global Redis. So Redis is Lua interpreter doesn't allow you to access like Redis stuff because he says all bets are off then but what you can do is if you want to access things you can give each function its own like global namespace instead for you your global namespace is this. And so we do that to basically eval you remember the thing where we said we want accuracy to be attempts divided by correct attempts that's or average divided by score, score divided by attempts. So basically any valid Lua that you put into an expression or this thing will be evaluated over here as well. And if anyone wants more detail on this we can go through it offline. The other thing you can do is leaderboards right. You get a score you update you update the leaderboard and then if the new rank is different from the old rank you you send a notification you publish what happened here. So yeah if the new rank is different from this thing you can just do a pubs up and then your listener can notify people when a leaderboard has changed and this is really really cheap to do. You don't have to keep polling the leaderboard or calculating anything or or any of those the rest of it is just some bookkeeping stuff that you need to do in order to get this get this thing working. So Redis is a really really really awesome data store. I think it's just going to get even nicer. I think you're going to see Redis become part of the plumbing of data stores and people will start to build more more interesting abstractions on top of Redis like they're doing with level DB. So one thing is Redis is fast which means that your code can be simple. You don't really have to worry too much and of course your mileage may vary about performance per se because Redis takes care of all of that for you. Read the manual. I cannot stress this enough. It's one of the most exemplary open source documentation you'll ever find every document every command even gives you like a shell in which you can type in the command and play with it. It's really really nice. It's very terse and it's extremely accurate. You will not find many mistakes in the Redis in the Redis manual. And don't fear Lua. Lua is your friend Lua is a snugly little programming language that can help you do things that your competition can't. So that's the number of ways your client will basically do it for you. There are two things you can do. You can say eval and eval SHA. So eval means you send a script across each time. What you can do is you can also say eval SHA which is you send a script and the SHA of it and then if the SHA doesn't change you don't need to send a script again. The script is stored in Redis and it saves you the network overhead of sending the script each time. Yes, you just change the script and when the SHA of the script changes your client will send or Redis will know basically. So the everything else is really simple. The node bits are very simple. This is just something that picks up from a AMQP from a RabbitMQ queue and calls a particular command on it and the other one is pusher.js which listens for notifications on something and sends it to pusher. So what you can do is when a client asks for a particular channel pusher can send that channel state to something listening here and you can say okay these are the slugs that people are watching right now and we will just send them out to pusher when it makes sense. So that is my presentation. I am happy to answer any questions that you have. Okay. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Thank you. Thank you. Yes. All right. Yes. Thank you. Thank you. Yes. Thank you. Yes. Yes. Yes. Yes lord. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Yes. Thank you. So what you are marking out all of our queries. Yes. Yes. Yes. Yes. is rank and you have a message data and and at the end of yeah sorry the node client when it the Lua script stops executing it will it will also tell you which are the nodes that got updated yes you have to have a separate client for it yes yes and then it executes the callback function which basically sends it out over web socket yes yes I think your key space notifications offer you a lot more richness in the in what it is that you can send on them I haven't explored BL pop and BL push for this particular implementation because I never had I started with the most stupid thing like you want pub sub just use pub sub and it it worked fine for me so I didn't really need a reason to explore those but yeah I side kick uses that so yes yes yes yes yes it's it's it's a tough problem keeping data in sync is the problem with denormalization this is why they tell you don't denormalize but if you want performance you are going to have to denormalize so there are number of strategies all of them are painful at one level one is that you lock down access to how data is changed to ensure that all of this goes together this is almost impossible because things can fail at the ready send etc etc etc a better way would be to have strategies to recover from failure a know when something has failed be know how many data points you have see build a system so that you can offer once and only once processing guarantees so in order for me to get back in sync I should say ok I had a 10 minute time a downtime on my ready server from this time to that time I just go back 15 minutes and just send all the events again and events that have already been seen before should be ignored the so now we have like the number of combinations that we can have runs into the millions so we can't implement health checks we can't implement health checks but if you have a smaller like set of slugs that you are dealing with you can implement health checks to run every some random specified period of time to to check whether the two data are in sync or not there are no really really good options it also the right one will depend on how many slugs you are looking at but mostly this is a good solution when you don't care if you are if you drop to out of 20,000 if you are not doing a nuclear power plant control systems or flight control systems it's usually ok to drop a couple when nothing no one no one will die so so normally they do the same thing for web page analytics like they don't really stress if they drop an event or two sorry let me let me just do this right what the hell happened what do we get sublime text to isn't doesn't OSX have the worst like fire problem success yes so the thing is Redis sets Redis hashes are very efficient for storing data but they don't allow they only allow you scalar values as the as the values you can't have a set as a as the value inside a hash so this is this this particular thing is so that so I have the slug which has all the data and if it's a set I say slug hash and then the key and I make a set like that so this allows me to have sets as to store sets as values as well and then the client basically knows that it has to put all of these together to get the consolidated data back in Lua yes as I said they have the wrong symbol for everything this is not equal to this is what takes all your time when you are like doing Lua is like no good I am sure there is a good reason but yes no no no I don't think that's what he says I think the first bit he totally says but I don't think he says that you can't use any other key then what I'll have to do is remove the the sets and arrays from from this or I mean it's not difficult to do this outside in node either I can definitely I already know what keys I'm going to need so I can just generate them in node and send them across that's not a problem this person is not insoluble insoluble problem but yeah seems I have not read the manual as carefully as I thought I had but I mean I remember the section from this I don't I didn't see this I mean I'm not on the mailing list maybe I missed that but it's not an insoluble problem you can do it outside and then send it to Lua a small node on performance this thing does it takes 2 milliseconds to update those five keys and 25 keys takes 3 milliseconds and 250 keys takes 7 milliseconds so it's it's really blazing fast so for our for our app we update around 55 odd keys depending on who the user is which you coaching class season etc etc and it's like it's 7-8 milliseconds fast enough and essentially you'll only notice the lag over the network the web socket on the way back no more questions alright sure yeah just catch me offline anytime.