 So the talk is Python load balancer 0 to 1 million requests per second. It's by Abhishek. Abhishek is an undergrad in triple IT. He's working in Robustist R&D team, where he has built a custom load balancer for Robust. Thank you. Can you guys see me at the back? Yeah. So this is my talk. It's a Python load balancer which I custom built this summer in a company called Robustist. Before I get into anything, I really want to tell you guys what an honor it is to present my work in front of you all. This is my first talk and my first conference, and I'm really glad to share this moment with you all. Before we get into anything, a little bit about myself. I'm a fourth year student at Triple IT Hyderabad. I'm doing a dual degree course, so I'll be passing out in 2017 with an MS degree. My areas of research interests lie in NLP basically, but I've done some other projects also. I'm part of the Language Technology Research Center in Triple IT, and my other interests include well football, and I'm a center forward for my team in Triple IT, and I'm also part of the organizing committee of an Adventure Club of Triple IT Hyderabad. Well, the world technical expertise I have is, I've done a few open source contributions. I have done a few internships for various companies, and I've done a few personal projects for myself, and well, this came about last summer when I was working with a company called Robustest, and that's where we built the load balancer. Before the summer, I actually had little or no idea to what a load balancer is. I did not know how the term itself was alien to me. Only when I started building it, did I realize it's vast and very necessary applications. Now, I must warn you that this is not a technical talk. I will not be getting into technical details of it because it's still under development, but I will surely make sure that I convey as much information as possible. So, what do you get when you hear the term load balancer? Anybody? Have you guys used a load balancer at all, or sharing server loads? Yeah. Anybody else? Have you used a load balancer before? Have you heard the concept? Well, as he said, it's when you hear load balancer with something, well, it balances load. That's all it does. A load balancer balances load. A load balancer balances load in such a way that it is distributed among all the different computing resources. Here, he said that it balances load among servers. The computing resources need not necessarily always be servers because many computing components can be there as servers or other computers, computer clusters, network links, disk drives, anything. The computing resources could be anything and the load balancer would share resources among all these. I mean, would distribute workload among all these. It aims to optimize resource use, maximize throughput and minimize response time and avoid overuse of a single resource. The load balancer is intelligent in some sense. It aims to distribute the workload in such a way that it optimizes the entire system. It makes sure that no single resource gets overused and it minimizes the entire response time of the application. Using multiple components increases reliability and availability. Well, this is true because, well, your burden on one single component reduces because you have multiple components taking care of your entire application. So the burden on one single component reduces. Therefore, the entire application becomes more available, more responsive and more reliable to you. Well, load balancers have multiple different applications and this is the basic architecture of how a load balancer normally looks like. But they come up with different applications, different techniques and I would go on to show you what load balancers are there and what different applications they give you. So here is a typical architecture of load balancer. So you have a load balancer which is communicating with multiple backend servers and there are multiple clients connecting to your load balancer to communicate with the server. Now, when I began this project in the summer, the way we all start coding is we have our local host, we have our load balancer running on local host, we have our servers running on local host and we have different ports assigned to them. So that's basically how I want to explain to you guys also because I want to give you a small glimpse of the experience of how my summer was spent. So here you see that the load balancer was on port 8080 and there are multiple servers. So one to server four running on ports 8081, 8082, 8083 and 8084. So now for a client to communicate with any of these servers, he has to go through the load balancer. What we do is the address of these servers are not made public. The only thing that is made public is the address of the load balancer and the port number of the load balancer. So if a client wishes to communicate with a server, he has to go through the load balancer. So here the client will say that, okay, hey, pass me through to S1 or he wants to connect to S1. So then the load balancer what he does is, well, he's intelligent. So he knows, the load balancer knows where the S1 is and what address is located at. So it manages to create a session between the client and the server S1. As I told you, there are many different applications for load balancer. Here are some of the load balancers which are there currently in the market right now and they all perform exceedingly well in their task. But well, what else do they offer? Because the thing is, when you're always creating, when you're creating an application, you need to make sure that it caters to your needs. Here, right now, for example, here HAProxy, BalanceNG, F5, all these load balancers perform exceedingly well in the task of load balancing, but what else do they give? So HAProxy, for example, it can give you a HTTP or TCP proxy for web-based applications. So that's what offers. And then F5 gives you a virtual load balancer along with the standard load balancer it comes with. And 8n, for example, it offers URL manipulations and Nginx gives you the ability to configure weights for each server. So, as I said, there are multiple applications for your load balancer and you can tweak it to fit your needs. So when we go on to the uses of the load balancer, we see that there are multiple uses for such an entity because even before the summer, actually I had no idea to what to expect or to what to build or how to do this, how to go about this. But when I started using it, I realized how important it is in any growing industry, in any application. So here, we see that it increases security. How does it increase security? You could see that the client could not directly connect to any of the backend servers because the address is not made public. To connect to any one server, he has to go through the load balancer. Therefore, in some sense, you're prohibiting direct connections to the servers. So for any client to connect to a server, he has to go through the load balancer. That way, you're increasing security of the entire application. It improves performance of the entire system by distributing traffic. The performance of the entire system is, well, it's increased because you're reducing the burden on the servers which are managing and maintaining the application level traffic. By reducing the burden on those servers, you make sure that your entire system is more responsive, more available. So that's how it improves the performance of the system. Now, it makes the entire system more fault tolerant. This is an important key concept, actually, which you want because the load balancer ensures that there is no one single point of failure. If you have a server which is unresponsive or which has broken down for some reason, and now the entire system is a collapse, but with a load balancer, all it has to do is it has to distribute the traffic among various other servers in the pool. Therefore, thereby, the entire system becomes more fault tolerant. Here are some of the examples of existing load balancers and the statistics they give. The entire traffic which is coming into the application is depicted by the green line and the server one and server two are depicted by the blue line and the red line respectively. As you can see, the sum total at any point, let's say five AM, six AM, is equal to the total traffic coming in. So the load balancer is distributing its load to server one and server two. Now, you may be wondering why server one gets way more than server two. It's probably because server one is bigger, or maybe because, like I told you, like I told in engine X, you give more weight to server one. So it's preferred more than server two, probably because server one has a lesser response time than server two. So you prefer server one more than server two. And one more key concept to grasp is that it is not necessary that the load balancer, all it has to do is to balance load. Because here you have this hourly mail statistics which a load balancer gives is, it's blocked some mail because it says it's spam, blocked some mail because it says it's virus, allows some other mail. So it also allows a moderation of the entire traffic as such. So whatever data is being put through the load balancer, it can also moderate it. Therefore you can use, you can tweak your load balancer to like fit your needs. And this is exactly what we did at Robustest because we had very specific needs, we had very specific goal and we had to, that's why we had to build the load balancer from scratch to fit in our environment. Well here are, well the benefits of using load balancer, scalability. That is the most important, the most foremost thing for any growing application. As your application demand increases, you want to be able to satisfy all your customers. How do you do that? You can just add new servers to your server pool. So when you, all you have to do for your scalability of the application is that you have to add servers to your server pool and just update your load balancer. Now it's going to start sending traffic to the new server in the server pool. It improves server utilization and maximizes availability. Well this is because, well no server is a small server. That's what the load balancer says. It's because no matter how small or how large your server is, your load balancer is going to make sure that all the traffic is being distributed among these multiple servers depending proportionate to their sizes. So if it improves the entire server utilization and maximizes the availability of the system. Load balancer can do other things than redirecting traffic. This I've already mentioned because I've told you about BalanceNG and F5 and all those load balancers. Apart from that, I do not know if you guys have heard but you might have seen this project called Mantrid probably on GitHub. Nobody? Oh well, there's another load balancer project called Mantrid upon GitHub. Well what it offers is it offers you a place, it offers you a scenario where you can hold the client when the backend servers restart. You can see how this is useful. And also right now I'm always talking about well servers but in truth the load balancer can communicate with any computing resource as such. So two years ago when I had another internship with another company, we had to do a disaster recovery or a risk management kind of application. So what we had to do was we had to mimic every IO operation on one device on one disk to another disk. So here also a load balancer would come into play because whatever IO operations you're performing on one disk, all you have to do is you have to mimic it to the other disk. If you're performing anything through the load balancer, the load balancer will be capable of copying it and putting it through to the other disk thereby creating a backup disk. So in case of a disaster, if you lose one disk you always have a backup disk. And also you guys might have heard that, well a couple of years ago when Michael Jackson died, Google redirected all queries regarding his death to a error page saying that well your request looks like a DDOS attack. It was because, well there were so many people pouring in for this information. So many people asking why, what was the cause of, what was the reason for Michael Jackson's death and so on. So you can see that even a load balancer has been capable of doing such tasks, like you can figure out if there are more, there's more requests coming in and so they could also figure out that your system could probably be under attack. So that way also increase security in that sense. So to get into my project, well the thing is we had to build the load balancer from scratch because it had to fit into our needs. So I'll be explaining the entire project as a whole and I'll be talking about how we needed the load balancer and then I'll get into the challenges we faced by building one. In the beginning when we started the project we did not actually even know that we needed the load balancer because initially we just thought that we didn't need load balancer, we just wanted to do some direct connections. But when we started building only we realized that the load balancer was very essential, it was a very necessary part of the entire application. So to understand this, let me go through the application as such. So this is Robustest, this is the application which was developed in Robustest, the company itself. So this basically it offers you a web interface, an interface where you can test your mobile applications on the real Android devices. So these devices which you see here, they are real devices which are connected to the server back there. So what you do is when you're, how to test your application is you upload an application or you choose from a list of applications that are given already and then you choose the mobile device which is present here. So once you're done with that you get into another page where you're given a small screen which shows emulation of the phone and which is a real device which is connected to mind you. So it's a real device what you're going to be testing on. So, and you can control these with your mouse and with your keyboard and you have statistics based on like memory usage, CPU usage and network and et cetera. So what we have here is you may have all developed Android applications or may not have Android applications, but you know that when you develop an application you develop it for your phone, you develop it for, you know it works on your friend's phone, it works on your phone, but not necessarily that it works on all the phones that are present there. So, well this gives you a place for you to like, well test it out on multiple different phones so that you ensure that well your application is working. So, well you may have figured out like where the load balancer comes into play in this scenario because it's going to be the one which is going to be communicating with all the mobile devices in the back end. So, when I was not part of the front end team, I was only part of the back end team. So, what I essentially did was I had my load balancer communicating with multiple mobile devices in the back end and I would test it with a VNC viewer or something on my computer. So, what I needed basically was I had multiple mobile devices in the back end and they were all running a Droid VNC server or VMLite server which is basically, Droid VNC server is open source if you guys want to look at it, please go ahead and look at it. So, the Droid VNC server or VMLite server which is running on the mobile devices and you had a VNC viewer running on your computer. Now, if you wanted to emulate the phone on your screen, you just had to give it the address of the phone, basically the address in the port number of the phone. Well, in my project though, we had multiple back end mobile devices connected to the load balancer. Now, the load balancer had an internal database telling the phone type and the various addresses of each mobile device. For each client to connect to one mobile device, he had to give in the phone option and the cookie. I'll tell you the reason for a cookie in a while. So, basically he had to give in the phone option and then what the load balancer would do is would look through the internal database and then it would patch it through to the mobile device. Initially, when I did the load balancer project, I was confused because all I really thought was that the load balancer, the initial and the only contribution to it on the whole system was that it was hiding the information of the back end servers. So, initially, you guys must have also figured out that probably it's going to be acting as a server and a client. It's gonna have a server support and a client support so that it's going to act like a server to the client and then act like a client to the server and then pass the information back. But this is not true because that means you're actively involving your load balancer in the communication channel. When you're doing that, when another client comes in and asks for a request, the load balancer is busy with the initial client already. So, that's why what we decided to do was we established a session between the client and the server. So now the load balancer is out of the communication channel and he's able to take requests from other clients. So here what we did was we had requests coming in and so depending on what type the client chooses for, the load balancer makes the case session between the client and the mobile device. And so you can emulate the mobile device on your computer screen and you can do whatever you want or whatever you want is fine. So, but the reason for the cookie was we had to create a sticky session implementation. That was also, that's what we wanted now load balancer because the entire application was built on a per device R billing. So when you're connected to the device and when you close your browser and you come back to it, you're essentially going to be connected back to the same device because well, there was a per device R building as I said. So, and also the fact that when you are connected with one device, the other clients cannot connect to that device because it's in use with one client already. So, when that was happening, the internal database had to be updated saying that, well, this device is being used. So, well, I can't connect to it right now and so on. So, this is how we, well, basically built it and this is the basic, just to the project and well, if you guys have any questions or queries, please feel free to ask me. I'll answer to it best of my requests. I mean, so my knowledge, sorry. And an ideal device. Sorry, I didn't get it, mobile device. So, well, you have a Droid VNC server. It's a, you have something called a Droid VNC server or VM Lite server which is running on the mobile devices and what you have is essentially, you'll be parsing your argument basing on, well, which phone type you want to connect to and you have all standard libraries, socket, OSRE, that's all I used for the project. It was all those only, HTTP request and make the session between the client and the device. That's all. How do you, I mean, if there are five instance of a device and you want to connect to, I mean, two of them are ideal and three of them are busy. How do you route between? Sorry? How do you route between them? I didn't get you. If two of them are ideal and... And three of them are busy and a new request comes in? Yeah, the database is always updated, right? So, suppose two are busy right now. So, the database is, as soon as your mobile phone is in use, the database updated. So, now when the new request comes in, it says that the mobile device is active so it can't participate in your calls. So, it's an actual database itself. Yeah, it's an actual database. Hello? So, you mentioned that you give the IP address of the load balancer rather than the servers for the clients. So, let's suppose a client is sending a request to my blog and there are 10 servers in the back end hosting my blog and load balancer is balancing the request. So, a client sends a request to my blog and the load balancer forwards the request back to one of the servers. How does the load balancer knows that it has to again send the request back to the client? Yeah, exactly. That's what happens is that when the client, when the load balancer gets a request from the client and now the load balancer figures out that which server it has to, till there it's done. So, after that's done, it establishes a session between the client and the server. Now, when you establish a session, again the address and everything of the server or anything is not given out. So, security is still maintained. It's just established session, it's also there. And the client and the back end server, yeah. Okay, so I had a question regarding if a load balancer goes down, how does? Yeah, make sure that load balancer is always active. So, we have to- That is basically this stuff, yeah. One more question, it might be an off topic, but I wanted to know how do you implement, I mean, if I have a use case where a client needs to always present in one particular server. But I can't get it. I mean, there is a use case where a client has to be connected to a particular server. Yeah. Okay, but there are many requests coming up and you want to do a round robin. But along with that, you need to have an IP hash. So, IP hash, IP hash along with round robin. So, how does that happen? I mean, is it- I've not explored such options, but I do know that there are multiple algorithms which the load balancers follow like round robin technique or lease connections time, lease response time, but I've not explored such options. Okay. Hey, can you able to hear me? Sorry? Yeah, I can hear you. So, have you faced this? No, no, no. We didn't do any of those because we didn't have to maintain. We just had like, you know, it connected to each server and each mobile device and back. We didn't, there was no reason for us to care for DDoS or anything of this. The thing is, it won't be there in our case because when one mobile device is connected, another mobile device, another server, another client cannot even request for that mobile device because it's already- Yeah, yeah. Yeah, yeah. But my load balancer does not work there. My load balancer is behind that. Yeah, not in the application service, yeah. So, why did you need to develop your own load balancer? What didn't work with the user commands? Yeah, because we also tried. Actually, we actually started with Mantret. We thought we'll put Mantret in or we'll, because that was open source and we thought we'll use Mantret as our base. But the thing was we had to handle data of the mobile phone and what was, that data had to be handled. That data was not handled. So, we had to develop our own and we also had to implement sticky session and so we did that. So, sticky sessions are handled by many of the load balancers. Many of the load balancers. The standard load balancers. But we had to handle the data or the data of the phone, what phone gives and what the VNC viewer gives back to the others. Okay. Yeah. So, to manage the response time of the load balancer, do you use any type of caching to store like static content or something like that? No, we don't. Response time, well, it depends on the server response time. If the server is more responsive, that means your load balancer is going to choose. I mean, you can choose so that if you have a least response time protocol as your primary algorithm that your load balancer follows. That means that it would choose a server which has lesser response time from the others. We did not do anything particular to this because all our phones are on this. And any particular algorithm you follow to distribute the load among the servers? No, we're not exactly. This is not exactly a load balancer. We're not exactly distributing a load in this. No, we're sending requests based on which phone type they want to access. Okay. So there is no balance of load as such, but then we have to, we need the load balancer nevertheless for distributing. Okay. Thank you. I wanted to understand what design things that you used in this one. Like for example, for reusing connections or taking the connection footprint to be very low. Sorry? So you are your service or whatever this proxy thing is, is facing the clients, right? Yeah. And you, so it's kind of a single point and where every connection would converge. All the questions come through there, yeah, yeah, yeah. So did you, any design thing that you used? We had different threads running for different connections and so on. I can't get into the technical details because it's under development right now and it's a company product. Yeah, so it's okay. What I asked about like design techniques, like for example, did you use socket to use? Yeah, OS socket, actually it's very basic socket programming. Do you do something for reducing the footprint taking or taken, so you have to respond back very fast. Yeah, yeah. That's the main goal. Yeah, yeah. Did you do anything? We just, we just implement session between those two and it was pretty responsive, it was pretty fast anyway. So we didn't mind after that. Sorry? Yeah, yeah, file this up for the roadbansers. No, probably we're going to explore. It's still in a development phase as I told you. So this is so much as I, as we have done, I've told you. This I did probably in two, three weeks we've finished this, we finished how much we could and then yeah, that's all there is. Engine X, no, no, no. This is all custom built. Yeah. Any more questions? Thank you guys.