 I'm going to talk on how exactly the application behaves when you're running the production. And of course, it will behave here. And when it behaves here, what they have to do with that. I'm not going to bore you if I bore you just spring in and ask you to stop. So this is a basic outline because we are running a couple of non-submers as a two at do-select, and that's where we are at work. And all right, so these are the issues which we faced in the beginning when we started running non-game production. Like, when you are running it locally, everything will look good. But when you run for production, all these things will get messed up. And I'm not abusive. I'm not using it as we are working here. So it is going to be on the basic stuffs. In general, I work with engineering at do-select. You could have seen me at the booths. The first thing is logging. Like, do not do this. Like, it might look cool. Like, too many logs. You log everything you see, like every possible thing, and you never do this. Like, that's way too much. I'll tell you why. In general, when you look at the log levels, they are like way too much. Fatal, info, log, errors, trace, all these things. It might look really cool. You'll think, fine, I'll do all these things. It is like, simply waste. In the short sum, if you're running on a unit server, two things which matters is log and error. There's two streams, standard output and standard error. These are the two streams which all these guys like, if you use Kibana, all these stuff as in for log repeat, these things are the ones which really matters. The others doesn't really make sense. And every single log write you do. For example, consults are real thanks over. Whenever a user connects, you log this much data. It doesn't really help. Every single log is a write to either standard output or standard type of stream. It's a write to a file if you're running on a unit server. So if you keep logging way too much on scale, it might affect your performance a bit. Trycatch does exist in Node.js. I've seen most of them, doesn't use this, but of course you need to handle exceptions. I'm not going to talk about any library or any external modules which you can use. I would suggest sentry. It's a pretty good platform to check all the exceptions. The main takeaway is you might have this tendency of logging everything. Let me log everything. It is like more information. Why not? Information is not bad, right? But this is the tendency to not to log everything. The more you log, the less you can find. For example, if you grab for some word or something, you don't really need unnecessary information, right? So do not log much. That's a key takeaway. And two things, log error, that's it. Don't use all these other tool stuff. It might look cool, but it doesn't really work. And the basis of monitoring, of course, when you run something on production, you monitor it. The obvious things are CPU and memory, right? And your memory graph doesn't always look good. It is like, you might have seen a reviewer in the graph like it looks like this, but in production, it never looks like this. At least for no one to give you like this and this. So don't worry about that. These are the obvious, the CPU and the memory if you need to monitor. And you need to monitor I.O. as well. I.O. chats, you may ask why. Can you tell me why do you really want to monitor I.O.? All right. Because when you understand how exactly that node works in its core, it uses LibUV that is used for all this. So you might have heard this as a cross I.O., no one is as a cross I.O. on this stuff. So LibUV is the one who is taking care of all these as a cross things inside Node.js. So you really want, for example, whenever this review will have mappings according to the operating system, for example, if you take Linux equal, equal is the queue system which just depend on OS, not on the node runtime. So whenever you would make a five records or something, Node just pushes to LibUV. And LibUV is the one handling all the concurrent records to your system. So you really want to know what are the, and of course, I'm just not clear using more for the sake of it. You'll use it for some purpose, right? For example, we use for our backup server for the cloud ID, where you need to do a lot of writes and reads to the file system itself. So here, the use cases, we really need to know what exactly is the, for example, initially, you know, to start off if we run on AWS, the micro instances with normal, normal hard drives. But when we looked at, like, there were these many write operations and read operations, we need to move to SSD because that is efficient. And the graphs were, like, really efficient. That's one thing. Configurations, the key taker is, leave all the configuration to the deployable books. Do not have all these configuration mess-ups in your application because it doesn't really make sense. This is a quick, quick thing. Like, please, you know, sneak up like something. So, how to find demo heavy config and config in your loop of the application? Add del config to giving node and what, the both files are the same so that in your local, it will try to get that del config if not in local config. So, this is a small thing. Containers, like, I've seen too many guys who use containers for the heavy config. Do not do that because unless you're until you really want to optimize it. For example, if you're running it on ROAs, you need to run it as a container, right? In that case, it makes sense. If not, just take us, because bundleite is the worst bear you can make, and don't do that, like, do not run anything on containers. It doesn't really make sense. Memory leaks, of course, when you run, how many of you have these memory leaks when running node-approaching? Yes, that is the obvious, like, we have faced a lot of memory leaks because, you know, the type of, that's how it believes. So, I'll tell you how exactly your, when you run a node application, you're all like, how exactly the memory stack looks like. So, outside this RSS, that's residence set, here I am. And except there are too many things, stack and heap. Stack is where all your privilege are, including data types are, and heap is where all the, for example, all the reference data types are. For example, if there's an object which dynamically grows, it's in the stack and it's referred to the heap. And your heap is what causing the memory leak. So, what happens in memory? So, in NodeJS, it's not like C, where, see, you need to exclusively free your memory. Like, it's manual memory management. But in NodeJS, the V8 runtime has automated memory management. So, it's a garbage collector. It's a pain. It's a trade-off. But, you need to, you need to, you need to worry too much about memory management. So, how it works is, so just, just a protocol inheritance. So, there's a heap object, and every single object is a, is parent, parent, parent, parent, and then there's a heap. So, when garbage collector runs, it will first check for all these objects which are directly reachable from the loop and is not being used. So, for example, I'm not giving you an example because there is no general way you can write a NodeJS application without leaking memory. Like, there's no, there are following patterns. There are a lot of stuff, but anything can, for some general things like, don't use global variables. Like, if you are using Glowshare, please make sure you know what Glowshare is or understand how exactly it works. If not, don't use it. Because that is what it keeps referencing, it keeps referencing to something and that thing will not be reachable by garbage collector. The collector thinks, okay, this is still being used, but it's not really being used. So, that is how your graphs will look like when you have memory leak. It goes from down to up. So, everyone would have seen these sort of graphs in your monitoring systems. So, this is one major thing to avoid when running, when running motivation. Of course, everyone does face it. So, these are the major things. There's no stuff that happens you need to follow to avoid memory leaks, but. And heap dump helps. So, of course, there should be a solution, right? So, heap dump is where a dump of your heap, like the heap memory. So, if it is explicit, it's not mine or something. It's very explicitly telling you what exactly the functions are, the complex trace. So, heap dump is, if you're running on a Linux server, I'm pretty sure everyone does. And you have these signals. So, usr2 is a signal where you can send to your process and it will just, there's a package called heap dump in NPM. You can download it, it's pretty safe because every single package you use in your load modules is your way. It's not the package owner's way. So, this is pretty good, we have been using it. And when you just put this thing, process.on, sig usr2, that's a Linux signal. Just collect the garbage and dump the snapshot on the same folder, right? And you get into the server to kill usr and the PID of what exactly your process is. And there you get, like, whenever you start the application, do this, get a snapshot. And then after maybe, you know, six hours or something, do another snapshot. And when you get into Chrome profile tools, like, you can upload the snapshot and compare where exactly is this memory leak happening. It's pretty simple. When you look at it, you'll understand. And too many guys are always over-opting my session, like, oh my God, I don't know, I might hit immediate users now. And I'm so worried, so I put so many machines. I use auto scaling in Node, if you have so much money, of course you can use auto scaling in AWS. But in general, when you have a machine, make sure you use all the resources. And if you know BI, the runtime, it's single throttle, Node.js is single throttle. Like, even though if you have four core machines, if you're just running a small node application, if you use only one code, either one of them, but only one, it won't utilize all the four codes. So in that case, the two scaling is everyone should know, vertical and horizontal. Vertical is where using all the codes across the machine. Horizontal is where you scale across machines, like put three more, four, three, four, five more machines and put a load balance in there. So first, scale vertically, use all the resources in your machine before adding more machines. So there is this module called cluster, which comes along with Node, it's not an explicit module. Use this cluster, and what this cluster does is, get the number of CPUs your machine has, get the thing, and this cluster will fork those many workers in your machine itself. So what happens is, whenever, the next, so this is a simple code, when there are workers, what it does is it checks, okay, this machine has four CPUs, I spawn four processors. And this thing will automatically run and split up on those four processors. In this case, you're not missing all the other three codes, if you have four codes, in general, when you run without the cluster, you use only one code, also the three processors. But here, since you're spawning process of all codes, the load balance is happening inside the machine itself. Then you can, if your traffic increases, then you can add more machines, and do the same there as well, and put an explicit load balance there. If you want, like find you things, you know HAProxy, HAProxy is a pretty good proxy where you can balance all the loads inside the machine itself. Just check it out. And so we made the same, so we thought, we'll cluster it up, we'll use the cluster module, and all this stuff, but one of our servers were a real-time server, so we use Sogarev, right? So in that case, Sogarev has a seven-way handshake, if you know. Like HDP has this normal handshake there. Sogarev has the seven-way handshake. So what happens is, in the handshake, the first two, three requests, we're already in the first mode, and we had four cores, we were using cluster, and all these was being load balanced inside the machine. So the handshake was heating here, and when the handshake is done, after a while, when this core is full, it's 100% utilized, the proxy was re-routing it to the next cores. So the problem was this handshake was broken. Like, because when you're making a website connection, you need to maintain that state with that core itself. Then it moves, the concept thing will be broken. So we're just thinking initially everything was fine, because it was just using the first over, the first core, but after a while, when traffic increased, so this was 100% utilized, and it started moving out, and those stations were breaking. So there's a rescue, it's pretty simple. So this is a web socket. So what it does is the TCP, the first network layer, and then on TCP and TCP, and then the web socket, connection is established. So after the web socket connection is established, the benefit is headers are not transmitted, only the data is transmitted as frames. So that's the benefit of web sockets. The problem is web sockets, handshakes, might arrive at a different purpose. The solution is maintaining TCP sessions. So this is a TCP session. So for example, what you do is, there is this module called the TCP socket cluster, which is, again, I say you can use it because we have been using it in production. Do not ever use some stupid shit module, sorry, some stupid module in NPM because you will regret it then, because when we were using this memory leak issue, one issue was in some other external module. And in that case, we didn't really know where I was taking like, my goal is find this, find this, find this. We were taking snapshot and stuff, and in the end we understood, okay, this is the module which is causing the issue. Please don't have to do this, that's fine. So, sticky session is where, just this TCP socket cluster. It maintains this, so it has a cookie-based TCP session. So it gets a cookie, and then it will maintain, okay, if this request is coming from this cookie, go only to this worker. Do not hook it to the next worker unless that session is being entered, right? In that case, that stickiness is being maintained, but between the code and also a load balancer, or you can have proxy or anything, kill the client, this complete connection section as in the session is being made in, right? These are simple conventions which you can follow. Do not fight over several millions or two space or four spaces, doesn't make sense because it doesn't really make sense. We wrote this in GitHub, GitHub does a bit better so we wrote this convention for Git, and just when we use the standard style, you know Ferros, he's the one who wrote the standard style, like two spaces, no semi-colonies, and all the butchers. Like, as you can just go through and check, and how much time left, that's all right. Thank you, you can ask me questions. And, good morning, man. Hey, hi, great job. I myself have faced the issue with socket island sticky sessions, but the NPR that you mentioned that only works with socket island sticky. Yeah, it works only with socket island sticky. Yeah, and what if we want to use any transport legacy or which one? You can test the code as in the sticky socket cluster and do it with different web sockets. Yeah, okay. I cannot recommend any other package which I have not really run in the code. Cool, have you ever tried it now? Yeah, man, I want to take a look. It's from Google. Just wondering, do you ever have to tune GC intervals? Yeah, so this Lord's... I'll answer that. The question was, I guess, do you remember? Is there a way to tune the GC intervals? See? No, I mean, I'm asking that. Yeah, yeah, we have to, we have to. So I'll just explain that scenario. Lord's GC is very weird. It has this maximum size of 1.5 GB. In the beginning, when we were running our servers, to be honest, the server was running on firewall and we... No, it was very small, but to be honest, that was when we were running. So this Lord's GC has this thing. Until the heap's memory is 1.5 GB that's hard-coded in the Lord runtime itself, it will not run the GC, right? So the tune was... If you really look at it, the GC pause... So Lord's GC has this thing. Whenever the garbage collector is running, your application is completely paused. You cannot make any request the application, but it was in less than 10 milliseconds. The garbage collector, whenever it is collecting, so they have this mark-and-sweep recently, Lord's runtime implemented mark-and-sweep. So they basically mark these memories and then sweep at the end. So it is like less than 10 milliseconds, which we really didn't need to care about. We looked at the metrics. It was less than 10 milliseconds, which is like ignorable, it's fine, because it gives the automated memory handling right. So that was it. And if you are running any application in RAM less than 1.5 GB, then you can exclusively set what is the maximum limit where the GC is running. So don't run GC manually because that doesn't really make sense. Yeah. Yeah, so why don't you say Lord's container is... What is the... I'm not going to be an opinionated judge here because it really depends. I've seen a lot of scenarios where people, they use corridors and they use containers because they want to use it. What is the whole point of using containers? Like when you are, when you have a particular infrastructure and you really need these containers to talk to each other and if it is really improving... For example, I hate this docker and the docker recently has become more of a product and not this thing. So if you are using RKT which is why corridors, then it's pretty good. It is pretty efficient, but when you run it on docker, it is really not efficient unless you properly tune it and all these boxes and stuff inside because docker demer is one big pain to deal with. So that is why I said if you are 100% sure, you can run it and make a net net fine. If not, it's unnecessary pain. That's it. Anything else? All right, I'll be here. You can tweet to me at DOLFPAX. Don't ask me what that means. And I'll be here if you have any questions you can ask me. Thank you. I'm going to be the doctor soon and I'll be there for you in a few minutes.