 Okay, this is the name of the talk, in case you are here for a different one. Don't worry, you can go to the one that you actually are going to do here. What I'm going to try to do today is show you, or explain you a few cases where you can have memory issues. I'll try to go through most of the cases. It's really hard to actually have a definite guide or something like you will see in a YouTube video. This one works, one link, mega, 4K, full HD fixes all the issues. It's really hard to get that, especially for memory issues. Why? Because they can happen for many reasons. And you can fix them on many ways. So what we are going to do is just try to talk about some of the cases I have seen. Some of them are worse than others, but that's it, yeah. My name is, I think you all know now my name and my birthday. That's my email, that's my Twitter, if you wanna reach out sometime. I don't tweet too much, but if you are into that, that's my Twitter. What the stock is, I mean, it just has to say, obviously this is more about talking and if you really want to, if you have a memory or you have something wrong with your application and you want to probably discuss a little bit more in-deep, we can just talk, later the talk, or I'll be giving a workshop at 2 p.m., just around the corner, about memory lakes and we can just run some examples and just all let you know how to fix them and how to identify them. So if you want to have something more practical, you have it. See you at 2 p.m. So this is the question I usually ask, why, because people are really excited to just raise their hand because hey, I have it and there's more people that are right that it is, yeah, yeah, you too, you too. So this is the question, how many of you have a memory lake or a memory issue? You can't remember the number of the times or you can't remember the last time. Okay, yeah, I mean, this is pretty interesting because there's something great about JavaScript and Node.js in general is that you really don't need to learn from the beginning about how to deal with these type of issues and I think that's what makes Node and JavaScript really a language that everybody can get and start and actually start to build stuff because you really don't need to learn this. The issue is that when you have to learn this, it's on Friday night, when production is down and someone is calling you, hey, why the server is, what application is not working, why is that? Or have you seen this application? Well, why this server, this service is not running. So don't worry, it happens in real life, it happens in TNT, it happens in all the companies. Do not worry, you're not alone. It happens to all, including the best one and I can't make sure, be sure that that's true. We have, at Node.js, we have been working with different companies. Some of them have products that you are using right now and with really critical process that they need to handle in Node.js applications and they're really bad and complicated applications. So don't not feel bad. It happens to engineers at big companies so why it cannot happen to you? So let's start with the most important part of detecting a memory leak or actually trying to fix it is one of the thing is check the heap profiler and the timeline. What the timeline is or what the heap profiler is. The timeline is just a tool and I think how many of you have used the crumbed up tools? Yeah, okay, you can see the timeline. The timeline is really not something that is going to tell you exactly what the issue is but it's pretty much for you to understand when or at what time it is starting to slow down because one of the things with Node.js applications is that you not only can do servers or press APIs or something like that. So the memory can happen, the memory leak can happen not only when you are receiving connection but the memory leak can happen when you are doing some processing or something like that. So it doesn't mean that the leaks or memory issues only happen on applications that you get load or something like that or customers just joining your application or something like that. That's probably one of the biggest issues too to try to understand that because you know something and you know that this could probably be an issue, it doesn't mean that it's going to apply every time. So you have to be really sure that you know your application, you know where are the bottlenecks and that's something that we are going to discuss later. So the timeline is just for you to understand at what time your application is starting to not work as expected. And the heap profiler is just to get all the information in all the nodes and at the roots inside your application at that time. Or at the time you took the heap profile. So when you are trying to fix this type of issues you really need to have a workflow on how to detect or how to fix it. And this one is actually a really good one. I call it TST but it's the 3D snapshot technique but it depends on the case because sometimes it's worse. So I change it for TTT that it means 3, no it's 4. So FST is 4 snapshot technique or 5 snapshot technique. Why? Because it really depends on the application but if you're just going to start digging and just start to understand what application is doing you really need to embrace that way to figure out things. And I didn't invent this, it was actually someone at the Gmail team, it's called Lorena Lee. So thanks for that. What this means is that you take a heap profile at the beginning of your application as a cool start. Why? Because you need to make sure how your application is in memory when it starts. Then you start to increase the load. So you just start sending more and more requests or whatever your application does. And after a certain point it could be one minute, two minutes, three minutes, you take another heap snapshot. And after more time, it could be another two or three minutes you take a different one. So why would you do that? Or why would you take three snapshots at different times? Well, it's because you really want to make sure which objects are the ones that are increasing over the time. And the only way to know that is knowing how your application is started and after a period of time which objects are newer or didn't disappear from the first snapshot compared with the second snapshot. And if you want to be completely sure you can compare the second snapshot with the third snapshot. So you are making sure that you are going to look only for the objects in your heap snapshot that are still alive, that are not being garbage collected. So that's a really good technique, you should totally use it. So there's something in the Chrome DevTools and I think how many of you have taken a heap snapshot before? And there's a bunch of columns that you see a percentage of for each column and usually you don't know what any of them mean. And it's fine, I mean, it takes some time to understand all of that and what is important. But something I really highly recommend is to check for the distance. There's a column called Distance in the Chrome Developer Tools when you check a heap snapshot. And something I have seen for a while is that the closest and the constructor that has the closest or the minimum, the closest it will be like two or three number in the distance column is usually the one that is responsible for the memory leak. So that's something that's not completely true for all the applications, but I have seen many cases and that's what I can usually recommend. If you see a big constructor and that constructor is really close to that route, that means that you should actually go for that and start to look on that direction because it's pretty sure that that constructor is going to have the memory leak. Yeah, so check for constructors. Usually check some of the call stack that's something that I'm going to recommend at the end, but how many of you name your functions or you just make function arguments and all that stuff? Well, when you don't name your functions and I think it was kind of fixed by the time and I think in node eight, something was implemented on VA that it didn't matter that you didn't name your applications because something in VA was more intelligent than us, so it was kind of generating names for unnamed functions. However, when you start to look at the heap snapshot, you start to see in the call stack that there's a lot of functions and they don't have names. So there's actually no way for you to track where this call is being done or how you can actually fix the issue. So I highly recommend just put a name, don't put an X or a Y or something like that on your functions, it's not really helpful. Seriously, when you are trying to debug stuff and especially for heap and memory, you really want to make sure that you have all the complete information. And this is only talking if your application is not minified or transpiled. If your application is minified or transpiled and you don't have source maps, then you are going to have a really good time trying to figure out what this strange symbol transpiled to my code. So have fun with that on Friday. This is something that people actually don't know is that you can generate redacted snapshots. What this means is when you take a heap snapshot, it takes, you are going to see all the strings. In most of those strings have sensitive data. So if you are going to send this heap snapshot to someone, just please make it redacted. So any of the strings, I mean, the content of the strings are not important for the person or the company or whatever or the service that is going to fix the issue or actually do consulting for you. But do it because it's probably going to have all your secrets of your application, including credit cards, information and all that stuff. So really take care of that. If you don't strip the strings in your heap snapshots, you are giving someone else information about your application and probably your users. So just be careful because you are sending information to someone else that is really highly relevant. And there's this one, card dumps. This one are really interesting because it's probably less use and usually when you use, the people who use card dumps, they usually are familiar with C++ and all the devlogger tooling that's out there for C++ or OC. But you can actually do card dumps on node two and it's weird because when I talk about this, people do not know that you can actually do that and actually have more interesting information because we card dumps, we can, in real time, it's like a travel time when you can actually know the state or application or something at certain point or a certain period of time. And that doesn't happen with the heap snapshots. You are the one that is trying to collect all the information and the memory at certain time. So card dumps are really interesting. They take a while to understand, especially if you are not familiar with any devlogger on C++ or OC, to create a card dump. Usually what you have to do is, most of the time just send a signal to a board or you can put on your code the process that a board. But what that means is that you have to modify your code to actually trigger the card dump. And the card dump is expected to be triggered when there's unhandled exception. However, you can do that or you can just use this option when you're running your application so you don't have to modify anything and it's going to award every time you have an unhandled exception. It's just way better to not modify your code and especially if you are on production, you cannot modify your code to actually get all the unhandled expressions or every time you throw something to do something weird. And something is when you get the core file with the previous one with the process that a board, you get more stack traces that are not really relevant to the problem. Why? Because you are modifying the code to actually make it break at some point, to actually abort. So don't actually do it if you really have to make sure that you are aborting at some point and you are not able to trigger. Because it's very normal. I mean, how many of you have had an issue that is on production? But it never happens on development. Yeah, it's like every day, right? That happens all the time. And usually, there's a few things that you can do for that, especially with the stress testing you can actually prevent some of those cases, but there's another talk. So yeah, you can use that, it's better. You don't get too much things on your core file, get only everything that you actually need. Then you can use AlolaDB as the vulgar, it's not database that happens a lot, that's confusing. You can use that to actually inspect and get all the information that you actually need from the core file. We can go more in-depth on that, but that's trying to explain how the vulgar works for C++. And I think that's too much. There's something called a node that is kind of plug-in for AlolaDB. What's the thing with this is that you can actually get a stack trace, JavaScript stack trace. So it's very useful. What I think is, I'm not sure if it's highly developed at this moment. Something that is not good about core files is that you have to make sure that the vulgar that you are using and the core file are on sync on versions. Why? Because if you are using a core file that has, for example, something in V8, or a different version of V8 than you the vulgar has, then you're going to have a problem because if there's something that changed between versions, then the developer is not going to be able to catch that. So you are missing information. You are making core files out of nothing. And it's not going to be worth it because you're not getting the whole picture for your application. And that's one of the things I would say. Just make sure that you are running the same versions between the vulgar and the moment you're building the core file. If not, then you are going to have a hard time trying to figure out why your core file doesn't have certain things. Because it's probably named different. Sometimes in V8, something in the structure just changed the name. And because of that, there's no way to know that. There's no way that the vulgar and the core file are going to link that because they don't know what happened. Promises, something interesting with promises is promises are really hard to handle. And something with memory is that when the promise is not the same when the promise finished to when the promise worked. So when you take, for example, a hip snatcher or something or a card dump, sometimes you get the promise finished. But it doesn't mean that the state of the promise can change during that time until it avoids. So that's one of the unknown things about promises. That means this is one of many. And I think you on the previous talk just saw that. So make sure that you understand promises really well. If not, you are going to have a really hard time. I think we all have with promises already a hard time. But I actually recommend a sink away. So please, if you are still not using a sink away, just do it. It's really important that you try to embrace this. It's way cleaner. It's much easier to debug. And it's going to be less painful. I actually really highly recommend it. So I have some tips for different situations. I know that people usually look for something to not do. So they can avoid this type of things. One of these is on server-side rendering. I'm not saying you should not do server-side rendering. It's just that you should do it properly. Why I'm saying this? Because we have seen companies doing server-side rendering that they have a really big abuse on how they do the transpilation or all the processes of the server-side rendering. And when you do that, you have to know that there's a cost. There's a trade-off. And what that means is that a simple Node.js application can start on 20 megabytes or 30 megabytes if it's too complex of a memory on a cold start. However, for a server-side rendering, it can start on 200 megabytes of memory, 300 megabytes of memory. And that's just starting. It's doing nothing. Why? Because most of the server-side rendering, all the components of everything that is being processed, is storing memory. So when you do heap snapshots of server-side rendering applications, you are going to see many strings or many templated strings on memory. And that's how pretty much a server-side rendering works. So it's going to be way harder to actually strip all those objects or strings or whatever the server-side engine is creating. So if you're using server-side rendering, make sure that you know what is the trade-off and you know the cost. And that's something people do not understand. Usually, and I understand that there's reasons for some architects to use something. But usually, memory is not that. Because we are so used to when something doesn't work, it's just put more memory, more cores, and let's make it. Because it's pretty easy in AWS just to increase that. So it doesn't matter if it's not working, just put more memory and more cores. And we will be fine. In case two, it's monitoring tools. Make sure that you are choosing the right monitoring tools. Why? How many of you have been using Dynatris at some point? Probably a few. Well, if you do a server profile or do a heap snapshot, you are going to, or if you see the call stock, what you are going to see is that Dynatris is grabbing almost every function, every call. Why? Because Dynatris needs to know what is happening with your application. But the way they do it, they grab all of that, every call. So you just make sure that I have seen people just blaming Dynatris because this function is taking too much time or it's way too common in the heap snapshot. But most of the time, there's agents on monitoring tools that they just grab the calls because they need to get information. But it doesn't mean that they are responsible. And that's something that you learn with CPU profiles and heap CPU snapshots. So just make sure that there's a few tools that cause an overhead on memory. But you really have to understand how the monitoring tool that you are implementing actually works. If not, then you are going to have a hard time and you start calling Dynatris, hey, hey, you just brought my application. And that's not good. Close your connections. And I think I have a really interesting case here. If you can see, let me close that. OK. I tried to create what I'm doing. Let me see. OK. I'm not sure if you can see it. Well, this is just a Mongo application. It's a press API. It's just pretty simple, nothing special. It has Mongo database that is being loaded with random data. And this is probably the closest exercise that I can find to explain people why you should close your connections. Because if we, this way, I'm going to mirror. So I can, if you see, here, I'm just going to start, then I'll start the Mongo. And just ignore the fact of the cut that just appeared on my screen. And 1,000 tabs on my, OK, this is unsolid. If you never have used that or seen that, it's a monitoring tool. You can do other stuff. One of these, this one has something interesting, is that you can see this graph is CPU for the x-axis and memory on the y-axis. So our application is here in 12 megabytes, between 12 and 14 megabytes. So I'm going to just add a bit more of some load. You see how Mongo starts to get crazy. And you see, that's, and it just got increasing memory. And that's fine. I mean, that's how it's supposed to work. If it recovers, that will be the natural way. But it doesn't recover. It never garbage collect the rest of the objects that it just created and doesn't need anymore. So if you see, and if I keep increasing the load, I mean, another 100 users, you see, it's going to increase, increase, increase, increase memory, but it's never going to recover. And this is a memory leak. This is actually a memory leak. Why? Because the application is not recovering. That's how you identify that you are having memory issues. So how do we fix this? It's actually very simple. You just do kind of close after you send the answer. And the issue was that you were not closing the connection after you answer. Now you fix it in the memory leak. So this is one of the things that I really make people understand is that it's just full of best practices. Don't think that if you're missing something and the application doesn't break, it doesn't mean that it's not doing wrong things on the inside because it actually is. And it's going to break later and really hard. And it's going to be way harder to recover and try to debug and understand what the application is doing. So the only issue here was it wasn't even Mongo. But some people blame Mongo for everything. I mean, I blame Mongo for many things. But at this time, it wasn't even Mongo. Mongo was fine. We are doing everything fine, except that we don't close connections because we think this is fine to have connections open. So where is it? Yeah, so close your connections. Please do it. You don't want a lot of things being kept, especially if you use Apollo. How many of you use Apollo? Well, you probably use it or GraphQL. When you keep your connections open, the objects created for Apollo or GraphQL, they are stored in memory. So you are keeping all of that and not getting garbage collected. Look for new service pages. What this means is the way you serve stuff actually matters for your application too. So make sure that you implement a way to be not attached to a certain platform, but you can actually try it on different ones. Why? Because a few times is the way you serve things. If you can actually try different ways, it's really helpful for you to at least discard the way you are serving things. Something interesting, and we have seen a lot, is review your package that are CSS2 or in JavaScript, or two JavaScript or something like that. Just please review that, make sure that you know what the package is actually doing. That's a lot of translation, that's a lot of compilation, that's a lot of things that are happening there in those package. If you are really not sure or if you think that's a cool package, just make sure that it doesn't impact your memory and especially performance. A lot of these package store many things on memory because they have to do templating of the things that you do and what that translate to, for example, JavaScript. So please check that if you have a dependency, just check it. Keep naming your functions. That's something I explained before. Please do it. It's really helpful. I'm telling you, you are going to be really happy to know that you know the specific function where the issue is. Again, if you're doing a lot of minify code, or if you're working with TypeScript, you really want to start using source maps so you don't get unreadable code in your snapshots that you cannot understand or fix. Yeah, and this one is the most interesting one because I have seen combinations of Node.js and Java back in servers and many other combinations. One of the things is I have seen Java servers or database that send information to Node.js services. And one of the things is we had a team that was blaming the Java team was blaming the Node.js team because it was taking too much time. It was taking four seconds to process many things. But we just jumped a Node.js to the call and we figured out that the Java service was actually taking the four seconds to process and get back to the service. So make sure that you actually understand where the bottlenecks and sometimes it's a network bottleneck. It's not your application bottlenecks. Or you can just blame the Java team. That actually works sometimes. Crazy, right? I mean, and I could just keep going and keep going and keep going on all the reasons and possibilities that you can have actually a niche with memory. But this is pretty much a whole view or a general view how to look for and what probably you don't want to do or at least try to avoid as much as you can and try to fix it later. That's it. Thank you very much.