 So guys, today I have one challenge here. It's my first talk in big event and my first talk in English. So probably I will make some mistakes. My apologies for this. And I start to study Elixir last year, and I create a backlog with some questions and topics that I already know how to answer with Rails and Ruby. Questions like how to deploy, how to debug, and how to use async jobs. And today I want to share a little part of my experience with you about debugging techniques with Elixir. So what's debug? Debug is a process of the five years. It's a common process in the day of developer. We know that isn't a easy process. Sometimes we face hard monsters too big. And in my opinion, this is the main motivation to study debugging techniques, because you can focus and solve the problem, and you can learn more about your language and your code. And Elixir is the Erlang virtual machine. Erlang has more than 30 years old. And when you install Erlang, Erlang ships with applications to debug and maintain. So in my journey, I learned tools to use to debug and when I need to use each one. And the system formation is a little bit different that I usually use in Rails. So I learned how to analyze this. And as Erlang is a distributed system, how to connect to different nodes. So nice to meet you people. My name is Eric Kich. I live in Sao Paulo, Brazil, the biggest city from South America. And if you visit us, it's a beautiful place to go to Ibirapuera Park. And I am the organizer to catch Pokemons near our office. And it's a hard job, because I need to back our office with life. And when I am not catching Pokemons, I work at Platform Attack. We are a consultant in Elixir. So we do Elixir code. We do design and review. Also, we do custom development. If you need something, talk to us later. Maybe you'll know us because some open projects, open social projects, like the Elixir lang and device and simple farm dance from Ruby ecosystem. Maybe you'll know the Platform Attack because Jose Valin. Jose Valin is the co-founder and Elixir creator. But one thing that you don't know about us, we export talents to the world. So probably you don't know our Brazilian counter singer from Poland, Jose Valin. She needs somebody. She's got a plan. She'll jump out of my lap. So when you see the next picture in Elixir, probably you'll know the soundtrack. And Valin, I love you. I hope that we are friends after this. So I start my journey in the Elixir guide, probably the same point from almost of you. And in the first page, I did my first example. I create a script file to print the hello world. And here I met the IO puts. It prints the output to the device. So I usually use a lot in Ruby. And I use everywhere in Elixir. So I use in my scripts. I use in the IXL. Until I face my first error some hours late because you can't use with some different types. And I searched this in the Google. And Stack Overflow answered me just, oh, just use inspect. I try it. It works. And I feel happy again. But what is the difference between puts and inspect? Puts use just char data, so strings or list of char sets and strings. And inspect use the inspect protocol that transform any data structure into readable text representation. Later I found this in the Elixir guide. You can see more in the protocols section. And you can use inspect in other parts of your code. And I thought, from now on, I only use inspect. This feeling is true, but not only because it works. Inspect have two great features that I want to share with you. The first one is the pipeability. So if you have this example when you try to multiply by 2 each number from a random list, but how do I know the input values? I can use this to break and bind in a variable the list and inspect it. But Elixir has the pipe operator and inspect uses this. You can insert in each step that you need. The inspect will print the values and pass through to the next function. So if you have a lot of transformation of your data, you can use a lot of inspect to it. Inspect has another great feature, the options. And I use it a lot to debug. One of them is the label. You can create tags, label, in your code. Because when you have a lot of inspect, you can feel confused with all of the outputs. So for example, we can create a lot of labels. This is only a master. Probably will be released in the version 134. But I want to thank you, the Michael Moskawa and Andrea Pat for this. Because before it, I created a lot of inspect, one with label, oh, the list. Inspect, now the list. So this helps a lot to debug it. Another option that we can use in inspect is the limit. When I am debugging a Phoenix application, I usually inspect the connection. And we have a lot of information in the struct. And we can use limit to see all what you need. So for example, if I want to see only the assigns, I can use limit five and check the information. Inspect has a lot of other options. And you can check in the documentation. So from now on, I am sure that I will only use inspect. Some developers prefer to be simple and only use inspect. Others prefer a more robust tools. So Elixir has the IXPri that inspects the state of a particular process. Maybe you already heard about pri in the Ruby. It's a little bit different, so let's see the IXPri in action. It's a similar example from the previous example. We are multiplying by two a list, given by the user. And to use the pri, we just need to require the model and call our function. So in this case, the function double. When we call, we start the pri. And things to keep in mind. The first one, we have access only to the Lexi scope. So it's bind and processing information. When you start the pri, the shell will be his set. So you don't have access to the variables that you bind before. And when you finish the pri, after respawn, a new shell will be started. So let's see this in action. So I start my example here. We have a pit for this process. When I call our function, we receive a message to allow pri. And this process will be the same, the same PID. But we don't have access to the variables created before. But we have access to the list information. And after we finish the pri, after we respawn, we create a new process with a different PID. So it's a powerful tool. And we can use pri with Phoenix. So for our example, we will use the Phoenix example from Chris. It's a set that uses the channel feature. And in this case, I will try to pry in the handling function that try to send the message to all the users. But when I try to do this and send the message, I receive an error. So we cannot pry. We need to use a EX shell. So to solve this, to use pry with Phoenix, we just need to start with IX. EX pry can be used with test, too. And we call the mixed test. We will receive the same error. And just use the EX shell to solve this. In the text example, after we run the test, we need to finish the process and call it again. EX pry is awesome. You can use it with Elixir and Phoenix. But it blocks the process. So don't use it in production. And EX isn't a traditional debugger. You can't navigate among the breakpoints or other things. So for this, we have the debugger. The debugger is a visual interface. It's from Elunger. And it's easy to start. We just need to call the command debug start in our shell. It will open the GraphQL tool. And you can set almost anything in the GraphQL tool. First, we need to run another command. We need to interpret the specific module. In our example, the RuneChannel. After this, we can see the module in the debugger interface. So we can use all the interface to set the breakpoints, for example, in our handle in the Lightning 44. And when we send the message, our process is break, so the second line. And we can inspect the value of binds and navigate among breakpoints. But as I expry, it broke the process. So don't use in production, too. And unlike some of the burgers, you can change all the values of the bound variables. It's because you can change the values of variables in your long journey. So PID reference binaries or port values cannot be changed. Until now, we saw some tools to use, but only in development. So now, let's see some other tools that you can use in production. And we can use this for code and behavior debugging. Before it, I want to stop and tell you a small story. I need to talk about the Erlang virtual machine, more specifically about multi-core schedulers and reductions. And it's very important to understand how it works, because the system information is a little bit different. When you saw information about CPU, for example, you can check that it's different from other systems. So Erlang, after release 13, has support to symmetric multi-processing. This takes full advantage of multiple CPU architecture. So what does that mean? For every core, the band virtual machine starts a thread that runs a scheduler. So we have a CPU with a scheduler, and it's run on kill. It's run kill. So each scheduler is responsible for a group of process, and immigration logic will keep the schedulers balanced. So schedulers work is a primitive way. Schedulers can stop and resume the process later and use the reduction count. It's a long story, so let's see an example to show these more in details. So the scheduler is working in the process, and after finish that process, it will get the next available process in the run kill. And this scheduler, this process will start with reduction count equals 0, and scheduler has a limit to process. Sometimes its limit isn't really it, so the process reduction count is increasing when the process is running. Can be a ship function or expansion function. When it's an expansion function, the process is preempted, so it stops running, and it plays at the end of the run kill. And the scheduler is still working, get the next one until finish all the run kill. So this is how the Erlang VM works, and it's important to understand it because the next tool is the observer, and the observer is a graphic tool for observing the characters of Erlang's system. As the bugger, we can start just with a command in our shell observer.start, and we have information about system, about memory, applications provides three, and so on. You can find a lot of information in Observer, but I want to focus in two errors and two features. So the common errors that we found in our application is the CPU bottleneck. So when we start Observer, in the first tab, the system tab, we can find information about CPUs and threads. Here we can see the available and online schedulers that we are using in our example. And in the tab load charts, we can see the scheduler utilization from our process. But you don't have the CPU utilization, and why? CPU is very hard to profile. The VM does a lot of work unrelated to process when it comes to scheduling. So we saw this when we understand how the fix works in the Erlang virtual machine. And to avoid going to sleep when work is low, the threads that control the Erlang scheduler will do busy looping. In our example, after some time, we can see this. So here we have only one scheduler that is working, and the other accelerates down. All the process were moved to this scheduler to save energy. So how do you identify bottlenecks? So you can identify with reduction count, because it usually means a high amount of CPU users. And you can find the reduction count in the observer. We have the process tab, and all the process has its reduction. So let's get one example. We have more than 1,000 reductions. And when we see more information about this process, we double click the process. We can find a different information. In this case, we have more than 2 million reductions. And why? This information is the cumulative information from the reductions. And we can change this in the process view to check all the reductions. With this, we can find some process that are using a lot of reductions to run the process and your functions. And we can check some separable bottlenecks with reduction information. The other common error is the memory week. Observer showed to us the total memory that the application is using and splits in some groups. We can find to the memory users and the size in the memory allocator tab. And the information for each group is different. So I want to list three mistakes to avoid to not raise a memory week. So sometimes we use dependence that use ETS tables. Or we store a lot of information there. And we forget that ETS tables are never garbage collected. So we need to do this manually. Another mistake is creating dynamic atoms. So dynamic atoms are cached forever. And the most common failure is the message queue. So we have a process that sends a message to another process. And all the message is recorded in the mailbox. And when we not flush or treat this message, we can raise a memory leak in this process. So now I want to show two other features that we can use in Observer to identify errors. The first one is the crash dump. It's like a kind of travel machine. Because when your process die, a dump file will be created in the application path. And you can use it, the Observer, to examine this crash dump. So with this file, we can have access to all the process that was running when it crashed, the memory allocated, and so other informations. It's very helpful to lead your debug process. The other feature that I love in Observer is tracing. Erlang has a lot of modules to help you. So you have the event tracing, dbd, and ttb from Observer. And we have the tab trace overview. So let's use the same example from chat, the random in function. So to use trace, we need to add the node in our example, myApp, localhost, and select what we want to trace. In this case, just the receive message from that process. And when we start the trace partner, we need to choose the module and the function and what we want to show in the trace. There are some saved maps. We will use the message color and start the trace. When we start and send a message to this process, we can see all the information that a process send to another. And if I want a message like hi, I can see this all in the Observer. Observer is a lot of modules to show this information. So we have 30-party applications. For example, we have the ErweBurley. ErweBurley has access to the system information, check the code and tracing. You can use it too. And we saw how to use the bug, sorry, how to use Observer in development. But we need to know how to connect to a remote node. And it isn't hard, but you may find some troubles. My first trouble that I see is the module Observer can be not available. When you release in production your code, you create a self-contained package that strip all the unnecessary applications to make the fine output right away. So probably we will see this error. The Observer is not available. And to solve this, you just need to add the application runtime tools. It's a low footprint tracing debugging tools. You just need to change your mixed file and add runtime tools. Another error that I face it is because it's normal to use a server without a graphical interface. Sometimes we use the Linux distribution with Erlang. And so of this, we need some steps. The first ones in name or remote node. When you create a package, you have the file VM args. For example, you can change the name and the cook. This is important because Erlang has a demo that acts as a name server. The Erlang port map dimmer, the APDM. APDM run in the port 4369. And when you start our application, passing a name and a cook, for example, your application will be registered in the APDM. And now all the nodes can see this application so you can check all the nodes from your Erlang network running APDM names. After this, you can connect another node to this Erlang network. So you can run observing in standalone mode. It's the best prets to use this in production and development because this minimizes the impact. So if you are trying to debug or check some time formation, it's a good practice to use this to not freeze your process. So to start Observer, you can use the same cook but a different name and can pass the information to it to start standalone mode. And your server will be registered in the APDM and you can check all the nodes again, use the APDM names. So after this, you can check all nodes that you can connect. In our example, we use this but depending off your configuration of your server, you will need to use a CCH port forwarding to connect your local PDM with the remote PDM. And if you want to do this, I blog posted this before so you can have all the code that you need to configure your server. Another tool that it's great for production and development is Cecil. And if you don't use it, I recommended to start to edit this after our talk because it's very direct supervisor, cache and progress report to Logger. So you have more information about all the Observers. For example, you have information about the configuration when it starts, when the process die and so on. And you can start this just flagging Logger Cecil reports as true or you can add the Cecil application to the mixed file. So what we see in this start point of my journey, so let's recap. Some tools we can use only for development. So IX, the burger and if you will try another tool, I recommend you to check if the tool isn't block your process because if you're using production, it's not a good idea. And we saw the system formation is different. If you try to see the CPU, probably we will see only without assessing your application, we'll see the same formation CPU. So we need to know that the schedulers can be configured, you can use schedulers, you can set to down or you can create another more scheduler than your threads. Crash Snap is our friend. So it's normal to not monitor all the time your nodes and when something bad happens, you need to check and all the information is in this file. And you can run Observer and other tools locally to connect to remote nodes. So you don't need to create another structure to see your production nodes. But this isn't the end from my journey. I have next steps that I am studying and I recommend you to check, for example, the applications from along there has a lot of applications as the Alizer, Observer, and so on. Tomorrow we have a talk about the Alizer, for example. Another repository that I recommend is Osmo Elixir there is a section for debugging with other tools. You can have some monitor tools like Elixir Matter and so on. And if you like, if you want to see more information about monitor, I recommend you to watch the measure your Elixir application talk by Hena at track two, so Hena is here. There are two other great talks in the Erlang solution channel in YouTube. The first one is the profiling in the bug Erlang system by Roberto and Martin. This will talk about how to solve a real problem from application, how to use the information that you collect to solve. And the other is the debugging complex system by Luis Felipe Gautier. So those talks are awesome. I recommend a lot to watch. It's a different point of view to debugging. And one book that I recommend is the Erlang Yanger from the awesome guys from Heroku. And you can find more information about the Erlang Virtual Machine and some bottlenecks that you can find in your application. So guys, I hope that this information can help you to debug your applications, can help you to find bugs easier. And thank you. And these slides will be in my speaker deck. Some questions guys? No? So thank you guys.