 Thank you everyone. I hope that everyone is able to hear me even people at the end Great. We do have quite many applications. We are using every day. I Would say that I usually use about 50 including the ones that are on the phone That we always we quite often forget But the question is Do you really know what they are doing? What the applications you have installed on all your machine linked with Hundreds of libraries, maybe not hundreds, but ten libraries linked with a program. That's what I would say. It's pretty normal Do you really know what happens when you run your application? I would say from my experience John with the research and development for this project that I Would say that you don't know but Why would you like to know what the application is doing? I Will start from asking you a personal question. How many of you? Happen to do the bagging at least once a week. Raise your hand Okay, quite many of you. How many of you do like the bagging? Okay, I was expecting that there will be less of you, but great But I am a fan of you Because personally, I don't like the bug. I prefer to add new things to my programs then to then to debug them. However The research shows us that a typical developer Spends between 50 and 70 percent of their time debugging the application The debugging in fact what they understand is debugging Itself writing test maintaining code although those kinds of activities I guess that with your experience with different project you would say that it's mostly true if you really Count count the hours you spend With your project what it means that if we try to find out a way to Limit the time we spent on debugging We are going to gain a lot of time to add new functionalities to work on all the things that just a bug And we have as it's 50% of the time we have way match to gain To make the things even more complicated I'm concentrating myself on complicated program programs what I mean by complicated programs those that communicate either with other programs running on the same machine those that are Talking by network that are communicating with devices Be it in embedded in IOT on desktop. I would say that most applications We write to those days are those complicated applications and we do have interaction of this Application with the external world and the external world is influencing the way our application behaves As is not the external world by the part of the world that we haven't written ourselves as reminder from your classes on Software quality what we can do to limit the bugging There's a the process part so issue tracking code reviews testing but also the party of tools to so you They daily use of the bugger the traces if you are little little deeper Benchmarks if you are into performance to find out how your application behaves with time and with your changes And the advanced tools that you may use there are tools for different purposes and as a as as As an as an engineer we choose the right tool For the right task and I will give you a short overview of the advanced tools You can use to debug a complicated application first if I want to understand The network connection of my application what I use is way shark way shark allows you to see all the network packets that are transmitted a received from your application The small difficulty may be that to understand the results you have to understand the network protocols You have to have the basic understanding of how the protocols behave How the protocols should behave in the case of the application and what could go wrong? so for for many protocols you can have you can find The analyzes that just dump the data that is being transferred, but for some more complicated cases you will have to go and look Into the the the specification of the protocol or Have have the knowledge from other sources If you want to know How the application is interacting with the Linux kernel in this case What you can use is this trace program What it gives it gives you all the system calls that are being called With the arguments and with the results That is very detailed. This is very detailed information very useful The disadvantage First business venture is the size of the output you get Even this is not a complete output of a less it just wouldn't fit on a slide even over 10 slides It wouldn't fit and you also have to understand what is happening With all those Cisco's we have to understand what they do and why they are called In that way understand the parameters. They are passed understand the the written values If you want to know what happens on the inside of your application, you can use Perth Perth is even more If it's different kind of a tool, but very very powerful. Here you have an Example an example output of Perth that is recording context switches for an application Extremely useful in many cases I won't be talking much about the Perth because it Values a separate presentation or maybe several ones. In fact, there's one later today. I found out in the program So if you understood Perth you can you can learn much about this and about the things that Perth can do for you However I'm a kind of hacker. So I know this is cause I have networking background. So I read the hex dumps of the network protocols But Not everyone does and it's not the most efficient way to the bugger program So I started to think how to optimize my time and the time of other people what we can do To use the super advanced tools that can give you great results that give you enormous amount of information about your tools But you have to be able to use them So what what we can do and I said I started looking into the problem What I what I found out that what we really need is a layer of tools and documentation that will take the output of the tools that already give us everything and we show We extract and show you just the information you need There will be probably need of some tutorials on application or example applications And I I wanted to test if it if if that idea is going to work and I and I wrote an example first Analyze this application you have you can see two eggs the red one and the orange one They look more or less the same from the From the external just the color is different But you don't know what's inside What I'm going to tell you tell you here what I wrote is an is an is an app that allows you to compare two Executions of the same program So what can it compare it compare the accesses to files the network the communication with all other services and Any other systems call what I'm using is the stress The stress the stress output Now why would you do want to do that? Well, they are quite many reasons You can have an exec you can have a Non-condition when the application is behaving correctly and non-condition when the applications behaves incorrectly You can have different versions of to an installed machine. That's the reason you may have made an upgrade of your system And then the application doesn't work anymore You have different machines you may want to see for performance reasons if you change something in the system What happens with the application? Why it behaves differently and for for the bugging You they change it behaves differently you want to find out how it's interacting with with with its world And finally for security reasons if you do have a golden copy of the Execution trace you may be able to see if there has been some malware and start inside inside of your application If they the behavior hasn't changed during in the time now I will Show you the the user interface of a program That you should know Why? The K calc you probably have seen on the same app a similar program What I did and is I started to trace what K calc is doing and What we do have with two different Execution of it Is that we have a bunch of Directories it's using The library the system cute and KD libraries nothing extraordinary. We would probably expect that it's using it It is opening quite many things in TMP and In v2 But I found out that there is a difference I show I'm showing read the difference between two executions that one of the cushion execution was using the view random and the other wasn't That was happening when I opened the setting setting window That is for some reason by some low-level libraries Initializing the random number generator. I wasn't expecting to see that in a program, but it does it does it We have if you can see how many configuration font files The the the Qt is using at initialization. That was also pretty pretty interesting to see I have also another trace with two different different machines when you have Very big difference between the two so it was pretty fun to just to see what it happens in that small application What I've learned because I've learned a lot by writing this program. I've seen that There is enormous number of accesses to different system services from the libraries and For some it I wanted to understand why they are there What do they mean and it took a little a little digging around the around the place to find out what it is Some were unexpected from the from the first of the sites somewhere quite quite natural Another interesting thing for me was that I Wasn't aware that Very advanced and new system calls in Linux added in the last year. They are really used by real applications. I Knew that those Cisco exist. They have been added recently and They really get used by the applications That that that was interesting that means that That the new stuff is really being used and I've seen big differences when running those tests on different machines even when I was running the same test on Two different machines that we're running the same distribution and There's more or less the same version of the same distribution So those were inked just by the This was just linked to that the packages installed on the machine That's how I started to understanding the problems that we often see on the on the mailing list and bug reports That someone runs an application. It doesn't work for them On a system that for someone else it it works. It works very well It's just that there if you if you look if you look into the Libraries accessed in the file access. This is there is a big difference in all those two So I've learned a lot. I may write quite many other tools like that because it's pretty interesting. I Would like to have your feedback which tools to understand your application better you would like to see What do you would like to know more about your application? Send me an email later or ask me at the conference. I will be really Glad to hear from you what we have real problems with the your real applications out for the So scored I have just managed to push it today While removing some big traces that were in my private repo So you can if you want to to look around you can get it from githlap. It's it's gpr3 the graphical interface written in pi cute That was also interesting to use and you can you can reach me by email on all the means Do we have any questions? Yeah, okay. The question is What is the what is the what are the tools that I'm using? for For this for this development the answer is that currently I'm basing myself on a stress basically for the for the first part I Will be us adding a perf Perf perf measures to in the in the in the in the future The future that is coming soon What what I'm going forward is to have Have a goal don't have don't so have Have an idea what I want to show and that from that Get the get the tools I need to again the parameters the options the data I need to get from those tools I'm not going to pass a parse everything that is giving me the s trace And perf on the application because this is going to be extremely heavy Probably not useful, but I would rather prefer to have smaller tools that would just give you an of a small part of an overview the The directed directed tool Yeah, okay, the question is how flexible is the algorithm of the comparison? At the beginning it was it was a proof of concept. I wanted to go fast So I wrote the whole application in python even knowing that on the performance side Maybe it's not the best investors ever What I found out that Use I was I started using the I wrote it a pretty pretty general way There are attributes and fields and and then they are they they are sorted So I'm I'm comparing that the two sorted list at the end with different attributes It's not exactly a diff. It's a it's a data structure And in fact, I found out that Even in python comparing in that a way it's fast enough What happened when I when I launched the demo? It has done the comparison online That that that wasn't an offline comparison. I was just the complete s trace files on my disk But the whole processing was done online. So it's fast enough. I told myself that I could rewrite the engine later on in C or I can try to interact with other engines that that other If I if I need that currently It's fast enough. So that's good Okay, the question is Is it is it a better way to just add a user interface to to s-trace or perf As I said not both s-trace and perf are completely open source projects They are one of the basic tools that you can you can download for your for your system I Don't know why there there is no user interface graphical user interface for strays I would expect the data's were never interested in writing one For perf there is some kind of Graphical user interface However, it's not that easy to write a graphical interface for perf because of the amount of data and the amount of Configural ability it has it's pretty complicated to do it It it may be one of the of their directions to go forward because it would it would be nice to have that too Yeah, the question was if I have looked into LTD and G Yes, I did for the For the reason of time I didn't prefer present you LTD and G System tap and 20 other tools that you can use That would be even more complicated to explain what they do The problem we have in the Linux world for me that we have too many great tools We can we can get nearly all we want But you have to know which tool to use and at which time and also it takes time to master each of them Because they work they have their own particularities. So yes, I did in jeans one of the other sources In there because it's also giving some other things that may be interesting. Thank you very much then And see you later on the rest of the