 Okay. Good morning, everyone. Welcome to the third talk of the lightning talks. We are having here adding performance counters to HTOP by Hisham Muhammad. Thank you. Good morning. So we're going to talk about performance counters in H-Top. A little introduction about me. I am Hisham. I'm the original author of H-Top, a project that started way back in 2004. I'm also the lead developer of Lurox, the package manager for Lua, and the confounder of Double Linux, which is the craziest distribution you've never heard of. At work, I am a developer at Kong, a pretty cool open source API gateway that's also written in Lua. Check it out. So, short introduction to H-Top. Well, it's an interactive process manager. The original goal when I started out writing it was just like the original top back in the day in the early 2000s. It annoyed me. I wanted to try to make a better top, quote-unquote, and by this all I originally meant was to be able to scroll. Nowadays, the voice versions of top have improved a lot since. They all scroll now. But, yeah, back then it was a novel thing, believe me. So if you've never seen it, the light's not too good, but this is how it essentially looks like. You can scroll your processes. You have three views of your processes. You can select things and change priorities and things like that. So, let's start talking about the topic of the day is metrics in H-Top. So, I started in order to produce like a familiar environment, I started with a default set of top. I think it's still the default set that it brings now to today, but which was essentially what I'm using here in my presentation style. Like PID, username, priority, nice number, the three numbers of memory because you can't never have a single clear number of memory usage on Linux. So, like, we cop out and show you like the virtual memory, resident memory, and shared memory. The state of the process, CPU percentage, memory percentage, which is funny because, well, if you can't show me a real number of what the memory usage is, how can you show a percentage? You have to look into the source code to see how the calculation is done. Spoiler, I stole the calculation from top. Like I said, like I just want to show the same numbers. So, in order for people, so that people wouldn't say that my tool didn't work. Total elapsed time used by the process, CPU time used by the process, and command name. It turns out that most people never go beyond the default set. I've met a lot of people who didn't even know about TreeView in spite of the fact that I put in a small, this screenshot doesn't show it, but like there's a bar at the bottom, like the last line. It shows like F1, F2, F3, F4, F5, and all of the main features. At one point, I made TreeView into one of these features, like F5, so people would see it. But still. So, as I said, htop has a lot more than these default metrics. So, if you do find the setup key on F2 and press it or use uppercase C, which is compatibility key with top, you can enter the setup screen and play around. You move around the cursor and choose like the meters, display options, the colors, columns. You can move there and you can mess around with the order of the columns in fact, even more columns that are available there. Fun fact, originally I put in the full list of everything that the process data structure would return to me on Linux, but I didn't implement all the columns, and I just put them on the list and they were returned like blank. Because I was wondering when people would complain to me that certain columns weren't working because I figured that some of them sounded like they would never have a use for it. There are some that are still blank and nobody has ever complained like in all these years. Some of them they did and I did implement. So, speaking of metrics that are available in htop and a lot of people don't know about, like one that I would like to take this opportunity to bring your attention is IOMetrics. They were inspired by IOTop and it have been available in htop for years. If you go to that list you will find IO rate, IO read rate, IO write rate. So, it's like the number of bytes that were read during the bytes per second. And IO rate is just the total, the sum of read and write which is useful to use as a sort column. So, you can see like whoever is doing IO and IO priority which is really fun because you can select like real time best effort and you have like highs and lows and you can control the IO scheduler for your processes. So, those are already available. But even then I was always see like people using like htop and IO top side by side which is nice but the functionality was basically already there. Just like with top I was like inspired with whatever they did and I tried to bring it over to increase the functionality. So, and moving on to new metrics which are the topic here. Recent addition to htop are the delay accounting metrics. This was actually a suggestion originally by Brandon Gregg, the performance guy at Netflix who has done like some really amazing work with flame graphs and introduced a lot of performance concepts to a lot of people. He showed up in the htop issues list and posted in issues with like a feature suggestion like why didn't you guys add delay accounting. And then another user, Andre Carvalho, he decided to pick up the task and he implemented those three metrics which are like percentage of CPU delay, IO delay and swap delay. What does that mean? Percent CPU delay is the percentage of time that your process is ready but it's in the ready queue of the scheduler waiting to run. IO delay is percentage of time it spent waiting for IO devices to return its requests and swap delay is the percentage of time that it spent waiting for swap pages, memory pages to be swapped in. Quick note for disrumentainers. To enable those options with like dash dash enable delay account you require libnl for netlink support. And to learn more like in detail about delay accounting I really recommend Andres blog post about it. So inspired by adding new metrics like I got inspired to work some more on htop again and consider adding some more metrics because I was starting using Perf recently. And Perf is a tool for Linux that makes use of harder performance counters which are a tool that modern processors have. So those allow for really low overhead performance metrics in which like the CPU will account certain events for you like cache events and behaviors, other behaviors like branch prediction and things like that. So the CPU exposes that functionality that functionality is managed by the Linux kernel and it's available in user space like and the main way you can get to those nowadays well until today would be via tools like Perf tools which again a lot of people don't know about. Essentially to use performance counters as a programmer essentially what you have to do is you perform a syscall to request a counter so the syscall will return your file descriptor which we will periodically read and you will get the numbers. But when dealing with that you have to be aware that file descriptors and performance counters are limited resources. So if you want to start opening like file descriptor for every process in your system the htop process might run out of file descriptors unless you like configure it as so. And same thing for performance counters like if you have multiple instances of htop running some of them might fail to acquire the file descriptors for the performance counters. But this implementation is based on tip top by Evan Rojo and Antoine Nodon from Arial. They have great academic paper describing hardware performance counters in detail and again I was inspired and based on their code which is essentially a top like measurement just for performance counters only. So yay for free software and being able to build upon the work of others. So what were the performance counters that are coming up in htop? First one is IPC instructions per cycle which because of the nature of superscalar CPUs usually since you have multiple pipelines you are allowed to have multiple instructions running at the same time. So if you are using all of your CPUs time that doesn't mean you are maxing out your processor's capabilities because if you are using a single pipeline and it might have up to say four you are essentially using a quarter of your processing capacity. So besides using only CPU percentages it's important to know what's the IPC that you are getting. There are also some absolute numbers like millions of cycles executed and millions of instructions executed per sampling rate. Percentage of cache misses percentage of branch mispredictions and branch mispredictions of course being a hot topic nowadays. And also lots of metrics for you to look at the usage of your L1 data cache which is an extremely important metric for when you are tuning performance of your algorithms. So like the number of reads in the cache the number of misses that cause you to go like another level deep in cache writes and write misses. And those are being like CPU features it depends on the CPU model exactly which ones are available but those are usually available in modern systems. So to make it all manageable visually I always said to people that to me HSTOP was more of an exercise in UI design than in systems programming because the whole idea of the project was to try to make consumption of this information more manageable. I had to add the feature of multiple screens. So now in the latest version of the code you can have multiple screens with multiple configurations of columns and you can just press tab to switch between the screens. So this required a larger change in the format of HSTOPRC so this will require a major version bump. So I have a couple of minutes so let's go to demo time if I can manage to. So well just to compare so if you go to like plain HSTOP if you go press F2 and go to the setup screen you would get columns, active columns and available columns and if you go to like the latest and greatest version if you go here instead of columns you have screens then you can have multiple screens and those can have different sets of columns from the list which now includes it's hard to read but the ones at the bottom are the ones that I've been talking about. So that means that when we go here and press tab you see that those are really dark and say like not available it's not or no permission because I'm running as a regular user those are root processes but like if we go here and try to do something let's see, let's do something that's at the disk so we see now there that fine now because I'm running through the disk it's got a lot of IO activity over there and so if I press tab now I see all the other metrics we got like IPC of 2 over there the number of cycles, instructions and the CPU it's currently running on, those kind of things and then as we update we get the L1 information for some reason my processor I'm not getting the right misses but I'm getting read and read misses and writes to the L1 cache fun thing, if we go here and try to do that again we see no disk activity at all because now this read is all going through cache if you see over there like the blue and the yellow those are the processor caches so if I go here it's root, where the hell is it drop cache, there we go and I tell the kernel to drop the cache the cache went down and if I try to do that again then I'm starting hitting the disk again and it's all pretty easy to see like the cache went down in the graphical meter and now we can see like the numbers of disk activity going on so this was an easy way to visualize those are the screenshots in case the demo didn't work because it's like the adapter, the VGA adapter so in terms of availability well everything's on GitHub and it's just a matter of packing the tables and making the releases and soon becoming to the distros so the IO metrics and delay accounting will be in the stable range like 2.1 which I'm going to be packing and uploading today and for the performance counters and the multiple screens since it's a breaking change I am going to make like a better release and package it as HDOP 3.0 it's going to be in the website as well everything is Linux only but patches are welcome for support for other platforms and thank you very much