 Okay, so I'm not a DevOps person or an Ops person, and currently I'm barely a Dev person. But so I came to this conference and I was expecting lots of old Unix ECIS admin types with Neckbeard and stuff like that, and I was pretty disappointed that this was not the case. So I thought I would try to inspire people to live up to... So what I thought is I would try to inspire people to go back to their true Unix roots and start growing Neckbeard, and I guess everybody's on board for that, right? Right, okay, sorry. So that's not actually what this talk is about. About a month or so ago we were having some sort of random discussion about latency tuning or something, and somehow I was on this mailing list, so I sort of stuck my nose in, and Kiran said, hey, why don't you do a talk? And I didn't know enough about it, so I said I won't do it, and then when I came here I thought this might not be useful to anybody at all, so let me do a talk about it. So what I'm going to be talking about is this thing called Perf. It basically is a profiling tool for Linux. It's very lightweight, it's sort of the canonical profiling tool for if you're working on kernel or low level stuff, but I think it could potentially be useful for you guys when you're facing performance issues. I'm just going to quickly walk through what you can do with it. How much time do I have? So the first thing you can do is called Perf.top, which is something very similar to the top command, but there are some differences. So what I'm going to do is because I don't do DevOps stuff and I have nothing that looks DevOps-y on my machine, I'm just going to play a video, just believe me that there is a video playing over here, and what I'm going to do at the same time is run Perf.top. So what you can see here is not what you would normally get with top. Let me make this smaller, yeah. So what you see there is not what you normally get with top. What you're seeing is basically like a per DSO per dynamic share object capture of what functions are taking the maximum CPU usage. So since I'm doing playing a video, video decode is obviously the thing that's the highest CPU user. So let's see. I want to see what this thing does. I can actually go deeper, I can go and see per thread, what per thread or per process, what is happening. What I can also do is go look at the code right there and see performance hotspots in the code. And we were lucky this is some of the SFMP code that isn't actually assembly. So you've got the actual source code, if your system is set up for it, and how much CPU you're spending and that sort of thing. So that's one possible usage of this. You can also do this another way, which is you can do Perf record followed by Perf report, which is the same thing in a sense, except for you store the data and you can analyze it in detail later. I won't talk about this because I don't think it's time. One minute. Okay, cool. The last one that might be useful to you is Perf shed, which is a bunch of latency measurement tools. I'll just show you a quick demo of this, hopefully that will work. So you ran this thing and you thought the latency was higher than it should be, and you're wondering why that is. What you do is you run Perf shed and you record all the information that you want while this thing is running. And then you run Perf shed latency, which gives you an idea of the average delay for that particular task to be scheduled and the maximum delay. And say if it wasn't two milliseconds, it was like 15, and you're trying to get like a 10 millisecond latency, you know that, okay, there's something went wrong. At that specific time, which is in the next column, which it's not showing up here. So once you know what time this event happened that caused you a scheduling delay, you can actually walk through what's happening on each CPU. Each row, each column there is one of my CPUs and you can see what thread was scheduled at what time and what was swapped out, what was swapped in instead. So that helps you tune latency and things like that. And finally, it's over almost, right. You can do syscall profiling, but we don't have time for that. And thank you for listening.