 But thank you, thank you. All right, so the slides are already online and ready to go. If you bring up the URL there, slides.unsupported.io, slash explore-estrace, at least when you're looking down at your laptops, checking your email, things like that, I always think you're looking at my slides. So it's helpful. If anybody wants anybody to get them, we'll leave it there for a few seconds more. And cool. All right. If you do bring them up, the cool part is as I go along, you all will follow along with me, OK? So like I said, your system calls in you a brief exploration of S-trace. I don't like staying up there very much. There's demos at the end. I'll come back up, but usually I'm down here walking around. It just makes it easier for me to kind of go through this. All right, so first off, hello. And I think a lot of people don't start off with at least hello, right? So hello. My name is Alex Suarez. I wasn't expecting an introduction, that's why this slide is up there. I wasn't sure how the audience would sympathize with Adele or Lionel Richie, so they're both up there, right? So again, hello. And also, thank you for coming out. There's a lot of great talks at this time slot, something I wanted to go to. It would just be weird if I didn't go to them, or I didn't go to this one instead. So thank you for coming to this talk. Thank you for spending your time with me. I really do appreciate it. My goal here really is to entertain you long enough to learn a little bit of something, hopefully walk out with a little bit more knowledge, hopefully know less knowledge. That would be weird as well. So how are we gonna get there, right? Our agenda for today, kind of the topics we're gonna talk about. First, we're gonna talk about why this topic. Why spend 45 to 50 so many, it's talking about a very simple, very simple tool, very basic topic. Then we'll talk about what is S-Trace? Experience level in the room can probably vary from 10, 12, 15 plus year admins to two or three year Linux users. So I wanna get also on the same page on what S-Trace is, the same Blingo, the same jargon. And we'll talk about system calls, which are kind of the core of everything going on here, and kind of the meat of it. After that we'll do some demos, because what's the presentation without demos? Chance things go horribly wrong, it's always fun. So, so far it's like good, Wi-Fi works, this is working, you know, so that's kind of cool. Then we'll do continued learning. I can't show you everything about S-Trace or by anything in 45 minutes or 50 minutes, right? It would be impossible to really cover everything. So what I'd like to do as part of my role, at Rackspace as part of my role here is to get you enticed just enough to go out and learn a little bit more and give you the tools and the opportunities to do that in an easy fashion, right? Make it easy for you to learn more, make it easy for you to go out and just pick up more information. Lastly, we'll do some Q&A. And by the time it should be lunch, so here we go. So why this topic, right? Again, why spend 45 minutes talking about S-Trace? For me it's two things. The first thing is about solving problems. At Rackspace, one of the things that we do is, one of the unique things that we do in a managed hosting environment is that we get to see thousands upon thousands of configs every single day. While most people have environments where they run 10, 20, 100 plus servers, most of them are the same. Most of the configs are the same. They know what's running on them. At Rackspace, in a managed hosting environment, we have thousands of configs every day, running everything from your normal lamp stack, very basic stock install, two very custom apps, to the Java app that has been updated in five years because developers know where to be found. And so we have all those bits of information, bits of things going on. What that does is present us with lots of opportunities, we'll say, opportunities to learn and to figure out things, right? A lot of troubleshooting, okay? And also, this talk is about going back to, phone's dead, there we go, going back to basics. Because we have so many configurations on so many different servers, we can't guarantee we're gonna have the latest and greatest monitoring or application software, right? We're not sure we're gonna have New Relic, right? Or App Dynamics, or any of those things installed on there, right, any type of monitoring. So this is a talk about a tool that was pretty much came out, its last really big push was in 93, right? I'm talking about a tool that started in the 90s. Pour it over from Sun. This is not the latest and greatest, right? But this is a tool that we can use to troubleshoot anything on almost any server anywhere, okay? That's why I called it back to basics talk. That's why I find it super useful for the admins that I work with every day so they know how to use S-Trace in a very effective manner. Because no matter what's on that server, they can understand the very, very base core of what the system calls are doing to troubleshoot what's going on, okay? So that's kind of why this topic, right? So what is S-Trace, okay? We'll cover two things. What does S-Trace do and how to use S-Trace, okay? First off, very simply, S-Trace can interrupt a process trapping the system calls and the return code and give it to you on the screen, okay? Everything that the application is doing and that way it communicates with the kernel, it's letting you know what's doing what those system calls are, okay? So, right, what are system calls, right? I've already introduced a new term, a new topic. System calls are those functions that the kernel exposes out for you to use from your application. Most applications don't call system calls directly. They call libraries, right? Things that are implemented by G-Lib C or different C libraries to then talk to that hardware directly, okay? So your system call is that kind of interface between what you're writing in code and what you actually happen with the system, right, with the kernel. Okay, whoop, there we go. Oh, there you go, yeah. So, very basic, right? Jumping right into it. We can use S-tracing first off by doing S-trace and then the application we want to trace for you to simply S-trace and W. We'll do all the examples you see up here, we'll do demos of, so don't worry about what the output is in the next couple of screens. We'll do demos of it and we'll kind of go through that step-by-step in the demo section. But what you might get is something like this, all right? You'll get the system calls that you see up here, okay? Hopefully, if you can't see it up here on the screen, it's a little small, I think. Hopefully, you can see it on your devices or your laptops, think of that question. That'd be great. Oh, I'm ugly, it's fine. They don't want to see me anyway. Yeah. If we can, I'd like, I'd prefer them down. Maybe not. Ooh. All right, I like that. A little better. Yep. And I'm okay with that. Like, any good admin? You don't know he's there? He's doing his job well. Oh, wow. Okay. Wait till something screws up in a second. Okay, so as an example, I'll put you in my get, right? We can build upon that command and then do, oh, phone, thank you, phone. We can build upon that command and then dash V to make it a bit more verbose and get all of the gory details. If you can't see the screen, notice up here this first line, the ZEC line. What's happening here is it's combining all of your environment variables into a small structure and not showing them to you. But with a dash V, it goes out and gives you all those gory details, okay? You know, you may hear sometimes that some like cron doesn't run with some environment variables, right? You can probably verify that by S-tracing and see what environment variables it does have. What environment variables do get passed to it, right? So it can really deepen down all the little gory details, okay? You see all your environment, you see your path up there, your home directory, right? Your logon name, all those little details, okay? We can also trace processes that are already running. This is useful for, again, for that Java app that no one wants to restart and your server has been restarted in three years. We see those, somebody has one of those. Those are fun. So we can trace a process already running, right? Also useful, an example we'll see later on for tracing like daemons like Apache. And you go have a web server, right? And it's already running, you don't wanna kill off. We can do that, okay? Now, in doing this talk a couple of times, this example always comes up, right? Big old warning. When you trace a process, S-trace sends a SIG trap or signal to the process. Some processes can detect that and just shut down or act unexpectedly. Usually these are proprietary installers, think Plesk, right? Something they don't want you to kind of reverse engineer, okay? Something that involves a serial number or something like that. This happens all the time with us. When you install Plesk, it doesn't go quite right. We wanna figure out what's going on. We'll S-trace it. And then they'll remember that we can't because it detects that and kills out and says we're being traced, we're stopped. So literally, your S-trace process sends a SIG trap. And your application is like, whoop, hold on, it's a trap, hold on. Now it may continue on and continue to run and give you what you want back or it may continue to just die, get a SIG stop, a SIG kill, right? It depends on what application might be coded. So keep that in mind. If it's something that you really can't have die off and it's acting just the way you want to, maybe look at other options, okay? But again, it's very, very rare that this happens. Again, mostly Plesk, right? It's kind of the big one we've seen all the time. Small applications can give you quite a bit of output, even just small ones like W or password or things like that. So we can save the output to a file, simply like this, S-trace-o, file and command or the PID we want, okay? Ooh, wait, let's figure out what happened here. That's good, thank you. All right, not a big deal. Where is, where is caffeine? Where'd you go caffeine? Is up there anywhere? All right, let me turn off the displays real quick. Well, I know what's going on here. It's on the S-trace-o, let's see here. Turn off mirroring and there is caffeine, awesome. Display, arrangements, mirrors. For people to present, caffeine is a great little tool against one of those things that just continues to, whoop, whoop. Where'd you go, Firefox? There we go. All right, caffeine just continues to move your mouse for you, quote unquote. All right, so again, we can direct the output to a file if we want to. Normally it goes to standard error. Just keep that in mind, not standard out, but standard error. Try if you try to pipe it and get the first-hand line, just like that. Just need to redirect standard error to standard out. All right, so, next. We can also follow child processes. This is, again, we'll see this a bit more explained in one of our last demos, where we S-trace Apache. For those familiar with Apache, you may know that the root process has a mainpid, but it spins off all the child processes that handle all of the connections. We don't want to follow the root process. We want to follow that child process. Those child petroleum processes are more than anything else. So we can do that as well as you see up here. And these commands are just building on top of each other, right? What we're seeing here is we're doing the dash ff to follow the children, dash o, to put it out to them a file, and then a pid for whatever, maybe the Apache process, maybe it's something else, okay? And the output's gonna look something like that. You'll see what you'll get from that command is all the files individually labeled with whatever name you gave it, as well as the pit number. This will say a pit number for it, okay? So, useful stuff. We've seen this used multiple times when the scenario is that Apache is segfaulting over and over and over, and we think it's the PHP file somewhere in there. What we can do is do that, strace, follow the children, and attach it to the Apache process. What we'll get is we'll see a process that they all attach, and I'll die off really quickly. And the last line of every single one of those files will be open, such and such, PHP file, and then segfault, okay? So, super useful and super helpful when you wanna show a customer, look, it's this file, all these processes die from the same file. Did you change anything? No, right. Except that one file, right? I've been on phone with customers for 45 minutes to an hour. You changed nothing, no, no. Show them that. Oh, you did change that one thing, but that shouldn't affect it, no. It's part of the framework. So, all right. You can also use strace to not only get the output, but get stats, get a summary of the system calls that were made, any errors, and how much time they spent on each system call. This is useful when, for example, I'm not sure where to start looking at a process, right? We've already seen here that the output can be quite big, and we'll see it again in the demos. So, it's helpful to see, okay, where are they spending most of its time? Is it spending most of its time malicking memory? Is it spending most of its time opening files, checking for other files? Right, we'll see that here. And what you'll see is the output looks something like, look at that, okay? You'll see here that you get a percent time that's spent, and it's percent time in system calls, okay? From the man page, it's trying to measure that from the time difference between going into system call and coming out of the system call, okay? You'll get the number of seconds it's spent. I believe that's wall time, if I remember correctly. And then the number of calls it does, the number of errors, the number of errors are, the number of, if I remember correctly here, the number of negative one return codes for an error, okay? Maybe the file wasn't there, right? Maybe something that's not going to be detrimental, but again, maybe the file wasn't there, maybe you can have permissions, any of those things would cause an error on something like that, like an open like that right there. Okay, all right, let's see here. And then lastly here, we have how a process can exit, we'll get that at the very end of that Strace file, useful to see if the process did exit out correctly, right? Useful, or if it was killed by a signal or with a different return code, okay? Now we can get the return code from a bash, a variable just easily, but we can save this and look at it later on, okay? So that's kind of a quick rundown of some of the flags I find useful and how to use Strace, okay? Our next thing here is going to be system calls, okay? This slide came up a little earlier, right? I want to make sure I introduce a topic or a term and define that very easily, very early on. And the same one here, right? A system call is simply that as a function provided by the kernel that the application uses to access, make hardware calls, right? To write a file, to get memory pointers, things like that, okay? Now there are about 441 different system calls. I'll show you how I got that number later on. Some of them are synonyms for others, so not quite that many, but there are quite a number of them. We'll go over a few of them today that I find useful, kind of the first ones I want you all to know to kind of really, that most programs are doing, right? Opening a file, reading a file, checking the stats of a file. We'll look at those very briefly. And also, this is one of those sections where I want to talk about finding out more information and learning a bit more, okay? The man pages, usually we hit man and then a command, we get a man page up, right? Now, which section of the man page we get can vary, okay? The man pages are built out of different sections. Section number two is for system calls. Now, fine and dandy, but there are some commands, or as you see here, a bash built in, that have the same name as those system calls. For example, here read, okay? If you do just man read, you'll get the bash built in man page. Not what we're looking for when we're trying to figure out what a system call is doing. So, simply, man two read, and then we get, you see up here, the system call man pages, okay? And this works for, you can do man two and then any system call and get the information back, okay? Keep in mind some may bring up the system call page first, some may bring up another one first, right? Stats are a great example of that. You run man stat, you're gonna get the man page for it, you're not gonna get the system call page for it, okay? And we'll see that later on as well. Oh, let's see here. I really regret throwing a 30 second time on my phone now. All right, a note on system calls. The good thing about system calls is that they all have a similar structure, okay? As we see up here, okay, you have a system call. I'm gonna do this real quick here. You have a system call, we have a read, then you have any arguments, right, that you might have for a system call and then a return code, okay? And the beauty is that all of these system calls have the same structure. They all have the command, any arguments, and then return value, okay? And from the man pages, you can look at what the return value should be, what it means, right? And what those structures are that you are doing inside those system calls, okay? All right, so let's cover a few of them that I find useful, especially when someone's just learning about S-Trace, kind of the ones you're gonna see almost every single program I'm doing. So let's start with this. Open it, whoop, there we go, there we go. Opening a file, right? Opening or possibly creating a file. Most applications are gonna want to open a file, create a file, things like that. What you'll get up there is you'll see, you'll give it a path name for it. You'll see any flags you wanna do, use of it. To read only, open, append, any of those things, right? And what you would say, you give it a path name and open returns a file descriptor, right? A file descriptor is simply just a small number that the program will use to reference that file later on, okay? And what can you reference that file would exactly be the read system call. Notice here, what it's expecting is a file descriptor, okay? Coming from the return code of that file which opened, if you will, okay? On success, returns a number of bytes read, okay? If you see that you're requesting to read 800 bytes and you got 400, well, maybe you had an error. Maybe you read to the end of the file, right? So you can look at your system call and see, hey, am I reading as much as I expect to be reading? Yes, no, why, why not, okay? And then fstat or stat. Again, this is not one of those where if you just run man stat or man, at man stat you'll get the, the, I believe the application man page. But if you do a man to stat or fstat, you'll get the, well, it says call, man page, there you go. Okay, and this returns the information about a file. This is what the stack command shows, right? It returns the I know the file and in a structure you can begin to read those variables out, right? For example, the I know number, the permissions for it, things like that, okay? And then lastly, yeah, lastly, we have the syscalls for mapping memory, right? To map memory, to unmap memory, get a pointer back, right? We've seen it where applications just go through and map a ton of memory and we can show it's doing that by looking at system calls, right? So yeah, again, those are some of the most basic ones or at least the most introductory ones, right? The four ones you're gonna first see when you start looking at different applications. All right, so that's enough of me talking. Let's do some demos, this'll be fun. You get a chance to see me type horribly, okay? Real quick here, so on the slides, if you don't see a quick video up here, just refresh your page, it comes up, sometimes on a second time. I have these videos up here as part of the idea that you take these slides later on and whatever demos I'm doing on the screen you'll see up here later on, right? Obviously the demos aren't being recorded, so I try to provide these for you all as a kind of base to what I did up here on the screen because usually that's hard to read and we'll go ahead and look at, for example, that's not my one at all. There we go. What's that? Anybody got some? No, that my friend is the 50 year. I do not have that. $36,000 bottle. There was two in the US three years ago and there's one that came to the US, Canada, last year. So they're very, very rare. That one in San Antonio, with kind of just a small little plug for San Antonio. That's after the presentation. All right, real quick I'll show you what the demo looks like here and then we'll switch the terminal so that we can get a better view of it, okay? When you hit play here, what you'll see here is just, I think it's quite small and we'll do it better, but again, the idea of what's going on here, it's all, it's a thing called ASCII Cinema. You can copy and paste anything here, right? Play it back, kind of step it through, all fun stuff as part of kind of take it back and use it later on. But for here on the screen, right? So boom, S trace dash W, right? Again, the first example here, how do you use S trace? Very simply we can run a program and run S trace around that trace program from the get go and we see here, right? Notice what we see here also up here is this first line that's exact line, where'd it go? I probably missed it already. Let's do this. This is easier, S trace W, two, one. There we go, ha ha. Again, everything goes to standard error, switch to standard out. Again, first part of almost any S trace you're gonna do is this first exact line, notice what we saw earlier that if we just do with S trace and W, we get very small condensed output, okay? All right, cool. Thank you all, appreciate that. Okay, so what you see here, right? It's this first line, very, very simple condensed, right? So building upon that as we saw in our usage, we're just gonna add a dash V to that. Again, real quick in case anybody's unfamiliar where I'm doing this part right here, it's not part of S trace itself. I'm redirecting standard error to standard out so I can just grab the first 10 lines of that with the head command, okay? Again, not part of S trace itself, just a tool I'm using. So, right? So that being said, we see now our first line here, a bit more, right? All of our environment variables, everything's expanded out for us, okay? We kinda wanna get the nitty gritty of it. Okay, here we have a program not running quite right. We think maybe it's not getting a variable or a path or something like exported properly to it. Somebody else can help you out, okay? And again, lastly, we see here our return code, okay? All right, so let's switch back to, make sure I'm going with the slides here, right? So the next one here, S trace dash C and W. I'm not gonna run it, this time I'm just gonna switch back to the screen here, S trace dash C and then W, right? And when we get this time here is we get the percentage of the time spent on the calls where it's spending all it's time, okay? So, if we notice here, 245 calls to open, all right? Stubbing back here real quick, take off the head here. We'll notice that a simple program like W spends quite a bit of time opening, let's see here. Quite a few files from PROC, okay? Just to get some stats for itself. It spends a lot of it's time there. Sometimes tech comes to me and says I wanna learn how to just get better at programming or get better at X, Y, and Z. Project I've given in the past is use S trace, trace your simple tools, rewrite them in your language of your choice to learn how to use it, right? To re-implement some things. So, you should be able to look at this and kind of get an idea of what's going on, what files it's reading, where it's getting stats, things like that, okay? All right, so, again, we see here where it's been all this time. Now, for example, this program I wasn't familiar with, I might start looking at, okay, let me spend some time looking at all the opens, right? Now, I'll show you a demo in that here in just a second here, let me show them all my slides, all right? We can do something like this here, S trace, S S O, password die out, okay? All right, and again, the O, local machine, do we try and cap on that now? We'll take a small amount of time to figure that password out. I've learned my lesson before. So, when I do a dash O, what I get is everything with the standard error now goes to a file and I can interact with the program as I would normally, like you see here, everything going in the standard out that should, okay? And I can look at my file and say, less password dot out, and see my file there, right? I can look through it, see what's going on, can have some fun with that. Okay, now, let's see here. Are these demos going too fast or slow? Should I slow them down? Good, all right, I got some thumbs up, all right. No one's saying no, awesome. All right, so, here's a fun one. So, we had a scenario one time where, sorry, let me back up a little bit. As part of my role at Rackspace, one thing I do is teach Red Hat classes, and part of those classes are to have S trace enabled for everything. Now, a question came up, why would we have, or not S trace, I'm sorry, wrong presentation, SE Linux enabled for everything. And part of that, there's an exercise that we're recovering through a password, where we need to relabel all the files. And a question came up, why do we need a relabel, we're just changing our password. And so, what we found out is that, part of this is going on, we said we do a, LS dash, I think it's I, Etsy shadow, right? We have, the ash I gives me the I node of that file, okay? And when we did a password, let's do a password again. Password, okay? We saw that the I node of Etsy shadow changed. So, hence, if we're in a covering mode with no S Linux enabled, it wouldn't get context. But we want to figure out what was going on there. So, the first thing we did troubleshoot that was, we did, I saw earlier, was S trace dash O, you know, password.out, password file, okay? Password command, we did that, okay? And let's say, we looked at all the files there, and we said, as a lot of lines look through, I want to look at the many lines, right? But we knew it had something to do with the file. We knew it had something to do with operation on Etsy shadow of some way. And so, the next thing we can do with S trace is to limit the types of system calls that we collect. Okay? This is, I consider this a bit more of an advanced or a next step, that's why I include it in my first part, but I want to include it in the demo, because I think it's super useful. And it's a way to begin to really dig down and filter out what we need, what we need. So, let's take a command here, I'll break it down for you. S trace dash E, trace equals file, dash O, password trace.out, and the password command. What I'm telling S trace here is to only grab any operations that have to do with a file. Open, close, create, things like that, okay? And so, let's do that, okay? And then, now all of a sudden, our area to look through is now 142 lines. Okay, I'm looking at just the output, or the file operations. So, password trace.out, right? And we see only all the opens, right, when we get. I'll shortcut a little bit here, because that demo took forever, or that kind of troubleshooting process took quite a bit of time, but what we found out was that, is that at some point, right, the password command renames a newly created file to Etsy shadow, therefore, the new iNode, okay? Therefore, if you have SC Linux enabled, and you need, or it's on a running system, you need to re-label everything there, okay? So, kind of a way, we try to troubleshoot our own things to figure out what's going on and balance them like that, okay? You'd be surprised by the number of applications that do just that, open a new file, and then copy everything back into place after the fact. So, kind of a fun one, again, that one was just like this, dash e trace equals file, and you can do that trace on different families, not just file operations. I can say just the opens, or just memory operations, okay, I can really narrow it down like that, okay? All right, and then we have another demo here. Again, that video there is the demo I just did, see a chance to see it later on as well, in case you wanna review it. Again, building out a little bit here, again, not quite a new demo, but again, the difference here. All we're doing is the iN-V here to get a bit more output, right? And then, well, again, what we see here is a bit more information, right? And right here is an example of what you might see as an error. We looked at dash c, right? Again, nothing that's gonna kill the process, nothing that's fatal, maybe, but maybe something that wasn't found the way it should be, okay? Sorry to say it again. So, the question is if I can filter out critical versus non-critical errors, and that's not that I've found. Just like when you get that dash c part, right? The, let's see here, s trace dash c dot w, right? That there's no, it's just returning back where it comes back as a negative one, right? So, not that I've found, there might be, but not that I've found. Go ahead and make my copy out like, maybe it's out there. There's a whole lot out there. All right, let's see here, okay? And then, one last demo here. This is a fun one. So, s trace dash ff o, our file, right? And then our PID. You see here the little hint here, hc-axis and file-axis. So, our scenario, we had a customer who had their Apache document root on an NFS mount, and it had been that way for a long time. Everything had worked just fine. All of a sudden, they brought new developers and then things started to take a turn for the worse. Actually, let me sync this up here real quick. There we go, okay. So, it basically got new developers and then, like I said, it, their performance took a turn for the worse. And we asked them, would they change, would they change, nothing, nothing, right? But something had to have changed. So, this demo here, I don't have it recorded. It's just, it's kind of a fairly long demo, but I wanted to at least walk you through it to see on the screen at least once. If you have questions later on, come up to me and ask me or reach out to me on email or whatever, we can do it again. But this demo here is, on the server, I have a Apache server running. And only thing I've changed here for this server is, I've lowered the match detection per child just so I can see things, my threads spin up quicker than I want them to, or quicker than they would normally, okay? All right, and then, what they had done is, let's see here, where'd it go? One second here, there we go. There it is, okay. All right, so let's do this real quick. I'm gonna bring up here and say, while true, actually, that's trace, all right, local boxes, while true, curl. Okay, fair enough, okay? So I'm just pre-trafficking my server here, okay? Now, what I'm gonna say here is say, system CTO, yes, I'm running REL7, or SIN7, status, HTTBD, right? Because what I care about here is the, where'd you go? I'm getting the PID of the main Apache process in this case here, okay? And what I'm gonna do here is say, Strace dash ff, again, to follow those children, dash o and we'll say slash temp apache, or let's say scaled dot out, and then I'll get PID number after that, and then dash p there, okay? Did I get the wrong, okay, hold on here. There we go, main PID, that's what I was looking for earlier. There we go, okay. Which is here is that because my max section is so low, I'm sparing up processes quickly, okay? And what we'll see here, if I just till this off real quick, is I go to temp, I get all these files here, right? All those are the patch processes that we're running, okay? Let's take one of them really quickly here, okay? So the setup here is that in order, if you enable HTT access on an Apache server, right? Every directory that you have in a path needs to be checked for HTT access, okay? And this customer had, as they called it, their directory structure very organized. What they meant to us, it was a very deep and very wide, right, just a lot of files, okay? So for example here, let's do this here, quick. And say curl, I'm not even kidding, it was about this deep. Sorry? Oh, never. So let's go back here and say slash, there, okay? Let me double check something here on my settings. That one's good, okay, am I that correct? I think I do, all right, let's see what happens here, okay? So while true, do that, right? Awesome. And while it's going on, temp, rm-rf scale, and then our strace command here, okay? So again, I haven't changed the patches settings too much right now, so let's go here real quick. Now I'll cut it off just in the interest of time and say we have, just picking any of these here. Now look at that, notice that for here, it looks for HTT access for just part of dub. I have it turned on just for that. Just so I can find it in here. But you know it goes through the path of getting these stats and not a big deal, right? Oh, let's see here. Now let's turn on, again, closing everything off, resetting everything. Let's turn on HTT access here. And this is gonna be in our VAR to dub html directory, right, where all of our path is. We're gonna come down here and say allow override all, because that's the easiest answer to get working right now. Not the best, but the easiest, okay? All right, we're gonna save that, systemctl, restart, htpd, systemctl. Every time I did that before, it says ctl instead. Oh, all the time, right? Tap complete, I'm like, oh, dang it. All right, so ctl, restart, htpd, okay. That worked, thank you. Timing, right? All right, so let's go back here and do this again. All right, systemctl, status, htp service, and let's get our main pit again. All right, and then we're going back to our strace here of that, okay? Again, as you saw before, attach, attach, attach, great. Now, if this was segfalting, you'd see attach, detach, detach, detach, and you'd find your error, but that's good here. Now, let's take a look at any of these here. Less, let's see your scale, that, nope. Oh, sorry, scale it out. All right, grab one of these, thank you. Now notice here is what I'm talking about here. What was happening was that, check that directory, that directory, that directory, right? Right here, right here, right here. And all of a sudden you're adding in a number of calls to an NFS system that you don't really need, right? Why did they do that? Well, the developer said they need to change some Apache settings, right? And there wasn't much, but they need, because that one change, you add all this overhead. But they wouldn't believe us until we showed them something like this, right? How could such a small change have such a big impact? Okay, so you have that many more hits to an NFS directory, that many more calls on it, your performance is gonna suffer quite a bit, okay? So, yeah, they wouldn't see it, right? Okay, it's just them. But now throw on their 20 rubheads, right? All in that same NFS directory. So, again, this is one of my favorite examples of how we can use Strace to show what's actually going on and where the impact is, okay? So, with that, I'm almost going to close here. Continued learning, my last section here. Like I said, there's no way that I can show you everything. But what I like to do is leave you with some extra bit of, I hate the word homework, but opportunities to learn if you choose to do. Up there on GitHub, you can find some code examples, very, very simple C code examples that will help you illustrate when I do a simple open of a file or read of a file. What system calls does that translate to, right? What does it translate to? So, I'm gonna give you a real quick example of that on the system here itself. And let's see here. If I close this here, shut that off because we don't need that anymore. And let's see here, CD, Strace code examples, okay? What you'll see here is some very, very simple C code. All right, hello world. As with anybody first programming out, hello world, right? But what we can do is compile that, GCC, let's see here, hello world C, okay? Good. And let's run Ada out, again the basic standard, just the normal file gives out there, hello world. But what does that turn into in system calls, right? We mentioned earlier that applications don't, bless you. Bless you to all of y'all. For some habit. Strace here and say hello, sorry, Ada out, right? What does those simple library calls try to be in Strace? There we go, right? We're gonna see here all the things it's doing just for a very simple hello world, okay? Now you can build upon these examples and say what if I write a file? What if I meld memory, right? And see what type of system calls you get back from that. So you begin to see as you title it together and you look at big applications, you see what they're doing, right? And you kind of get an idea of what's happening in there, okay? So again, just a real quick, small code examples. We look at the open file, go through and try to read a file, okay? And then again, following through with this, it's real quick here, GCC, open file, let's see. That's fine, old code. But if we do Ada out, file. There we go, right? But if we do this now and say if things learned earlier today, Strace-o-my-file.out, back. Again, now we get to see what those small opens and reads look like when it comes to what's actually happening at the system level, okay? And that's a way y'all can go and learn a bit more about that, right? Small code, find other code online and do the same thing, okay? It just helps you sort of to visualize what's actually going on code-wise to what's actually happening on the system itself. So, all right, let's see here. All right, so towards the end, I was to kind of jump back to the beginning here. Let's do this here on my phone because like any good talk, right? Show you what I'm gonna talk about, talk about it and then tell you what I told you. So kind of in summary here, we talked about why this topic or why I find it super important for text to understand and why I wanted to share it. Hopefully you all found it important and found useful, can find useful walking out of here. Talk about what is S-Trace, talked about how to use it, right? And then we talked a little bit more about system calls and what those mean, okay? We did some demos up there, we did some horrible typing, at least I did. Things worked for the most part, which was kind of cool. It's very rare that it happens. And then we talked lastly about continued learning, right? How can you all take this from this presentation out into your next troubleshooting adventure, we'll say, okay? So with that, again, because I'm not used to having an introduction, which is kind of very nice, thank you very much. Real quick contact information, except by the end, my name is Alex Juarez, I'm a principal engineer at Rackspace hosting. My role there is a lot of training and mentoring of our front-end support texts. So doing things like this for them daily, kind of on a one-on-one, or doing things on a weekly basis with brown bags, kind of what I do there at Rackspace, okay? You can find me online. And then my friend Jill was nice to write me a bio, which was right up there. So again, might use to have this up there. So real quick, that's all. And then lastly, Q&A. Oh, we talked about whiskey. Let me talk up here real quick, so I can see you all if you have questions. Questions, correct? Yep, yeah. So question was, S-Trace included in most distributions. Yes, S-Trace was poured over from Sun in 91, and then kind of merged together for a 2.5 version in 93. So it's been around for a long time and all the systems I've seen, it's on there. Yeah, question right there first to come back here. Questions, how can you S-Trace a cron job? For me, so it wouldn't be the cron job itself that I'm S-Tracing. I would be S-Tracing while the descriptor if I want to run or S-Trace out the Damian Fort and follow its children that spins up. So that's how I attempt to do it, yeah. So, but usually it's going to be a script that I'm doing itself and seeing its environment variables that it's having, so, okay, question? I wanted to S-Trace something like Java, but it's really, it's called Unibasic by Dynamic Concepts. So this is a language, it kind of stores its source as a P code, sort of like a Pascal like P code. And I'd like myself and the developers really are trying to figure out what in God's creation this thing does have the time. And so how would you, would S-Trace be an appropriate tool for that when I said Java, you kind of laughed and I understand when you look at the Java, JVM argument list? Yeah, so it's the question that you could use S-Trace to create something like Java or kind of reverse engineer some code, right? And it really depends on how it's implemented. So like I said, it's something with an Installer like Plesk that would trigger for it, might just kill out. And really, it's not to reverse engineer the code itself but to see what's happening on the system. Maybe what files you're trying to access, what memory you're trying to mallet, right? So it's not necessarily to reverse engineer that code, but again, more to figure out what ultimately is doing on the system. Is it quick? Yeah, so if it's using incurses, would show that. Do we have something I'm not sure? I haven't come across that in my experience, at least we have troubleshoot stuff in the past. Usually what we're troubleshooting is gonna be, for example, what's it called? Coyote, right? So it's web backend or something like that. We're trying to troubleshoot. We might get, look at that a little more, but I haven't seen it using incurses or kind of in that experience before. So yeah, question, different? Question of the work with D-Trace. I do not work with D-Trace much. I know it's out there, just haven't used in my day to day. So, so. And you have the questions, questions in the back. Okay, that's a great question. So the question is, I'm just tracing a process and I see follow-scripters are already out there. How do I know what they are? I'm glad that question came up. This is fun. If you look at the PID that you already have, go to PROC, let's say maybe, let's see, your CD PROC, right? All the directories in there are those PIDs? Are PIDs, right? So within that, and actually this is a talk I submitted didn't get accepted, so talk to the scale about that. If I go into any of those directories, those PIDs, if you will, let's do one because we all are pretty sure what that is, right? And I go in there and look at the directory FD or follow-scripters. It will give me the follow-scripter to the file, okay? So that's where you can tie us together, okay? Any other questions? You're very welcome. All right, any other questions? No, cool. Thank you.