 Beautiful. All right, let's do machine introspection. So sometimes your computer is misbehaved, sometimes something goes wrong, and very often you kind of want to know why. And so what we're going to do is we're going to look at a bunch of tools and let you figure out what went wrong, what is currently going wrong, and how you can prevent it from going as wrong in the future. So generally, whenever you want to do computer introspection, you're going to need privileges. So in general, the kernel will not let you look at arbitrary state, and so very often you need either to be part of a particular group, usually wheel, which is like you can act as root group. There are other groups like power, audio, video, optical, this, etc. that are used for various more special purposes. Probably just add yourself to wheel and it will be fine, but if you're paranoid about security you might want to add yourself only three groups that you need. In particular, we're going to be using the pseudo command. So the pseudo command is basically a prefix for whatever command you want to run, and then it will run it as if you are the root user. So as if you have all privileges that are possible to have on the machine mostly. So for example, I can run date as myself, or I can run who is, I guess, who am I. So it says I'm John, and then if I run pseudo, who am I? And I'll ask for my password, which I can then type, and now it will say that I am root. Notice that it prompted me for my password. You can change that setting if you want to. So I can run Vsudo, pseudo, Vsudo. That's so long, I think. Fine. No, I think you might be stuck. So Vsudo lets you edit the pseudo respile, and it will check that it is tactically correct after you save it before applying it. This is very nice so you don't accidentally lock yourself out of root. Notice here there's a line that says root is allowed to run things as root. In addition, wheel, so people of the group wheel are allowed to run any command as any user of this line means. You can also disable the password check if you want, otherwise it's going to prompt you for a password, and it does so again every so often. But if I run pseudo a couple of commands in a row, it'll sort of remember my password in some sense, remember my authorization from the first time I read it, but after a while it times out. Whereas if I combed it out this line instead of that one, then now we no longer prompt for passwords at all when you run pseudo, assuming you're in the right group. But I would like it to keep that, and notice what I did here with pseudo-su. So su gives basically that you start a terminal as a different user, and by default it's root. So pseudo-su means pretend that I'm root and that I'm starting a new terminal as myself. So it gives me a prompt that is me as root. And here I can remember whatever I think I want and immediate this root. Okay, so the first thing that you want to do when something goes wrong is you want to figure out what happened. And usually the way you do that is by looking at logs. There are a lot of log files in your machine. Most of them are in barlog, at least traditionally, that's where things work. So barlog, as you see, contains logs from all sorts of different things. Usually the file is indexed by name. There are a couple of ones to be aware of. In particular, xorig.log is handy for when x fails. There are usually folders for specific subsystems like cups. And normally if you're in a web server, for example, the logs will also be here. There's also a kernel log. The kernel log used to be in barlog and now has sort of been coded in stupid ways. So there's a command called demessage which gives you all the messages that the kernel has printed since boot. So if you run demessage, it will show you all those messages since the last boot. And of course from the data wrangling lecture, you'll remember that we can grep through these logs if you want to move for particular patterns. Finally, there's the system log. And this is basically where everything is being logged nowadays. Some programs will also log elsewhere like to barlog. But in general, the system log is where you want to look. Most systems today, the system services are managed by systemd which has its own journal demon. The way you look at the journal for your program or for your system is by running the journalctl command which gives you the log since the last boot. It gives you a log in less by default and you can scroll around which is kind of nice but you can also of course type and do data wrangling on that too. There are a couple of things to be aware of with journalctl. So first of all, if you run journalctl without nothing, it gives you all the messages since the last boot. You can do dash u and then give it a unit like log in d. Great, no entries for log in d. Systemd log in d? Great, because of course. So this gives me all messages made by systemd log in d since the beginning of time. Which might not be what you want. You might want only the messages from this particular unit since you started your machine. There's the dash b flag which has only the last boot. You can also say only the previous the last boot or the third from last boot or the first boot you know about. And so this flag you can use to try to debug what happened in previous reboots. Like let's say you work with your kernel, you try to reboot it, now there's a new log start and you want to check the previous boot log. You would use this flag. Journalctl also has this annoying property that if you try to pipe it through things sometimes, like if I do dash b correct into... Oh it doesn't do it here. So sometimes journalctl for various stupid reasons decides to crop your log messages and just show it dot dot dot. So you sometimes want to give full if it starts to do that. There's also a dash in 100 which is or whatever number you want which is shown with the last end lines. This is similar to journalctl, pipe through, tail and 100. Those are basically the same. Alright let's see that something is wrong and you want to figure out what is going on in your system. The first thing to start with is the top command. So top shows you all the processes that are running on the machine, how much CPU they're using, how much memory they're using and a bunch of statistics about your system. Most people don't really run top anymore but it is nice to know that it exists. Instead most people run htop which is basically the same thing but with a nicer interface it shows you how much load there is on each core sort of more graphical representations of things. In addition you can do things like here you'll notice that it shows all the threads of all the programs. You can press T to switch to a tree model where it shows which things are some processes of what. Which is really handy to figure out like if there's some particular thing that's messing up what started that program. Which you can look at through this or if that's all you really care about you can use ps3 which shows you the same things or the tree of all processes and where they stem from. If you want to know the process IDs of them you can do ps3-p and it will print the process IDs as well. This is handy just for figuring out what's going on and why is it running. Oh yeah the tree-moded htop which is press T to turn it on and off. Sometimes if you want to know what those programs are doing like I see a bunch of weird programs here and I have no idea what they're really doing one thing you can do of course is look at their standard out if you have them open in terminal but if they're running as a service in the background you might actually want to check the logs and then it's useful to know about journalcql-f which shows you the last few messages and then it will keep your terminal open as new things are logged it will print those new log messages. The equivalent thing for dmessage is dash w which says for wait so this is wait for more messages to be printed and just keep printing them as they come similarly if there's a particular file like in var log let's say that I want to see new xorg log messages tail as a dash f go fly which will print the file and then watch that file for additional changes and if they come then print them as they come you can also do less plus less plus f xorg I don't like less but you can doesn't seem to I don't like things taking over to my terminal because we sad right the other tool that's really handy if you're looking for resource usage is a tool called dstat which I currently apparently don't have which we're going to fix right now so dstat is a program that monitors all sorts of different subsystems on your machine and then just prints out information about them these can be things like network traffic, disk traffic the number of interrupts your CPU is handling context switches, the number of processes the CPU utilization it has a bajillion flags for monitoring different things so you can watch only certain CPUs watch disk, watch paging swap space all sorts of stuff I think there's dash all dstat dash a but this is really handy just like seeing a real-time snapshot of how busy your computer is and what it's busy doing if you're running some program and it seems like it's not making progress open dstat and see if anything is going on there's probably some there are also more specialized tools for things like looking at disk space so for disk space, df is the most common utility df shows you for all of the power systems you have on your machine how much space is used, how much space is available and where is it mounted by default it prints the number of bytes which is a little annoying because if you do dash h it will show you that in normal numbers so in this case you'll see that slash this partition on this disk it's the total size use space and available space if you want to know what in a particular directory is taking up a lot of space there's the du command so du stands for disk usage but it does not show you disk usage it shows you the disk usage of particular files so for example in my home directory I could run du star and actually let's do this in where do I call that directory ah photos photos okay there are two directories here but let's say I wanted to figure out how large it was which were more important I can run du star and it will show you how large each of those three pieces in terms of all of the files below it as well dash h gives you a human readable symbol and you can also do s to get a summary so if I do yeah so the summary will not show you the things underneath you which is the summary of the arguments you give there's also a tool called dust which does all the same things but you even much nicer much nicer interface also is better weird hours if you're looking at network connections there's a tool called ss so ss is fantastic for looking at what is connected to what in your machine by default when you're in ss it shows you all connections on your machine all protocols that the network stack supports so basically all open sockets normally you will do something like dash t so dash t shows you all open tcp connections you'll notice that currently I have no open tcp connections you'll just wait on my laptop but if I did something like put a there'd be an image it will now show me that there's one tcp connection that's been established qsize which ip connected to which port where now there's no longer a tcp connection I can also look for listening ports so if I give a tl it will show me basically all the server ports on my machine hopefully there should be none but if I start additional programs like for example if you're running dropbox or something on your machine that's going to start up a service or if I run it's a relatively new laptop hence there's no stuff netcat is a really handy tool for just piping data it's mostly used by hackers but it happens sometimes actually useful dash localhost port 30 6000 and now there's right so now it will show me that there's one port listening on my machine if I do dash p it will show me which program is doing that listening if I put n it will not show me that this is the x11 port the actual port number and so this is saying the program in C with this PID is currently listening on port 6000 so this is handy for just figuring out what is running on your machine and if you're like me then you want the list to always be empty if you start a browser you'll probably be listening on port 2 okay so that is networking so let's go briefly over configuration there are a lot of ways to configure your machine people focus on two primary tools that you end up using all the time the first one is when it comes to networking the IP command is just really handy the IP command lets you configure pretty much everything that has to do with networking on your machine it comes installed with most modern distros the basic command you want to use is IP adder so IP adder shows you all the interfaces you have what IP address is there about to so in this case LO is the loopback interface it's the interface you can use to talk to your own machine without going over the network things that start with E are usually Ethernet ports one way or another so they're wired connections and things that start with W are usually wired connections so in this case you'll see that my Ethernet cable is state down which means that it's not currently connected whereas my wireless port is up and it has the following IPv4 address and also the following IPv6 address if I should care about that you can also change all these so IP is actually a really powerful tool for changing the network settings it's basically what is used under the hood to set up your network in the first place the IP command is notoriously hard to use in general you want to do things like IP help and then a command basically it will tell you the grammar it uses to parse the commands which is not the best form of documentation but you can do things like IP I think you need to like help adder set help adder adder help great so this is the how the grammar it uses to parse IP adder it's awful and then add and help comes last instead of first sometimes I think it's like this it's all sorts of stupid but if you look at the man pages those are a little bit more helpful because if I remember correctly they have some examples not that the examples are particularly useful but it gives you some flavor of what the command lets you do and normally lets you do things like bring into this up, down change the IP addresses change broadcast addresses, those kind of things in addition the one thing you should know about is IP route so IP route tells you if you're trying to basically how your computer is going to communicate with other machines so if I send out a packet that's going to go to anything on this subnet so this specifies a range of IP addresses then it says that anything that I send to any of those IP addresses is going to go to this IP address and anything else, so default anything else goes to this IP address so this is basically the way network is configured that any network that is on the same network as you connected like the same switch or router is going to be connected through just directly, you can just send packets to them directly anything else you have to go through basically your router so in this case my router is this IP address so anything that is not going to something that's on the local network goes to the router and then it fares it out often if your network is if you end up in a weird network situation IP route will tell you what to do there's also the ping tool which is really handy way for figuring out what's wrong, usually the order operations is first you try to ping a host in this case that worked so ping basically sends small packets to that server and sees whether you get the replies here it says that I got reply if you don't get reply then try an IP address if you do get a reply from a IP address but not from a name it means your DNS resolution is working so translating domains into IP addresses is not working if you don't get an answer from either of these it means that your local network is working and then you might want to try to so remember here this is my router you might want to try to ping your router in this case that worked so my connection to the router is operational and so if something is broken it would be broken on my sorry it would be broken on the router but if I couldn't talk to an external IP the issue is somewhere in between this is a really nice way to bisect your way down to where the problem lies and if you have IP issues etc.resolve.conf is the thing that defines which name servers you're using so this is a handy file to configure if you want some other settings than the default and finally for system communication if you have services that you want to run in the background like Cron we talked about already if you want to run some kind of web server or an SSH daemon or any kind of service you want to run in the background of your computer those are managed by systemd so systemd is the system daemon and it manages your system it just keeps growing and water encompasses but the basic idea is that for every service on your machine there is a service file so for example there's a service file that I don't know the box went and so the service files basically define a unit which is a service so we look at those with journal CTL there you can do dash u and then a unit to limit log messages to only that and it basically describes how you run that service and how you shut it down later you control them through systemctl systemctl status shows you all the services that are currently running and what IP addresses they have depend on what you can use systemctl start and restart followed by a unit to stop a particular service or restart a particular service if it produces errors and just use journal CTL to figure out why you can also set services to either run it boot or not so if you type enable and then enable the service it will be started on boot if you type disable in a service it will no longer be started on boot if you type your own systemd unit it's not terribly hard and then they go in etc. they're pretty easy to write but you need to look up what unit files are like they're not terribly painful it might be worthwhile to figure out how to write one if there's something you would like to run on boot oh yeah one last thing if your boot feels slow systemd analyze shows you how long your boot took if you give it the blame option it will tell you how long which thing took in starting up your machine it's a handy connect alright I think that's all we're going to do program introspection as well I think we're going to leave it there actually why don't we move program introspection to the next lecture and if you have something you can keep going or we can end early I have lots of other tools I can talk about I don't have anything else like schedule that's a good question there's a program called console on your machine so if you just open Spotify and search for console you'll see it and that's where you can see all the system logs it's probably a good way to program and if you want the I think instead of systemcdl I have a program like launchcdl but like but like there's launchdl I'm sure it's painful it's one way I have both and Mike is the one Windows is actually getting better with PowerShell you can do a lot of this the syntax is pretty annoying I just want everything like I don't want to have to open a program like ever the only program I have to open is my browser everything else in terminal is alright so if I'm going to keep talking I don't have any things I can open but I have to install some programs here so mlocate is a package you can install in most systems and what it basically does is if you run updatedbe it will scan your entire file system and create an efficient search index so I can do something like locate 6858 and it will show me all the files in my machine that are related to 6858 that contain the string 6858 so just given fcf like we talked about before I can use this to fuzzly search directly onto the list of all files in my system look at 8658 and I can do so very efficiently so for example I want all that and I can do this it's a really handy like locate is really handy for finding things quickly because it doesn't actually have to if you use find for example that has to walk your entire file system every time where as locate is just string search what's useful to know about is dmidcode so dmidcode basically parses all of the firmware on your machine and tells you what different things are like what hardware you're running and all the capabilities that they support this is probably not useful unless you're doing like firmware upgrades or you're curious about what exact cpu model you're running but it is kind of handy for those kind of things like for example here it says my cpu is an i7 specifically it is this i7 it has all of these features supported, what its clock speed is like cache slices and everything this is also a program called elastopo which will show you the entire physical layout of your cpu including all caches it's pretty handy for doing performance debugging it has a single cpu package that contains four different cores each one with two cpu, two logical cores this is hyper threading and it's showing me that I have an instruction cache of this size an data cache of this size an l2 cache of this per core and a shared l3 across all those and also the layout of my PCI plus it's pretty handy you can get it to print out a text only version it also lets you have something called hwlog bind which lets you run a program on only certain cores or only certain CPUs or only certain hyper threads which is very handy if you're running benchmarking stuff you might have noticed that when I ran dmid code without sudo it tells me something about slash sys permission tonight so slash sys is something that's worth knowing about slash sys and slash proc are special parts of your file system that are managed by the kernel none of them are real files they're just sort of meta information that the kernel exposes and sometimes they do set properties that the kernel wouldn't otherwise have to set so for example if we go into sys what kind of finding here they know let's go to devices and then something like yeah classes generally because you want to go if you don't know where you're going because it shows you things by classes so here for example Intel backlight here I can like cat out the max brightness of my screen background I can cat out the actual brightness but also I can do things like echo 500 into actual brightness sudo and now the oh sorry I can't set actual brightness and now you can't see this but my screen just didn't and so this is a way to basically cause the kernel to do things by changing parameters there are lots of cool things to do here like change queuing behavior in the kernel disable various safety features like a SLR you can turn on additional safety features you can break your system entirely so be careful about what you do here but it is worthwhile to know about it slash prompt contains information about all the processes that are running so let's take some arbitrary process like this editor which is 6905 by cd into that directory so this is slash id of a particular process and if I ls you see there are a bunch of different things to show me information like I could cat cw or so lsla this will show me that the current working directory of that process is homejohn if I cat command line it shows me this is the command line this program is run with and so this is basically the files that htop and top parse to produce the information that they have and you can do all sorts of things here like mount the entire memory space of this process but it's all very unsafe don't do it it's also worth knowing about slash boot so slash boot is the where all the files that are run when you boot your computer go it's also usually where the configuration of your boot goes so for me so for me this is the configuration file here I can set basically which options like it when I boot my machine and what each of them do in boot there are two primary files you need to know about it's the vm linus dash linux file which is basically the boot image of linux that's small enough to fit boot that then boosts up the rest of your linux installation and there's init ram of s which is basically the same thing it's not the same thing they're two different stages but for all of this purposes you can think of these two files as like the boot image for linux so they are the things that set up all the hardware in your machine and then when the hardware has been set up these are the files that then start your actual linux kernel these things matter if you're like running a custom kernel or something like that it's just useful to know that these files exist I have way to get this installed I have no idea IP tables is apparently pretty much built in firewall that uses kernel rules to filter into the traffic I'm not going to go through all of how IP tables work because it is a bit of a pain it lets you set rules for packets that look a certain way that are for a given port, that are from a given IP, from a given range to a given range, buy a new program and sort of basically manipulate them in however you might see fit the one normal thing to do with this is basically set up a firewall drop all packets that are for any port that's local to me unless I specifically allow that port you can have it rewrite packets if you want, you can have it redirect packets if you set up an EPN for example, it will often use either IP tables or IP routing or both to ensure that all of your packets go through the EPN and no packets can leave the EPN so it's a useful thing to know about, but hopefully not something you'll interact with too much directed person I think the last thing I would mention given that we got into EPNs is WireGuard Sure, why not WireGuard is basically an entirely new EPN implementation that is implemented from scratch and works as a very small opinionated piece of EPN software it does not allow a lot of integration or anything, it just is supposed to work and be secure and be fast and in my experience it has been it's pretty easy to install, pretty easy to set up and just works beautifully the way you start using this is you run this command starts my EPN and this stops the EPN and that's all the commands you need all the key handling is using private public key pairs and so it's just so much nicer with MIT's EPN or basically any other EPN solution which is often a pain to set up a Linux WireGuard just works but you do have to set up this over yourself but yeah, I think that's all I wanted to say about machine introspection although I can probably say more unless you have more questions anything you wonder about how you might figure out about your computer there are also nice tools like I don't want to see more of these questions about introspection across the 3D, I'm not sure what shell do you use I use Zshell I just wonder if you all could argue in front of us because I would like to switch Zshe tries to be like Bash and that's the drawback for Zshell the advantage is that it's an advantage in the sense that there are things that run over Bash run over Zshe but it means that Zshe doesn't have quite the same flexibility as Fisch does to do things a different way whereas Fisch is sort of saying we're not going to care about compatibility with Bash we're just going to do things in a more sane way more human friendly way that said, you wouldn't really write scripts in Fisch although you can it's more written for like humans writing commands or loops or if statements are low line whereas I think Zshe's slightly nicer to write scripts for in part because it's closer to Bash it's cool can I mentally think of it as like the eye on Fisch being made by Fisch I have I use Zshell but I think the make on Fisch is pretty much identical to Fisch is this pretty much like a lot of the things that Fisch introduced were copied into modules that nowadays you can have even in Bash I run Fisch without any plugins because I don't really need any whereas I think you probably wouldn't do that in Zshe to me at least my recommendation is to just choose one and look into all the features that you can be using don't stress too much about what to say like figure out using things like syntax highlighting like things that are like quality of live stuff so I really like the Fisch sort of statement of purpose which is finally a command line shell for the 90s it was released in like 2000 and something but it's like catching us up to the 90s so oh it works like this so yeah it's pretty neat Fisch suggests commands as you type based on history and completions just like a web browser watch out Metscape Navigator 4.0 glorious VGA color given that you're going to put some effort into configuring your shell you can probably add Zshell and Fisch to behave more or less the same except Zshell is like almost Fisch is ditching with expert trying to be slightly more human friendly one of the things there might be an equivalent for the sensation I'm not sure so Fisch has this thing called so they basically have an expansion to aliases where aliases are not just one command is equivalent to something else but something expands to something else yeah that's default I think they call it global analysis or something like that and it happens it expands while I type so for example you'll notice if I type ls and then space it turns into xr because I have alias ls.xr but it will actually show me the true command like if I type print which is an alias I have it expands into the full command and then lets me go back and edit arguments to that thing and I've found that really useful that it actually shows you the command I suppose to hide it behind other areas you can also do things like save that a command wraps another command so aurman which is a package manager wrapper I use for arch it takes basically all the same arguments as pacman plus a bunch of other ones so I can tell Fisch that any argument that works on pacman also works on aurman so in my Fisch group I have this thing and this basically improves the autocompletion so it understands that these commands are the same for anything that exists in one do you have a question? is there a reason why you still use arman and then they stop developing arman is no longer publicly developed but the maintainer is still using it he was just sick and tired of all the criticism he got for it by people who didn't know what the aurman was around and could buy packages are just in this important position where you can find a package for pretty much everything but it's in the aur and people are writing tools to make it easy to install from the aur but part of the point of the aur is those packages are not entirely safe to install so you shouldn't trust the stuff that's there and so he got a lot of flack from essentially new users who didn't know what was going on saying like your tool doesn't work but so it is still being kept up to date so it works when new releases come but he's just not taking public feature requests it also has a bunch of nice features like if you do a system upgrade and there's been a posting to the Arch Linux use feed since last time we ran an update it will show you that new update I think that's all I have great