 Welcome back to 353. Thanks for joining me today. Sorry about the lecture scare yesterday or before. That was one of the weirdest experiences of my life. Apparently you can shovel snow, pinch a nerve, lose all strength in your legs and then you can't bend over anymore. So apparently that's something that can just happen to you after you are 25 and older. So you have that to look forward to in your life. But yeah, that was not good. So other piece of news. So if you can, in the Discord, there's that Sylvite channel. If you want to check out that tool, it was made by someone who has taken this course, one of your fellow students. So if you could check that out, that would be nice. It's there to hopefully help you get some feedback and he is eager to improve that tool should help you kind of manage your time. So this lecture is when things start getting interesting because after this lecture, we get to figure out how to create processes. And then we have multiple things happening at once. So your world is going to change from this point onwards. So recall that a process, we should know kind of what a process is right now. It is the instance of a running process or running program. So we argue that there are some things that have to be in the process. So it has to have its own set of virtual registers because it should be independent of any other process running. It shouldn't just change registers randomly. Otherwise no program would ever work in the history of ever. They also, we argued, would have virtual memory. So they have their own independent view of memory. They can even use the same address, but the operating system has some magic where each process will actually access different physical memory and they'll be completely independent. Even if they access the exact same address and in virtual memory, that's where your stack and your heap live. So we can add something else to the process. So we can add the open file descriptors. We kind of talked about them yesterday being the standard file descriptors that every process by convention has open. So zero is standard in, one is standard out, two is standard error, you'll get used to those numbers. So that is also tracked within a process. So we call whatever file descriptors it has open by default. You probably have three open by convention and they're all the standard file descriptors. So what we use to keep track of a process in software in the kernel is something called a process control block or PCB. And since we're in software, not hardware, it's not printed circuit board. It is a process control block in this course. Specifically in Linux, there is a struct called a task struct that you can browse on GitHub. That is the process control block for Linux. Some things it contains, so that this will be common across any kernel. It would contain the process state, like the CPU register. So if it has to switch it back and forth, it has to save what the registers currently are somewhere and it would do that in the process control block. It would have some information about scheduling, whether or not to run this process, maybe how long it's been running for, things like that, memory management. So keeping track of virtual memory, what memory it has access to, what memory we should not let it have access to, IO status information, so that would be any input output, likely the file descriptors it has open and anything else associated with input or output and any other type of accounting information or other things it needs. Great thing about open source is you can just go to GitHub, you can see everything that is in it. It is really massive, there is a lot of information in it. One of the other types of accounting information that we actually really care about is each process gets a unique process ID to keep track of it. That process ID will never change as long as that process exists and it will be guaranteed to be unique. So this is the process state diagram and it describes the lifetime of a process. So our first goal here is to, there's some mechanism that we'll explore today to actually create a process and the kernel might have to do something to set it up, maybe set up its virtual memory, maybe mess with its registers, something like that and then as soon as it is able to run, so after it has been set up, it would transition to this waiting state. You could rename waiting to ready if you want so it is not actually executing on a CPU yet but if the kernel decided to actually run it, it could. So if the kernel actually decides to run it which is the job of the scheduler that we will see next week, not this week so we can just assume the kernel just makes random decisions. It would take that process and then actually run it on a CPU core and then we would actually see some results from it. So it would run, run, run over and over again until eventually your kernel decides that, hey, you've had enough time running, it's time to give someone else a turn in which case it might transition back to this waiting state or ready state where it's not actually running but we could run it again if we wanted to. Other states that we can go from the running state in order to actually get rid of a process and end it, we could take this transition to the terminated state and we saw previously what the actual system call is that does that, anyone remember the name of the system call that ends a process? Yep. Yeah, exit group or exit either are fine so that is the only way to terminate a process so we actually know how to actually go from running to terminated. The next weird one in this diagram is sometimes you'll be running, you'll do a system call and well, sometimes it's really, really slow so if I'm, say, trying to write to a physical file, writing to our SSD is much, much slower than just writing something to memory so while it's doing a system call the process might go into a state called blocked, it is just another way of saying it's waiting for something but in this case blocked means it's waiting for some IO to happen or it's just waiting for something to complete and we can't actually execute it even if we wanted to. So blocked, this basically just means this process cannot run anymore, it's waiting for something to happen and then eventually whenever the kernel completes whatever operation it asked for then the process will go back into this waiting state where we can actually run it again and then whenever the kernel decides it could go into this running state so any questions about this state diagram? So this is the entire lifetime of a process. All right, we're good and ready to get into some fun. So this is kind of a primer for lab one which I'll release either after the lab session on Thursday or Friday so you'll have a week to do so lab will be short. You probably use like, has everyone used like the task manager or activity monitor to see everything running on your machine before? Yeah, everyone's done that so that's basically what you're going to write for lab one. So what you're going to write is going to be specific to the Linux kernel because each kernel has a different way of doing things. In Linux you can read the state of a process using a proc file system so everything in there will look like a file but it's just a way for the kernel to present information to you where it doesn't have to come up with a billion different system calls. Remember we only have a few hundred system calls for absolutely everything. So it just creates some fake files that you can access and you can just use the standard tools on them and they will work just fine. So in the proc directory that is all of the kernel state you can ask a whole bunch of interesting questions by just going there and figure out things. So one of the files actually will tell you what CPU is running, it's running on so if you wanna see what you're connected to remotely is running you can actually use this file system. So one of the things in this proc directory is just a bunch of directories that look like numbers. For every number that's there that represents a process ID and inside that folder or directory will tell you a whole bunch of information on that process. In there there is a file called status that contains the state of the process and it's name and everything and that's what we use for lab one to actually display what's going on. So, oh yep. So by the files aren't real, what I mean by that is that they don't actually exist on the hard disk or SSD or anything. So that's why they're not real. They don't actually take space on the hard drive, they take zero bytes and here we can just look at one real quick. So for instance, if you look at the proc directory they kind of look like files but their size, except for that one, if we scroll up this is their size, it's all zero bytes. But since the kernel controls everything you can read and write these files and the kernel will generate the information for you on demand. So like for example, you can do this on any system you can look at this file called proc CPU info and it will tell you what CPU you're actually running on. So here it doesn't have a good name but tells me all sorts of information about the CPU if this was a more standard system it would say like it's an Intel whatever or an AMD whatever. But you'll learn more about this as we go through the course not terribly important at the moment. So process creation. So we could create processes from scratch that would probably make sense to you that I have a system call where I just in one step I create a process and I start running a program that I tell it to run. That might make sense to you and that is what Windows does. So you tell it what program to run sets up the process control block for you loads that program into memory puts aside any other information it needs like it's global variables or whatever sets it all up so it's ready to run and then it runs whenever it decides to but Unix, so Mac OS and Linux actually you can't do that it actually decomposes it into two steps and gives us a bit more of a flexible abstraction although initially it will be a bit weird but trust me it is actually quite useful. So instead of actually creating a process we could clone it. So that is what Unix does. So how it works is it pauses the current running process so it goes into that waiting state where it can't run anymore and then we copy its entire PCB into a new one and now that new process is an exact copy of the old one and including variables including virtual memory locations and everything like that and then the job of the kernel is to make them independent after that point and because programmers need to distinguish which one is which so if you have two that are exactly the same you probably wanna know which was the original and which was the clone so to distinguish between them Unix always keeps a parent-child relationship between them so whoever created it is the parent and then whoever is the newly created process is the child and this is where the start of the weird Google searches begins. So they could execute different parts of the program together but there are two independent processes so it might look like we're in two different if statements or something like that but your world hasn't changed when we argue about a single process but now we can have more than one process at a time. So what we'll see at the end of the lecture is well after we fork what we could do is just tell that process to load another program and execute that instead set up a new PCB yep. So the question is when we create a new process does it get moved to the kernel? So the kernel keeps track of all the processes so whenever you create a new process that will be done in the kernel and then you'll see the result of it and we'll have an example very shortly yep. So just take that PCB and then create a new one that's the exact copy of the old one. So yeah the question is why would you do that? So for some things you like for web browsers for example processes are a good way to keep things isolated cause they're completely independent from each other so say your web browser or someone opens a new tab you just fork and well I haven't said you just clone like a previous tab and then you don't have to reload all the libraries and everything again like everything's loaded for you and then you can just after you complete the clone then you can just render a new website that's different than the parent and we'll see why that's effective as we go on when we learn about more things but turns out that cloning something is actually really really fast too so it doesn't really matter and sometimes it's very beneficial. So don't worry about that. For now we will see let's just see how to actually clone a process. So word I accidentally just said was fork. So fork is the system call that creates a new process and like I said it is a copy of the current running one. So the function prototype is just fork takes no arguments and then returns an integer. So the return value is actually the process ID of the newly created process and if it fails to create a new process it will return negative one and if it is successful so if we have successfully created a new process that is a clone of us then in the new process that is created fork will return zero and then in the parent process that actually created the new child it gets a process ID that's greater than zero and that will be the process ID of its child. So now it can keep track of its child and you actually know who's who but after the fork completes like at the time of the fork both processes exactly the same the only way to distinguish the two is the return value of the fork. So now there's two processes running they can access the same variables but they are completely independent they are separate and the operating system why this is fast is something that is called the copy on right optimization where you don't have to reload like the standard C library or anything again you can actually share memory between them and copy them really fast as we go through the course we'll figure out how to actually do that. So on POSIX systems you can find documentation using man which is the short for manual so we'll be using the following APIs today so fork, execve and then wait will be in the next lecture so you can use man to look up documentation so just man number and then the function name so some of these will give you weird results so for instance if I do I think it's man fork no not man fork man execve okay so for these these will actually tell you all the information about the system call you need or you know it's 2024 you can just use the internet so they're also available on the internet but if you don't hate you can use the man pages. All right so let us dive into an example and where things start getting actually fun so here is our program can everyone see that okay so in here after main well we know because we S trace that main is not actually where we start executing things some things that will happen before main is it will open the standard C library loaded and all that fun stuff but now we're going to put a fork in it and immediately we're going to call fork now after the fork we have two processes are exactly the same while right now we didn't declare any variables in main or anything so they'll just start executing after the fork independently the only difference is going to be the value of this return PID so I'll have two processes and then in the parent return PID will be greater than zero if it's successful and give me the process ID of the child so in here I have a bunch of print statements so I will print what the value is of the return PID and then I also introduce a few more system calls here so there is a system called called get PID so that gets this processes process ID and then there's another one called get PID so that gets the process ID of your parent so all of these things should be consistent if there is a strict parent child relationship so in here I return or I print off the value of the return PID which since this is a child process it is going to be zero and then the child's process ID well should be the same as the return process ID from the parent since it should represent this new process and then I can check in the child what its parent process ID is which should be the original one so any questions about this? Yep, no the get PID and get PID they're system calls that are part of the C library so their system calls are given to you so in here let's run it and ask any questions because this is when your world starts getting weird whoops I need to recompile it so if I run it this is what I get and in fact I got really lucky in that the kernel switched what was running randomly so here I got printed the parent return PID which should be the process ID of the newly created process I had and then well the kernel decided to switch over to the other process and execute that so it switched over to the child process and then it printed return PID which is zero so if that makes sense it's the child the child's PID is 6879 which is exactly matching what the parent returned and then the child's parent ID is 6878 and then that child process terminated it was done the kernel decided to switch back to the parent and execute that so the parent's ID was 6878 which makes sense because the child's parent ID should be whoever created it and then finally we got the parent's parent ID which is 4700 so any questions about that? It's okay if this is a bit weird Oh yep. Do I have any questions? Yep, yeah so it kind of looks like I went into both if statements at the same time right which should be impossible but the reason why it looks like I'm in both if statements is because I have two processes running so one executes one and one executes the other so I can try and draw that so luckily your reasoning about a single program or just single process hasn't changed but what has changed now is that there may be multiple processes running so in this case what was it? So in this case what I like to do is write to the side what process is actually running cause we can have multiple processes running so initially I just pick a random number say this is process 78 to match what we saw but with less digits it would execute main and then the first thing it does is call fork so we will create a new process that is exactly the same as process 78 and is actually executing this line so we would create let's say we create process 79 and it would also be doing this fork call so at the time of the fork they are going to be exactly the same but we don't have any variables or anything like that so we don't really have to argue about much but the only difference is that in process 79 well that's the newly created process so what will fork return? Yeah, zero, right? So that's the rule you're gonna have to remember for this course the newly created process always gets zero from the fork and then in process 78 eight fork would return the process ID of the newly created process so it should be 79 that gets returned from this fork now both of these processes at this point are completely independent and then you can use all the rules that you know and love from C and arguing about C programs so we have to argue about two different processes so say we have the child running first so if the child runs first which is in orange here well it would create a local variable called return PID and then assign the value zero to it and then it would check this if statement while return PID is not equal to negative one so it would fall through here, check does the return PID equal to zero? Yes, so it would execute these three lines of code and say it goes all the way to the end it would fall through return zero and then this process is done so this process 79 is responsible for printing all of these lines then let's say it's done now we can only run the parent so it gets the value 79 back from fork it would now create a local variable called return PID, assign it to 79 then we would check, does return PID equal negative one? Nope, check here, does return PID equal to zero? Nope, so it would fall through this if and then it would print these three statements and then return zero and now the process is done so we did not violate the terms of C we did not execute both sides of the if statement we actually had two processes one executed the if, the other executed the else so for a single process still acts as you would expect C to have but now we have multiple processes, yeah yeah, so forked forks whatever is running it so as soon as you call fork you create a new process that's a copy of whatever just called fork so your questions were, yeah it's not executed again we'll see weird things we can do where I see where you're going, yep no, 79 does not call fork so in this originally whenever we start the process like whenever we start about it we just argue that one initially only one process is running main and then this one process calls fork and then 79 or 78 creates 79 so 79 only comes into existence after it returns from the fork so before that it didn't exist, does that make, yeah so in this, so a process is just a instance of a running program so whenever we run a program it's actually running in a process because it needs virtual memory and all that stuff yep, yeah so I'll make an example after this because it has virtual memory they should be independent so they can change them without affecting the other but I'll go through other questions, yep outputs oh yeah, so the question is about this PIDT data type so it's just a typed up so usually we want things to be more easily readable and try and express what the number actually is so there's this PID underscore T which is basically just an int but it's just a more expressive way of saying this int actually represents a PID but yeah sorry I should have explained that but they mean the same thing it's really an int with a different name, yep oh that's a good question so parent, parent, parent ID it is some random value, what does that refer to? any guesses as to what that would refer to so this parent process is what I actually ran so whoops, it is called fork example so this is fork example so the question is who created the fork example process so any guesses as to what this PID is, yep terminal very close, very very close any other guess, yep VS code slightly colder but still in the right track, yep the kernel, which process so processes are managed by the kernel not part of the kernel though, yep ZSH or a shell or something like that so that is the answer, that is the process or the process that I'm typing these commands into so it makes sense that it would be something like ZSH or bash or your shell and in fact, because we know about the proc file system now we can actually see exactly what it is so I can do look at the file 70 or 4700 and look at the status and see its name hey, it's ZSH and I can see who created it so it has a process ID, it has a parent process ID and I can follow that up as long as I want so that also brings us to an interesting fact here so if the only way to create a new process is by forking or cloning well, guess what? In order to run my process ZSH I was originally a clone of ZSH at some point which, fun to think about, yep yes, yeah so the question is why did I see like one line from the parent and then all from the child and all from the parent so at this point when we had two processes like right after the fork that differed in the return value we have two different processes and your kernel gets to decide what runs and the kernel can make any decision it wants so me seeing that output was just random so between the process I'll always see like these three lines from the parent all in a row in that same order and the child I'll see those three same lines in order but I can't argue about anything between processes because it's out of my control the kernel just picks what process to run it can just stop it from running run something else, stop it, run something else it gets to decide, yeah in theory I could have seen parent child, parent child, parent child I could have seen any combination between them but between in a single process it will all be in the same order yeah so in this case I actually got lucky so what typically happens is the parent will just run to completion and then the child will run cause it's not doing that much so yeah like I said I bet so I got three parents and then three childs but if I run this over and over again likely I'm gonna see it in that order over and over again but I don't have to yeah so this time I got really weird the child printed it all first and then the parent and then before that I got the same thing I got originally parent, three children, parent but I can't make any arguments about that and that is where the fun comes in yeah so the parent's return PID is one higher than its actual one so that is the process ID of the child you just created and typically the kernel will assign it sequentially so the parent was running so it was running 75, 87 in this case so if it creates a new process the next number available is probably 79, 87 yeah the only difference is from fork so fork essentially you can think of it as it kind of returns twice but it returns only once in a different process so it creates two processes in one process fork returns the process ID of the child if you were the one that created it if you're the newly created process you get a process ID of you get a return process ID of zero all right yep yeah so that's a good question is there any way to communicate between processes and we'll actually have a lecture for that it's inter process communication so yeah right now all our processes completely independent they can't communicate so we need a way to communicate between them which we'll see in a bit yep wait at the back so when would the fork command fail so the fork command would fail probably if your computer is catastrophically failing if it's out of memory or something like that or maybe you've run out of process IDs because there's only a certain number or you've run out of some other resource so usually if fork starts failing you're kind of in the shit no good so yep yeah so that's a question that we can do at the end of lectures is this the fork that's referred to in a fork bomb and the answer to that is absolutely so also to demonstrate that they are completely independent let's create a variable called x x equals to one and then after the fork let's just print off the address of x so this print line I just highlighted how many times is that going to print twice right it should print once from each process and because they are exact clones of each other the address of x should actually be the exact same so here so I can see from one hey the address of x is whatever edd04 and this one's edd04 so I can go ahead and go here and change say x equals two and then in this one x equals three and then here I can print off say the value of x so what should I see when I print this out yeah the two different values of x that I assigned even though this could run in any order so if I go ahead and run that I should see x equals three and x equals two no matter in what order they run in because the kernel makes sure that after the fork even though they look exactly the same after the fork they are completely independent of each other one change does not affect the other if they are so they're actually updating the exact same memory address so if they actually weren't independent sometimes I would see like in the parent process if they actually access the exact same physical memory well what might happen is that the parent updates the value to two and then if it's the exact same location the child comes and updates it to three so sometimes the parent might actually print out three instead if they actually shared memory or anything like that but no matter how many times I run this I'm going to see x equals two coming from the parent and x equals three coming from the child because they are completely independent at that point so questions about that so you can see same address same everything okay so next question might be how do we actually start executing a different program and that is a different system called called execve so that replaces the current process control block replaces the process with another program resets the process control block and starts executing that so execve has the following api so path name so full path to the program to load so on linux that'll be an elf file and then there's like an argv array so that's what you get in main all the arguments that get passed to you and then there's environment variables same idea as execve or argv we'll have some we'll have some code examples with exactly what this looks like and then it returns an error on failure does not return if it's successful so let's just go into a code example so in execve whenever we start executing main we just always assume that we just have some process running it with just a random process id so here we print off i'm going to become another process and then in here what it takes is an array of strings or an array of character pointers so argv is the command line arguments for it by convention the first argument is just the name of the process uh but it doesn't have to be in fact and it doesn't have to exist in fact that was actually an exploit at one point where someone assumed that you would always have like the first address would always be valid and some and that was actually a mistake in the sudo command that you might have used during your vm so they screwed up that check and then turned out you could just access sudo as any user and just have full control over a system so that was a fun bug but this is generally the way you do it the environment variables um it's basically just like uh not quite a hash table but just like key values that we can set for a process but we'll just make it null for now and then for execve we're going to turn this process into one that can actually start executing a different program so in this case the full path of the program i want to execute so say i want to do ls the default location is user bin ls and then i feed it the arguments and then i check if execve returns an error if it returns an error which it returns negative one it means it failed to load that program or something like that it failed so all these functions if they return negative one and fail they set a global variable called error no and then we can use this p error function to actually nicely print out the error force with what happened so we can say execve failed and then in the case where execve is successful this process is now transformed into one that starts executing ls and it actually is no longer executing this code anymore it is running ls you can think of it as it's running main of ls now and it this line here will never actually execute so there's only two things that will happen execve will transform this process into ls so it will maintain the same process id or it gets an error and then i see an error message so if i run this i see i'm going to become another process and then i see the results of running ls because i just transformed this process into ls and it started running ls so this is basically how your shell works so instead of create process that starts running another process if i want to run another process i could combine fork and this execve so i could just fork make a clone of myself and then transform the child process into whatever i want so any questions about this yep yes yep yep keeps the same process id so this process that printed all the ls stuff if this what was originally executing main was process 100 ls would have been process 100 there all right any questions about this yep yeah so what's the difference between these two so why do i have ls here and then ls here so this is the path where the actual executable is and this is supposed to represent whatever the user typed in their shell to actually start that program and they might be different so sometimes some programs will actually read this and do something different so they might actually share the same binary um in this case it doesn't actually matter i can just type whatever i want here and if i run it oops that's not the program doesn't doesn't matter it doesn't use it some things will use it by convention so if i did like ls dash dash help it usually tells me if i go to the top here it gives me like a usage thing like i should do ls option file turns out that if i it actually just uses that to generate help messages so if i change the first argument to wtf and then the second one to help and hopefully when i compile it i should get the same help message but if i scroll up what tells me about the usage tells me what i what i typed so just a convention it uses doesn't have to match but if it doesn't match you might be confused but you know in the shell you can like alias stuff and whatever and sometimes it's helpful but you can kind of screw around with stuff if you know what you're doing and confuse people so don't worry about that too much all right any other fun questions all right i have a fun question to you because someone mentioned something called a fork bomb so here's my question to you what would happen if i did the following so everyone likes wall loops right it's like wall one so that's wall true what happened yeah yeah yeah so that's the right answer so in this well if i have wall true bad things kind of happened so first time through assume i just have one process so then that process would create another process that's the exact copy of itself so now we would have two processes and then they'd be right here at the fork now both of those processes are going to go through the while loop again and then both of those processes are going to create another process each so now i have what four processes and then each of those four processes assuming that it switches every time iteration of the loop those four processes are going to create four more processes so one each so then i'll have eight then i'll have 16 then i'll have 32 and then i will create a lot of processes so yeah there's a question before of like why would fork fail might fail here so if i actually run this what will happen so let's just also just print oops let's see if fork starts failing so i'll just put in a print all right oh god oh you're making me do this all right so if i run this i should see just a lot of bs right and let's see let's put the process id here too i get all right oh jesus why why why do i do this so if i run this my screen should just fill with stuff my system probably will not be happy right three two one oh yeah it's fail it's starting to fail but um there's still a lot of processes running on my machine and that's probably not great so my machine is actually smart enough to or the kernel running is actually smart enough to stop creating processes after we've exponentially created quite a lot so i can still use this machine there's just a lot of processes trying to run over and over again in this while loop but i can still use it on some machines that have a kernel that is not configured to do that well eventually you're going to run out of memory or run out of process id's or something like that and then your computer will not no longer be usable and this is called a fork bomb so exponential growth just keeps creating processes over and over again and thankfully i can kill them all and it's smart enough to kill all of them because i have a very smart kernel but if you run this on you know some other machines it might not behave so well and you might screw up their system all right well that's a good thing to end it on so uh just remember i'm pulling for you we're on this together