 All right, welcome back everyone. So, new week that we get to start exploring processes. So, all right, come on, settle down, settle down. Volume. Okay, so where did we leave off last time? So last time, we looked how to transform a process and actually run another process. But we didn't actually get to the point where we actually create a process. So finally we get to do that today and hopefully I have a better chat setup so I'll actually read it now. So now I have time, so that should work better. So we had that initial question before of like why the hell am I doing this thing of replacing a process and that's the only way to run a new program. Why don't I just create a new process from scratch and tell it what program I want to run? So if you did that, you would load that program into memory, create the process control block just from scratch out of thin air instead of just copying the curve, instead of doing what we'll see today. So this is what Windows does. So Windows actually has a create process system call like thing and what it will do is it will just create a new process for you out of scratch and you tell it what program you want to run. But Unix is a bit more flexible in that it decomposes this process into two steps. So the process creation, which is separate than running a new program and we'll see that that's actually a useful thing to do and it actually gives us some more functionality. So what Linux does instead or any Unix is instead of creating a new process from scratch, it just clones the currently running process. So what it does is pause the current running process, copy its process control block into a new one so it makes a complete copy of that running process, reuses all of that information so any libraries it has open, any files it has open, all of its contents of memory, everything like that, they're all copied and then the only difference between these two processes after the system call, which is like a clone, is the return value of that system call. And Unix is maintained this like strict parent-child relationship so whatever process initiated the call is called the parent and the other one is called the child because it created it. And that's the only difference and there's a strict parent-child relationship and every single process on Unix follows this. So every single process will have a parent. And then the nice thing about this is if we don't want to continue running a new process that's the exact copy of whatever was running minus its return value, well if you want you can optionally have it create a new process by doing that exact VE system call and then replacing the current running PCB and starting a new process. So it's a two-step process to create and run a different program. So this is kind of what a typical process tree looks like of that parent-child relationship. So everything at the top is a parent and whatever is below it with that line is its direct child. So if you have a virtual machine or you use a Unix system and you do top or h-top, something like that this will be your parent-child relationship and there is only one special process and that is process with process ID 1 usually just called init and the kernel's only job when it's finished booting the actual hardware all it will do is create a new process with process ID 1 so each process gets assigned a number which is just the way to identify it it will be a unique number so when the kernel is done booting it will create one process with process ID 1 and it will load typically whatever is in like S-bin and init or something like that so a program called init and that process is responsible for creating every single other process on your machine. So in this case we can see that okay if we want to follow the tree to say Firefox or something like that well init create a process called systemduser which is a weird thing basically that's just a manager of another manager of processes for your specific user and then you might have like a desktop environment like the actual your actual desktop and it would be something like gnomeshall or something like that if you're using one flavor of Linux and then through that that would be running as a process and then you would go like pick applications Firefox and launch it and that would launch Firefox as part of what it does when you click that Firefox icon and that would be running as a process and then Firefox might have a bunch of different other processes also called Firefox which funnily enough there's going to be one new process for each tab on Firefox basically how they architected Firefox so each of these processes would be a different tab for example and then another way you could go is hey if I opened a terminal then it might have a process like the terminal server and then within that I have my shell which would be zsh or bash or something like that and then next would be whatever command I started executing so if I'm running something like htop that would be the thing running so that would look something like this if I go in here and you probably can't see that at all and I should probably get out of there so if you do something like this so if you do something like this this will show you all the processes running on your machine and try and draw like a tree on that command thing so here we can see process ID 1 all the way on the left there you can see it and then see whatever processes it creates and you can go ahead and explore every single process on your machine so these are all the processes that are actually running and for lab 1 you are writing a simple version of this program so you're going to explore, you're going to see all the programs running on the machine and print out their names as part of lab 1 so this gives you a bunch of information all for lab 1 and you're going to write off this process ID there and you're going to write out not what command got run that the process represents but what its name is so so if you run htop that would be in your dev container if you set up lab 1 already if not, oh yeah, if you haven't logged on the github site for the lab please do that because I have to manually create your repository if you leave it to the last minute and I'm asleep you are definitely screwed so please just do it as soon as possible all you have to do is log in it shouldn't take more than like 2 seconds so you could do it now and then yeah so that's about lab 1 so if you wanted to explore knowing a bit more about this parent-child relationship you could explore see what process created which and next we'll get into how to create those processes but first note that as opposed to before linux terminology is a bit different, I just showed you very generic terminology, linux terminology is a bit different so there's this proc file system which is what you'll be playing with in lab 1 so there's like slash proc and then there's a bunch of folders and files in there that you can explore every entry in proc that has a number, represents a process ID and you can actually poke around and see all the information about the process you want so if you wanted to look up the process state like that state diagram before there's a file in there called status and one of the lines in it just says state that will say what the state of that process is so you could check any state of a process just by reading that file and looking for the line state and in there will be one of those five states running or runnable so linux doesn't make any difference between running and waiting or ready, it just says both are running and then there are two types of sleep or blocking, there's interruptable sleep which means you could wake it back up and it might be able to do some useful work and then there's in interruptable sleep which means no matter what it won't wake up it's waiting for a network connection or something like that so even if you woke it up it couldn't make any progress so it just kind of stays around and then optionally you have some things where you can manually stop and start a process that typically aren't used that much but you can do it and that's that stop state and then the last state is this weird term zombie so we will learn what zombies are in the next lecture and it will get a little bit weird the terminology or kind of gruesome but it's always kind of fun so that's basically how you translate it for linux and you can poke around that proc file system and you will get more familiar with it in lab one so like I said before just so you have it on unix all the curl does whenever it boots up and initializes itself and it's ready to run at programs is all it does is run a single process so it will just run typically a knit which would look for an sbin and knit typically on your file system and that is responsible for executing every single other process on the machine if that process does not exist then the kernel thinks you're shutting down and it will exit for you and shut down your computer so basically if that process exists your operating system exists if that process is gone that means you want to shut down the computer so the curl would go ahead and shut down the computer for you so for linux there's a bunch of different things that could implement in it you could even do your own in it if you really wanted to but for linux it's typically something called system D but there's a bunch of other options that are subject for debate and on some other operating systems there might be other like weird tasks to kind of do some bookkeeping things so some operating systems like windows would make a fake idle process basically to keep track of how much idle time your CPU is just sitting around for and then finally these magic numbers so again these magic numbers for file descriptors are just convention and there's something you should probably remember if you're mucking around with any unix process at all because you have to manage these if you're creating processes and make sure that they actually are what they are supposed to be so every standard convention is that when you start a process there are at least three file descriptors open zero one and two file descriptor zero is standard input file descriptor one is standard output and file descriptor two is standard error so if these are only your file descriptors where you read and write bytes from while we kind of have some knowledge as to what our terminal emulator actually does because these just deal with raw bytes and it doesn't really matter past that so whenever you type into something like a terminal while some of the jobs it has to do is actually translate your key presses into bytes and then writes them to standard in so your process can go ahead and read them and then everything that you write to standard out or standard error well it has to be on the other side of that to read those bytes out which would be typically ASCII characters and then it actually has to draw those bytes out for you on your display so you can actually see them and then part of the other fun that we'll get into with them is you can mess with the file descriptors and redirect them to other things if you just don't want to display them if you want to send them somewhere else the power of Unix is you can go ahead and do that so if you really want to see like if you're debugging maybe you have a bunch of open file descriptors and maybe something went wrong so in that PROC directory there's also this FD directory and that tells you all of the open file descriptors for a process so if you use a PS command that will show a list of processes matching your user there's lots of other flags and in fact if you don't like them well you're writing basically PS for lab one so you could add additional features to that if you want to see anything else about a process but for example if I run a terminal emulator and see what the file descriptors are I can read what standard in out and error are and they'd be something like devTTY which is like a terminal device that you don't have to know and let's go ahead and see just for proof of concept what is actually running on my terminal so I just happened to know because I went through the lecture that my shell is running as that process ID so if I go there and look at the FD directory I can see what the file descriptors are so there's a few other open ones but you can see here that they're a little bit different 0, 1 and 2 are actually all the same thing and there's devPTS thing which looks a bit weird that's a pseudo terminal which you don't have to know the details of it but it's just a fake device that I can read and write bytes to and that's basically what your terminal emulator does but you can see it has a few more open things and you can explore this all you want to kind of get the nitty gritty details of what's going on you can also use this command because you can do that if you see has anyone ever seen that error message before like oh some random process file open and you have no idea what that is well on Unix because you have this you can figure out what it is so for example you can use this LS command or LS of which will tell you what processes have a file open if you really want to know so one thing I could do is I could see what processes actually have libc opened which basically is probably every single process on your machine but if you want to you can do something like that and it will show you everything that has libc open and is using the C standard library so that's kind of a cool thing you can do you don't really need to know how to do that but it's kind of the power of this model and having everything nice and visible for you wrong button okay so that's a little demo of that you can do that for any file on your machine you want so this is actually the meat and potatoes of today so you can find documentation using man whenever you do these labs like if you do man execve you can get some documentation for it today we'll be that cloning process it's called fork so it's called forking and then we'll see a wait system call in the next lecture but you can always use like function name to look up documentation and sometimes if it isn't what you expect you might have to give it a number so for example if you do man fork without any number I believe it gives you like documentation for awk or something that you've probably never used or never heard of so if you want the actual like system call wrapper you have to do man to fork or something like that so that's just something to pay attention to but so let's talk about fork so its API is quite simple it creates a new process which is a copy of the current running one and the reason I introduce this so early is this might actually hurt your brain a little bit because now processes don't start at main and keep on executing now you can create a new process in the middle of another running process so we'll have to probably get used to it and practice it again and it has a following API it's nice and simple where it doesn't take any parameters whatsoever and it returns just the process ID of the newly created process so it's like a C standard wrapper so if it returns negative one that means an error happened you didn't create a new process but typically you always assume it returns successfully otherwise your computer is probably dying in some respect and you probably have much bigger problems but typically we won't see failures we'll see zero returned if it's the child process and we'll see a return value greater than zero in the parent process where that return value is the process ID of the child so the child will get a new process ID as part of being created and then the parent will get that process ID as a return value and the child will just get zero and that's the only way to distinguish what the original process was and what the new process is is just the return value of fork otherwise they are exactly the same and we'll see over and over again what I mean by exactly the same so now there's two processes running and the operating and they like literally exact copies so they can access the same variables but they're completely separate and independent copies of each other so if you modify X in one process it won't touch the other process they're completely separated from each other so we'll see mechanisms behind this because this seems like really slow because I mean it's an exact copy so like even all of the contents of memory will be the exact same which might seem really really really slow like hey if it's Firefox and it's like 10 gigs or something like that because browsers you know use that much RAM well I don't want to have to copy 10 gigs just to create a new process that would probably be really slow we'll see techniques to actually speed it up but for the purposes of you using it it's like an exact copy of all the memory contents as well so here is our example so fork example we'll have one process in each branch so luckily we start reading programs from main as usual so if we execute this program it would start executing it main and then the first thing it does is call fork so fork would create a new process and now right before this return value we would have two processes and now in the parent it would get the process ID of whatever it was just created say created process ID 101 and then in the child which would be process ID 101 it would get zero return there and now there are two processes that can execute it's up to your kernel whether which one to execute or if it wants to run them both in parallel that's up to the kernel you don't get to decide it's up to the kernel and then at this point well in I'll assume that there's no error so that this branch wouldn't be taken so now there are two processes one the child process would be an exact copy so at the time of the fork when it's an exact copy there's nothing defined or anything so there's not really much to copy and then it would return zero from fork so there'd be two processes when I execute this the child would come into this branch and then print off all this information about the child so it would print off whatever fork returned and then it would print its process ID which is done through another system called called getPID so that will return your process ID and then you can also check the child's parent ID independently through another system call so that's getPid so getP parent process ID and then in the child it would be done this if statement and then fall through to the end of it and then return zero which would call exit and then end that process one second and then in the other case where the parent there'd be another process that would check if the process ID is zero it's not it would be the process ID of the child then it would come in this off parent return whatever it was in this case I said 101 then print off its process ID and then its parent yeah so the question is assuming fork doesn't have an error and I get past that then both branches run simultaneously and the answer is there's two processes and they may or may not run simultaneously and you may know what order they run in in fact you definitely won't know what order they're in so it could be random depending on whatever execution and this is the stuff you'll get used to in the course where you have no control and you don't know which one is running first and it is completely up to the curl to the side so after the fork right after the fork returns that's the new process so there's a new process you can think of it right after the fork and it would get a different return value and then otherwise the two processes are exactly the same yep so the two processes are exactly the same except for the return value of fork no so they'll be different so let's go ahead and run this so this is the start of like unfortunately this course will get into threads and processes and we have to argue about this stuff so like you said some questions you might have is like oh what order does this run in funnily enough this will actually execute in the same order like 99 times out of 100 but it's like that 1% chance if you actually rely on it running in order that you run into trouble yeah so I'd be running a process that starts executing main first whatever is currently running so the process currently running that fork call so like so like normally if I ran like hello world or something like that and we ran this we would expect this to create a new process and this to start executing at main so this would be our process and then in this case this process would call fork and it would be a copy of whatever called fork at the time of the fork so let's go ahead and run this and we'll do lots of examples so if I run this I see both sides of the if branch but that's not one process executing both sides of the if branch because that's still impossible we only see both because we execute we have two processes running after the fork so in this case I see parent return process ID so this would be the return value of the fork so in the parent I got process ID uh 31074 so a way to check is we can see the difference between them so the child got returned zero from that fork call and then the child's actual process ID through that getPID system call is 34074 which makes sense because this number the parent return process ID and the child should be exactly the same because that's what we define you know that's what the return value of fork should be so those are the same we can also check that the parents or sorry the child's parent ID which is whatever created it is 34073 which is the parent's process ID so that is what created it and then you could go further and say well what created this process that I just ran on my shell and the answer to that is well this process 22347 and then since we know what PROC is well we can figure out what it is so PROC so that's what created that process so my shell created that fork example process and then within fork example it created a new process and then that printed out those child lines and then that process printed out the parent lines and then they both executed and exited and then stopped so any questions about that yeah yeah so what's our question that if we write something like like that yeah so if we write this so this actually has a special name it's called a fork bomb because it is so the first time through this loop it would create a new process and now there's two processes and now both of them would go back to the beginning of the loop then both of them would create a new process each so that we have two new processes then all four of those will go back and then create two new and we'll have 8 then we'll have 16 then we'll have 32 so one fun thing you can do to your friends there's like a shell command that will do this and it looks really really weird and yeah if you do that it just creates processes over and over again exponentially and then eventually your computer will run out resources and crash so if you want to provide some protection against it that's what you have to do as like a kernel so some newer kernels will actually detect things like this and be like no no no I'm gonna kill you before you start replicating so this is something you can do but alright any other questions yeah so that's the return value of the fork so here I'll show it another way too so we'll go through this like fairly fairly slow so first we'll assume we're running this program and we are running it as a process and I'll just say for simplicity that it's currently running process ID like 100 so if I say I have process ID 100 that something else creates for me and starts executing I start executing main so I'll put a blue marker that says where each process is currently executing so before I'm created I start calling main I haven't made any call yet then process ID 100 is going to call fork and now process ID 100 is going to move down here and as part of that assuming it's successful it's going to create a new process so we would call it say process ID 101 and it is also it is also right after the fork so right before the fork there are exact copies of each other in this case there's nothing to differentiate them at all and then the only difference after the fork call is going to be the value of return PID so in process 100 what would its return value be from that fork yeah 101 because that is the process ID of the newly created one so it would get 101 back as the return value fork and process 101 would get zero back as that return so this is after the fork so it is not a copy because it happens after the fork the only thing that gets copied are the state right at the fork not after so after they're slightly different and the only way there difference that return value so now at this point we have two things executing we process ID 100 and 101 at this point you don't know which one is going to execute first so if you want to like be if you want to be really complete you would have to argue about you know if this runs first then I'll see this if this runs first then I might see this and you'd have to do that for every single line because it could get swapped out at any time and we'll see that a little bit but now I assume that if let's say for example it chooses to run process ID 101 well it's going to execute as we know normally it would check the if conditions and everything so in process ID 101 here whoops come on yeah so it's in process ID 101 is process ID greater than 0 hope no right so it would fill this and in this process it would go to the else and then check is the return PID equal to 0 and this case it is so it would go ahead and the next thing if it's still executing would print or would execute this print f which would execute the child thing and then eventually it's going to print off all the statements and then fall out and then it would just return 0 and in which case at that point it exits the process and that process is no more anymore and then next we still have another process executing so that's process ID 100 so in this case instead it would reach the same branch again so it would reach this if statement but now it's checking the return PID which is different between them and it would actually check that hey it's greater than 0 now because it is 101 so it would print off the parent return whatever and then the other prints and then because it's in an else if it would just jump to the end because it's done it went into the if branch then execute return 0 and then it would be done so everything is a copy and they're all independent so we can see what that looks like too okay so first here so first I'll demonstrate a few other things so here whenever I run it I always see parent first instead of child well the only reason I see that is because I put this command here which is like a you sleep system command or system call which tells the current process to just go to sleep for a little bit and wait so it's just kind of wasting time after it prints which actually shouldn't do anything and you actually don't know why I put this here yeah this is not here for the reason I thought sorry so we'll see why that's there in the next lecture but it doesn't have to be but sorry that's confusing yeah don't worry about that sleep for now but if I run it over and over again I always see parent first although there's nothing stopping child from going first and I am getting very unlucky so like I said like yeah oh sorry you can't see anything okay whoo okay so if I execute this program again and again and again well I always see parent first instead of child and in this case I'm just getting super unlucky okay I guess this year I just get unlucky so you can see them in different orders but it's really really really rare because typically the way the scheduler works is it gives each process like a little slice of time and in this case it's not really doing anything so by the time it calls fork it would probably have some time left to execute and then immediately continues executing after the fork that original process so like like I said like 99 times out of 100 you'll see parent first yeah yep they can be stopped mid if statement as well so if I wanted to I could go ahead if I want to like see issue I could sleep here so you sleep will put this process to sleep first for one second so in this case no matter what order they switch tend to switch really really fast I should see parent return process ID and then it will be in that block state and another process can run so that child process will probably run print it three lines and then we'll see the parent do the other ones so if we go ahead and we so if we go ahead and compile that and run it now we see that hey that first parent line happens then it would switch to another process do those three prints and then finish the last two prints of the child yep yeah so it could stop in the middle print f depending on how print f implemented because it may or may not do a system call or may split up the input if it's like too big but it would get interrupted at some point if you have lots of processes running then they may just if like there's a line break or something it may get interrupted in the middle and you see like it split off weird so yeah this would happen yep yep yep so the child doesn't exist until fork is done so the way it works is only one process is calling the fork in this so when I initially start here like when I start running main I only have my process 100 and then process 100 like 101 doesn't exist yet so process 100 will call fork and that will create process 101 and then the only way to differentiate them is the return value of fork so in process ID 100 it will see 101 returned and then in the newly created one it will see 0 so you can think of that so after fork there's going to be two processes and the only way to tell them apart is the return value of fork so there's two things that can run and you don't know what order they're going to run in yep so this is an example of concurrency or parallelism we'll get to that later but you don't know what it is like you know we won't have two things running some chance if I only have a real CPU for example but it could run concurrently your operating system or kernel is the one that switches between processes that we touched on a little bit and to see that they're actually like complete copies of each other well let's make some variables that do something more interesting so I've int x equals 42 and I need to switch that so I've int x equals 42 so if I'm truthful I would be like hey at the fork let's print so if I do something like that and I run this what should I see I should see x equals 42 two times because it's after the fork if I move it here what should I see yeah x equals 42 one time because it's before the fork so let's go ahead and make sure that that is actually true so if I go ahead and do that I see x equals 42 probably from the parent yeah it would be from the parent then I see x equals 42 from the child and they're completely separate but I mean I just made the variable it's not really that compelling so what would happen if I did something like so in here my parent always runs first and my parent always runs last so let's say in the child I do something like x equals I don't know give me a number 13 so in this case I have x equals 13 in the child and only the child and here I'll remove that so anyone has a guess what I'll see printed yeah yeah ideally if they're completely independent well here before the fork I have x equals 42 and then that exists before the fork and then I would fork so both processes would have an x called 42 or the x with the value 42 and then there'd be two independent copies after the fork so if I change one it wouldn't change the other so in here in my parent it wouldn't change anything and then print off x hopefully equals 42 and then only in the child it would print x equals 13 and go ahead and print x equals 13 so let's see that so if I do that I see x equals 13 and that change happens after I print it so it definitely happened first I definitely changed x equals 13 before the other one printed what the value of x is and I can see it's still 42 sorry you had a question? so with a fork they're exact copies of each other and they copy even the memory contents so they're exact copies so this seems a bit weird right but this is how it is any questions about that or any like other examples you want me to do with that yeah so the question is if I have a file and it copies everything including those open files well would I have problems with two things writing to the same file and yeah you would which is funnily enough that's why I see both print so we both we know that the print f's just write to file descriptor one and that's why I see them both on my terminal because it's a fork they're exact copies even their standard outputs so both processes standard output go to the same thing so that's why I see both of them okay so we can see even like and by exact copy I'm going to mean like exact copy so even if I so maybe it's okay that their exact copies in terms of like the value of x's the same but like what about its address so if they're like exact copies down to the byte should they both have the exact same address even though their values are completely the same all right hands up same address hands up different address okay that would make sense that they have different addresses right kind of well if they were exact clones they should be exact clones like they should even have the same memory address which seems a bit weird but if I go ahead and run that yeah it complains whatever they have the same memory address even I told you they are exact copies of each other and they're completely independent yeah it's the operating system so someone already solved the mystery this happens because of the magic that we like touched on before virtual memory each process has its own virtual so virtual addresses so they're not real physical addresses so as part of the black magic of your kernel well in this process this address would only be valid for this process to point to a specific location in memory and in the other process this address would actually point to a different physical location of memory even though they look exactly the same yep so the question is when the first fork happens do they point to the same physical memory address so the answer to that is initially it does and that makes it fast but we'll get into the implementation details as to what's actually going on there yep we'll get there okay any other quick questions yep well we can do that so the question was is the address completely randomly if I created it after the fork would they be you know completely different so in this case if I move the x back one it's not copied as part of that fork and then I have two processes that create x as their next thing they do so if I go ahead and run that I see that hey their memory location are exactly the same and why is that well if I do int x it's created on the stack and right before that fork happens well there's that main function has a stack its stack pointer is going to be at some address and at the fork it's an exact copy so at the fork they actually both have the same stack pointer address so if they both do the same step of just creating one more variable on the stack they'll have the same address yep so we'll get to that each process has its own like virtual memory that's actually mapped to valid addresses yep so here I can make them diverged like they can completely diverge if I have you know if I create a y here in this process it would have a y and the other one wouldn't they're completely different now and in fact to create a new process I could do like the execve here in this one and it would be a completely new thing yep this is the only way to make them do different things because that's the only return value you get from it you just get a number and that's it yep so the question is if I just malloc do I get the same address and the answer is it would probably set up the malloc library and it would all be the same addresses so if I do the same step I get the same thing alright well we'll leave at this for today we had nothing on in the chat which was kind of weird but cool alright just remember pulling for you we're all in this together yep