 All right. Good morning, everyone. So today we'll be a bit different. We will do a lab for primer So we'll do or I won't do you will do a mini version of lab four so if you can do it the lab four is not much harder than this lecture, but As the caveat, I'm not going to be doing anything. You're gonna be doing it So we're gonna we're gonna come up with a solution together and this will also somewhat help for The quiz because we're going to deal with processes and make processes and do fun stuff So what are we doing? Today, this is our task. We want to send and receive data from a process and actually communicate with it And this will also help you if you ever need to glue things together with Python Essentially, we are making the Python sub process module So our task for today We want to create a new process that launches whatever programs in the command line argument And then we want to send the string testing with a new line to that process And then we want to receive any data it writes out So we want to send data to it and get data from it. So we're communicating with it both ways So that sounds pretty cool. How do we do that? So I'll introduce some API's that we're going to be using today. The first one is called pipe so Jesus that jackhammer is annoying So the first one is called pipe and It takes as an argument. Oh, sorry. I missed the it there. It takes an array of two ints That fills in a file descriptor and it's a C function. So it returns zero on success negative one on failure and sets error no So a pipe is basically a one-way communication channel using two file descriptors so if pipe succeeds it fills in the array of two integers and at index zero that is the Read end of the pipe so you're only allowed to do a system call read on that file descriptor and then File index one is the right end of that pipe So you're only allowed to call write on that file descriptor and the way it works is anything you write to it Will pop out of the read end so that's kind of useless for a single process because you're writing data You read but because it's a file descriptor and we know when we fork we can copy file descriptors And we know standard file descriptors. We can use a pipe to communicate between two processes As long as we Do some fancy stuff that we'll be doing today so you can think of it if it's easier as like a kernel manage buffer and again anything that's Written to the right end will pop out of the read end and the kernel will handle that for us So we saw exec Vee. It's kind of ugly to make our lives a little bit easier. There's C wrappers To make our lives easier and one of the C wrappers that will make our life a little easier is called exec LP So instead of taking a full path It just takes a file and then an argument. That's not an array and it just ends with a null null Pointer so this is the exact same as exec Vee It does not return on success and it will return negative one on failure and set error. No, so it skips Having you use a raise and it also searches for executable using your path environment variable So you don't have to specify the whole stupid, you know path to some executable starting at root All right, so next API will use is this close. That's how you close a file descriptor So you're done with it and the kernel can reuse it So again return zero on success negative one on failure sets error. No, that will be a common theme And it closes it that file descriptors no longer usable You can't read or write on it and then it frees up that file descriptor again file descriptors are just a number So the curl can reuse it again So if you close file descriptor zero Typically open will use the lowest file descriptor so it would just replace zero So that's one way you can fiddle with file descriptors another way is This one dupe and dupe two So it essentially makes a copy of a file descriptor and again This one's a little bit different. It returns a new file descriptor on success negative one on error and sets error. No so it copies the file descriptor old file descriptor and Sets a new file descriptor so they're both pointing to the same thing So in the dupe version We just copy, you know, we take the old file descriptor and we just get a new file descriptor back And both old and new are pointing at the same thing But in the dupe case, we don't get to pick what number that is and the dupe two case we do so Dupe two is also going to be atomic So what dupe two will do is it will take this new file descriptor argument If it's already open and a valid file descriptor It will close it for you would would free up that number and then it would do the copy of course If we have multiple threads, for example, if we have to argue about stuff for like your quiz or any multi-threaded Program, there'd be some race condition there if it has to close in an open and another thread couldn't come in and swoop That file descriptor so dupe two is atomic and that's why it's you know a special system call So it'll make new the old fd and the new fd refer to the same thing. So any questions about this? So there's make two file descriptors point to the same thing and with dupe two we can change the number All right. Now we know enough to make a solution so If you want to follow along whatever it's in the lecture 16 directory That that's how you build it. It's already built and we're going to create an executable called sub process That just takes a program as an argument and again We want to send the string testing to it and read whatever data writes out to standard out All right, so There is a small skeleton So the first thing just make sure that we have two command line arguments otherwise it returns e in Val which this means invalid arguments and Here I just set up some arrays because we'll probably need them and create a fork and then in the parent I give these Pipe file descriptors and the process ID of the child and the child I give the argument for the name of the program So Where should I start? So let's go back to our goal Because you guys are doing this I am not so we want to create a new process that launches from the command line argument So if we fork we probably want to turn our child into that That's probably a good first goal and then we want to send the string testing to it and then read whatever result comes back out of it So what should I start with? Yeah Okay, so Comment was write some code to figure out if I am the child or parent. I did that so I have a parent function and a child function and You know when I make it I have a bunch of unused file descriptors because I don't set them to anything yet Or I don't use them All right, so first off So let's see if I want to run it with this I expected to you know Run that you name program and then I should be able to capture whatever output it gives So if I run that right now, oh, whoops, I already write some code. Oh, yeah, I already started you off There we go. So in the child I just execute that program So that makes sense Cool, what should I do next? You name yeah Yeah, so we could execute oops We could execute any program we want What's gonna happen if I execute cat right now? Anything I type should be come out So if I run cat I get input-out-put error Yeah, the fuck the file descriptors are still a bit messed up. I didn't do anything with them, right? So there's two things it's trying to read from standard in which is the same standard in as my sub process And it's like and now I don't see anything available to read. Yeah Okay, so let's set up the pipe between each other. Let's capture some output first. That's sound good to everyone So if I want to create a pipe between the two and essentially share file descriptors Where should I create my pipe? Should I do it before or after the fork? After the fork so you want me to do like here I'll do pipe with out pipe Like that sorry It is an error. It's already two. Yeah, so it's all right, too Okay, so I'm going to just put everything so I wrote a check error function that basically just checks if the return is negative one And then we'll just exit the program with error. No Just to make debugging a lot easier. Oops All right, so I do that Okay What's My next plan So I created a Pipe for an out pipe. So After here, right, I'll have out pipe fd 0 Which is my read end And then I have out Pipe fd one is my right end Sorry No, so they'll get set from the pipe call. Yeah Yeah, so in this case if I have my pipe after the fork, how many pipes do I have? Two right after fork. I have two processes. So I have two pipes and There's no way I can share because they have two independent pipes now, right? So I can take advantage of the fork So if I move this up So now how many pipes are there one pipe, how many processes are there? To so I have one pipe in two processes. So then I can connect them together. Hopefully, right? Okay, so that seems a bit better So I have my pipe and then here in the parent in the child I pass, you know out pipe file descriptor and a pipe file descriptor to both of them. So I both have In those functions, they can both access both of the file descriptors but there's only gonna be one pipe here and It's the same for processes because whenever I forked it copies their exact clones of each other They're independent, but all their file descriptors are the same and file descriptors are essentially pointers So they have all the same file descriptors Sorry. Yeah, so they're they're kind of it. So there's one put so first We're just gonna deal with getting output from that process So we have one pipe we have two processes and we just need to connect them together somehow Yeah, yep, but I forked after the pipe so it's caught When you create a new process everything's the same thing at the time of the fork Right and this is how it's gonna work. This is good how it's gonna work on quiz three, too So everything right before the fork at the time of the fork They're exact copies of each other with the only difference being the return value of that fork call So right after the fork, I will have two processes one will come into This and called parent and then one will come into child and called child But at the time of the fork they're going to be independent or they're going to be complete copies of each other So because I create a pipe before the fork, they both have those file descriptors. So This would be, you know If I want to give them a number there's going to be the standard file descriptor 012 So this one's going to be file descriptor 3. This one's gonna be file descriptor 4 like globally Because it just uses the lowest number So after we fork both processes are going to have the five file descriptors They're going to have the three standard ones and then two from the pipe Yeah, so we're going oh, why do I have in pipe fd? So we're going to be giving data to it and then reading data from it. They're each going to be pipes Yeah, they're all one way So I'm going to have a one way into the process and one way out of the process So we're just going to deal with the one way out of the process first Yeah Yeah, so I just initialize it at zero, but it gets updated due to the pipe call So the pipe it's going to use you know the pipe fd argument and then update them if it's successful No, so by default whenever you create a process, right? There's the standard file descriptors There's file descriptor zero one and two And then the way Linux says if you open new file descriptors use get the lowest number first So pipe's going to open two new file descriptors and it's just going to do lowest first So I'm just I know that this is going to be file descriptor three and this is going to be file descriptor four It is pointing to the read end of the pipe Sorry Yeah, so um So the comment was can we make the read end point to standard output? That's close So what do you typically do if you know if you're a process and you want to write some information out Or you want to say hello world or something like that. What did we do? What system call did we use? Right, right. So if we want to capture the output Then what we would want is for the right end of the pipe The child process That end of the pipe should be its standard output, right? So it would write to that and then In the parent process we should be able to read from that to get data out of it Right, and that's our one-way communication channel. We want to Send that process's standard Output to us and we want to read from it So we can do that with a combination of dupes So in here if I want to do that, uh, let's So I could do a dup If I want to overwrite a file descriptor so Let's say I want to overwrite Which file descriptor am I trying to replace here if I'm trying to capture its output Standard out so I should probably replace standard out and this is just It's just a little bit more readable than seeing one. I wouldn't really care if you put one But here so what file descriptor do we all want to replace with standard? replace with standard out out pipe fd one right So So if we do that That gives us that's a little bit closer. So now this would replace Essentially, it replaces the original standard out, which would be the terminal or something and replaces it with the pipe fd And then overwrites whatever was file descriptor one originally So if we do that, let's see what happens if we run your name. So if this pipe is connected Should we see output now? Cool So we did something That's cool. We should probably check that if we have errors or not Oh, there's comments in chat too. Cool Yeah, so pipe's just a one way communication channel. All right So let's make sure dupe didn't have any errors. So it doesn't have any errors. We're capturing it And we should be nice and clean up all of our file descriptors. So we should also just check error We should probably just close out pipe fd Zero close So this way we can clean up all of our file descriptors. So Let's read through it again. So At the time of the fork we have five file descriptors. Yeah Yep So I don't need it because now there's two file descriptors pointing at the pipe I don't need file descriptor four for example So here here, let's go through it. So here at out pipe fd. I have five file descriptors open, right? So I have the default zero one two and I have file descriptor three and four Okay, so then the fork happens. So both processes have the exact same set of file descriptors Let's argue about the child. So if we go in the child Here it would be a dupe two of what I say that was four So this makes fd four four and fd two Point to the same thing Right after this line So at this point I'm just cleaning up my file descriptors. So close this would close fd Three and this would close fd four and that way it's nice and clean So my new process whenever exec happens, it's still it just has this three standard file descriptors It doesn't have any additional garbage open And we're going to see why this is important quite soon. But yeah No, so if I close zero then when that process spawns zero will be an invalid file descriptor No, I never close zero So I overwrote zero in this line But I and then I closed additional ones I'd never close zero Like I yeah, I closed three and four So four and zero are pointing to the same thing Yeah, yeah, I just killed a pointer. So now now I just have one file descriptor pointing to it and it's the standard Your standard out. Yeah We're being Nice Yeah, we're we're very nice programmers here Okay, so We don't see any output. That's a good sign So how do I read whatever the hell just happened? How do I get that output back out of it? Yeah, so I want to I'm communicating with the parent So I should be able to Get data somehow. So what should I do here? A for loop Sorry, we want to print out what the child sent to us. So it should it dumped information to the pipe see Okay, let's Yeah, that's that's the class dog Um, so what do you want to do? Sorry? So let's see. So here's our pipe Right and this is right And this is read so In here we have The child And then we have the parent Okay, so the way the child is is the child's file descriptors are one two three After we clean them all up and then It is we leave this alone original original And then file descriptor one points to out pipe Fd One which is essentially This So it's filling up that side of the pipe Okay, and then so in the parent, what do you want to do? Sorry So are you But we want to capture it. What about if we want to do something with it? Yeah So standard in So in the parent I have So here I have one two three. So these are all original Because I haven't screwed with them yet, right? These are all original and then I also have I also have out pipe fd Zero and out pipe fd One Yeah Get my standard into out pipe zero Yeah Yeah, that seems that seems logical, right? Why am I We don't want to touch the parents file descriptors. They're all good It's a file descriptor. We can read from it. So what happens if we just read from it? All right, who wants to tell me how to read from it? Read Okay Whoops Read Let's see Oh Oh semicolon. All right Whoops Hey that compiled statement with no effect Yeah, that compiles Yeah Yeah, I mean why wouldn't that work? Can I do this null semicolon? I can do that reads just a function pointer Yeah, I mean, but if we just do read it the compiler gives us a nice warning. It says statement with no effect So, I mean it's not going to do anything But I mean it's cool. It runs still it compiles so So read so it needs a file descriptor what file descriptor we want to read from Out pipe fd zero. So that's the read end of the pipe So if the child process is writing to the pipe anything Anything it writes to the pipe comes out of the read end if we read didn't okay It needs a buff and size t Help my brain. I hit my head this morning. So I don't know what to do Buffer 10. Okay, we'll use this magic number All right, so we'll use this magic number Yeah, so that's a question. Do I have to check if it's ready first like what about they I get there before it writes any data and well Is read a blocking system call? Yeah, so read will just sit there until it has data That's remember our example that just read read and wrote It was essentially cat It just waited for data right it just waits there So that's fine. So that should be fine if we wait there So read needs a buffer and then size of buffer so that's Missing a bracket All right, that's kind of cool So what should I do next so that seems to compile it did something Yeah, so there's something So we'll take a little bit of shortcut So if it you had to read more than 4000 bytes, you'd have to do multiple calls and fill and do the buffer We'll assume everything's pretty short and fill and fills that so we'll be we'll be a little bit lazy But in general you should do better than this Yeah, although first I should probably check my errors, right? So I don't even know if this is valid. It's I'm just throwing it out. So what does read return? Anyone off the top of their head Number of bytes read I heard okay bytes read Okay, well I should probably check my errors because it returns negative one if it's invalid. So I'll just Do my check error function just to make sure So check error again just checks us the return value is negative one if it's not negative one It just returns it doesn't do anything if it is is it captures the error No Prints the message with whatever you give it and then exits the process So here let's check that we actually read something So that's still good. Hey It didn't crash or do anything. It's pretty cool Let's check printf And I'm allowed to do printf because I didn't screw with my the parents file descriptors So I can still do printf because remember we know printf. It just calls right on file descriptor one So if I screwed up file descriptor one I can't you if I printf it's just going to go to the pipe and then go off to some nether regions So this is why in the parent. I did not touch them So here I read Let's see. Uh, was it you let I think that's the right format bytes read Maybe it's ld Tape has int. Oh, okay. It's just an int Well, I'm just seeing the number bytes read first. So I read six seems right Because if I run it It's linux one two three four five and a new line So read so we have some evidence that we actually We actually read something and we've read something from another process. It's kind of cool, right? Well, let's wrote out. Let's just write out whatever we wrote. So I'll give you the shortcut for this. So this is a weird format specifier So this is if it's not a c string. So it is percent sign dot star s So by default we you kind of know what this means, right percent sign s It's a c string, but is the output from this process a c string? No, we've No idea if it ends in an all or not. So this format specifier Just lets it be a string and you have to say the length of the string So the length of my string is the number of bytes I read And then buffer So then I can got So I'll just output the string from my buffer given the number of characters I read. So if I do that I My parent process should have actually captured the output from essentially the sub process or the child If I do that got linux Wow, so we actually are communicating between processes now It's pretty wild, right? Yeah, yeah Yeah, so so how this worked So how this worked is in we created a pipe So we create a pipe So it had a right end and a read end and then we forked So now we have two processes a parent and a child They both have the same pipe all the file descriptors are copied at that point in time Then after that they're all independent So in the child what we did is we replaced file descriptor zero with The right end of the pipe So now whenever that process So whenever that program prints the standard out, it's just going to do a right one and now Instead of right one being whatever is a default. We set it to the pipe So whenever it writes to standard output, it's actually writing to our pipe now And then we're just nice. We close the extra file descriptor so that That process just has the three standard file descriptors open and not some additional garbage And then after we exec Yeah, so after we exec it just starts running that program it You know it does a read on file descriptor zero in this case It doesn't use standard input. It just displays some message and that writes to file descriptor one like standard out So it writes to the right end of our pipe and then in our parent process We just simply read that data So we're communicating between two processes here. That's our one-way communication channel out Okay, who wants to guide me through the next one So next one we want to whoops We want to send the string testing with a new line To that process Yeah, so another pipe Luckily enough I already created a pipe an in pipe fd So that's pretty good. Should I create my pipe before or after the fork? Oh Oh, oh you said before okay, okay All the jackhammers are screwing with my hearing Okay So now we have two pipes So we already set up the out pipe properly because we're set it up so that we We set up so that we read from it and then we set up so the child writes to it So for the in pipe, we should be doing the other way around So the child should be reading from it and we should be writing to it to form that communication channel So let's deal with the I'll remove these and let's deal with the child first So what should I do for the input pipe in the child whoops pipe not pi pi would be good though Yeah, go ahead Yeah, so I want to connect the standard output. So I probably need a dupe to call because I know I want to replace standard input Which is just going to be zero. So what's replacing standard input? so I in in pipe fd which zero, right That is the read end of the pipe so whenever now whenever this Process executes whenever it reads from standard in it would be reading from the pipe Okay, I will and then we want to be good and close our additional pipes this Oops, why am I recording macro stop? all right Now we fixed our pipes Cool. Well now we still have a bunch to do. Okay, so now we want to Write information to that pipe So this was Okay, well if I want Give that process Some data. What do I want to do? Just write and I should write to it first Before I read from it. Otherwise that doesn't make too much sense Hey, right to the pipe Oops, right to the in pipe. So let's be specific and then read from the out pipe So cons car Message so our message was going to be testing with a new line Okay, well, I want to write to it. So I should probably check by it's written And then I'll do the right system call. So What file descriptor do I want to do here? in pipe fd one Right, so that's the right end of the pipe And then I need a buffer. So I want to print the message and The size is whatever the string length is So that should do that and I will check for error check error bytes written Right just to make sure so If I execute it now And I use sub process with if I use u name doesn't matter. It's not going to read anything anyways, right? so More interesting is probably cat. So if I do that got testing So I wrote testing to it And then I read testing from it Because cat just doubles whatever so Everyone get that okay, so We have forgot something Sorry set exit it stops earlier. So it does stop a bit earlier um Are we good parents? We want to kill it We want to wait for it We create a zombie. So we should actually be fairly nice So if I want to wait for it I can actually where do I want to wait for it? Yeah, after the right and before the read probably So I just let it finish and then I'll read whatever data it has So how do we be good parents? So yeah, we can do waitpid So if we want to waitpid we need Somewhere to put it so waitpid It wants a process id which is childpid so And then stat location. So that's w status and zero So we want to waitpid for it And then we should of course check error so it takes a int pointer and sets it to the status of the child whenever it exits So Yeah, so man So if we look at it We call wait status Wow this Suspends Okay, that that that doesn't help Essentially it just writes the exit status there and then you can check it So first we'll write some asserts W if exited so this is just stuff you'll know from the beginning So we want to make sure that the process has exited six That it's exited and then if we want to get its exit status we use w exit status So that gets its exit status so we can double check it actually exits properly so What's an exit status that means everything's all good? Zero so we should probably just make sure that our program's all good and nothing bad has happened So let's do that So now when we do this on cat Uh-oh spaghettios We broke something So I have to control c and it's sitting there And the cat's still not running But we broke something our weights in the wrong spot So no our weights not at the wrong spot wait after read Could but I don't want to So what is if it's cat what's it going to do it just reads Over and over how when does it stop? End of file or when read is zero or it's closed Did we close it? so the in the parent The parent still has The right end of the pipe open So how does cat know it's done if I still have the file descriptor open It doesn't know what your process is going to do you could write data to it at any given time So after I write to it here What should I probably do? Close So let's check error close And what should I close one So now if I close that that means hey I'm done I close that file descriptor and it should Then be able to send you know that end of file thing which is just read returns one which essentially just means it's done So that file descriptor cleanup is going to be a fairly important thing in lab four because you're going to be essentially this Right if I do this over and over again and type That has what three pipes in it. That's lab four So if something freezes Then it's probably you didn't handle your file descriptors properly So here I'm actually fairly messy with my file descriptors. I could be a little bit nicer where At the beginning, which ones do I know for sure? I'm not going to use at all Well, I don't use this one at all So I should probably close it and I also don't use outfd I don't use outfd one So I don't use the right end of the pipe. So those are two ones. I don't use I should just close them immediately Just to keep things clean. So I don't have this stupid issue where You know file descriptors stay open and it just kind of looks like it hangs there because it's waiting for data So I cleaned up So I cleaned up in pipe fd zero and cleaned up in pipe fd one Then I cleaned up out Pipe fd one. I should probably clean up out pipe fd zero just to be complete So I'm done with it here. So I should probably close it right after I'm done with it So now I'm clean so here In main I created two pipes and then Oh, sorry in main. I created two pipes. I can delete this now I created two pipes and then the child I closed all four of them So all four file descriptors. So I close them here close them here after I shuffled them around And then in the parent, I also closed all four of them. So I closed the ones. I know I'm not going to use immediately And then I closed the right one right after my right call and I closed the read one right after my read call So I cleaned up everything nice and this is Kind of why they tell you to you know make sure you malloc on free because there's some things it's much it's very important So now if I do this Subprocess cat Cool and also the uname one works As well still so you're just sending a data that it doesn't care But I can run a whole bunch of other stuff ls ls works too. I can capture everything from ls I don't know so true is also a program that doesn't do anything just sets a one as an exit code So that's fine Then false. Oh, well false. It always exits with error code one. So triggers that assert So that's pretty cool And then as the next things that I'll leave you off of to fix is hey, what about if I do, you know, something like this invalid huh It kind of failed silently Why would it fail silently? Yeah Yeah, but why can't I see that a problem happened So it tries to run the program called invalid Which doesn't exist So I could also instead of invalid I just do all right. Well, we're at time anyways, but that's that's our solution. So Close So you'll have to fix that in lab four But but lab four if you understood this today You just do it a bunch more times So if I do cat pipe cat pipe cat Well, I'm creating three processes two pipes. I'm just connecting This to that and then this to that That's our one-way pipe Cool. Well, let's remember I'm pulling for you. We're all in this together