 Okay. Welcome back everybody. We are going to continue with where we left off last time. And if you remember, we were talking about how to get a networking layer that we can build interesting distributed applications out of. And basically, layering is our technique here. We're going to build complex services out of simpler ones. And so we started with the IP datagram service, which is basically a way of getting unreliable messages to route from one point to any other point in the network. And from that, we're going to build up something more interesting. Okay. And the reason we need to do this is the physical layer or the link layer that's this next layer up is pretty limited. So packets are of limited size, which is often called a minimum maximum, excuse me, transfer unit of 200 to 1500 bytes maximum, except in very special circumstances in server rooms and so on where it can get larger. And routing is limited to within a single link. And so that doesn't help us much if we want to go across the planet. And so our goal is basically to show how to go from the physical reality of packets, which are limited in size, unordered, unreliable, only machine to machine, only on the local area network and asynchronous and insecure into an abstraction that helps us build interesting systems, which is arbitrarily sized messages, ordered, reliable, process to process rather than just machine to machine, routed anywhere and synchronous and secure. And our first step down that path was basically wrapping the IP datagram with an additional piece of information to basically give us a process to process routing. And our example of that was the UDP protocol, which is IP protocol number 17. And basically we took the IP header, which was 20 bytes that was machine to machine. And we added another 232 bit source and destination 32 bit entities here, which include a 16 bit source port and a 16 bit destination port. And basically that minimal header gives us the ability to route from a process on one machine at one IP address to a process on another machine and another IP address. And the way we do that is by supplementing the physical name with a port. Okay. And the important aspect of this UDP is extremely low overhead. We just added basically eight extra bytes to the IP header. And oftentimes UDP can be used for streaming services that might be audio or video, etc., where we want to have low overhead. Unfortunately, you can often use this in an antisocial way if you're basically shoving way too much data down the pipe and blocking everybody else out. And so we're going to talk about something a little more social when we get into TCP IP. So the problem with UDP still is it's unreliable. So if we send a message down from one process to another process, we're not guaranteed that it's going to arrive. And that checksum helps us identify corrupted packets, but it doesn't help us make sure that packets arrive. And so basically any physical network can garble or drop packets in one way or another, just because errors due to noise and other things can cause that transmission to fail. And good examples of that, for instance, is in a low power scenario. You might want to try to be transmitting as low power as possible just to keep your energy usage down. And so then you're going to start dropping packets and you need to do something about it. The other part that's kind of often not thought of in terms of lost transmissions is congestion. And so this is a situation where the data has arrived properly at some intermediate hop. But what actually happened was there wasn't any buffer space at that intermediate router and the router was forced to essentially drop the packet. And we're going to talk about congestion as we go forward as well. And as I mentioned, when I said UDP is not necessarily social or is anti-social, it's because you might ignore the congestion problems and go ahead and try to shove all your data through anyway and cause everybody else's packets to be dropped. So we want to get reliable message delivery on top of this unreliable IP datagram network. And so this really means, among other things, that we need to make sure that packets actually make it to the receiver. And we want every packet to arrive at least once and at most once. Now, why do I say at most once? Well, as we'll see, one of our primary techniques for reliability is going to be retransmitting on failure. And it's quite possible that we have an erroneous assumption when we thought that the packet didn't make it. And so we said it again. And as a result, we end up with two packets. So that can be a problem. And we can combine this with ordering. So if we have a bunch of packets that go forward, and we want to make sure they arrive in the same order at the destination that we sent them, perhaps the same mechanism for reliability and ordering can be used, and we'll see that that's true. So that's actually going to be using acknowledgments. So as a simple example here, how do we ensure transmission? Here we have a communication between A and B. So A sends a packet to B, B sends an acknowledgement. So what did we find out at that standpoint? Well, we found out that the packet arrived at B. And basically, the receiver is just sending this act to tell us that the packet arrived. And if we're checking checksums, etc., we can essentially say that the packet arrived correctly. Now, what happens if the packet is transmitted, and it doesn't arrive? Well, eventually, A is not going to see the act. And so it's going to time out. And it's going to send the packet again. And eventually, we get an act. So notice that this technology that we're using here is going to let us make sure that the packet arrives at least once. But as you can imagine, it may not help us with duplication. So it's quite possible that this packet was sent, and it got delayed for a long time. And so it didn't really get lost. And then A sent it again, and then B gets two copies of it. Okay. So some questions about this is if the sender doesn't get an act, does that mean the receiver didn't get the original message? No, because I just told you that it's possible that the packet will still arrive at B later. What if the act gets dropped? Well, if the act gets dropped, then we might keep resending. That's another problem that might come out here because acts are unreliable. And if you remember, when we talked about the general's paradox, a couple of lectures ago, the issue there was really that messages going along with messengers in both directions were unreliable. Okay, so we need to do something a little bit about this to deal with the fact that our acts can get lost and our packets can get duplicated, etc. So how do we deal with message duplication? Well, the solution here is to put sequence numbers into our packets. So the sequence numbers are going to be a monotonically increasing number that we put in the message. And so if a message with the same sequence number that we've already received shows up, we know it's a duplicate. Okay, and so the receiver is going to check for duplicate numbers and discards if it's duplicate. Now, the requirement here is that the sender needs to keep the copy of all the messages it hasn't gotten an hack on yet so that it can retransmit. And the receiver needs to keep track of all of the sequence numbers that might still be in flight so that it can throw out duplicate messages. And what's hard about this second requirement here is the receiver needs to figure out when it's okay to forget about received messages. It needs to somehow have an idea that those messages with those sequence numbers are no longer in the network. So here's a very simple option, which is basically our sequence numbers are exactly one bit. And that fits either zero or one obviously. And so what happens is we send a message at a time. And a starts with sequence number zero and it sends a message with sequence number zero to be. And when it gets an act back on zero, it'll send sequence number one to be and then it gets an act and then it goes back to zero. And assuming that we don't have messages that stay in flight for a long time, this one bit sequence number space is enough to help us with this duplication problem at the receiver. Okay, so the pros of this, of course, it's very simple and it's a very small overhead. The cons are it's poor performance. Now, why is this poor performance? Well, if this is a long path between A and B, and we're sending one packet at a time and then waiting for the requirement, or for the act, excuse me, the acknowledgement to come back, what we know is that this path is mostly empty. So all of the buffers between A and B are mostly empty because we're only sending one packet at a time. And if you think about this, this little thought experiment, what we really want is something more like wire latency times the throughput here so that we can fill up all of the cues on the way from A to B and on the way from B to A to maximally utilize our network and have maximal throughput. And so this alternating bit protocol is definitely not helping us here. It's just not allowing enough packets in flight. And if the network can delay for arbitrarily long, so if a packet could go from A and then take a, I don't know, or run around Australia and come back before it goes to B, then the one bit is not going to be enough to suppress duplication either. So let's put some more bits into the ax. And so here's a simple idea here, which is a window based acknowledgement. And it's a windowing protocol. It's not quite TCP and you'll see why in a moment. But here we have A and B. And the idea is we're going to send up to n packets without an act. And that means that we have to have log n bits or the ability for n different sequence numbers. And this is going to allow us to pipeline because we can send a packet with sequence number zero one two three and four and they can be in flight. And then we can have they can get in queued at the destination side, assuming that the queue is big enough to hold n packets. And then we can have the acts come back. And sort of when we see an acknowledgement zero, then we can reuse that act number and so on. Okay, and so an act really says, in some sense, receive packets up to sequence number x, time to send more. So acts are serving a dual purpose here. Basically, they're giving us a reliability aspect, which is concerned confirming that the packets actually received. And it's allowing reordering to happen here. So if you imagine, for instance, that these packets were sent out zero one two three four, but then they get reordered in the network. And then they arrive for arrives first. Well, we know, because we're expecting sequence number zero, we know that there are some other packets that are either lost or still on the way. And so if we wish we could in fact not acknowledge packet four until we get the ones before it, so that we make sure that we have everything in order. And and the sequence number really says that everything up to that sequence number has been received. And if a packet gets garbled or dropped, the sender times out and it resends something. So the question here is will be still act for packet four if it failed to receive packet three. So the answer to that question is really dependent on the protocol. So you could you can design him multiple ways you can act for immediately, even when you don't have three yet, because the the sender can know something about it. Or you could wait to act for until you get three. And that's what that's doing for you is it's making sure that you don't overwhelm the queue at the destination. Because if we act for too early, and we're still trying to wait for zero one, two and three, and that starts sending five, then we may run out of queue space destination. So it depends a lot on your queue management as well. Should the receiver discard packets that arrive out of order? That's a question. It's very simple to do that. So if we get packet four early, we could just throw it out and assume that a queue will eventually retransmit. The other thing is we could just put it in the queue and keep track of the fact that we haven't received everything up to there yet, and fill it in as we go. So these are different options. Okay. And so the alternative is to basically keep a copy until the sender fills in the missing pieces. And it reduces the number of transmissions, but things get a little more complicated. Okay. And what if the acts are garbled? Well, we just time out and resend. And what's good is if we time out and resend, say packet three, but packet three was already received. The destination knows that packet three was already received. And it could throw out that extra at the destination while still sending the act, because it'll assume that the reason it was retransmitted was there wasn't an act. Okay, so this is this is a basic windowing protocol because we're sending a window of packets of size five here. And when it becomes a sliding window, what happens as soon as we get packet zero, and we send that up to the application, then we can go on and acknowledge say five, etc. Okay, and that's now we're getting into what TCP does. Okay, are there any questions on this so far? So this is not TCP, by the way, this is just basic windowing. Okay, are we good? So what does TCP do? Well, TCP is a stream based reliable protocol. It's IP protocol six. So if you go back to the IP header of a few lectures ago, you'll see that there's the IP protocol number, which is eight bits. And so a six fits in there for TCP. And the idea here is that you stream bytes in and they stream out in the same order the other side reliably. And so what you see here is that, you know, A, B, C, D, E, F, G has already come out on the other side. But the rest of the alphabet until we get to S is somewhere in the middle here. And so I'd say stream in stream out. And basically, what's important to notice is because it's stream in stream out, there's no explicit message boundaries in the high level API, but rather just you put some bytes in and they come out in the same order. And you might put in ST and then UV and then WX as separate calls on one side, but they may come out together on the other. And so it's really only the guarantee that all the bytes that go in come out in the same order. Okay, and it's up to you to put any message boundaries on that if that's what you wish to do. So some details here. So fragments, the bytes stream into packets inside. So the packets are entirely hidden from the API. And so if I put in a really large amount of data all at once into into one side of TCP, it will separate it into a bunch of packets so that it meets the minimum transfer unit size maximum transfer unit size, I kept saying keep saying that. So that none of the packets in here are too big, it'll fragment. And the other side, it will put things back together. Okay, and so we want to avoid exceeding the maximum. And inside IP itself may do additional fragmentation. So both TCP and IP may fragment and everything gets put together back at the destination. It's going to use a window based acknowledgement protocol based on this, the fact that we're sending bytes in and bytes out. So it's going to be a little different than the previous one where we're putting acknowledgements based on sequence numbers based on packet IDs. Here we're going to put them based on bytes. Okay, and so in the window is going to reflect storage at the receiver so the sender doesn't overrun the receiver's buffer space. So we want to make sure that we don't go ahead and send packets off that then are going to have to get dropped at the receiver because there's not enough space. So that's basically part of this protocol is to make sure that we don't overwhelm the buffers. Okay, and the other thing that we want to do is we on the flip side, we would like to fill all of the cues up between point A and point B so that we get the maximum throughput out of this. Okay, and so TCP has kind of got these almost contradictory requirements one don't put in so much that you get dropped packets at the destination or even in intermediate routers. And on the other hand, make sure that enough can get in there so that you're filling all the cues up. And so that's where an adaptive protocol has to come into play. We'll talk about that in a second. And we're all automatically retransmit all the lost packets and adjust the rate of transmission so that if we start losing packets in the middle, because all the cues are filling up, we slow down. And that's where we become good citizens so that if other people are going through that router, and there's a link that's shared that's lower bandwidth, we will automatically slow down to adapt so that we're good citizens and everybody gets to share the bandwidth. Okay, and so that's fundamentally how TCP is one fundamental difference between TCP and UDP is that TCP is a good citizen protocol. So I'm going to show you an example here in a moment where we actually walk you through some of the sequence numbers. But I want to talk about the space of sequence numbers. So the space of sequence numbers are based on bytes that are transmitted. And so they start at an arbitrary point that's randomly chosen when the connections made. And so at the destination, or the source, excuse me, we basically have those sequence numbers that have been sent. Okay, and these represents bytes that have been sent. There are ones that have been sent and are in the network but haven't been acknowledged yet. And then there are the ones that haven't been sent yet. Okay, and so that's the space of sequence numbers in this space again is byte based. Okay, and the receiver basically has sequence numbers that have been received and given up to the application. And notice that there are some that were sent at the sending side that have been received and even given to the application yet, but the sender doesn't know that because it hasn't received the ax yet. And then there are things that bytes that are sitting in the buffer at the receiver side. And again, those are ones that have been sent but then not acknowledged yet. And then there are bytes that haven't been received yet at the receiver. And so we can see that there's some that were sent some sequence numbers that were sent at the sender that aren't received yet. And then there's of course, the ones that haven't been sent and they haven't obviously been received yet. Okay, and so the sequence number space is fundamentally byte based. Okay, so here we go. I want to show you an example. Now, the sequence numbers arbitrarily start at the, at the time of connection, I'm going to say they started at 100 here just for illustration. And this is going to represent the receivers buffer, okay, window. And what happens is over on the right here, I'm going to show the acts that have been received by the sender. And so right now, what we're going to say is that we've acknowledged everything up to byte 100 or sequence number 100 and we can take another 300 bytes. Okay, and so the the receiver is is advertising its window fundamentally as it goes. Okay, and so if you look here, we just sent a packet from the sender side where the starting sequence number is 100. And it was of size 40. And it showed up at the receiver and it got put into the buffer. Okay, and what happened is that buffer now has from sequence number 100 up to 140 or 139. There's a chunk of bytes there. And so what do we acknowledge? We acknowledge that up to 140 is the acknowledgement, which basically says that we have received up to 139, and that there are now only 260 bytes left in the window. So notice that we filled up 40 of our bytes and that's taken off our window size here. And so why is this useful? Well, this is useful. If we're we don't have an application that can accept data, and we fill this whole buffer up, sender is watching that. Okay, and so here's another case where we send a packet with sequence number 140 that gets put in here, it happens to have size 50. And we acknowledge back 190 and we only have 210 left. Okay, here's an example of sequence number 230 and size 30. So what happened here? Well, what happened here is a packet got lost. Okay, so the sender had sent this packet, but it never showed up somewhere. And so this this packet with sequence number 230, the sender sent it happily, thinking that the packet with sequence number 190 had already made it. And it didn't. And if you notice what TCP IP does is at this point, it only acknowledges up to 190. Because that's the only continuous chunk of data that's been received. And notice also that the remaining buffer space hasn't changed. Okay. And so as far as the receiver is concerned, it's still saying that I've only gotten a contiguous set of bytes up to 190, even though it's got a chunk in its buffer that's already been received further on than that. And notice that because we're advertising 190 slash 210, the sender is never going to send too much data such that we go off the end of the buffer, it's always going to send up to 210 bytes from sequence number 190. Okay, and so let's keep going here. Here's another sequence number 260, still acknowledging the same way, sequence number 300 still acknowledging the same way. So notice from the standpoint of the receiver, it's not getting acknowledgement for the data that it sent. So this seeming problematic, right? And it knows right now where the problem is, it knows that it was the packet from 190 that it sent that didn't get in there. And so at some point, we get a a timeout at the sender, and it retransmits the next packet that was missing, which is this one starting at 190. And now we fill everything in. And the the receiver says, Oh, now I have everything contiguously up to 340. And you can send me 60 more. So so coding to data basically helps you with the so the data doesn't get lost. Okay, so it as you're encoding your data, you're basically making sure that it's less likely that transmission errors are going to occur. Okay, and so it becomes less likely to have these retransmissions. The, and then we're finishing up here, we have a sequence number 340. And then finally, sequence number 380. Okay, and notice at this point, at this point, the receiver is acknowledging up to for sequence number 400. And but saying I have no buffer space. And so this is the point at which TCP IP, the receiver window is fully shut down and TCP IP will no longer transmit anything from the sender because it knows there's no space at the receiver. And so this is in fact how when you tell that or SSH into a remote machine, and you you start typing, this is the flow control that will make sure that that that channel doesn't try to transmit more than the receiver can get. Okay, any other questions about this? Okay, now, something which I we're not going to talk a lot of we're not going to talk about today is there's also a enhancements to this protocol called selective acknowledgement, which is negotiated typically between source and destination. And if that exists, then it's possible to say more clearly, for instance, in the case of a timeout, exactly which packets are missing. So if you have multiple holes, the selective acknowledgement, you can actually say, Well, I'm missing a packet here and a packet there. Another question that's on the chat here is, what happens when you run out of sequence numbers? Well, when you run out of sequence numbers, it just wraps around a zero and it just keeps working. Okay, because basically the sequence number, you're not going to have four gigabytes of data in in flight at once. Okay. All right. Did that answer your question, Max? So overflowing is not bad in this instance, because as long as you don't have more data in flight, then you can support by 32 bits worth of sequence numbers, and it doesn't matter if they wrap around. So I don't know, what else can I explain here, Sebastian? Can I, is there one piece in particular that is not making sense to you? So the exact time who the question here about who times out at this point, this is the sender times out, because it's seeing that it's only been getting acknowledgements for packets up to 190. And it knows that it's been sending everything else or sending other things. And so at that point, it the the timeout happens, and it starts retransmitting from 190. So that's the sender is timing out. Okay. All right. Is that good? Any other questions? specific questions? Okay. So so what you see here on this diagram, just to reiterate is this is the receivers window at the top. At the left, you see packets coming into the receiver from the sender. And on the right, you see packets going out to the sender from the receiver, and they have acknowledgements on them. And what's interesting about this is if you look at the packet format for TCP IP, in fact, TCP IP is a bidirectional protocol. And so packets that you send to the other side, you're also putting acknowledgements in the header for things you've received from the other side. So packets simultaneously acknowledge in the header for data received at the same time they're sending data. And so it's efficient from that standpoint. packet sizes are basically determined. The question here is how do you determine packet sizes? Again, the answer is based on based on the MTU, so the maximum transfer unit between the source and the destination is one of the things that will chose to make packets small enough to not get fragmented in the middle. The other thing is, if you've, if data has been sitting around buffered in a socket for too long, TCP will send it off as well. And so there's a dynamic kind of protocol here for deciding. Alright, so now what about congestion? Alright, so how long should timeout be for resending messages? If you wait too long, you're wasting time if a message is lost. So in this instance here, if we waited too long to do this retransmission, then there's a whole bunch of data that's just not being sent in our cues all the way from source to destination are going to shut down and our bandwidth is going to be bad. Okay, but if it's too short, then we're going to duplicate and send retransmit too often. And this is bad, even though we're about to get an act, we're sending duplicate packets in there. And what's a little bit counterintuitive here, perhaps, is if we retransmit too early, thinking that we're making the connection better, what we're actually doing is filling all those cues up and making it more likely that things get dropped in the middle because we run out of buffer space. So you got to be really careful to not retransmit too much. Okay, and so there is a stability problem here, which is more congestion means the ACA is delayed, which means unnecessary timeout, which means more traffic, which means more congestion, which is closely related to the window size of the sender. If it's too big, it means you're putting too much data into the network. So the the what I haven't shown you here is the sender has its own notion of how much data it's allowing to be outstanding. Now it can't have more data outstanding than the receiver's buffers, but it could have less. And it could choose to do less based on the fact that there's too much congestion in the network. Okay, so basically the sender needs to try to match the rate of sending with the rate that the slowest link can accommodate. So the sender actually uses an adaptive algorithm to decide the size of n, which is how much how many bytes are allowed to be outstanding. And the basic technique is, you know, the goal is to fill the network entirely between the sender and this receiver while still being a good citizen. And the technique is to slowly increase the size of the window at the sender more and more and more data until the acknowledgments start being delayed or lost and then you know you've gotten too big and you've got too much data outstanding. And so TCP has a solution called slow start, which is one of many possible variants on this, which is you start sending slowly, that's what it means. And as long as you don't get timeouts, you basically increase the window size of the sender more and more data gets sent before acts by one for each act received. And then the timeout causes you to basically decrease your window size by half. So this is called additive increase multiplicative decrease. And that's why you get this sawtooth. You get this sawtooth behavior in TCP where you slowly ramp up the amount of data you're sending and then you quickly go down and then you ramp it up and so on. And that sawtooth is a pretty good way of adapting to the size of the cues. And I actually put up a paper. There's a link off of the main schedule page about a Van Jacobsen paper from way back when when this first version of this additive increase multiplicative decrease was put together. Okay, but that's more 168 topics, which will CS168. So we'll move on from that. Now, if you remember, now we've got a secure TCP connection, which we had when we looked at web servers way back at the beginning of the class, where what happens is a socket that the client wants to make a connection with a socket at the server. And to do that, the server sets up its own server socket. And so the client requests a connection off a given port. And then the server generates a brand new connection between the client and the server, which is, as you remember, a five tuple defines this uniquely, which is the client address client port on one side, the server address server port at the other and a protocol in the middle, which in this case is TCP. Those are the five pieces. And as a result, we have a nice reliable byte stream between client and server, and that in turn allows us to do things like web servers and so on. And that port, which we just introduced as part of putting a protocol on top of IP, so that's why this is TCP IP, has many well-known values like 80 or 443 are typical web ports, 25, send mail, etc. We talked about those earlier. And network address translation allows basically there to be even more connections between clients behind firewalls and servers than you might suspect given the number of IP addresses. And I'm not going to go into that in great detail, but network address translation works roughly like this. If you have a client that's behind a firewall, they try to connect to say port 80 and they have a port of their own that they randomly get. And then when it goes through the firewall, it gets translated to the public address by basically having a mapping between the client's port and a port at the firewall side. And so these connections actually when they go through the firewall, these five tuples are dynamically adjusted on the fly so that there can be a whole bunch of client addresses that are private addresses behind the firewall, and you do that as long as you have unique client ports to identify them. And so that's what network address translation does. Okay, but let's look at this connection now that we have TCP IP. Clearly to set TCP IP up, we have to somehow come up with the sequence numbers and basically establish the overall protocol between source and destination that I was just talking about. And so the goal here is to agree on a set of parameters between them, which is basically a sequence number for both sides, for instance. And we need a good starting sequence number, which is the first byte in the stream, but we don't want to be too predictable about this because if somebody in the middle were to know what the sequence numbers are, there have actually been demonstrated ways in which somebody can insert themselves in the middle of a yellow connection here by knowing what the sequence numbers are that are going between the client and server and then basically causing a man in the middle attack. Okay, so we got the sequence numbers need to be unique enough and hard to predict to avoid hijacking connections. Okay, in some ways of choosing the initial sequence number are things like, well one of the things we have to worry about is making sure that the sequence numbers are randomly far away from previous connections so that packets that are already in flight might not show up from a previous connection with the with a new sequence number. Okay, and the simplest way to do all of this basically is to suit a randomly increment previous sequence number and choose a new one and there's a bunch of protocol implementations that basically have a pseudo random choice. So a question that's on the on the chat from a from the previous slide here is when I connect to the internet through firewall is my client ip port visible to the server always the same so the answer is the ip when you're connecting through a firewall here the ip that the server sees is always the same because that's the ip address of firewall. Okay, the port is usually different even without a firewall because the socket connection between the client and the server needs to be unique and so this client port is usually randomly chosen even when you don't have a firewall so that's okay, no problem with the question there but certainly when you've got a firewall in the middle there's going to be yet another thing that's scrambling the client ports. Okay, so let me just show you briefly this handshake and then I want to move on to something else but basically the setup is a three-way handshake where the client sends a packet with a with a sin bit set and a proposed sequence number the server accepts it sends back a sin for the reverse connection as well as an act for this forward connection and a proposed new sequence number and then finally we get the act back and so after we've gone sin act then we have established a two-way connection with x and y sequence number starting points okay and so that's the that's the three-way handshake okay and there are some interesting issues with this where if an attacker sends a whole bunch of sin packets to a server but then doesn't do anything else with it the big problem with that is the server if it's not clever allocates a bunch of memory for every pending sin and as a result you can cause the server to run out of memory and so there's a bunch of tricks that you can do in terms of caching and cookies and so on that people use to avoid that all right and then closing again the host sends a fin when it's ready to close host to acknowledges that that direction is closed but it may have some additional data to send its direction and then finally it sends a fin and we get a fin act and now we're totally closed okay and we can retransmit all of these in the process all right so now i want to move on so now that we've got a reliable way of making connections how do we build distributive protocols well in general uh once we've got a reliable connection we're still going to do something like message passing because we set up a reliable byte stream but then we do our own message boundaries maybe with a link followed by some data and then we've got message passing between a sender and a receiver and so now we've got reliable messages from point a to point b and we can abstract that away to think of this as send a message to a message box at the receiver where what's a message box well it's a unique for instance ip port combination at the destination and so now we can think of once we've quickly you know under the cover is established a tcp ip connection then we can start sending messages to it and receiving messages from it and this abstraction of a message box uh for the center and receiver now allows us to start doing things like oh i don't know building a two phase commit protocol like we were talking about last time for instance okay and so most of the distributive protocols that you are going to run into are talked about in terms of sending messages uh and and so we're going to think of it that way now moving forward as well and we're going to see what can we do that's interesting with a message base protocol so to do that we need something possibly a little more uh powerful than just a raw message okay because this raw messaging is still a little too low level so we've got reliability which is great but uh if we're building something interesting out of the sending messages we still have to wrap up all the information uh into a message at the source uh and then at the destination we have to decide what to do with it uh you know if a message comes in then what well we unpack it and decide what to do maybe we need to sit and wait for a bunch of messages to arrive um and even more subtle but still kind of uh painful is what if we have machines with different byte orders like big endian versus little endian orders okay so you learned about that in 61c where big endian order is something where a 32-bit integer is four bytes in a row where the first byte is the most significant byte and a little endian architecture is one with the reverse where the first byte that comes over a pipe or as in memory is the least significant byte you know if you've got two machines and they're trying to communicate and they have different endian orders that's a problem okay so uh you know uh computer science has gotten a long way by being lazy in clever ways and a very clever way to be lazy is to use a remote procedure call on top of our messaging okay and it basically is something which calls a remote procedure on a it calls a procedure on a remote machine okay and uh you could think of a client call something like remote file system read from file rutabaga and somehow through the networking and so on it gets translated automatically into a call on the server which calls the local file system and reads from the file called rutabaga so this abstraction of a remote procedure call is going to let us basically think and the client side like we're just calling procedures even though they're going all the way across to a server and coming back okay and so here's the rpc concept a little more basically here so the client is calling function f with a couple of variables v1 and v2 and what really happens is they call on their local node and it goes into something called a stub so a stub is a is a piece of code that gets linked into the client and it is special code that takes these arguments v1 and v2 and bundles them in some network transparent way and sends it off to a server which where the server stub receives it and unbundles them and then makes a call to the local version the remote version of the function which has a return value which now needs to be bundled back up to send back okay and then the clients double unbundle it and receive it and return it to the caller and so the caller called f of v1 v2 and it got back r and that actually went across the network okay and so if we think about this from a network standpoint there's actually a machine boundary being crossed here two machines and some networking going on in the middle with some packet handling and so on and so the tcpip with the mailboxes and messaging abstractions are all happening at this layer and then we're bundling stubs on top of that and as a result the client and server thinks they are calling procedures okay and what's good about this is not only does the client not have to think anything about messaging but the they don't have to worry about endianness because the stubs automatically transmit transform values in sort of from machine a into a network neutral format typically and then they're unpacked back into a format that b understands and vice versa okay that's all done automatically by the rpc now the implementation of this is it's a request response message passing under the covers the stub provides the glue on the client and the server the client stub is responsible for what's called marshaling the arguments and unmartialing the return values so marshaling is taking values and putting together into a network packet and unmartialing is taking them apart okay and so this depends on the system but it's may convert values to some canonical form serializes objects copies arguments passed by reference and so on so this marshaling process is potentially complicated but it's being done automatically by these stubs I showed you earlier here okay so the question here is is the client stub code something that you have to write and the answer is no I'm going to tell you about that in a second question here is what is what's the difference between rpc and a generic serialization set up like google protobufs so the google protobuf is helping you with the the network independent formatting but it's not helping you with the packaging automatically something up so you think that you've got a function here so you could have the protobufs kind of on the back half of this if you want it okay now let me tell you a little bit about this so the equivalence here with regular procedure calls should be pretty clear so the the parameters get put into the request message vice versa the result is put into a reply message the name of the procedure is passed in the request message the return address is basically the return address of the client needs to be included in the the connection with the sender to somehow make this work out properly so the answer to the question about do you have to come up with the stub is no there's a for whatever rpc technology you've got there's something called a stub generator which is a compiler that generates stubs for you the input is an interface definition which basically defines all the functions that are going to be available remotely and what the types are of their arguments and return values and then if you put that into the compiler and the names of the functions it will generate the stubs and then once you have the stubs you get to link them on the on the client in the server and voila you have a remote system that will looks like you're making procedure calls now some cross-platform issues here one is as I said these these idl languages know how to convert to and from canonical forms and tag things so that you know how things are encoded so that there are too many conversions if you can avoid it how does a client know who to talk to at the remote side well that's what's called binding and so you basically need to translate typically the name of some remote service into a network endpoint name like a port and an IP address and so on okay and so that's the binding process so any rpc has to basically talk to a binding service which is just like dns but for the rpc and it's another word for naming at the network level and you could either have static binding where every name for a given name it always maps to the same IP address or you might have dynamic binding which is a little more flexible happens at runtime so dynamic binding most rpc systems have that use dynamic binding via some name service and the name service is again just like dns but it's for the rpc so it's a little bit more complicated than dns because we're actually registering services like file service or something that's a database or whatever with this service and when we connect to a to one of these services we're actually asking for the details of how to connect to that service remotely okay and why do we want to do this dynamically well for one we can figure out who's permitted to contact the access the service itself and deny them upfront and also fail over if one hard piece of hardware fails we can dynamically switch over to a new piece and perhaps the clients only see a little blip in their service and we can also if we have multiple servers we can have some flexibility of binding time to choose one which is what big web service is like google and amazon dynamically choose between a set of possible servers that are outstanding at any one given time to try to balance load and so on and we won't talk about that for now okay so some problems with rpc so rpc is not a panacea for everything among other things non-atomic failures so when we're on a local node and we're making a procedure call mostly what we have to worry about is the machine crashing and if the machine crashes then basically your don't care anyway because the application is crashed with rpc you now have a problem where the requester client side is making a request but the receiver side crashes in the middle and now you have failures that are not atomic it might be kind of in the middle of processing some procedure call for you okay and so there's many different types of failures here you could have a user level bug causes the address space to crash machine failure cause causes processes on the machine to failure some machines compromised by a malicious party causes one of many pieces to fail and if you have these types of failures if you don't have rpc the whole system is going to crash or die and that's almost better maybe from a what really happens standpoint because you know whereas after rpc you have machines compromise or crashed in the middle and now we have to start worrying about how do we make a system distributed across multiple nodes behave well even when some of them are failing or being malicious and so now you can kind of see where why we were talking about distributed consensus and Byzantine agreement it's one of our tools in the box another is going to be replication and hashing which is going to be our topic for the rest as toward the end of the lecture and next time as we start talking about distributed key value stores but these non-atomic failures which could be distributed across multiple nodes can easily give you an inconsistent view of the world did your cache data get did your data get cached in one place but not in another or is a cache out of date or did the server actually do what you requested or not so these kind of interesting errors start coming into play but you know what but we're going to confront them because you know the advantages of distribution is that you can basically potentially get more reliability if you can fail over to things that are still operating when other things are down okay so another problem which is probably obvious but I'd like to say in any way is that RPC is not performance transparent so the cost of a procedure call is much less than the cost of say doing an RPC to another process on the same machine and is much less than network RPC okay and there's lots of overheads in RPC marshaling stubs kernel crossings communication et cetera that make it a very different animal from a procedure call so it's it's why it's an RPC not a PC and so programmers need to be aware that RPC is not free and you don't want to do RPC just for that the heck of it you'd like to do it in a way it gives you some benefit of location transparency when that's a useful thing okay and caching can help you but now you start getting into some consistency problems which we're going to address coming up okay so what is RPC useful for well RPC is useful for all sorts of things lots of services are actually exported as RPCs where you make remote procedure calls to services okay and as I'll talk about in a moment the NFS file system is was one of the first systems that really had a solid RPC as its underlying implementation platform so here's another way in which RPC is useful so if you think about communicating cross domains you know how to how to address spaces share with one another and we've talked up till now about things like shared memory and we've talked about semaphores and monitors etc maybe reading and writing for a file system and pipes so these are all things that you know you're familiar with another is doing a remote procedure call between two different processes is a possibility and what's interesting about doing that is if you use RPC between one process and another then you can transparently move that process for load balancing reasons to another node somewhere else in the network and your code will keep working just a little slower okay so RPCs can be used to communicate between address spaces on different machines or on the same machine and services can be run wherever it's most appropriate access to local and remote services kind of look the same from a semantic standpoint the performance is a little different and there's lots of RPC systems out there okay if you know what to look for there was a really large complicated one called CORBA the common object request broker architecture you use RPC those of you that have windows boxes use it all the time when you have two applications that communicate with each other oftentimes they're using DCOM which is a distributed RPC there's RMI which is a Java remote method invocation architecture and so on so there's many RPCs and one thing I can't leave the topic of RPCs without talking briefly about micro kernels so all of the kernels we've talked about so far in the term have all been this monolithic structure where what I'm showing you here in light blue is the part of the system that runs at kernel level with extra privileges and the applications running on top are basically user applications a micro kernel structure is one that's the kernel space is much more limited okay there are something that handles address spaces something that schedules threads so basically that's giving you processes and threads and then an RPC handling system that basically is able to forward messages from one user level process to the other and that's it that's the whole kernel and what you do instead of what we did in the monolithic case is things that used to be in the kernel like file systems are now processes running on top of the kernel and an application that's reading and writing from the file system is actually does so by performing RPCs through the file system excuse me through the through the micro kernel okay so when an application is doing a read to a file it's basically making an RPC call that goes through the micro kernel and comes up to the file system and back okay and so why do this yeah why not system calls well the problem with system calls is in the application in the monolithic case an application makes a system call it's now in the domain in which the file system is running whereas the an application makes a system call here that system call is merely into the micro kernel the micro kernel doesn't know anything about file systems okay and so the equivalent of what used to be a system call interface the POSIX system call interface to the file system is still available but now it's a library that makes RPC calls into a file system that's actually in another process okay and so why split the OS this way fault isolation so the the original reason for building micro kernels was basically to make a system where bugs in basic kernel services like windowing and file systems and paging those bugs might crash that component but the system itself stays up okay so it's a fault isolation it also enforces modularity okay so you have to come up with a really good interface to the file system even with the kernel okay and it's location transparent because if you have a particular thing that's running on a particular micro kernel and it's being bogged down because too many other things are running then you can just migrate it to another system and the RPC can be transparently forwarded across to another to another node and you'll still get service okay so the question also about what's the priority for the micro kernel in terms of scheduling system services yes they can be scheduled at higher priority as you can imagine there's a whole question about how to get the right trade-off between the system services that might be overusing the file overusing resources you don't want them to prevent anybody else from running and so on so there's a delicate change there in terms of what priorities everything's at and that's a whole other discussion okay so now let's move on now so once we've got a good messaging service and maybe we use RPC to give us transparency of location and transparency of hardware type then we can start talking about building storage where the storage is actually out in the network somewhere okay and this is what we typically call network attached storage and you're all very familiar with this now usually when we started with thinking about network attached storage way back when it was basically on your local network now of course network attached storage can span the cloud and the distance you go is much further but let's leave the distance of how far away our storage is for a conversation in a moment but let's actually talk about network attached storage a little more abstractly here and one of the questions that comes up is once we put storage on the network suddenly we can have multiple clients easily talking to that storage and we can now start worrying about consistency so when one client does a right when does it appear to the other clients okay that's called consistency so when changes appear to everyone they appear in the same serial order another question might be availability which is how do I know that I can get my result for sure because clearly if the the last hop network is down then nothing's available but maybe there's a partition in the network somewhere that happens to cut off part of the storage service but not all of it is my data still available and so we can start talking about availability in terms of our storage now and we can also talk about partition tolerance which is how tolerant is this system to the network being arbitrarily chopped okay so consistency availability partition tolerance okay so again consistency has to do with how do rights appear to other people and in what order availability is can I get my results for sure and partition tolerance is how tolerant is the system to maintaining its semantics even when parts of the system are partitioned and there is a it was originally a conjecture that was that you can only get two out of those three consistency availability partition tolerance at once can never get all three and it was informally called the cap theorem and it came from a brewer okay Eric Brewer who was a professor here for a long time and he was really conjecturing that this was true and then there was a call for a bunch of papers a while back and a bunch of people put in theorems about proving the cap theorem and it turns out that you have to be very careful about what your assumptions are and to find things very carefully and then you can prove the cap theorem okay but there are many different proofs all depending on what your assumptions are so we're not going to go that far in this class but we're going to say loosely speaking that the cap theorem says you can only have two of these three things you can have consistency and availability but not partition tolerance or you can have you know availability and partition tolerance without consistency and etc and let me just give you one example of this so how can I have say availability and partition tolerance but not consistency well that says I put a cash on the client and I cash all the data and it doesn't matter what link I cut in this I can always get my data but I'm not going to see other people's rights and so it's going to be inconsistent and so on okay so with that in mind we can start talking about how do we build the distributed file system so distributed file system here is one in which the client is separated from the server over the network and maybe we do reads across the network and we get our data back across the network and so it's basically some mechanism for transparent files stored on the remote disk to be gotten to users and in principle what we can do is we can mount the remote files into our local file system so for instance slash home slash oxy slash 162 might be mounted on the laptop so that when you go to this with your file browser and you go down these directories and you look it turns out the moment you cross this boundary you're actually talking to a part of your local file system it appears but really those are that's really on the remote file server okay and so the typically in a remote distributed file system case you can often mount file systems directly on the client they look local but your queries go remotely and so how do we do that well here's a picture that I've shown you kind of before of the interiors of an operating system and really it says that somehow in the file system layers we have the ability to have file systems that are both local and remote okay and the way that this is happens is there's special interfaces inside here called VFS that make it possible for us to mount block devices and file system devices that are either local or remote and everything above that layer doesn't know the difference okay and so this virtual file system switch or VFS kind of has this idea that if you have users the client processes above and they do open and you know opens and reads and writes being the POSIX interface and closes they will get the exact same interface regardless of whether they're talking to say an MS-DOS file system a FAT file system or an EXT2 file system the same APIs will be useful and that's basically due to this VFS layer okay so VFS is a virtual abstraction similar to a local file system gives you a virtual version of super blocks inodes etc and it's compatible with a variety of different file systems including remote ones and it basically gives you the same system called interface above that's used for different types of file systems but VFS was originally designed in a UNIX environment so even things like the MS-DOS file system below have to fake out the existence of inodes and super blocks to the VFS layer in order to be compatible with that so that's a little bit strange but basically it's a VFS common file model in Linux basically has a model that file systems have super blocks and inodes and so on there's several different types of objects that are very similar to fast file system and if you know how to produce your system in a way that looks like it has these items then it can be linked in with VFS and pretty much any file system that you can think of has been designed to fit in with VFS okay so with that in mind that means that the client that's using remote file systems thinks that it's just you know going into the kernel with the POSIX open read write close standardly but those actions the file system are getting translated remotely and so let's ask ourselves what does that look like and so a simple idea would be that when we do a read well maybe an open actually opens the TCP channel underneath and establishes the RPC system and now when we do a read that's an RPC that goes to the server and it returns the result which is data or when we do a write it's an RPC that goes to the server and it returns an act which is data and it just looks like a procedure call or you know an extended system call if you want to think of it that way okay so it's a remote disk and no local caching or maybe there's some caching at the server side but in this simple example we're not trying to deal with client caches and the question might be okay the advantage of this is that it's extremely simple server basically provides a completely consistent view of the file system to multiple clients we build this RPC system call layer to basically just call the regular file system calls that are at the server side and we can implement the VFS to basically translate calls across the VFS layer into the right RPCs and we're done the different you know the downside is that yeah this is simple but it's so simple that it doesn't perform well going over the network for pretty much every read and write is going to be painful okay and the server becomes a serious bottleneck when we know that reads are often cacheable and we would like to have some cache that's on the client side in this particular simple view of the world isn't providing that for us so what can we do that's different well let's put some caches in okay so if we put use caching to reduce the network load in practice we're going to use the buffer cache actually at the source and the destination so really what we're saying is let's figure out how to do this in a way that doesn't disable the buffer caches caching of items across the network okay because in this previous kind of hypothetical we were somehow getting no advantage of any buffer cache in our RPC system so let's put it back okay so we put some caches in the system we've got some at the server side and there's a question of what's what are some examples of distributed file systems in use I'm going to show you NFS in just a second so the NFS file system is very common AFS is common you've probably been using that anytime you mount your home directories at Berkeley there are many other types of file servers that are out there and so you know so distributed file systems like this are definitely in common use especially in the local area so so the problems with the simple ideas here are what about failure where the client caches have some data that's not committed at the server and the client fails then that might be a problem so remember we have this delayed write issue even when we don't have network file systems it gets even worse because the latency is longer between the client and the server and we have more opportunity here to lose data okay so let's look at an example here cache consistency is is yet another problem here so if we try to read what happens is it's not in our cache to start with because this is a cold read the read might go across the network to the server it reads something off of the file system into the server's buffer cache and now it's going to return to the data and we'll put it in our buffer cache all right and so now subsequent reads are very fast because they're in the cache okay and now if we write we might write it in the cache here and notice what happens if this client crashes now we haven't updated the server and we certainly haven't updated the client and so that data is just plain gone so not only does the client not know about the data but it's lost for all time so what we really want to do is make sure that when we write it goes through to the server it gets written on disk it's probably in the server's cache as well and now we get an act back and at that point maybe we get an act before we allow our right to go forward and as a result if there's a crash the data's not lost okay but we still have a consistency issue because notice that we're still reading from this cache and client one hasn't seen the updates so the fact that we did get the data to the server is good from a durability standpoint but it certainly isn't helping us make sure that the client the other client sees our updates so this is an inconsistent problem with our data and this is just the beginning of issues that can come into play when we have more than one client talking to the same file service okay so notice also that a read from f1 at the client gives us v1 a read from f1 excuse me at the first client gives us v1 and the second client gives us v2 and so that's another example of our inconsistency so how do we deal with that? Well, let's see if we can deal with failures for a moment what if the server crashes can the client wait until it comes back and just keep making requests well the changes might be in the server's cache but not in the disk are lost yeah we just saw that so that might be okay maybe what we need to do is make sure that every write is always flushed all the way to the server before we go forward so what if there's a shared state across rpcs client opens the file then does a seek server crashes the client now tries to do a read at that point if the server has forgotten where we're seeked to where we have seeked to then we've got a problem okay or what if the client removes a file but the server crashes before an acknowledgement is the file gone so this is just you know many interesting failure problems start coming up so one way to to combat that is what we call a stateless protocol which is a protocol which all the information required to service a request is in the request so a great example of where that might be is rather than having the seek pointer in the server we make sure that every read just says what offset from the file do we want to read from okay all right so that would be an example of a stateless protocol even better would be one where it's idempident which really means that we can keep resending the same client request over and over again and the result will be the same and it won't change the result on the server okay so if the client times out without hearing a reply from the server it just reruns the operation again and it's safe if we've made sure things are idempident so if you remember with HTTP what happens in the simplest case where you go to a server and nothing's happened or you don't see an update everybody knows about getting a refresh well that's an example of of a stateless protocol and in fact they're cookies that are stored on the client that are resubmitted with each request and that's kind of a way of keeping all of the state that we need for our actions to be kept at the client side not at the at the server side and as a result the protocol is stateless so the NFS file system network file system was one of the first really popular network file systems it's been around for 30 years now and it defines an RPC protocol for clients to interact with file server so how to read and write files traverse directories etc it's stateless to simplify the failure cases keeps most operations idempident so even removing a file so if you were you know you you could return so if you try to remove a file and you don't get the act back and you remove it again you get back an advisory error saying the file's missing to basically allow idempident retries okay you don't buffer writes on the server side cash and you reply with an acknowledgement only when modifications are reflected to disk so one of the ways that the original NFS kind of tackled the lost state problem is it made sure that all writes are flushed to disk at the remote side which was unfortunately slowed it down quite a bit but gave you a nice reliability semantics here was the architecture and if you notice we've got VFS right here in the middle which basically says that there's a system called interface which goes through VFS as we just mentioned and that VFS layer may go to all types of local file systems unix and msdos and ntfs and otherwise otherwise if it's an NFS mounted file system it'll go to the NFS client which uses the XDR version of RPC this was one of the first very popular ones and that RPC goes across the network and makes remote procedure calls to the NFS server which now goes into the VFS interface and talks to the remote file system so VFS the original VFS layer was actually part of NFS as was the original XDR RPC was part of NFS from Sun Microsystems so there are three layers there's the file system interface I told you about the VFS layer and the NFS server layer just showed you that the protocol is RPC for file operations on the server reading and searching directories manipulating links etc and it's right through caching which basically means modified data is committed to the server before results or return to the client so you get you lose some of the advantages of caching on the right side because the performance of right can be long and the one thing I haven't told you about though is you need some mechanism for readers to eventually notice changes okay and we need to fix that the servers are stateless so every request provides all the arguments for execution I told you about rather than assuming there's a seek pointer at the other side we do something like read at a particular i number in position not read at a file okay and there's also no need to perform open or closed on the file because the client is no knows something about the format of iNodes and so on at the remote side everything's idempident which means you can retry it and multiple requests at the same time I already said that the failure model is somewhat transparent to the client's system in fact you can even mount NFS in a way that if the server goes down the client just locks up until the server comes back and things just transparently keep running that wasn't great as a transparency because you really want to know that your server is down so that you can do something else and so in fact most people mount NFS now with what's called a soft mount so that if the server goes down your process just fails with a with a read error or a write error rather than just locking until the server comes back so let's talk about consistency here I want to make sure we get some consistency discussion before we end today's discussion so the NFS protocol is a weak consistency protocol the client polls the server periodically to check for changes if the data hasn't been checked in the last three to three 30 seconds it knows it needs to check and then when a file is changed on one client the server is notified but the other clients wait a little bit before they find out so in this scenario we showed you earlier where client two had basically already written a new value to the cash and client one had an old value in its cash what happens eventually is this client will poll and say well is F1 still okay or has it not changed since I last saw its update and at that point the server will say no no no there's a new value and it'll get updated in the cash and now the client gets new data but notice that this is a weak consistency so it means that data takes a little while before all the other clients in the system know that the data is has been updated okay so that kind of leads us to an you know is this something that you could deal with all right what if you're making some changes to some parts of a file and then you try to immediately compile it on the other side what happens is weak consistency okay and that leads us to this question of sequential ordering constraint yes I see on the chat somebody says tears yeah fortunately the time timing is set up these days so that you mostly don't notice this but what sort of cash coherence might we expect from the file system okay so here's an example on the local file system where you have client one two and three where a read say we start out with a in the contents and a read gets a and at some point client one starts writing B and client two is reading during this time frame and at some point maybe it gets a or B and then eventually it writes C and then what is client's three C well it might see parts of B or parts of C and so on and so this is actually a bit confusing right it's sort of what does it mean to see parts of B or C especially if you're talking at block level where a right might span multiple blocks okay so what might we actually want assuming we wanted a distributed system to behave exactly the same as if all of the processes are running on the same system then what you'd want is if a read finishes before a right starts you get the old copy otherwise if a read starts after the right finishes you get the new copy and then otherwise maybe you get the new or the old copy that's kind of a semantics you get under normal circumstances but for NFS if a read starts more than 30 seconds after a right you get the new copy otherwise you get a partial update you can see that this these semantics might not be quite what you want and so this 30 seconds is too long okay and the various versions of NFS as it's gone from version one to two to three to four have introduced new mechanisms faster updates and so on to try to make this more consistent and we'll pick this up next time with the Andrew file system which will be another option on this and then we're going to move into talking about distributed file systems in particular we're going to talk about the cord protocol all right and anyway so for now in conclusion we've talked about how to get something like a reliable byte stream between two processes on different machines over the internet you know allowing us to do read write flush on the byte stream it uses a window based acknowledgement protocol which we talked about and congestion avoidance automatically adapts the sender window to account for congestion in the network and basically this is why TCP is a good neighbor protocol because it automatically adjusts the amount of data that's outstanding flying through the network to try to on the one side fill as many of the queues as possible from destination to get high throughput but on the other hand back off when other people need that end with we talked about remote procedure calls in some detail which is the calling a procedure on a remote machine or a remote domain providing the same interface as a procedure and automatically packing and unpacking arguments without the user having to deal with that that's in the stub and it's there's a special compiler and a domain language IDL that lets you specify what the arguments are and return values are for the various procedures you want to call okay and the other thing about RPC is it adapts to Andeanness and other details at the destination between the destination and source we also talked about what a distributed file system might look like giving you transparent access to files stored on a remote disk we talked about the virtual file system layer which allows you to attach multiple different types of file systems into one system such that the same positive user API still works and we started to see the interesting problems with cache consistency keeping client caches consistent with one another all right and we showed you how the NFS action works which is a polling system we're going to see some better semantics with AFS next time I'm going to leave you now and NFS is still very common now yes it's in common use that was a question on the chat so I'm going to leave you guys and we'll pick this up on Tuesday and we will go to much more distributed file systems on Tuesday you have a great evening and have a good weekend talk to you later now bye