 Hello, everyone. Welcome back. It's been a little while since last time, hopefully Hopefully you haven't all been longing for this for so long that you're now desperate for some more Just before we dive back into where we were first there was a interview with me on the co-recursive podcast about my research about rust about doing live streams And a bunch of other sort of things related to what I do. So if you want to give that a listen, I highly recommend it It was a lot of fun to do. I will also be at RustConf this year later in August So if you are at RustConf and you enjoy these videos and feel free to stop by and talk to me Even if you don't enjoy the videos, you should still feel free to stop by and talk to me We're going to continue this time with the TCP stream we've been doing for the past two streams We might not finish all of TCP because it's a pretty large protocol But I think we'll at least have gone through the interesting design challenges that we had And I want to make sure that we switch up what topics we talk about And so I don't want to do more than three on TCP Because this is not going to be a final usable stack in the end regardless Of course if someone wants to take it further, they should feel free to But I think we're going to switch to another topic after this stream And when it comes to another stream, you may or may not remember that there's a page that is Essentially a way where you can vote for what stream topic we're going to do next I know that the last time a bunch of people voted, some of these options were not here And so I recommend that you log into the site and just go to this URL It can put it in the chat too, give me a second There we go I will also put it in the video description if you're not watching live And here, the idea is that you can vote on what streams you would like to see You can also put them in order And it will basically run a rank choice voting algorithm to figure out what people overall prefer Currently it looks like the next stream we'll do is a second open source contribution stream Which would be really fun, but there are lots of other interesting ideas here That it would be interesting to hear your input on and sort of discuss what it is that you want to see If you have other ideas that aren't listed here, then feel free to reach out to me on Twitter or email or some other way And I'll consider adding them into the list Alright, with that all said, I think it's time to continue with TCP It's been a while since last time, so let's do a little bit of a recap of where we are Both for your sake and for my sake Of sort of where the implementation is at and what we were thinking of doing next So, I'll post a link to the interview too, yes I can I'll put that in the show description as well Here we go Twitch chat can't see YouTube chat if that's supposed to work That is supposed to work, but there is something weird about restream and chat sync I think they're preparing an update, so hopefully that'll fix it for next time But it's something that I sadly can't do much about Right, so let's recap where we were We have this Rust TCP library that we've been building And if you remember, it basically sets up a sort of artificial network link Using Tuntap drivers in Linux If you go back and watch the very first stream, you remember that we sort of used this Linux mechanism To give us a way to essentially write our own networking stack on top of just IP And we've written essentially, if you look here, we have sourcelib Which implements both the stuff needed to handle packets and send packets So this is this packet loop business that just handles all the incoming packets And decides how they affect the underlying TCP streams that we are maintaining So we have a network interface You can create a new one that's going to create a new Tuntap link You can bind to get a TCP listener, just like a standard net TCP listener And you can also accept to get a new TCP stream And when you have a TCP stream, the idea is that you can read and write from that TCP stream So these are very similar to the standard types you would find in standard net Currently, of course, we don't support handling actual data All we do is we open up the connection and then we close the connection again There's no exchange of data, that's not something we've added And so that's going to be the main topic for today We're going to implement the necessary stuff in order to read and write data over TCP Rather than just handling connection, set up and tear down We also don't have a connect method, I think, on interface So currently, all you can do is set up a server Ideally, of course, you should be able to connect to a server that's running elsewhere This would be if you want to make an HTTP connection or something And so adding a connect method would also be good, I think So we have this main.rs, which is what uses our library and what we use to test it And if you recall from last time, it really just creates an interface, binds to a port And then accepts a packet, reads basically no data from it So it asserts that it read zero bytes because our connections are always immediately terminated And so this server, if we try to just run one So if I run the server, there's now a TCP server running on port 8000 on my machine And if I run... Yeah, so my ton interface is the one that's generated by our server when we start it Is running at local port this It's not even true. I guess we'll find out And if I try to connect to it, you'll see that the server says it got a packet Established a connection and then in theory it should just be terminating that connection So when I terminate the input here, say that I'm going to send no more bytes Then the server prints no more data and the connection has been terminated And I can do this again I guess let's do this So this echo is essentially saying send no bytes So dash n means no new line either So this is sending an empty string to the server So the connection should be terminated straight away And you'll see that here indeed the server got another... Or accepted another connection, established that connection Read no bytes from the connection because we didn't send any bytes And then terminated the connection And so in theory we could keep doing this And luckily our server does not crash And the interface is, as I showed you Just something that looks very much like what you would use with standard net So the real challenge today is going to be to make read Actually be able to read data that the other side sent And for us to be able to do stream.write in order to write data back to the client So I should be able to do something like echo foo Actually let's see what happens if we do this So if I run this and I try to echo foo What happens? Yeah so currently it crashes with... We assert the data is empty because we don't handle data And so this assertion is something we're going to have to deal with And similarly a similar assertion would fail if we tried to write data on the server side Because that's also something we haven't added support for yet So that's what's going to be our challenge today Is to implement those features So let's take a look at where that happens So if you remember we sort of split the library into the source lib Which is the thing that runs the main packet loop The thing that receives packets from the network interface It parses out things like the port and whatnot And ultimately it calls into TCP connection So if we switch to the TCP packet You'll remember that we have this connection struct That contains all the various things that are defined in the TCP RFC So remember that we are following RFC 793 Which is the TCP RFC And it defines... where is it? Down here Further, further, further, further, further Closing a connection, where is our diagram? The diagram is further up, isn't it? It defines... where are you? Where is the giant TCP state diagram? Well, so certainly here it defines a bunch of variables So we need to keep for the state This is things like the sequence numbers for data we've sent Related sequence numbers we've received from the other side So that we can acknowledge the sequence numbers Or the packets that they send us But there's also this diagram Which keeps track of all the possible states that the connection can be in So initially you start out in the closed state When you decide to listen for connection You're in the listened state When you then receive a SIN Which is essentially a connection request from someone else You then go to this state, etc And all of these states have various things you need to keep track of Such as the current sequence numbers And so this is why our connection has a state And the state is basically all of these states Map to these states We also keep track of the sequence number space Of the stuff we've sent and stuff we've received And information about the other side And currently we also have this... This is sort of one of the last things that we added in the last stream Which is incoming bytes and unact bytes So incoming is going to be bytes that we have received from the other side But that we haven't returned to our caller To TCP stream read And unact are things that we have sent to the other side But the other side has not yet said that they have received it So the reason we have to keep this around Is in case we need to retransmit it We can't just send bytes out of the wire And then just sort of forget about them Because it could be that that packet gets dropped And this is of course what TCP is all about Is giving you reliable transmissions So if a packet gets dropped you need to resend those bytes So the receiver actually receives them and in the right order And that's what unact is for here And you'll notice there are a bunch of to-dos here That these may or may not be important to us So the send sequence base What we did last time was essentially abstract out All the states that's needed to keep track of how much you've sent And all the states that's necessary to keep track of all the states you've received We might want to add some methods to this To make managing these variables easier But for now I think we can leave it alone Connection has a bunch of methods That basically corresponds to outside events So this would be things like receiving a SIN on the network interface And you'll see accept is the thing that gets called When we receive a packet for a connection we have not yet established So imagine that the network interface is sort of in the listening state On a given port and you receive a packet from a peer that you've never seen before Then accept would be the thing that gets called So that you can establish that connection So accept does the TCP handshake And sets up basically all of the initial state that's needed So write I think... what is write? Write is just if you want to write a byte out Like you want to write out a TCP packet Write takes care of that It sort of sets up all the IP and TCP headers that you need And then sends out that packet So we use this to send out things like resets, acts Any kind of packet we want to send to the other side goes through write On packet is the main event handler So essentially our network interface whenever it receives a packet It's going to call this on packet So it basically turns into an event loop And it's going to call on packet on the appropriate connection So remember there can be many connections open at any given point in time So if we remember back to LibRS LibRS keeps track of this map from Quad Which is a local IP import and a remote IP import to a connection So it is going to use the packet headers to determine which connection the packet is for And then it will call on packet on that connection And this handles all of the TCP state transitions that we've talked about in past streams Let's see Right, so here you see is the assertion that we failed earlier when the other side has sent us data So we are in one of the states where the other side is allowed to send us data But currently we just require that that data is empty So in reality what we should do here is accept that data and then push it into our Or make it available to read calls And also currently we have no mechanism here for dealing with writing out bytes that we want the other end to receive So that's also something we're going to have to add So if we go back to our Lib So on TCP stream we implement read which looks up the appropriate connection So remember we take this lock to make sure that we only have one thing operating on a connection at a given time And then we check whether there's data This code in theory supports there being data for us to read So that's good Yeah, this basically just looks at the queue of incoming bytes So bytes that we have received and then reads out as many bytes as it can And down here we have the this is the condition variable that we added last time So that if there aren't any bytes to read we have a way to block Right, so our code down here What it's going to have to do is stick bytes into the incoming byte array It's a vecdq of u8s I think And then it's also going to have to wake up any readers that are waiting to tell them that there's now data for you to read Our write currently just adds bytes to unact But of course it's also going to have to wake up It's going to have to do something in order to make our packet processing code actually send out that packet Just adding bytes to this is not sufficient And we're going to have to figure out a way to do that Yeah, we're going to have to figure out what to do there too Not sure how to do that Alright, oh and shut down, we'll need shut down too Okay, I think what we're going to do is we're going to start by implementing read Because I think we have most of the stuff that's necessary to read And it'll sort of get us into the flow of things And then we'll implement write later because that also requires dealing with wakeups and retransmissions So let's see where we're at for reads So for reads we already have all the stuff that's necessary to get at the incoming data array All of this is done So if there are bytes to read Then we copy out the bytes that we can read If you remember from last time, the double copy here is because we're using a vecdq So a vecdq is like a ring buffer So the bytes that you can read might be in two separate locations in memory You should go back and watch the previous stream if you don't remember exactly how that worked But this is us reading out as many bytes as we can into the buffer provided by the user And then basically erasing those from the ring buffer so that that space is available for later packets that we received And then we return the bytes that we actually read And if we ended up not reading any bytes, specifically if this is empty, then we block So the thing we really need to support here is this assertion we need to deal with So this is an established connection and data Where does data come from here? So data here is the body of the TCP packet Which is what's going to contain the data that the other side wanted to send us So what do we have to do when this happens? Well, there are a couple of things that have to happen When we receive data, we need to stick it into incoming, right? So that part is easy That is just, what was it called? Incoming.extend Data So that part is easy enough But now we also need to wake up waiting readers So if someone tried to read in the past but went into this condition, then we need to wake them up In addition, we also need to make sure that we act the bytes that we received So remember in TCP, we always let the other side know that we have received bytes that they sent us So this, if here, that is just us saying that if we ever get anything from the other side and we are in the established state We're immediately going to shut down the connection Now in reality, of course, we don't really want to do that anymore We don't want to shut down the connection immediately because we want to be able to accept data from the other side So this business, we don't want to do anymore This business is really going to happen only when the user calls shutdown So shutdown is a command you can call on TCP streams to say, I have no more data to send Think of this as like, when you drop the TCP stream, the other side has to know that you did that But you can also say, I'm done sending you data but you can still send me data And so we're going to have to figure out what to do about this business But here, clearly, we also need to make sure that we acknowledge to the other side that we indeed received that data So currently, what we do is when we receive the packet, we sort of check that it's a valid packet And check which of the bytes we should really adopt In fact, I don't think it's even all of data here Stuff we haven't read So for example, I think I need my drawing program here to show what's going on So let's see Imagine that, so we have, this is our machine, this is the remote machine, and there's a channel between us This is the TCP channel Imagine that in the past, we have received bytes 1, 2 and 3 Those we've received in the past And then we get a packet from the other side that contains the bytes 2, 3 and 4 Now, when we receive this, we have to make sure that we recognize that We have to make sure that we recognize that we don't want to read these bytes again because we've already read them over here Instead, we want to make sure that we only adopt this byte This byte we do actually need to receive and add to our buffer This needs to go here Now that I did something stupid and I meant here So we might actually, in TCP, we might get a data segment where only parts of the segment contains data that we haven't seen before So if we just, currently what we did was dot incoming dot extend all of data But that's not okay, it has to be only the stuff that we haven't read already So let's look at how we might do that Let me see It's been a while since I looked at this code too So the received sequence space is the stuff we received from the other side And let's see So receive next is the, remember that the sequence numbers are naming bytes in the byte stream, right? So this is essentially, this is going to be 1 for the first byte, 2 for the second byte, 3 for the third byte, etc And so we only want to read the bytes from next and onwards So if we go back to the drawing Let's imagine that this is sequence number 703 This is sequence number 704, etc Because remember, sequence numbers start at a random offset Okay, this packet from the other side We're going to be told that the first byte of this is 704 Because 2 was 704 And our next pointer is going to be pointing to here Our next pointer is going to be 706 Because in order for this to be the state of the system We must have previously received all of these And when we receive them, we set next equal to 706 And that is how we can recognize that here when we receive this packet, we only want this byte And so then the question is, how many bytes do we skip off of the packet we receive? Well, this is the sequence number of that packet And so we have to skip next minus sequence number bytes Does that make sense? And my guess is in fact, remember how the TCP spec at the bottom has this really convenient overview of what you have to do on all possible events And so for us, we're going to have the segment arrive state So this is a new packet arrives So what do you do if... It's down here somewhere We already do all of this You can ignore the reset bit for now Ignore the sin bit The ac bit we can ignore I think Now we also need to update the windows Process the segment text So let's go back to here And see what it is we currently do So it's actually pretty important to do these actually in order As we have learned from messing things up in previous streams So let's see what stage we're at here So check the reset bit Oh sorry, even further up Up here So state is listen, this we've already dealt with If the state is sin sent That apparently we just like ignore So sin sent we can only be in if we've done a connect Which is not something we've implemented yet so we can ignore this bit Otherwise, first check the sequence number So check in the sequence number is what we're doing here We should arguably label these to keep track of where we are in the protocol spec So this is implementing all of these checks Second you check the reset bit Which I don't think we currently do anywhere Like we don't handle resets Which is fine Then check the sin bit So checking the sin bit in theory We should do up here somewhere, yeah up here So we basically just don't handle the sin bit at all currently So this business we also don't do because getting this is an error And we just basically don't have any error handling currently Like here it's basically saying send a reset to let the other side know that you weren't expecting that Check the act field though, that we do do down here If the act bit is off, drop the segment and return So no act, great, so we just return If the act bit is on, which is what happens in the remaining bit down here If we are in sin received, say if we're in sin received Then we do this business If we are in the established state Interesting, yeah so this is We're still in the business of checking the act field, the act bit is on because otherwise we returned If we're in the established state or if we're in fin weight one or fin weight two Because they say in addition two, right So all of these three are doing this business So that's this Any segments on the retransmission queue which are thereby entirely acknowledged are removed So here we're going to end up doing some pruning of self-unact Because if the other side sent us acts for bytes that are still in our retransmission buffer Then we can now safely remove them because the other side has said that it's received them So this is going to be relevant for writes Users should receive positive acknowledgement for buffers which have been sent and fully acknowledged That's fine, this is basically telling the when people did a right you can now say that it's been written So this would be relevant for things like flush So like if no if unact empty And waiting flush notify So if the caller has called flush Then at this point if there's nothing in unact then we can say that the flush is completed If the act is a duplicate it can be ignored Because of this I don't think is something we currently do So I think here we're going to say if sec n is less than self-send una Oh this is probably not actually a less than is it I feel like this check is also broken because of wrapping I think what we want here is actually wrapping less than Right so if this then we can ignore the act Oh interesting Yeah so this is basically more error handling that we can pretty much ignore Let's leave that out for now This is related to updating the send window So if you remember from the earlier streams One thing that TCP does is it limits how many bytes you can have outstanding Like if I've sent a thousand bytes to the other side And the other side hasn't said that it's received them yet There's a limit to how many more bytes I can send Like I'm not allowed to have more than a certain number of bytes in transit Of course when the other side acknowledges something Then now my window can grow again as in I'm allowed to send more bytes And so this is also something we're going to have to deal with for writes Update window This is the groove box dark hard color scheme Okay that's fine And then if we're in fin weight one or fin weight two So that's the check down here If we're in fin weight one If our fin is acknowledged Then our fin has been act So enter state two and continue processing If we're in fin weight two Interesting for the processing for established state So we don't actually want to adopt the data here quite yet Right so this is still handling the act Fin weight If the retransmission queue is empty The user's close can be acknowledged Okay so here So that's also not something we really need to worry about Close weight we're supposed to do the same as for established So this should also include state Do we even have close weight yet? No we do not great So there's a bunch of states that we don't currently deal with I'm just going to ignore those for now We ignore the urgent bit So now we get to process the segment text And the thing after that is checking the fin bit Right so here we check the fin bit that's the next bit So here what we want to do is process the segment text And so here we're going to bring back this if Right because we only handle the segment text if we're in one of these states The drawing software is called MyPaint Right so what do we do? Well Right so it's just saying that when we're in the state We're allowed to receive data Because the connection has actually been established And we can adopt Text from the packet into the buffers Either we can put a limit on how many bytes we allow to buffer Or we can just adopt all of them Like read all of the bytes out of the packet We're just going to ignore push Once the TCP takes responsibility for the data It advances receive next over the data accepted And adjust the receive window Right so Let's bring that in here Because it'll be handy to have the docs here for our own reference Right so here we need to send an acknowledgement of this form Is the requirement And notice this so this is something that keeps coming up in TCP That you don't necessarily want to just like send an act immediately Because if you're about to send some data You might as well include the acts Which are just changes to the header With any data that you're already about to send In our case though That's not actually something we're going to do So this is already I think what write does Let's double check that So our write function Sets TCP sequence number is set to send next An acknowledgement number is sent to receive next Right so send next receive next And set the act flag So all of that is already handled by write So write will take care of that for us So if we just call self dot write Nick this That will send that acknowledgement That will automatically set the right acknowledgement We will however have to do this business Which is update receive dot next to match the data that we just received And of course we're going to have to actually adopt the appropriate bytes So how do we do that? Well receive dot next is the next byte we're supposed to receive So recall that this was going to be What did we say? Self dot where is Receive dot next Minus the sequence number So minus the start of the packet If I'm reading that right I think that's right Maybe as you say It might be necessary We also still have to wake up the waiting readers Now from memory we already set up a way to do this So on packet returns available To indicate what is now available So if we go back to live Where's our call to on packet Right so this already has the required mechanisms for waking things up So really all we have to do is make sure that the returned a Availability contains the read flag And then the packet loop, the main event loop Is going to make sure to notify the necessary readers So here self dot availability And self dot availability Already checks incoming So it will already set the right thing So we actually don't need to do anything here As long as we extend to this it is now no longer empty So when we get to this return This availability is going to contain the read flag And then the packet loop is going to wake up any waiting readers So in theory this should now already let us do that And of course now that we have read those bytes We now set Let's see self dot receive dot next To be equal to the sequence number Plus the length of the data So it now includes any byte The receive next is now bumped by how many bytes we received And so here there are a couple of ways to do this We could either just add the number of bytes that we read Or we can just set it to the appropriate value And we're going to do this by saying Second plus Oh question Oh keyboard this is the filco majesto touch ninja So this is going to be self dot This is going to be second Wrapping add The length of the data And wrapping add Keep in mind that here if it includes So remember that fin and sin are both extra bytes So if the packet includes a fin byte Then we need to add one So if tcph dot fin Then one else zero I expected U32 found U size Sounds not right Let's do a check How long have you been specializing in networking? You know so much about it I don't think I've really been specializing in networking But it does help that I've written a tcp stack before Although a simplified one like this one It really just comes down to Learning how the protocol works The size for values of type U8 cannot be known At compile time This has to be a reference 341 As U32 Great So in theory we can now receive data correctly We do however want to make sure that the connection Is shut down once the other side doesn't have any more stuff to send Like we basically want to make sure that the other side knows that we're done So the question is how do we do that? Well, what does write do? So does write set the fin flag? That's a good question To terminate a connection do you have to keep sending the fin flag? I don't think so I think what we actually want to do here is We still want up here to So the moment the connection has been established We want to set the fin flag And write out the fact that we've set the fin flag Although do we want to do this? Probably not This is something we're going to have to deal with later So this to do that we need to make sure that we Resend the fin if the other side doesn't receive it It's definitely something we're going to have to deal with But not something we quite need to deal with yet But these two lines are essentially We're going to send a shutdown The moment the connection has been established Because we don't support write yet So we might as well do this We don't support write yet So immediately send and a file Of course ultimately this will not be here So we'll do this But in theory we should now be able to receive data I think I guess let's find out Nope If I run this Actually no we also need to modify main So we're going to do Buff is going to be something like this If n is 0 Then we break Otherwise We print out the data that we got In fact we could print this as a string If we really wanted to Technically it's binary data But right so We read into the buffer If we got no bytes then that means the other side is hung up Otherwise we print out the bytes that we got Then we try to read again So let's try that Yes I did do a master's in networking Although that was in wireless networking So it wasn't quite this If you end up dealing a lot with networks TCP is so ingrained in all of the networking stuff That it just becomes sort of second nature To know at least at a high level how it works Like how the various bits and transitions work People study TCP a lot for things like congestion control And how it behaves in various settings And that I know a lot less about But I do know sort of how mechanically the protocol works Okay so let's try now To echo foo to our server It crashed The question is why did it crash Slice index starts at 1 but ends at 0 Oh I know why this is So remember this business So remember how the sin flag is considered a byte In and of itself So imagine that we received the very first packet From the other side That packet has say the contents foo But it also has a sin flag set So at that point self receive next Is going to be 0 It's going to be the first sequence number Which is prior to the sin And the sequence number is going to be That same value Although that shouldn't actually matter Slice index starts at 1 Where is this? So this is at on packet Oh that's unhelpful Why do I not get the line number in on packet Hmm Reading data From this Good old print debugging Received our next sequence number And data See what we get when we do this So this crashed and it says Reading data at there Okay so it got an empty sequence of bytes And received next Is 1 Oh interesting Interesting So this would be the sequence number of the packet Is this Ah don't do that The sequence number of the packet is Um Is this The sequence The next byte we're going to receive is this one But if this byte Is the The sin bit Then this is telling us read one byte from this Like skip the first byte in data But there is no first byte in data Because that first byte is the sin bit Which isn't a part of data So how are we going to deal with that Well I think what we're going to do is If the TCP header has sin set I guess we're going to do Unread Data at And that's going to be equal to This We're going to decrease it by one And now we can do this Let's see how that works Still crashed, what does it crash with now? Same thing, huh? Hmm So why is this misbehaving Let's try that again So we got Reading data from index one Oh it's the fin, it's not the sin That's throwing us off The sin, actually we don't even need to deal with The specialty because we always read to the end of the packet If there's a sin there We'll just end the Unread data at will start at zero And that is just correct What we need to deal with is the case where there's a fin Because that's saying you should read one byte for data But there isn't actually a byte in data Um So instead Ooh How do we want to do this? This is only a problem If the fin byte is the first byte As in the fin byte is the only byte If the fin byte is at the end Here, let's draw this again It's gonna be helpful Um Let's go over here Alright, so Um How do I explain this? There's a data stream Which has some bytes Let's make them not very Interesting bytes One, two, three, four And then logically in TCP There's also this like virtual zero byte Which is the sin And there's a virtual last byte Which is the fin Right? Um Now imagine that we receive A packet such that This is the sequence number of the packet And also next This is also received.next So at that point, let's say that that packet Spans these bytes Okay, so we receive this packet And what we're gonna end up computing is Um That Next minus C is Zero So we're gonna read zero Dot dot Now data starts here Data is gonna be this Because that first byte is a virtual byte But because we are starting from zero Starting from zero is fine Because that's still gonna get us all of the bytes So the sin is not actually a problem Even if we don't do any special casing of the sin bit The fin bit is also not normally a problem Because Let's imagine now that we get another packet Over here That contains these bytes So it contains three and four and the fin bit is set And let's say that this is Next and also seek Okay, so at this point We would do the same thing We would say next minus seek Which is zero So we're gonna do zero dot dot Zero dot dot is gonna be From zero of data To the end of data Because data only spans these Because fin is virtual So that's still gonna give us the right bytes Where this is a problem You might have spotted this already Is if this Is seek And next So data is actually empty Data is entirely empty Because the fin byte is virtual So what are we gonna do Well, we're gonna do Sorry This would be if next is here It's the actual problem So next points here Because we've already received the fin byte So imagine that this is Like the other side resending the fin Right So the sequence number of the packet Is the start of the fin And our next is beyond the fin Because we've already received the fin in the past Well, if this is the case We're gonna compute Next minus seek And that's gonna be equal to one And so we're gonna do one dot dot But data starts at zero So there is no like One is gonna point to here Which is outside the bounds Of the empty data array And so this will only happen If we see a retransmission of fin Where fin is at the start of the packet So With that in mind What we really want to do here is If Unray data at Is Want Actually This might be a problem For even if seek is here The problem is if the next is at the end Right So I think what we actually want here is If Unread data at Is More than Can't actually use more than No, we can Is Greater than or equal to data dot len Is the only case in which it's a problem So here Sort of as a note to ourselves I think we want to assert That unread data at Is data dot len Because this can only be the case If Must be that We must have Received a retransmitted fin That we have already seen Next points to beyond the fin But the fin is not in data So in this case Unread data at should be set to zero Let's try that It's still crashed Where did it crash now? It crashed at this assertion Yeah So unread data at Is where we should start reading And that should be after the fin And the fin is located Logically at the end of data Which is Data dot len Is the start of the fin Plus one is the end of the fin Which is where we would start reading So this should be true Now what? Oh, sorry, greater than Is the problem They could both be zero Which is fine Like in the case of sin for example So that's fine Ooh Let's see No act Got connection No more data Okay, so clearly Something weird happened Let's try to We'll bring back Our Our friend TCP dump Wire I don't remember what command we used T shark Alright So we're gonna start the server And start T shark And we're gonna do this And see what happens Let's see So First thing that happened is The connection attempt to the server The server responds So the connection attempt is a sin The server responds with an act for the sin And a sin So it acts one The other side acts It then sends Four bytes of data We Respond saying we're done sending data So this is the point where We immediately send that we're not gonna send any more data They send us We're done after Five bytes So this is Sin plus four bytes of data Plus the fin And the act is two because they acknowledge our fin Then we act Then we act their data Then we act their fin The real question is So this does imply that We're receiving all of their data So the question becomes Why does it say that we read zero bytes of data? That doesn't seem right So let's here say Reading from Of And see where those bytes go Reading from four Well That doesn't seem right So reading from four implies that We're just reading from the end Which is not what we want to do We want to read from the start So this four seems unfortunate Because this should really Whoa That just happened In theory this should be zero Instead it's saying How many bytes we should read Which implies that We have for some reason already Updated receive next Is that true though We do that there Why do we do that there Does the spec say to do that there Where is this This is After checking Up here somewhere Not the reset bit But probably first check the sequence number That's fine I don't think that line should be there Check the act field None of the sets to update receive next Because receive next Shouldn't be updated until here I don't think that's true Specifically I think this should Only happen Ooh When should this happen I think this should only happen After we have Received that data Yeah this line is definitely not right Oh I guess When we receive the sin We do want to say that we should skip the sin bit Fourth check the sin bit I don't think that's true I think this happens further down I think that It is only At this point Where we actually Decide To read some data That we're going to add to it Because now receive next Is going to be equal to Oh yeah It's this business And also of course If Seek Although there is a question here of When we get the initial seek From the server we need to make sure that we set Next to be One I'm thinking of when the connection is first Set up then we get a packet that Contains the seek At that point our next should be One Yeah this is not going to be quite right We're going to have to deal with that here too So the initial Seek packet I see If tcph.sin Then Self.receive.next Should be equal to The sequin number Wrapping at One Is So this is Got Sin part of Initial handshake So here we want to assert the data is empty Because it should be The sin should contain no data For memory I don't think that's necessary Check the sin bit Backbit is off See this line makes me think That you can't send bytes First packet of the handshake Which is unfortunate Right So let's try that Although now Now this is going to be wrong because now Receive.next is going to be Oh I see in the subsequent Segment in the next packet Receive.next is going to be One right immediately after the The sin and Seek.n is going to be One The first data byte So that should still be right And this Oh This should be sin And we should never get Here with sin set So this should still be right Let's try that Search and fail left equals right Destination and Source slices have different lengths Wait where did this happen? TCP stream read Well that's Progress But why is it not giving me the Line number that's kind of annoying So this is happening In the implementation of read So that suggests that it actually Got some bytes to read so that's good It would be useful to know what line It was on though But apparently it doesn't want to do that So it's one of these that are failing Oh yeah this is definitely wrong So this code is where we are Reading We're reading bytes out of Incoming and into buff Right And We're going to copy Yeah that's definitely not right And we want to copy Stuff into buff And we're going to have to copy from Remember Let me draw this again It's useful for reference So we're using a vecdq Which is a ring buffer And a ring buffer Is essentially just A list like this A vector It has a Head pointer And a tail pointer And We have stored Like so When you read from it You erase the old pointer And you stick it to where you're now reading from When you write to it You write at the tail And then you get rid of the old tail And you update the tail pointer Of course what can happen When you have a ring buffer like this Is You can have The head And the tail Be here So now you have byte one here and byte two here And so if we're now going to copy We can only copy from contiguous memory So we actually need to do two copies We need to do one copy From here to here And we need to do one copy from here to here If we're copying Into buff Which is here We want to make sure that this bit Goes to Here And we want to make sure that It goes to there And we want to make sure that This bit Goes to Here Currently though that's not really what the code does So currently The code decides How many bytes is going to read out from the head So the head is the From the head to the end of the List Of the ring buffer Like end as in Right most part of memory And that's going to be Either as much data as is in head Or as much space as is in buff So imagine that the ring buffer has like a thousand bytes in it But buff is only 20 bytes long Then we can't copy a thousand bytes into buff It has to be the min of the two And so buff is going to Copy from slice As many bytes as we Allow it to read And then This is where Right And so the question now becomes How much should we read from the From the tail Oh does it require these to be equal That's silly Okay so This is going to be buff Dot dot hread Right so we're going to copy Hread bytes From head into Buff And then we're going to see how many bytes are left So this is The length of buff minus the number of bytes we Read from head, the minimum of that Which is how many bytes we have left to read Into buff And how many bytes we can read into buff And then we're going to copy Oops nope And this is going to be I guess hread To hread plus hread Is going to copy stuff from the tail So let's see if that works better Ooh Read four bytes of data Got bytes Okay so our main program now actually got some bytes Out That's good It doesn't look like it terminated the connection though So remember Back when this used to work Where's that Ah Much further up apparently Oh I can't Oh because it's in this buffer It used to say no more data Right so the server Still needs to realize that the connection went away Which it's not currently doing In fact we can do this here Like this no more data is no longer printed Even though the client Clearly went away Like it terminated So this means that we're now missing the signal Of data to get Which seems unfortunate Right there's no No more data here It does get the four bytes But this suggests that the second time it tries to read It doesn't get to hear This suggests that Is receive close is not returning true So why Is that Well in theory Actually let's look At what the packets we get are Let's pull up Our trusty t-shark again Run this So It Sends to us Fin and we act that fin Right so this suggests That Over here we are probably Entering this So now the question becomes Why Is the Reader Not Like why Does it block So my guess is that it ends up down here In the second call Although we could figure that out right So here Trying to read Connection has gone away And this is Connection still active Connection still active Oh Why does this say That's weird So it says Read some amount of data Print this Oh that's because I'm stupid Right we want to keep reading We don't want to keep accepting Let's try that Great connection is still active Red bytes Don't know why prints got connection again Oh That's also because I'm stupid Let's try that One more time Run this Print this And now we'll see we got a connection We're trying to read connection is still active We're at four bodies of data And this of course is foo and a new line Then it tries to read again The other side hung up We read zero bytes of data Which indicates end of file And then we recognize there's no more data Great, and the other side seems to be Totally happy with that So with this I think we actually have Reads Clearly it works to just read We can try this if we want to be a little bit fancier Here's what we're going to do This is going to get real fancy Let's have this Print out What it got from the other side So this is going to be Stir from UTF-8 We're going to unwrap Because that's fine So this is just interpreting the bytes We get as a string I kind of want These ignoring weird packet things To go away I also want These to go away Ignoring weird packet Let's try this Let's clear out the screen a little bit Now what we're going to do is connect Great, so we're connected We got zero bytes I do hello, enter God packet Reading, hello Great, so it actually got The string that we sent Now what happens if we try to send another string To Zubar Ooh Zubar And then I'm going to terminate the input So control D sends in a file And indeed it says no more data And the connection has been terminated So reads now work even across multiple packets Yay Excellent Of course here there's no packet loss So it's kind of hard to It's hard for us to check whether all the logic is right But it does mean that we can actually now receive Data that's sent over TCP In fact, here let's try something Real cool So curl sends HTTP requests Right, like web requests So let's see what it sends Let's clear the screen And let's see what curl does Uh Well it does nothing could I type the wrong IP address But let's see Boom Read 80 bytes of data Get slash HTTP, host, user agent Accept no more data And curl says empty reply Great, so this now actually works with real TCP clients Right, this is the kernel Sending like real HTTP requests To our server We can't actually respond with anything But at least we got the data that we wanted Sweet So we now have working reads Let's go ahead and Commit this Implement reads So now it's time for writes Because you also have this notion of Retransmissions Because we don't really emulate a faulty link We might have an easier time Than you otherwise would And there will almost definitely be bugs In what we implement But we at least want the basic thing to work I think what we're going to do Is first implement Do I want to implement shutdown first Hmm No Yes Maybe Yes So remember how We have this ugly little Hack up here This can probably go away Kill that So we have this ugly little Hack where we immediately send Essentially a shutdown We send a fin byte To the client immediately Because we know that we don't support writes Well let's do a little bit better than that Let's make it so that In main We can do Stream.Shutdown Net Ooh That's a good question Shutdown Net Shutdown Write So instead of Us doing this internally in the library We're now going to have This happen externally Like you should be able to shut down The right side of your pipe So basically saying to the client I'm not going to send you any bytes This means that this is going to go away And this has to be something that the client Can actually trigger Do you implement time wait Hmm Depends how you define implement Time wait Is here Unclear that it does the right thing We don't have any timers yet So I'm going to go with no But the state is represented Right so this business Which we currently do immediately The moment any connection is established We're instead going to have The client be able to Issue this command Basically So how are we going to do that Well fin is a little weird It's not an actual byte right We can't stick it in an act Because it's not a byte But it is something that we sort of Have to retransmit Now If we go back to our state Diagram Here somewhere Should have just kept it open It's pretty handy to have Where is it There we go So close is the Operation that a client can make When it says it wants to send no more data Right so that's when we send our fin And so what we do know Is that if we're ever in a state Where If we're ever in a state Like fin wait one Fin wait two Or closing although Can we even be in closing Yes it is possible for us to be In closing I don't know we I don't think we implement closing currently It's probably it's own kind of broken But certainly if we're in Ooh that will actually be possible Now Which we're going to have to be careful about But if you're in fin wait one or fin wait two Then you know that you've already That the client has requested That you close the connection So the question really becomes Down here in shutdown What do we do Well certainly we're going to have to Get access to Down here We're going to have to sort of Open up the connection right Get the lock for the connection And then we're going to have to do something On that connection Right Like C dot finish Something How do you inline compiler complaints Um Neo-ving Language server Um which gives you this Or you can just use Ale which is also fantastic Um Yeah so the question becomes What do we really want to do here Like how do we want the client to signal That they want to shut down Uh well We could have a close function right So it's called close in the spec We could just say that there's a close function And then What we want to do with close we can do In here Um so the question because What do we want to do on close Well Uh We certainly want To indicate that the Connection should now be closed We might even want to send another packet Um So here let's add in Right um Closed Right so the This closed is really the fin byte Which is this virtual byte at the end Of the send stream right Uh so down here Uh I guess on Here On connection We're gonna have What do we have two of these Close Um what is close going to do Close Um close is going to Is certainly going to say Self.close equals true Right Uh and in fact We can again look at the spec See what it tells us to do So when you get a close Uh Nope too far down Segment arrived, segment arrived Status call, abort call Close call If you're in the close wait state Uh actually there are more here Uh if you're closed Like if you're trying to close the connection that doesn't exist That can't happen to us right Cause you have a tcp stream Um item And so you can't like You wouldn't be able to call shutdown in the first place Uh if you're in listen Uh We probably want to deal with this at some point But let's ignore it for now Uh That's interesting So sin sent This is if you try to shut down a connection That has not yet been established So in that case you can just like Ignore the fact that you sent a sin You don't need to send anything And when the other side replies It'll just get an error Um Sin received Right so here the question becomes What do we do Well We're going to go through the established state Which is you queue The fin as if it were a normal byte Right you queue it as if it were at the end Of the send queue Um And and only when Only essentially only when we're allowed To send more bytes And we've reached the end Where the fin would be Then do we send a fin segment Right so um Let's see here so remember how there's a window To transmit Uh let's go down Here somewhere Um so imagine That these are the bytes we're trying to send One Two And then logically we also want to send Here right so when Close is called we're going to add a fin To the end here Um but imagine that the other Side only lets us send One byte at a time or Two bytes at a time So we send these two So even the close has been called we can't send The fin yet because we're only allowed to have This is the window right We're only allowed to have Two bytes outstanding Now when the other side acts This byte for example Let's say that this byte is act Well at that point our window Extends to These two So at this point we can We can now ignore this has already been received Right And we can now send Two and three but we still can't send the fin Then at some point it's going to Act Two and now the window Is three and fin And two can be Ignored And so now we can send Three although we've already sent Three because it was already in the previous window But only at this point are we allowed to send Because it's within the window So even though close was called much Earlier we can't send the fin Until we're allowed to Is that The reason you're trying to make tcp Module development for rust even though Already have it on cc++ to be More secure no so The reason we're implementing tcp and rust is just Because it's an interesting use case In writing low level code It's Not actually Like The intention is not For this to be like a production ready tcp Stack it is more because it's interesting To develop But hopefully this makes sense right So we're only allowed to keep a certain amount Of bytes in transit To the other side so even though we want To send a fin we might not be able to do So yet this is also why We have this The I don't have it open but the Byte list is like Bites like these Right so when We're only allowed to send one and two but The client like tcp stream has already Written three we need to keep This around somewhere right Until the time when The window extends to here And we're allowed to send Right and so that is Unact and the fin is sort of Logically in that same unact But it's not a real byte So The question becomes What do we want to do on close well We sort of want to send stuff if we Can Of course the problem is this What we've introduced here Is this notion of time that we Didn't really have before we might Not be allowed to send anything yet So we need there to be a timer running That like decides that hey Now is probably a good time To send something Like We can't just do self Right here right I would not be okay So instead what are we going to do well We Can just we can imagine That there's just like always a timer running And so we're going to set self Close to true and then we're going to Wake that sort of Timer up to let it know that there are Bites to write out that would be One way to deal with this How do we want to do that though It's a good question So at this point we have to decide What our timer is going to look like So with this notion of time It sort of means that there always Has to be There has to be something that deals With time and that Schedules when we're going to send Stuff or we send stuff But where do we want that Timer to be The problem we have is that This timer also needs To Access the connection States we need to take the lock on the connection So you can't like Do or write while the timer is Also writing stuff out as an Example The easiest way to do this Is probably to have The Packet loop So we have this like on Packet loop So the packet loop Just Spins trying to receive things from the Nick Ideally So this is a to do we have from earlier Is we want this Received to have a time out So that we can like Block on receiving a packet Or a timer expiring From memory when we looked at this And actually a way to set a time out On receive here so this Is going to block forever Which means we can't do the Timer management in the packet loop We actually need to have a separate thread that manages timers So Let's do that Where Is it We What does interface new do Interface new Spons a thread What does the packet loop And I think we're going to need A timer Handle I need a Thread to handle here for Interface This is going to be the timer Handle So Ooh Actually that's going to be a problem That's not going to work The timer loop needs to also have access to the Interface because it needs to be able to send packets Huh I don't know how we're doing this So let me try to explain What the problem is When we open a ton top interface We only get one handle to it And that is the handle we need to use Whenever we receive packets But it's also the handle we need to use Whenever we write packets out to the network And The Timer thread Also needs to have this Because the timer thread could Totally decide that it needs to send a packet But the The packet loop Is the thing that owns that interface Right And in fact it's blocking on it The main loop is going to use Nick dot receive And it's going to be blocking on receive Is using that interface And we can't have the timer The timer thread also try to Use this at the same time At least not safely Probably So the question is how Do we set a Timeout for that Well I guess we're going to look at ton top See what we get So there's receive Here Let's look at what receive does Receive So receive Just as a read on the underlying File descriptor And what is the underlying file descriptor In fact Real question is can I get the underlying File descriptor Yes indeed I can So a file descriptor If you're not well versed in Linux And Unix Basically Everything that you can interact with Like files, network sockets Pipes Have this notion of a file descriptor So this is a number that the program keeps Track of that Is mapped to the file The network socket or what not In the kernel So whenever you do a read or a write You're really just doing a read You send a read system call to the kernel You tell it I want to read from this file descriptor It translates that to oh that is this TCP socket A ton top interface apparently Is the same thing It's underlying it's just a raw File descriptor And we can in fact Extract that raw file descriptor And if we want, if we look at Here let's look at Look at the rust Source code because that's fun Ooh Why is my internet slow It's somewhat disturbing Really? Why? Okay that was weird Alright Let's look here Lib... No it should be lib standard Sys Unix Net So these are operations This is sort of how Rust implements Various operations on Talk is like TCP stream for example So there's a There should be a read Set timeout So there's a set timeout function That Basically when you call things like receive timeout On In fact I can show you Let's look at TCP stream So TCP stream has A Where is it? You can do set read timeout And set write timeout on TCP streams This is the same for any kind of file operation And you'll see that Self.zero What is Zero here TCP stream is really a Netimple No, well it's going to be the same for TCP stream It's a netimp which is The implementation Of TCP for this platform And ultimately this ends up calling The set timeout function For the given operating system So on Unix This is basically what that does It Parses the duration Into an appropriate Like Structure that the kernel Expects And then it does a Unix system call to set Options, set a timeout On that Socket. And so the question is How can we do this a similar operation Like what is the system call we need to make Giving the file descriptor Of the Tuntap interface So that we end up setting A timeout for it Well That is a good question Let's go here And do Man read No, man to read Let's call Right, because remember Looking at the source For this We saw that receive Is really just calling Read on the underlying file descriptor Right Where was the FD, oh FD is in fact A file That's even better Is there a way for me to get at the file Right, so if FD Is actually just a file All we need to do is figure out How rust Sets a timeout when it does reads On files So instead of looking at TCP stream We'll look at file I don't know if there's a timeout For a file actually This might be the problem we run into Yeah This probably doesn't have a read timeout Let's see what read here does That is a Very good question On whether it's possible to set A timeout For read On Linux It should be Probably an iOctol To be honest But documentation for those are terrible Let's Set Timeout read I guess I could select over it I was hoping that was a nicer way That's awkward Okay that's fine We don't actually need this to be a high performance version So what we're going to do is use Nix Nix is a great A great crate that wraps Various low level kernel features In particular What we want to use here is The I guess it would be under sys probably Select This one So Select is a System call on linux That Let's you Essentially It Notifies you when A given file descriptor is ready To do an operation on It does not actually do the operation But what you can do is give in In this case A file descriptor that you want To read from And it will return When you would be able To read from it without blocking But it also lets you give a timeout So this is saying Let me know when I can read from this Or when this timeout has expired I was sort of hoping there was a nicer way To wrap this but it doesn't really look like it So So That is what we will end up doing So cargo tumble Let's add in here nix 013 And here We're gonna have to Issue this call We want to read from Nick But we want to not Block But we want to Make sure that We'll wake up when The next timer Has to be triggered Yeah, it's very similar to Selecting over channels Where one of the channels is a timer channel There are many ways to do this So epol is the other way That you often use on Linux On macOS I think you would end up using KQs Select is Not what you want to use in very Sensitive cases because it has some Issues It also has the Downside that we're now going to end up doing Two system calls, right? We do one for the Select and then we have to do the receive But we're just going to Ignore that for now So what to select take In File descriptors The highest file descriptor Set This is awful But I think we may have to Actually, is there a nicer one That I can easily use Probably not Ooh, maybe Great, let's just use pole instead Great Even better Because we don't actually care which of the So when you're implementing like Heavy duty network IO For example, you probably really care Whether you use select or pole or epol Or KQs or any of these mechanisms In our case, we don't actually care Because it's not super Performance critical But we do We just want whatever has the easiest Interface to use And so pole is going to take All the pole fd's What just That's fine, don't care about that Timeout is the number of milliseconds That pole should block It's another good question Whether milliseconds is good enough But we'll see So pole fd Alright How do I make one of these Great So we're going to say pfd Is going to be Nick's pole Pole fd new Right, and we're going to give We're going to need to give a raw fd As Raw fd And we also Need to give it event flags Which is probably just Yeah, we want Is there data to read Right, so we want Nick's Pole Event flags Pull in That gives us a pole fd Now that we have a pole fd We can pass that to Pole So this gives us a pole fd And now we want to call that on I guess This And we want to give a timeout And what is that timeout going to be Well, in theory it will be Whatever the next timer is I think what we're going to do here Is we're going to do something a little bit stupid Instead of trying to cleverly manage timers There are lots of ways to like Manage a set of timers and wake up When the next one is due to expire But instead what we're going to do here It's a little stupid But we're going to do the stupid thing Is we're just going to wake up every We're going to wake up every millisecond And then decide what to do This could totally be more clever But we're just not going to be clever And as you see The call to pole will block Either until the file descriptor becomes ready So this would be that we're allowed to read Or there's a single signal Handler is being called We can ignore that for now Or until the timeout expires And of course there's also You can give a timeout of 0 And the timeout of 0 Makes it so that it's going to return immediately And basically tell you Whether or not you would be able to read Okay so we're going to Wait for only one millisecond It's going to return a Nyx result It's kind of awkward And I think Remember that This function is supposed to return an IO result What pole gives us back is Not an IO result, it's a Nyx result And I don't think this Implements into IO error So what we're going to do Is do MapError E We're going to do Okay what can I do with an Erno Can I turn it Into an IO error That's all I really want to know Implement from Erno For IO error Great! So in theory I should be able to do E dot What was it called As Erno Unwrap We don't expect there to be any other type of Nyx errors And then We're going to question mark So as Erno unwrap is going to give us A Nyx Erno One of these That implements into IO error And so therefore we can use question mark So that I think should work fine Great And now Depending On what the Return value of this is So pull The function will return As soon as any event occurs Alright So the question is what is this return value Should really be documented here Maybe we should submit a pull request But for now What does pull return Of the field R events is an output parameter So What do we get back Oh it's going to modify the pull fd I see So this is going to be Really This And we're going to give in This It's going to be some n there And I don't exactly know what the n is The n is Okay That's fine So what are the return Values On success a positive number is returned This is the number of structures which have non zero R events fields Value of zero indicates The call timed out and no file descriptors were ready Great So Here If n is zero Time out Or I guess timed out And we're going with timers I guess we should Assert That n is not equal to minus one That shouldn't be possible Minus one would be an error Which should already be handled by Nix If n is zero Something timed out Currently we're just going to have that Do a continue Which means that if we get down here That means that The file descriptor is now ready And so therefore we can do this Right So poll fd Essentially what this means is When the poll returns If n is one That means that this poll fd changed Which basically Implies that this event happened We could double check that by checking The events that happened For the file descriptor Which in theory should return Things saying that yeah, you can now read But here because we know that The only way that we can Get to here is if it is now Pollable we don't really need to Is readable Like there's data available That basically means that we don't have To make this check at all Because there's only one possible way You could get here with n I guess we could Assert each That poll in the set But we're just going to ignore it Great So If I do like Timers If I do this That's not what I meant to do If I do this, what happened Why is it fetching the Oh because nix You don't have to Set non-blocking on the fd No you don't so this is one of the reasons That poll is nice Is because you just tell it I want to read from this file descriptor And let me know if it's not Available for reading So keep in mind that poll Is not I'm polling this file descriptor It's not like do a read It is tell me when I can Read Which is a little different Cannot find the value of pdf That is because it is pfd Ah pfd Somehow pfd is a lot harder to type than pdf Ah And of course we need to assume That we're on unix So that we can use As raw fd Um This So this really just set Um Close equals true right Close equals true And in theory it should also Ammon some timers To do But we don't actually need to deal with that At all currently Because the timers should just take care of that Uh which means we don't need This function anymore And Uh shutdown can return Okay immediately Uh and Now this needs to include Closed is false Yeah so now You see there are just like timers firing So often Which implies that our thing is actually working Um If we want to Make sure that the other stuff is still working We can just not do the Not print timers Uh and then check that This command in fact still works Which it does Right so this is implying that this is all working the way it is Because the We now have a sort of an entry point And Um And so what are we actually going to do Well there are a couple of things we could do Um Really what we want to do here is When a timer expires then like Do what the timer tells us Should happen uh when it expired Instead what we're going to do Is something a little bit stupid Um And you're probably going to dislike me for doing this But We're going to do it anyway We are going to do Where's my Yeah here Bear with me because you're all going to hate me for this So We're just going to loop over all of the connections And we're going to call this on tick method And on tick is going to get to decide Basically whether it wants To do something Um and so the idea here is that the The um The connections are Going to keep track of their own Like when stuff has to happen They can just check that in on tick Um and they only really need To keep track internally of When they want something To happen we'll see how we end up using That um So if we now go to tcp We now we have on packet And now we will also have I guess here we Can now finally give in Uh this I guess where is my on packet Uh Yeah uh It will get these Yeah that's not what I meant to do On packet So this is going to be On tick Uh And ooh you are right This will also check for availability Um Mmm we really need To Oh we do don't we We have to Drop the lock before we notify Mmm Okay so Like this is going to return an availability So now it for example If tick means that we got to send some Uh that can never happen Great So the availability cannot change because of this I think To do don't die on errors Uh if I guess if one connection Terminates it doesn't Because of what happens in on tick We might not want to just exit But I'm not sure So the question is what do we do in on tick Well What we really want to do is We want to retransmit If there is something to retransmit Uh we want to send new data If there is new data to send Of course you could also imagine We could do this proactively Like if we see that we have data That we're allowed to send we can just send it Uh what this means is Uh when you close The connection You might as well just call on tick Um and you'll see why in a second So what on tick is going to do Uh is It is going to Decide If it needs To send something It's going to send it Basically all we have to do like Most of what the timers are Is that and in fact we look at The r of c Um All right We're also supposed to enter Fin We also need to figure out what to do with this But that's a separate issue Uh If fin enter Fin That's a question for later Um but there should be I think there's a timeout, is that what they call it User timeout That's not what I want I think there's another timeout That's the wrong kind of timeout User timeout Segment arrives Segment arrives Segment arrives Segment arrives Status call Abort call Close call Receive call Send call Yeah so this is What do you do when you're asked to Uh write data That's fine Open That's fine I specifically want retransmission timeout Those are really the only timeouts That matter to us Okay So in on tick Is see whether we're allowed to send stuff And if we are then we should send it If we're not allowed to send stuff Um then we should check Whether there's more stuff Uh whether there's stuff that needs to be Retransmitted So here what we'll do is keep track Of when we last Send data Uh so This is gonna be Last send Or I guess Last Uh Last send And that's gonna be a Do we have time imported here So last send is gonna be An instant How do you set up unit test for this project So unit test for this project shouldn't actually be That hard Because you can just You could imagine that you just set up The R U R TCP stack as the server TCP client And see that they can talk to each other And then you do the same in the reverse The biggest problem here is that In order to start a server or client You need to spin up a ton tap Which requires A particular capability Or root access And you don't really want to run your unit Test as root So the other way to do this would be To test just the TCP stack So not actually use ton tap And basically Make our library Be generic over The underlying transport So that it doesn't have to be a ton tap Like You could basically test unit test this By Just unit testing connection And constructing the packets So that would be the unit testing you would write The integration testing is a little bit harder Just because of permissions The title just says Last live stream It's really annoying To be changing the YouTube title When I Upload the video later It will have a more descriptive title Okay, so we're going to have a Last send Which is going to be time Instant now The chatbot Is partially broken In the sense that I think You can't see comments from YouTube And people from YouTube I don't know whether you can see comments from Twitch I can see comments from both So At least there's that This was a problem last time too And I don't know what's causing it But I did see that I'm using restream.io So I can stream to multiple places And apparently they're planning Like a giant redo Of their chatbot So Okay, so what do we want to do here? Well We want to make sure that We need to start actually maintaining our window Because that's what determines Whether or not we're allowed to send something And in addition, we need to make sure That the sin and the fin Are also included In Included in retransmissions So we might need to resend our sin Now maybe we just ignore that Just because it's easier And it's not all that interesting But certainly for data packets We may need to retransmit them And we certainly will have to wait For the window to grow Alright, so Where Is If self-TCP thin So this is probably wrong Because this is going to update sin.next Which means it won't work for retransmissions So that's something worth keeping in mind And we're going to have to figure out Finns and such But for the time being Let's first just deal with what happens In ontick And we can deal with the other stuff later So Where's our Send So basically to decide Whether or not we're allowed to send stuff We basically have to see That there's space in the window Do they actually document This test? I think so That's the interface Managing the window Currently we don't manage our window size At all, which is probably fine For the time being This we will certainly have to update And here I think this The other RFC we had gave A better estimation for how you compute this I think the first thing we're going to do is Actually let's go to the top And look for Data communication Center of data keeps track of the next sequence number to use And the variable send.next The receiver of data keeps track of the next sequence Number to expect The receiver of data keeps track of the next sequence number The sender of data keeps track of the oldest unacknowledged sequence number And the variable send.ne So this Is So we have this unacknowledged Bytes Vecdq UNA is going to be the sequence number Of the first byte in that And send.next Is going to be The next The bytes we are going to Send next If any Yeah Where is the part About the window though That's fine So we'll have to send empty packets Every now and again just to make sure To keep alive the connection To guarantee that When either TCP is a zero window The reopening of the window will reliably be reported Yep, that's fine Like certainly setting the window size And managing the windows is pretty tricky Here Things that we've already sent Are Self. Send. Neck So Data that we Have not even sent once yet Is all the stuff that is In Unact And that is at or beyond Send.next Right So let's draw this again Because it will be easier Down here So our Unact buffer Logically I guess let's label these Because it's helpful For So This And when I draw the arrow here I mean this element This byte The sequence number of this byte Is Sender.next Sorry No, that's false The one thing that's annoying about this tool is erasing Stop erasing This is UNA Unact The first unact sequence number is this one The moment that something is act We're going to remove it from the beginning Of this array So that this invariant always remains true Somewhere in here We're going to have a pointer For Next Right And Next is The next bytes we are allowed to send Sorry Not allowed to send Next is the next byte we want to send So the next byte that we haven't Sent previously Now There's also going to be something like This is going to be The Send.window I think We should double check this This is the sender window It could be the receiver window Actually No, I think it's the sender window So this is how many bytes we're allowed To have outstanding When ontick is called This is basically the variables it's allowed to look at Right In this case it should observe That Currently We are allowed to send these bytes Right We're allowed to send these bytes Because that would still keep us Within the window So if we do send these bytes Then the unacknowledged byte is still one And now Send.next is going to be here Instead of here And at this point we can't send Seven because seven is still Outside the window At some point if we get In If we get Ack Is equal to let's say UnactPos2 Then now This is going to be Unact These are going to be deleted The window Now allows us to send Two more things Now we're allowed to send this And so we're going to put Send.next here And send those So there's like a sliding window Of what data we're allowed to send So here How are we going to do this Well Unsent Is Actually Number of bytes sent Is Self.send.next Wrapping sub Self.send Dot Unacknowledged Is that true? Yes Yes that should be true So this is the number of bytes That We have sent but that have not Been acknowledged Unsent is Self.unact.len Minus Unact So these are bytes that we have Not sent yet I guess These are the number of bytes We have not sent yet At this point We sort of have an option To decide what we want to do And It's going to depend on Whether we're allowed to retransmit yet If we're not supposed to retransmit Then So here let's see Self.lastsend Dot Elapsed Trying to remember what Retrans Two minutes is recommended For the retransmission interval when the window is 0 Want to see where they recommend How to set transmission So Send window So that is the send window The sequence number which the remote is willing to receive The range of new sequence number Which may be emitted by tcbLyce Between send.next And send Unacknowledged plus the window Minus 1 So this is the same that I showed In the drawing on the other one When the tcb transmits a segment Connecting to data it puts a copy On the retransmission queue and starts a timer When the acknowledgement for that data is received The segment is deleted from the queue If the acknowledgement is not received The segment is retransmitted The question is What is the retransmission timer What should it be set to Duplicate the number of retransmissions It's fine So TCP also technically Says that you should keep track Of retransmissions times Per packet you've sent Rather than per byte It's like If I send 5 bytes and then 5 more bytes If the retransmission Timer expires for the first 5 bytes I should just resend those I shouldn't resend 10 Even though potentially I can So this is also something we may have to keep track of Transmitted What is the timeout Here we go, retransmission timeout Because of the variability of the networks That compose an inter-network system And the wide range of uses at TCP connection The retransmission timeout must be dynamically determined One procedure For determining a retransmission timeout For Determining a retransmission timeout Is given here as an illustration There is probably a better way to compute this I'm pretty sure I've seen this in fact Measure the elapsed time Between sending a data octet With a particular sequence number And receiving an acknowledgement That covers that sequence number The measure of the elapsed time Is the round-trip time So the recommendation here Is basically estimate what the RTT is The round-trip time is And then Keep sort of a moving average Of the estimated round-trip time And then the round-trip time Is going to be Some Multiple Of that RTT Okay So they want that to be SRTT Hmm So this means that We need to keep track of when we Sent each byte And when that byte was Act It's basically what they're telling us to do We probably don't want to keep track of every single byte But instead every Every time we Send something We should keep track of the sequence number And when we send it The time out appropriately So let's go up This means that we're going to have to keep A little bit more Data here Specifically On the connection On the connection Let's Start another Timers Gonna be a timer struct Struct Timers It's gonna have a last send It's gonna have a Um Huh It's gonna have Something like Sends Or send Times Which is gonna be Let's say a vecteque Of Uh A particular sequence number And A particular And when that sequence number was Sent So every time we Send a sequence number We stick it into send times And when it gets act Then we take the difference between Um We take the difference between When we sent it And the current time And that is an estimate of the RTT Uh and then we do SRTT is a Time duration So This is gonna be timers This is gonna be a new timers With last sent Equal to that With uh send Times Is a empty one of these Uh and where Uh SRTT Is Time duration From Minutes To From I guess 6 2 times 60 If Self thought Last send Elapsed is more than This to be The retransmission timeout is U bound is an Upper bound on the timeout For example 1 minute Okay So not 2 minutes 1 minute Is more than Oh I see they want A lower bound and an upper bound Um so this Is Uh 4 is this If Waited 4 Is greater than 1 second And waited 4 Is less than Self timers The Smoothed RTT Times Beta I guess 1.5 Magic numbers are great Uh so in this case We've waited for what we believe The timeout should be And so at this point Retrans, we should Retransmit things Otherwise Uh We should send new Data If we have new data And space in the window So here we really If we're gonna retransmit Things the question becomes What do we retransmit? Well we Retransmit this is where Technically the tcp protocol Probably wants us to Only retransmit the The packet that Has timed out rather than as much as we Can. Um but instead What we're gonna do is we're just gonna send As much as we're allowed to retransmit Um So what that means Is we're gonna do something like Um Self.write of course Self.write is broken we know that It doesn't work for retransmits but that's something We're gonna have to fix Um and that takes the nick And it's gonna take Uh I guess Allowed Is gonna be Self.send.window Right what was the What was the Computation here For how much we were allowed to send Managing the window We're allowed to send From This Self.send.window Which means we don't really need it Uh so here We're gonna write everything From We're gonna retransmit everything From Unact But let's look at that in a second Um everything Um This many bytes Right so we're gonna send basically We're gonna because we're retransmitting We're just gonna retransmit as much as we possibly Can And that might include some new data Right uh and then at this point Um Self.send Next To become Self.send. Unact Unact Uh wrapping add Self.send.window Sort of this code is still a little bit inaccurate But does the rough idea make sense We're supposed to We're supposed to do retransmissions We just send basically the full window again Uh we could be more conservative here But instead we're just gonna We might actually here want to send like Only as much as we'll fit in a TCP packet for example Uh but that's not currently what we're doing Um and then Now self.send.next Will be Actually that's not even true Uh self.send.next should now point to The next byte that we want to send Although This might not actually be accurate But we'll get back to this code There's more to lay out the rough structure Of what it's gonna do Here um if Unsent is zero Then we don't actually have anything to do Right If we don't have any bytes that haven't been sent And there's nothing to retransmit There's nothing for us to do There's nothing for us to send here So we can just return okay Uh otherwise We should just send out As many bytes as we are allowed to send Right So we're allowed to send Self.send.window Minus Um Actually Minus Unact Right And if Allowed equals zero Then what do we do Well We can't send any data can we Um Like if the number of unact bytes we have Are the full window Then we're not allowed to send any more bytes We need to wait for the window to open up And so in that case We can't do anything And we're not supposed to retransmit So we do nothing Only in this final case are we allowed to send some bytes So here We're allowed to write out to the nick Uh Self.unact Uh Uh From From unact To Unact Plus Uh Allowed Now the observant among you might notice that Like there are some out of bounds conditions here That are going to be annoying to deal with But that we will actually have to deal with Oh we just got a lot Of people claiming That there's PHP None of this is PHP If you are saying PHP you're in the wrong channel Um Right so in this case We are allowed to send stuff and we do have Stuff to send and so in that case We just write out as much as we're allowed to send Uh And at this point self.send.next Is Uh Self.send.unact Plus Unact Plus Actually no It's not even that it is Self.send.next plus Allowed Sort of My VMRC is public there's in fact a previous stream Where I go through my entire like editor setup And stuff so if you search For that you'll find it Okay uh and we have to deal with Fins that's sort of separate So let's actually now tidy this Up a little um if we Need to retransmit what do we actually get To retransmit well um We get to Retransmit up to the window we might want To have a lower limit but because this is a We're only going to be sending small things Regardless it might not matter all that much Uh and we might Get like IP segmentation and stuff which Is nice um So here Right remember Unact might not have enough stuff to Fill a window so Send is going to be Or I guess Resend is going to be Uh The min of Self.unact.len And the self send Window And that's how much we're going to send Um let's also fix Right while we're at it So right What is unwritten here 1500 Ah great So it will only Write out at most 1500 bytes so that's great This is not quite right the updating Of Send.next Is not actually Right Um because the because it might be A retransmission right Now how do we Want to deal with that We could of course update Self send.next in every caller Um But we do call right in a Decent number of places The sequence number of course is also Not necessarily send.next Uh in a retransmission Maybe the trick here is going to be Giving in the sequence Number of the first thing Then I think we can write Write properly So the sequence number Is not going to be Self.send.next The sequence number is going to be Whatever the initial sequence number is The acknowledgement number is correct And now this can actually Update self send next Because self send next Is going to be Um let's see Let Uh Last sec Is Sec Wrapping add Well this really Minus one Next sec really If next sec Not quite And of course if it includes A If it includes a sin Then we wrote one more byte Um this is going to come back To bite us Because these aren't actually true Actually no that is true It just resets the flag at the end That's probably fine I think what we want here is Um okay Let's ignore that for now Um but we only want to Update send.next If we actually have bumped A send.next So in this case Um if Next sec is Uh I guess if Wrapping less than Self Send next So if that's less than Next sec Then Next is next sec So this means that in the case of A retransmission Uh we won't update Send.next to be less To be like whatever the last thing We retransmitted was So the idea here is that sec Is the sequence number of the first thing In payload Or potentially in sin Really Like including Counting the sin Is a different way to phrase it Um great Okay so now This is going to have to decide What to do Here this is going to be Self Send next Uh that's false C send Next right so Uh when we send the Initial sin to establish the connection Our first Sequence number is going to be Whatever we decided the initial sequence number was going to be Where else do we call Right here Uh same thing Self send next In fact for all of the All of the ones that were there already Um they already assumed this To be the case And they send no bytes so that's fine So this is sending an act In theory this could like also include Sending data Um so we could Make this just be tick For example Maybe just tick to Piggyback Act on data Uh This as well self send Next Because it's sending the Sequence number here is the Sequence number of Uh just past the fin Uh this does mean That we don't have a way to retransmit the fin So we'll have to deal with that later Uh what this does Let us do those is in the case Of our retransmission This is self dot Uh send dot on a Right oops Um that is the sequence number of the First thing in an act How this interacts with Sins we'll have to look at Um And down here is the same thing Uh No that's not true Here it is In fact Send Next Yeah so the Uh It should be the case That self send next Points to the first byte in Unact that we have not sent yet Uh and here Uh the bytes we want to send Is The min of Unsent and allowed How do you like rust Compared to php They are very different languages I highly prefer rust Uh php Is a very dynamic language Which is nice And it's nice for simple Web setups Um I used to program A lot in php I don't think I would use it for anything anymore Um right So here Do do do do do This should now work for retransmissions That's great Right so When we first start Sticking stuff in Oh right we haven't added anything to write yet Great So now if we look at Um lib What actually happens when we try to do it right What do we do here Oh I guess we're not actually dealing with writes yet We're mostly just looking at shutdown At least that was our initial intention Um So for now We're actually going to assume that Um We're going to assume that there are never any bytes to send We just want to get shut down to work first Which means that the only thing we could be Asked to retransmit is the fin Um Or asked to transmit is the fin Um The question is how are we going to deal With retransmitting The fin because it's not in Unact So we need to keep track of whether or not We have sent the fin Uh which we know Because Oh interesting So We have a couple of pieces of information here right We have self.closed Which is Um Has shut down been called But we also have Self.state Um And self.state Can be one of these Which indicate That we have sent the fin Right so if we go back to Our um diagram Um When we send a fin we end up in fin weight One If we have sent a fin we're in fin weight Two or closing Or time weight so in any of the states Fin weight one and down In all of those cases we have sent The fin right And so we actually Know whether or not Uh I guess Um Let's add a thing here Um Have sent fin So that is In Let's see In state Or established we have not sent it yet In fin weight one fin weight Two and time weight We have sent it So In theory we should be able to use that to determine Whether we're resending a fin or not So let's see So these are the bytes we're going to Resend Oh wait did it say Um actually I don't think that's True Thinking back to Thinking back to what it said on close I think it said Enter fin weight one regardless Like no matter Whether or not you can send the fin Um Close So close and established Yeah in any case Enter fin weight one So it's actually Not sufficient We need to have a separate mechanism for determining Whether we have Not sent the fin Because there also has to be like An if here right of Like if Uh Need to Resend Fin Or I guess this is really just Is the fin within the window That's really all we care about here So if Um If resend Is less Than I guess this is a No This is not that If resend Is less than the window And Self-closed Then We also set the fin Include the fin Or I guess can we Include the fin Right so resend is how many bytes Are out That might be zero in fact in our case It will always be zero And if The window has space For us sending the fin And we're supposed to send the fin Then we send the fin If unsent Is zero Right so this is tricky So here we need to know Whether we have Already sent the fin Because if we haven't We should send the packet if we can Specifically we should send the fin If we can Interesting So one way to do this Would be and not Self-closed Um But unfortunately The downside of doing this Is that this will Send the fin again Each time It's not actually true because it Is not correct because The fin should not get There aren't multiple fins Which this would imply Um So I think we may actually need To keep track Of Essentially the last sequence Number Ah here's what we could do We could keep track of what the sequence number Of the fin is So that would be up here In connection I guess we could do Closed It's a little awkward but we could do it this way Right so Uh Closed at is going to keep track of the sequence number We used for the fin If we have sent it Um so here this is going to be Closed at is none Because initially we haven't closed it at all Uh Right so if there's no data to send And we have Closed the connection Uh and we've sent the fin Then we have nothing to do Uh if Send is less than Allowed And self.closed And self.closed At is none So this is if we're allowed to Send more than what we're already sending And We're supposed to send the fin And we have not yet sent The fin then We send the fin And in this case uh Closed at is going to be Set to Self. Send.next Wrapping add Um Unsent Right that's where we ended up Sending the fin Because no more data is allowed to be Written once we've closed Right so the The actual structure of this is The bytes and unsent are followed By the fin And so self.send.next is Before these things are unsent So plus unsent is after those And that's where the sequence Number of the fin will be I think that's right I think that's right I mean it's probably broken Um There's also Moving to fin wait one So the old hacky code That we had was this business So this Set fin to true Hmm It moved us to fin wait one So that means That that is something that Um Shutdown is going to have to do Right it's going to do This Close true State fin wait one And from memory this was what Close was supposed to do Right Uh technically we should Check here for Um Check for That As long as the second fin is not Emitted nope that's not We don't do that It's going to be if C dot state Is equal to state Established or Estab I guess Uh if State do we even have that state Yeah soon received If no send has been issued There's no pending data to send The form of fin segment and send it Oh actually no we don't even That's also fine Um So if we're in Uh state soon Received Uh or we're in State established Then we move To fin wait Uh and If we're in any other state Other states could be close Wait and close wait is also Fine but also we don't support that state Um so soon received And established fin wait One Is also fine for us to do this And time wait Would be an error So fin wait one And fin wait two are both fine In all cases we move to Well Actually in the others it doesn't matter If we're in these nothing needs to Happen uh and if you're Any other state Uh then we really want to return Uh An error On the form of New Error kind Uh who knows Error closing That an error kind I can use Unclear actually Let's do connection Ported Might not be the right one but Um closing Actually maybe Maybe this is already Uh documented in shutdown What does tcp stream Shutdown say Um the Second call not connected Okay that's fine So shutdown is just Going to set up the state to Be right and then the tick is going to take Care of actually sending out the fin Is the goal to create as an exercise Part of a tcp stack that you will use Instead of the inbuilt kernel one Sort of so I mean what We're doing is really just Uh Building a tcp stack because it's an Interesting way to learn low level Programming and because you learn tcp In the process um I don't My goal is not to use this for anything It is more as sort of a An educational experience Um But of course the goal is that it Actually works uh and in fact It does work for reads now And hopefully with this change it should Be able to We'll see whether that turns out to be true Or not That is a lot of errors And they're probably All because of my stupidity And this should be Timers Of course there's a bunch of stuff we don't Maintain here either Um conversion Between numeric types Ignore back off get speed That's not quite how it works So this is Uh this is the third Video in a series so if you want to Learn about the underlying like setup For how how we're doing this in the first Place um you should go back To you should go watch the first Video in the series which is on youtube Um so I would watch these actually In sequence I won't Actually go back to explain how the earlier Parts work in this video because It's already explained in the earlier video Um Expected slice Found U8 That's a separate issue Let's see here Oh it doesn't have the state does it I think this is going to have to do See close And then we're going to have to In here Add in a close The result This It's going to do Self Self connections Hmm 268 I guess RTO is going to be Duration from No Yeah from nanos Actually Ah that's annoying Problem is we have to do arithmetic on time Which is Kind of awkward Um RTO found U16 Yeah Oh that's a good point I can add stuff to my Twitch that's a good use Like add links to the other things That's a good point Expected U size Found range 2 Ah so here's the other problem We have which is Remember how unact is A ring buffer You can't Index a ring buffer like this The ring buffer is really two Two segments right The sort of head segment And the tail segment potentially Um The real question is How do we want to do this Oh this should say recent This also has to Set Close that So the question is How do we want to break up this right I think actually what we're going to do here Is have right Support taking Two Buffers and a length Yeah that's what we're going to do So it's going to be Payload 1 Payload 2 And Lend And then we're going to do Unwritten write Payload 1 Dot dot lend Calls Hmm We want to make sure it doesn't write More than we're allowed to write Because of the window for example right So here what we want to do Is we want to write From payload 1 as many bytes were Allowed to or that's there And then we want to continue writing from Payload 2 if we're allowed to Which I think means That we're going to have to do Something along the lines of First Write as much as we can From payload 1 Which is going to be Min Of Lend And payload 1 Dot lend plus Payload 2 dot lend And then we're going to do P1 lend It's going to be the min of these We're going to write Out Written So written plus equals This Dot dot p1l And then Lend Minus equals P1l Sorry I'm going to explain this once I've written it Just to make sure I actually Understand it myself Okay We can So the idea here is We can write out Whichever is less of how much data There is and how much we're Allowed to write So that's now going to be lend So the question first becomes How much can we write out of Payload 1 while we can write Whichever is less of how Long payload 1 is and what our limit Is Limit instead of lend Limit is a better name For it anyway Okay so we write out some stuff From payload 1 As much as we can basically Now the limit is decreased By however much we ended up Reading from payload 1 So if we wrote It can't even be that This has to be written So now Once we write some stuff from payload 1 Now the limit has been reduced By however much we wrote To payload 1 Right? Now we try to write more From payload 2 So There are sort of 3 cases here Either we exhausted Payload 1 or there are 2 cases Either we exhausted payload 1 And there is more room Or we did not exhaust payload 1 To the limit If we used up the limit Then the limit here is going to be 0 And so this is going to be 0 And so we're going to read 0 bytes From payload 2 If we read some bytes from payload 2 And now we're going to continue Writing from payload 1 Now we're going to continue reading from payload 2 The amount we're going to read from payload 2 Is however much the remaining limit is Or payload 2's length Whichever is smaller And the result is we wrote this many bytes out From the 2 payloads Ok I guess this is like max data So what is size here Size is How many bytes are there in a packet Well It certainly can't be more Than The length of the packet But it could be less In which case it's the TCP header The IP header As much data as we can write As we have Yes Let's see Can unwritten.write End up writing fewer bytes than you ask it Yes This is why I do limit minus equals written Rather than limit minus equals p1l So I think this is still correct Like Unwritten.write can end up writing Fewer than p1l bytes Right It could be hitting the limit of buff Which is of limited size Or limit of unwritten Which is buff So that is why the limit is Decreased by how many bytes were actually Written out And if we then try To write out more bytes here Unwritten would still then say I wrote zero more bytes So I still think this ends up being right Okay What now fails Lots of things Great 1, 9, 3 Right So now this is going to be Of course now all of these have to Include more stuff in them Which is a little annoying But it's worth The cost Because Now Here We can now do Headtail is self Unact Dot Where's the place where we get the head H Tail As slices As slices And then we can do HT Resend So this will write first from h Then from t and no more than resend Which is what we wanted in the first place And the same is going to be the case here Where this is going to be HT Self unact As slices Except here We have to skip forward By an unact Which is also going to be A bit of a pain We want Self dot unact And unact So We actually need to Mutate these to skip Which is If It's writing to an It's writing to an in memory buffer That will only later write to the actual Output, yep That's right So that's why I think it's okay If it wrote directly to the output You're right, then it could write too much Alright, so Here We need to Basically Remove stuff from the beginning of HT All the way up to an unact If H dot Len Is greater than or equal to an unact Well, actually let's write out The verbose version first Then H is equal to H An unact Dot dot And T remains the same Because we just trimmed off that many things From H Otherwise Skipped Is equal to H dot Len H is equal to an empty one And T should be equal to T An unact Minus Skipped Dot dot And this can probably be Simplified Somehow But I'm not going to bother Let's Draw again Because this is a little bit weird So We have our ring buffer again Actually, I don't want So we have our ring buffer And let's say that The This is the head And this is the Tail Let's actually write these out One, two, three And four So with this setup Let's say that An Unact Is three So what that means is We really want this Slice Because we want to skip three If we're going to send only the stuff that's new Only the four here is new All the other stuff is stuff That we've sent in the past So we only want to send four So The question becomes how do we get a Like when we do Dot ass slices Right Then what we get back is A A reference to One, two And a reference to Three, four Where this is H And this is T And we want to turn that Into H is nothing And T is four That's what we want to end up with Right And so the way we do that If we look at the code we just wrote Is if H dot lend is greater than or equal to unact So if Let's say unact was one So if unact was one If unact was one Then all we have to do is skip The first one elements from head Right and leave tail the way it was Which is what this does H is just set to be Skipping and unact from H And T is left on the loan Otherwise, head is going to be empty So if an unact is greater Than the length of head Then head is going to be empty no matter what And so in that case We're going to set Skipped Is equal to two So we skip these two Right So that's here Head is going to be empty And T is going to be Skipping however we have left That we need to skip So an unact was three Skipped was two so sort of left Like how many are left One so that is this one So we're going to skip this one as well And so T is going to be Starting from one And onwards Which is starting from here and onwards Which is four And with that It should be possible for us to pass H, T and send Because send is sort of how many bytes We are allowed to send I think that's right Since the multiplying time Which we're going to ignore for now 298 Recent Expected U size Found U32 It's probably fine for this To be a U32 to be honest But it will be easier If it's a U size So let's just make it be a U size This should Not We don't need to do any of that anymore Because the first byte here Is the first byte of unact Whose Sequence number is The unact sequence number 289 That's still the timer Mismatched type Expected result We're certainly going to need to We're going to need to maintain These like last send stuff as well And those are going to have To be updated here That's not what I meant to do So here This has to return okay In the end We have to figure out What to do about these timers So when do we It might be the last send is not sufficient Because we might have to keep track Of Last send for It's going to be a little awkward I think we have to keep track of every send Because imagine that you do a send And then later on it gets Act what is now your last Send So I think you actually need to keep track Of when you send Every packet Does that And that has to be updated When you When you retransmit Hmm I think the trick here I know what we're going to have to do We're going to have to make I think What this is going to end up being is a B-tree map From sequence number To When it was sent Because we can use the B-tree map To look for all the sequence numbers Before or after Of course it gets weird with We can figure that out We're also going to have to figure out what to do here In order to estimate the RTT Because we don't want to use Retransmissions to estimate the RTT Only the original sends Otherwise If I resend a thing 5 times That doesn't mean that the RTT is 5 times as high So I only really want to count RTTs for From the From things I didn't have To retransmit Because I also don't want to do it Have it be the last send Because imagine that I send something The retransmit timeout Expires so I send again But the first one actually arrived It just arrived a little bit late So the other side sends me an act That I receive immediately after the second send If I use the second send Is when I send the thing that was now act Then my Estimate of the RTT is going to be very small Also wrong I don't know how real implementations handle this Like What do they use for the estimation It's probably a lot of My guess would be there's a lot of research in estimating RTTs I think it's just hard Which means it's fine if we do a bad job of it In fact we're probably going to do something stupid Which is great Okay so we really need to maintain This send times Business Send times And I think what we're going to do there Is Is Down here Whenever we write something out We're going to do self Send times In search The first sequence number And time instant Now So every time we Send any packet We store the sequence number we sent In that packet And the time when we sent it And then What we'll do later is in the code That deals with when things Have been act We can then sort of walk send times And use that to update the RTT And now because we do this in write It means we don't have to update all the places That send packets We can just do it here Of course This might have some problem when we send resets And such but We can probably just ignore that It does Mean that weighted for is a little Weird I think This is going to be like the Minimum Yeah This is going to be Send times Dot values Min dot Yeah Right so this is Actually no Map Map TG elapsed And we want a Max So this is What is the longest We've waited for anything that we've sent Although that's weird too I think it should only be For the currently Unact I think what we're really going to do here Is What's the interface On btree map for like Next higher value It's like a range thing What do they call There's a range That's what I want So I want send times Range And I want Self send Unact Dot Next Dot Map Time instant elapsed So that is How long is it Since We sent The last thing that is not Weird of a sentence but this is like Find the Find when we sent the earliest Unact things And see how long it is since we sent that And if The time that has expired there Is more than the time out than retransmit And if I guess This is like How do we want to do this conditional though This is like That should Retransmit So we should retransmit If Waited for Is some I guess we could even do If let Some Or is waited for If If there Is no such timer Like if we have no running retransmission timers Then of course we shouldn't retransmit Because there shouldn't be anything to retransmit Otherwise we should do it If Waited for Is greater than One second So this is the lower bound And Waited for Dot as float Sex So as float sex gives you The duration in Floating point seconds So this is like Seconds but with decimal points Is more than 1.5 time self timers SRTT As float Seconds If should retransmit And we should Retransmit things And here Can we include the fin Then self closed at Is it gonna be some What is it gonna be What is the sequence number of the fin Well it is Self Dot It is the start Of The Unacknowledged buffer plus the length Of the unacked buffer Yeah that is the sequence number of the fin Which I think is the same as down here I hope maybe That is not terribly Surprising Sort of want an assertion here to check that these are the same I feel like this should actually Be the same So let's down here do like a Actually no We can just make it be this Right because the fin just should always be that The start of the unacknowledged window Plus Plus the length Of bytes that we are That are in that window Should just always be the fin Down to fewer errors We need to import Btream app that is luckily pretty easy But timers Sometimes 292 Fine TTelapsed Assexflow What's it called Duration What is it Called it is called Oh man Stop with that Where am I on I don't actually remember what I'm on here Duration Beta So I guess I'm on beta And on beta it is called What exactly It's called Assexf64 Which means it's not going to compile Because it's a feature right Override set Nightly that's too bad Pretty annoying So we're going to have to do Feature 292 Sure 306 more numeric conversion This Of course needs to be Wrapping add As does this Which also means it has to be SU32 Probably 316 That's fine 321 Numerical conversions are The most fun So Getting pretty close I expected U size Found U32 Great Now we just need to fix the borrow checker 32 So why is this complaining This is complaining because Can our connection is mutable Values Mute Try that Variable does not need to be mutable 107 Okay 252 U44 That does get reassigned That is true That's fine That's awkward I'm borrowing into Self-unact Which means that I can't call self-right That's pretty annoying We can't actually fix that easily It could be That self.write Should just always read from Self.unact Rather than us like passing it in Everywhere We give a start And an end Instead of giving those slices Because currently here H and T are borrowing self-unact But self.write needs to borrow Self mutably Which you can't do I think we're just going to do here Zero And then down here We're going to do U size And now Right is going to change a little bit It's going to change To be Instead of taking payload one And payload two It's going to take Start U size Limit U size I guess offset And we're going to say Let mut H Mut T Offset Then offset Offset And then now this is going to be H And T These are all going to be H T Oops, let's see how that works And now anywhere that calls Right It's now just going to do This Of course now we have Problem Here Oh no, as long as you set limit to zero It's not going to send any bytes anyway Great, even better So it doesn't matter that it's using Unact behind the scenes Zero zero is always send No data anyway Alright, how about now Great How much faith do we have That this will actually work on the first try I'm skeptical It didn't crash immediately Still got data And then it crashed In on tick Which called right Well that's better Than I thought it was going to be I guess actually We sort of We might as well have Close call Self.onTick It can't do that Doesn't have the nick It's fine Okay So Right is failing somewhere Slicing Starts at one but ends at zero It sounds an awful lot Like Finn In fact Because Right knows the sequence number In theory it can compute the offset I don't think it needs to take the offset Right because Here Self.send.next should just be pointing to The end of Unact Here We want to send from the start of Unact Which is zero Here we want to send From Next Which is Unact plus number Of Unact Here we want to send from next From next from next Okay great So we don't actually need to take the offset Instead The offset is going to be Self.send.next Self.send.next Self.send.next Self.send.next Wrapping Self.onA Right No sorry Seek Because that is the starting So the sequence number of the Starting byte And we subtract the unacknowledged Index Which means if you're starting to send from the start of Unact This will turn zero and the offset is zero Otherwise it's going to be some larger value Which means that anywhere the calls Right does not need to give This argument This Is going to send to Unact Which Wrapping Sub Unact Is just the same This Starts from Next Although Next here could be pointing to the To after the fin These could both be pointing to In which case Right would crash because it's trying to look up past The end of Unact This does not need this This does not need This Does not need this This Does not need that So here We're going to have to look at If And what do we want to do here We sort of want to observe that We need to We need to handle the fin somehow This will also have a problem with With the sin I think We need to Special case The two Virtual bytes Sin and fin So Let's make the Offset and the limit Both be mutable Actually the limit doesn't need to be But the offset does Because we never read past the End of the input stream regardless Just leaving it to do for myself later So if The If the sequence number Is If the sequence number is the sequence Number of Good question So we have two sort of special cases We have the sequence number is equal To The Sequence number of the sin In which case we need to like shift it one Right Although I don't think that That would only happen if you retransmit A sin which should not happen So the real The real issue here is if the sequence Number is equal to the sequence number of Fin Because it'll be off by one And so therefore If we have the Sequence number Of the fin And if This sequence number Is that sequence number Plus one Then Seq minus equals one No then Offset minus equals one No then equals zero Right so this is If we're Being asked to Send bytes starting After the fin Then We need to set the offset to zero We should not be reading any data Or another way to say this is Here Although that's also just annoying Let's see whether that does the right thing Of course it doesn't If I do this what happens The crash is where It's so unhelpful that I don't get The line number here Alright it's time for some print debugging Right From I guess Seq Limit From So seek Limit Self Unact Then I also want to print out here Using offset Alright so what do we get Write sequence zero Limit zero From there that's fine Right so that's writing out the The initial sin Write sequence one Limit zero From there This is writing out the fin Let's print some more stats like Um This is self tcp fin And then let's not Do that here but do that Here instead In this What do we get So first we write out The sin At sequence number zero Great then we write out the fin At sequence number one And the question becomes what happens at sequence two Two Where sin is false and fin is false So when that happens It's using offset one Oh So we're correcting subtracting one But the sin Is the sin Unact is that why The wrapping sub is not picking this up Base this Self Send unA I think unA is not being updated correctly Base zero Zero base No base one So one base one So what sends this Fin Or rather why is this not being picked up Fin closes none for all of these How is this fin being sent though That's the real question Oh do we still have our hack in place Is that why Ah we do don't we Should not be there Let's see how that turns out Ooh That looks promising So let's see So first we write out the sin Then we write out Uh Just an empty packet Unclear why Oh this is an act to the Other sides Something I think this is just an act we decide to send Yeah this is just Us deciding to act something Unclear why Here We do that again It just seems like we're just sending Acts without needing to Um Then oh we're acting No no no this is us acting the bytes that they Send us Right because then we get the bytes right after Then our shutdown Happens and so we send Our fin And then it tries to send Oh this is the retransmission timer Kicking in That's also problematic Right so we don't actually reset Our retransmission timer Anywhere So whenever Something is act Right so here we want to make sure That we Remove anything from Right so this is We got and actually let's look at the RFC again Down here This is when we receive Where is it Check the act bit Check the act field Any segments on the retransmission queue Which are there by entirely acknowledged are removed So this is the trick Here We want to Basically remove So self Unact Drain We want to drain everything Up to Act N Wrapping sub Self send Right so this is So this is We got an act for something We want to remove anything from the Unact queue that has now been Act and then we also Want to deal with any of our timers So So I guess Here what I want is We want to Walk the Btree map So that We want to make sure that we keep track of Remove any of the timers That are relevant right So Self timers Send times I guess retain That's going to be A sequence number And A Send And if is between wrapped Self send On a Seek Act N So for all the timers If the sequence number for that Timer is In the list that Has now been act Then Then we want to remove the Timer However we also want to make sure that we Update our RTT Estimate Right which was Self SRTT So RTT is Send elapsed And self The smooth RTT Is Actually I think what we want is SRTT should just be an f64 As 6 f64 Then this should be Um I don't know what they recommended as the alpha 0.8 Sure 0.8 times the SRTT Plus 1 minus 0.8 Times RTT as 6 f64 Now what Cannot find value That's because this has to be self I guess this might as well just be Send elapsed 437 Really? There isn't a retain on btree map That's interesting btree map Huh That's awkward Why is there not a retain on btree map I guess we will do it the Stupid way btree map new Actually let um old Is Standard memory place Self timers send times btree map new And then we're going to do old.extend um No Self timers send times extend old into iter filter map seek and sent Which is essentially implementing Our own really inefficient Then none else some seek and send timers srtt This has to be 1.0 with 39 rtt So here this closure borrows self But we're also using self here so we need to Do some sad stuff like UNA is self send UNA So this is going to be UNA This is going to be srtt Otherwise the closure would not work But now how much is going to break Ooh that broke a lot Uh Drain upper bound was too large Really this one, huh Mmm So back for That last was that prune in this Again self send UNA And self Unact Oh it's the It's when we get the act for the sin If If self unact We only want to do this So act for 1 So it ignores that And then somewhere down here Act for 2 Great so we still haven't sent any data But now shutdown works Increase font, sure Better Great So now the trick Now the real question Is going to be Can we implement right This doesn't actually need to do anything Blush We don't actually have to Have a way to do flush yet But that's fine Um Let's change our main To try to do Stream.write Hello How terribly wrong do we think This is going to go Crashed Alright so So all of this is establishing the connection We receive The data from the other side First that's fine And then we send Write with a limit of 5 So that would be Our hello And also fin set to true This point fin closes Six which is Zero is the sin One is the h and hello Two is the e Three is the l Four is the l Five is the o Six is the fin that follows the o So that's right Uh We try to write 7 Um Which uses offset Zero This should also set limit to zero If we try to write Trying to write Uh Following fin Alright let's try that again Alright so it still did Something weird In fact it's printing Doing the same thing a lot Why is it deciding to Is the real question So it is certainly Writing out the data It's just doing it a lot Let's make our life a little bit easier Uh and have the Timeout thing be Like a second Instead Okay so what actually happens here Uh This is all the received stuff So that we already have dealt with Here's when the timer first fires After the write And it wants to write out 5 bytes That includes the fin Why is it 5 Alright yep So it's using offset zero base 1 In that and it's supposed to Write out 5 3 4 5 this is the hello Then it tries to write out from 7 And then it's just writing nothing That's all fine Although the fact that this stays Full means that we never get An act from the other side So let's Pull up our trusted t-shark Start this again Start this Run the client Alright so what actually happened there So we Sent Sequence 1 And act there 5 So this is the food that they sent us Plus their sin and their fin No just Plus their sin Then we send Our fin with a length of 5 So clearly It got the data Also why are we sending these We're sending those because We never get the act for this Why are they not Acting the data we're sending them That's certainly what's missing Because our packets Look well formed I guess let's try Pulling up wire shark And then run This So what does this get So this is the food that we get from them Then we This is us acting their sin again This is us acting their food Then we send a Fin act that contains Hello Data 5 bytes But why oh why Do we not hear anything from the other side Isn't that a problem coming from NC Well it does Certainly look like netcat is not doing the right thing But why is it not Doing the right thing I mean we could certainly do Dash I don't see why that should Matter Not yet implemented Great Why I don't think we want that Oh I wonder Whether If not so Let me just try a little Hack here That makes a difference I have no idea why It's not Why is it not printing the It's clearly getting the data That's what's weird I mean we could do We could enable debugging and see Whether that makes a difference but Sure I guess Debug and verbose it is I guess not Give me more verbose It's just not printing the contents What happens if we do curl instead Let's see if curl is better Behaved here No It shouldn't be firewall rules because The packets are going through We see that in T-Shark Let's see what happens In the case of Like if I do This is really weird because Very clearly The packet is going through Right T-Shark here When I run this We are getting a retransmission From them This is their SIN This is our ACK of their SIN This is Them sending What are they even sending Foo in a new line Foo in a new line This is us acting their SIN again Because our implementation is stupid This is Us Acknowledging Their foo This is us Sending our hello And also terminating The Connection This is us just sending ACKs This is them retransmitting The fin That doesn't seem right There is no fin yet I think this is the kernel from A previous run or something But why are these just Being ignored It's very weird indeed Let's try to just Like leave this running for a bit And see if that changes anything Certainly at some point we should try to Resend the data as well when the timer Expires But our code there could be broken I don't see Anything fascinating I mean it could be It doesn't like that we send The fin along with the data But I don't think that should matter Remind me why this doesn't work again This crashes us because Not yet implemented on 510 This is we got a fin Early so This is something like We're in the established State and got a fin In which case we have to Go to close wait Don't really want to implement that Shouldn't matter though I don't know why this is happening Let's do Here's what we'll do Let's do an s trace of that process And see what it gets It's loading a bunch of stuff Where does it start doing networking things It connects It reads food It writes food to the network socket And then it just pulls on the network socket So this indicates that the kernel Never thinks that it's getting data to read So it's not netcat So it is the kernel So why is the kernel not delivering That data The kernel not delivering It's certainly like The protocol interactions all look right And the data that Here we are receiving And the data we're sending here looks Totally fine The only thing I can imagine is that It's not okay for us to include the fin With the data So I mean we could try not doing that I suppose So In our right No In our on tick So this is just If Unscent Is Zero So now we should only send the fin After everything else has been acknowledged Oh did I Run it with the wrong flags No Forgot to run the T-shark So if we run this now With this what happens So here This is them sending data to us This is us acknowledging that This is us sending data to them And notice there's no fin And then we send the fin So weird I don't have a good answer for you It looks like all the Let's pull up wire shark again That's not what I wanted T-shark Apparently not Let's Try Because that did not make a difference And I don't think push should be Necessary So why are we getting no axe from the Kernel Here's one thing we could try If we do this Then This And then ss-t What's the thing for Choosing an interface ss-show only one interface Alright ss What is the I specifically want to know Oh fine Ss-t-i Destination Say 0-1 Let's see So it says It has act 5 bytes No It has sent 4 bytes And 5 bytes have been act Are we hacking too many bytes Last send, last receive, last act So it is Receiving segments It's just not adopting them Hmm So it's just ignoring all the incoming segments Because it's saying there's nothing new in them Or it's deciding that they're not valid Okay That's the only thing I can think of That it's deciding that the Segments we're sending here are invalid Why is it deciding that It's the real question Hmm I wish there was a good way to just like log The kernels Linux kernel Log drop packets Reasons why the network is dropping packets What's in here Start this Start that Rx packets Because it is getting packets There are no Rx errors There are no CRC errors No packets Dropped So it's getting all of them This is what's bizarre And it's getting more and more bytes But it's just kind of ignoring them Why is it ignoring them Let's look more carefully at these They send us a sin Sequence number 0 We send them an act Saying that the next thing we Expect is 1 And including Our sin with sequence number 0 They respond Accing our sin Giving sequence number 1 is the contents of this packet But there are no contents Length is 0 and there are no flags And act equals 1 Because the next thing they expect from us is 1 They then send Us With sequence number 1 Length of 4 We respond With sequence number 1 Acknowledging 5 So the next thing we Should receive is 5 Because we were expecting to receive 1 Then we got 1, 2, 3, And 4 Because 4 more So our next expected sequence number from them Is 5 So that act is correct Which is also why they're not doing any retransmissions Then we send Sequence number equals 1 Which is still correct from our side Length 5 And that They just stop responding to Hmm So it's a weird that they're not sending any I guess the keep alive Intro is like 2 minutes So it would be surprising if they did Are they ignoring our bytes And crucially It seems like it's not dropping Any packets But if you look at the Output from SS Up here The receive queue is also empty The receive window Is large enough to hold This initial window is 10 And we're not sending that many bytes So we should be fine There's just like nothing Here that looks weird Unless they think our Window is too large Hmm I don't know I also don't know how to Make progress without more Insight into how the The kernel is Deciding to drop our packets Which this is not giving Me Are there other Except none of them are being dropped And there's no reset being sent back And I think there is Also If we look at IP link Is there an IP Stats I wonder Monitor Seems not helpful I guess we could try sending It would be kind of stupid if this helped But what happens If we send More bytes I mean this is Clearly somewhat Of a desperate move I don't know What we're going to do then Is I'm going to commit this Because this Looking at the traces Looks like we're doing the right thing Which makes me think that What's actually going on Is that there's a TSP extension of some kind That we basically need to support And we don't And that's causing the kernel to drop What is TSP metrics I don't think that's going to Help us Sadly Because really what we need insight into here Is the Linux Kernels TCP stacks Like internal steps And I don't think that's something We're easily going to get I might end up debugging this sort of Separately but I think We're at a point where Me debugging this any Further is not going to be Interesting And we also got Like basically most of the way we want it So we know that reads work We have shutdown working on its own And we have writes at least Producing what looks like the Correct protocol messages So I think we're actually going to end it there Even though it is a little bit Unsatisfactory Because in theory I think we covered All of the All of the sort of Interesting aspects of this I think the only thing we're really missing Is the implementation of flush But flush is also pretty Straightforward So the only thing we're missing For flush is the ability to block Until there's nothing left And really all we'd have To do there is add Where's my TCP Available We would just add another available flag That would be like flush And we would set that flag In availability Only if Unact is empty And then we would have A condition variable wake up Just like we have elsewhere Come to think of it We don't do wake ups for writes here In a real system You would have a limit To how large you allow your buffers to grow And currently we don't do that So here there's definitely Currently our writes do not block And our flush does not block Which is not realistic I don't think IP metrics Like IP TCP metrics I don't think that's going to make a difference It doesn't print anything more From the other stats So I think it's really bounding The size of the buffer for write And implementing blocking For writes and blocking for flush Which are basically going to be the same thing As we did for read Like implementing the appropriate availability flags And adding a condition variable For flush and a condition variable For write Both of which are essentially The same as what we did for read And of course that we didn't actually get around To implementing connect My plan was to do connect next But given the issue we're currently seeing I don't think we're going to get past What we're currently seeing without some like Deeper level network stack debugging Which I think is going to be mostly orthogonal To Rust and more like digging into kernel things Which I don't want to spend a bunch of time on now So I think we're going to end it there We have now a TCP stack That can do reads and writes The writes don't work for some relatively Weird reason But the fact that Wireshark and T-Shark both show Like a valid TCP session With data going both directions Means that we're like If not there very close And I think we have All of the components we believe we're going to need Including the timers and stuff For now managing all of this Like segmented TCP retransmission It's definitely not the most Efficient implementation you're going to get But I do think we're in a point where The Basic structure Lends itself to writing A totally correct TCP implementation This is likely going to be the Very last TCP stream I might do one like much later If there's a lot of requests for it And if we can get past this The kernel dropping our packets In the meantime What I would like to ask is First of all if someone wants to Dig into this more I'll push all the code To the Rust TCP repository So feel free to like Check it out on your machine Try to run it, try to figure out Why this is happening Separately from that Please do go to the live coding page What stream you would like to see next There are a lot of ideas here They would all be interesting to work on Like just drag them in the Priority order you would like to vote for them And then we'll see what ends up Being the next stream My guess is the next stream will be In about two weeks But I will announce it when we get closer And with that Thank you everyone. Do you have any questions Before we finish? They don't have to be about TCP This project at all But like anything at all that I can That you want to like Talk about before I sign off Need to wait For the delay on the stream Doesn't look like it Great, well thanks everyone For coming out and I will see you On the next stream So long, farewell, Auf wiedersehen Goodbye