 Well, all right. Nice. Hi, everyone. Welcome back to another crust of rust This time we're gonna be talking about smart pointers and interior mutability. So This is something it's a little bit of a vague topic, but I want to talk about Some of the types you come across a lot in the rust world that maybe you have some passing knowledge of but they're just They're so pervasive and you need to have knowledge of them And these are the things like arc RC ref cell mutex cell The D ref and as ref traits the borrow trait Maybe the things like cow and sized if we get to it. I don't quite know and I figured the best way for us to try to understand these is try to implement some of them ourselves and So that's what we're gonna do As before I do a lot of these streams I have post all the recordings on YouTube afterwards So if you have to leave during the course of the stream or something just jump on there There are also a lot of older videos online that you can go look at Follow me on Twitter if you want to provide input on these episodes or new episodes Or if you just want announcements of upcoming upcoming streams and to that end I actually recently added a Sub-channel to the restation station Discord server so restation station is a podcast that is sort of intended to be a rust community podcast And I figured it would work pretty well to for it to have sort of audio visual content as well And so the discord right here both has integration with all of the live chat that goes on during the stream So in theory people can see here. I'll post the link to the discord in that chat for those of you who may want to move And it has announcements for upcoming streams So here you can see this announcement of my stream Ryan Levick who's also a rust streamer has a stream announced here And hopefully we'll get more of the rust streamers on to this one server So that there'll be one place where you can get announcements for these streams rather than having to like go all the way around And hopefully we can have some useful discussion here as well about about the streams the content of them and where to move forward And of course go to my YouTube channel if you want to see some of the past videos I also do these sort of for those who haven't watched them before I do these longer programming videos as well That you can also find here such as porting javas concurrent hash map to rust So if you want some more in-depth longer videos, go check those out All right, so today we're going to be implementing well, if we get to it RC ref cell and cell and also discuss ARC and mutex And maybe also look at things like as ref D ref Borrow and cow We'll see where we get to there. I think we're actually going to start out with Cell because it's the one that is Has the least amount of quirkiness to it So you'll see the and as before like for these crust of us there they're intended to be a way for you to Understand that the sort of somewhat more intermediate concepts and rust and so I'm not assuming you're an expert here If you have questions, please ask them other people will have the same questions as well and so just like post in chat and I'll make sure to check it every now and again and That will Hopefully this will mean that whatever questions you have get sort of persisted in the stream that in the recording of the stream That other people might watch later So if there's any questions you have other people will have them to feel free to ask them And I'll try to try to keep up with this the chat as we go All right Great So you'll see that the Rust standard library has this move this over there Has a module called cell And this module as the top level common says is shareable mutable containers now This might sound a little bit like a Weird concept in rust because in rust you have the notion of a shared reference This is just like the ampersand and you have a mutable reference or an exclusive reference as it's more aptly described as Which is only one thing has that pointer and so therefore you're allowed to mutate the thing that you have an exclusive reference to And so a shareable mutable container sounds a little weird, right? This is something where you have and you have a shared reference to something so someone else also has a reference But yet you're still allowed to mutate it it should immediately set off alarm bells in your head But this module is various container types that allow you to do this In a controlled fashion under the constraints of where where it is permitted This is often referred to as interior immutability So it's a type that externally it looks like it's immutable, but it has methods that allow you to mutate it The primary two ones well the primary three ones you have are cell ref cell and mutex mutex is not in cell It's in sync because it it Use the synchronization primitives that are provided by the operating system or by the CPU to make those operations safe So it doesn't really belong in cell But it kind of belongs in cell and you can really think of a mutex as a type of cell a type of interior immutability We're going to look at cell first because cell is is It provides interior immutability in a kind of interesting and very rusty way And it's a good segue into some of the more advanced things we're going to look at first Any questions about interior immutability or like exclusively shared references just before we dive into cell Hopefully the Hopefully it should strike you as odd that we can mutate things through shared references and rust like that that seems antithetical to what shared references sort of imply Even though in reality there there are ways to do this All right. I don't see any questions. So let's move forward with cell Okay, so a cell I can sell be used for recursive type storage Depends what you mean by recursive type storage. You can store any type in a cell if that answers your question If we could outline why using one versus the other it would be great shared mutation sounds fine But they seem to be a number of specialized tools to a similar problem. Yeah So one thing you'll see as we go through this is what are the restrictions of the different cell types the different interior immutability types so cell Ref cell and mutex have different restrictions on what things you can stick inside of them and how you can use them And generally like the farther you go towards mutex the freer you are to put whatever you want inside But the cost the overhead of doing the the required logistics to make the type work out also increases So I'll discuss that a little bit as we go through the types Box does not provide interior immutability if you have a shared reference to a box Then you cannot mutate the thing inside the box Is there a way to tell that a supposedly immutable struct has some some stuff inside it that is mutable like a cell? No You do not know Externally from a type whether it has interior immutability All right, so let's dive into cells. So cell is kind of interesting If we look down at what cell provides you'll see that perhaps unsurprisingly You can create a new cell and you give it a value of some type T You can also change the value by calling set and you'll notice the set has a an immutable reference But it still allows you to modify the value that's contained within the cell It also has a swap method which lets you take References to two cells and swap the values are inside of them There's a replace. There's an into-inner that consumes cells So assuming you have ownership of the cell which of course means that there are no shared references And then we get down here and you'll see that cell Where the type is copy has a get method and you'll notice the get does not give you a reference to the thing inside the cell Instead it copies the thing that's inside the cell and gives you a new copy of that value But it you do not get a reference inside the cell And in fact if you were to look through all the different methods on cell You would see that there is no way with cell for you to get a reference to what's inside the cell You can replace it you can change it and you can get a copy of it But you can never get a pointer into the cell itself and this turns out to be important think of it this way if if there's no way for you to get a reference to Cell to the thing inside a cell then it's always safe to mutate it Right because if no one else has a pointer to it then changing it is fine Does that make sense? Think about that for a second if no one if we know no one else has a pointer to the value we're storing then changing that value is fine Right and that is what cell tries to provide just by virtue of the of the method signatures that it provides It never gives one out and therefore it knows that no one has it The other restriction that cell has in order to make this safe is that cell You will see does not implement sync and what this means is if you have a reference to a cell You cannot give away that reference to a different thread and the reasoning for this is pretty straightforward If I had two threads that both have an immutable reference to the cell a shared reference to the cell Then both threads could try to change the value at the same time and that is obviously also not okay But if you have both of those restrictions if you know that there's only one thread that has a pointer to the cell Then you also know that if if I have a shared reference to that cell Then no one no one has a shared reference to the value inside the cell so it's safe to change it As long as I don't give out a reference to what's inside the cell There's a lot of words, so let's check in on whether that made sense before we continue All these guarantees are at compile time for cell by the way Alright, let me just open a window Question why can't we borrow as mute more than once for RC if it's own there's no RC here And there's no borrowing so this with a cell There's no there's you never use an exclusive reference in general with a cell If you have an exclusive reference to the cell you can get an exclusive reference to the value inside But at that point you can't get or change the value, right? In the example you mentioned you say if there's a single thread There's no need to worry about multiple references to a cell in that case. What benefit does using cell provide? So the benefit that cell provides is that you can have multiple shared references to a thing for example You can store Usually cell is used with something like RC where you want you want the cell to be stored in multiple places Or pointers to it be stored in multiple places like in some data structure like imagine a graph where some of the things might share a value Right, then you might have multiple references to a thing But because a single threaded you know that you will only be using one of the reference at a time And so what cell lets you do is in safe code. Let's you mutate that value Cell should usually just be used for small copy types. Yes You notice that you can only get the value out of a cell either if you have a mutable reference to it In which case why you probably don't need the cell at all or if Or if the value is copy and so you generally want to use cell with Types that are copy and that are relatively cheap to cop to copy out because that's the only way you can get their values Is there a sync version of cell? No, I don't think you can do this just with the type system if you don't rely on sync Okay, so let's try to implement cell ourselves. That's usually a good way to like Understand any of these things so a new lib and we're gonna call this Pointers why not Okay source lib we're gonna get rid of that we're gonna do a mod cell And we're gonna do All right, so we're obviously gonna need a pub struct. I guess Pub mod cell So we're gonna have a cell type. It's gonna hold a T We're gonna have to figure out what's inside here. Let's for now. Just assume that it's gonna be a T and We're gonna implement for cell It's gonna be a new Which takes a value of type T and returns the self and that gives us a cell that contains the value that was given We're gonna have I guess this is gonna be pub We're gonna pub fn Set we're just gonna take an immutable reference to self and a value T and It is going to do self dot value equals value. This of course currently will not work, right? We're trying to we're trying to assign to self dot value, which is behind a shared reference And so we can't modify it And we want to do a get self Which is gonna return a T And this is gonna be self dot value This is like the basic API we're going for right, but remember that part of the part of the call Well, okay, let's let's try to figure out how we might actually do this So at the core of almost all of these types to provide interior mutability is a special cell type called unsafe cell So if you go back to the to the browser, let me zoom in a little bit here that might help If you go back to cell you'll notice that there is one called unself Unsafe cell which lists itself as the core primitive for interior mutability and rust and core cell is Unsafe cell is is totally unsafe to use it really just holds some type and you can random you can get Raw exclusive pointer to it whenever you want and it's up to you to cast that into an exclusive rust reference When you know that it's safe to do so It's sort of a building block right so here we're gonna use cell unsafe cell So the value here is gonna have to be an unsafe cell. That's the only way that we can actually From an in from a shared reference that we can mutate something through that shared reference is by using unsafe cell Is there a classic example of when someone would want to use cell? It's usually used for smaller values like usually things like Numbers or flags that need to be mutated from multiple different places So for example, it's often used with thread locals, right? So with a thread local You know that there's only one thread accessing it and you might want to keep some thread local state like a flag or a counter or something but the thread local only gives you shared a shared reference to that thing because One thread might try to get the thread local multiple times and then sell as a good way to provide mutability to it Why does cell have an ass pointer method that gives you a pointer a raw pointer to the thing inside the cell Right, but trying to bring that back to a shared reference would be unsafe And so it's fine for cell to expose the raw pointer because you can't do anything with that raw pointer unless you write an unsafe block All right, so The value here is going to be just unsafe cell new value. So that's fine. That's not too bad and Here what we're gonna do is so unsafe cell has a get method and the get method takes a Shared reference to self and gives you a raw exclusive pointer to T right And so what we're trying to do is this And similarly here, we're gonna do dot get Star, right? So the code I'm writing now is currently incorrect and we're gonna see why it's incorrect so here we're trying to dereference a raw pointer and the compiler is telling us that that is unsafe and Rightly so right where we have a shared reference to this unsafe this T And there's the compiler doesn't know that it's okay for us to change that value It doesn't know that no one else is currently mutating that value under us for example It doesn't know that there's not some other thread somewhere That's changing this T that we're trying to dereference at the same time and so if we write unsafe here What we're doing is we're telling the compiler I have checked that no one else is currently mutating this value And if we just did this like this is just wrong that what we're currently doing is simply wrong Well, how do I want to do this Because even though we have said unsafe here, so the compiler accepts it the code is just wrong There's nothing preventing some there's nothing preventing currently the following from happening That's it's something useful to do this in a test-driven way, right? So let's do a Simple test here bad So this is gonna use super cell It's gonna do cell new Of 42 and then we're gonna do like a Threads spawn Threads spawn won't actually let us do this, which is a little awkward. Let's do like an ARC new of a cell new We haven't talked about ARC yet, but it basically lets us Share a reference to something across thread boundaries x.set 43 and In this thread we're gonna do x.set 44 sync arc There's a little bit of setup here, but Clone x Right, so currently nothing stops a developer from writing this code Right to start two threads. Well, it actually will be prevented, but that's annoying So here we haven't written anything here to prevent this from happening right to have two references to the same cell and then two different threads both called set at the same time and This unsafe is us just telling the compiler. That's fine But it's not fine right if two threads try to write to a value at the same time What value does that thing now have it doesn't have a well-defined value? And so this is not okay instead what we need to say is we need to Basically implement not sync for cell T Right, we need to tell the compiler that you can never share a cell across threads This is what we talked about for cell when we initially looked at the API for it The compiler has support for the syntax, but it's nightly only The way you get around for this for now is you basically stick a value in there that is not thread safe And guess what is not thread safe unsafe cell itself? so It is not sync and so we actually already get this Implementation because unsafe cell is not sync and therefore cell is not sync. So this is Implied by unsafe cell Which means that this unsafe is actually now okay? and this code will be rejected and If we tried to run that code, let me Get the compiler a little larger Fine, let's do this is this is just to get it to compile Err, fine. I was hoping to do that later But if I now If I now try to compile the test You'll see that it says unsafe cell cannot be shared between threads safely and specifically tells us within our cell type Sync is not implemented for unsafe cell, which means that cell is not sync And so even though we tried to pass it to multiple threads that did not work as intended This is a little bit of a roundabout explanation. So let's pause here before we move on This was a lot so let me try to walk through it one more time from the top the cell type allows you to modify a value Because through it through a shared reference because no other threads have a reference to it And so you can't have multiple concurrent modifications and Because you've never given out a reference into the value you store and therefore you can replace it just fine Right. So because get here returns a copy of the value that's stored inside We didn't we never gave out a reference and so even if we change the value We don't have to invalidate any references because there are no references outside And this is why this code doesn't compile because here that we're trying to mutate the cell from two places at the same time And to give another example of what what wouldn't work and what cell defends you against Bad to So here I'm just gonna have a single thread and show you why a single thread can also go wrong So here we're gonna do something like a Vec So here imagine that I do first is X Imagine that cell allowed you to get a reference out Right Then I could do this to get a reference to the first thing inside the vector and then I could do set Vec blank or whatever right and Now even though this is single threaded if I now try to say print first Even if I now tried to even though this is single threaded This is clearly not okay, right cuz here first is a pointer to this 42 Once I call this set that vector is gone as a first should be invalidated So we can't allow this code either and the way that we don't allow this code with with cell is by get not Returning a reference get only returns a copy and we never give out a reference which means the set is always safe all right, let's see if What we've done so far is makes sense Can you unsafely implement sync to show your test failing? Yes, I can Impel T sync for cell T unsafe So we can we can say that in fact we could if we want to Here's what I'm gonna do here. I've removed all the safety restrictions. So now these tests are both gonna pass Or sorry are both gonna compile right so if I now try to run the tests and run bad to Oh Right, I need to actually mark them as tests. That's a good idea So let me try to run bad to here Ooh That's interesting That should definitely not oh, I wonder it's because it doesn't actually get Deallocated Let's do box new instead For a string string is good. Oh So we're gonna replace it with an empty string and then try to print out the original string Why does this work I Think what's happening here is that is that the Even though the memory has been Deallocated the pointer is still valid. So if I try to do this that might why is it not printing this out? Yeah, so see here for example So in this test we create the string hello, we make our cell point to it We get a reference to that string. So this should now point to hello Then we change that value and now suddenly The pointer that we initially took out is now pointing to world instead even though that's a completely different allocation And so this should print hello, but doesn't Yeah, it's basically the allocator didn't release the memory So the pointer is still valid But you should hopefully you see why this shouldn't be okay, right? Because this is a pointer to this memory and once we change this once we allocate some new string here Then this memory should be de-allocated it goes away and this pointer isn't valid It happens to still be valid because of the memory system and that's why this isn't crash But if this was a larger more busy application, that would not be okay so we we want to disallow this and the way we disallow that is By never giving out a reference Right, so we want get to only work when the type is copy and then we give out a copy of that value And now this won't compile because you can't get that reference in the first place and This case the case where you have multiple threads. This one is probably not gonna fail easily Like it's gonna be hard to write this as a test that fails but the maybe the way to see this is What's a good way to demonstrate that this is broken There isn't really a good way to demonstrate that this is broken even though it is because the so the two threads are both going to Modify the value in place and the problem is you don't know what value. It's gonna be set to Maybe the way to do this actually is To have this be a Array It's gonna be zero a thousand twenty four zeros and this is gonna set it to be a Thousand twenty four ones and this is gonna set it to be a thousand twenty four twos and we're gonna do So we're gonna have one thread that tries to set the whole value the whole array to be one and one That's gonna set the whole array to be two Let's make this larger Take a little bit longer And then we're gonna wait for both threads to finish And then we're gonna print out The value that ends up being stored in there, so that's gonna be x.get. It's awkward I guess for I in start get reprint See what happens if we run this that doesn't seem right X dot get is not an iterator like this then So we scroll up here. We'll see that let's see if we find any that are broken All are all of these set to one because if so that undermines my point They might all be just awkward Actually, here's what we're gonna do. I'm gonna do Sort Sorry, this is Great Okay, this printed all of them as twos Sometimes though this really ought to Not do that make it a little larger Stack overflow you say I'm just trying to make it large enough that the threads start interleaving Apparently it won't let me do that. Okay. I guess in that case I'm gonna have to Argue why this is problematic. So the problem here, right is this thread is gonna be writing out this long Array of once this thread is gonna be writing out this long array of twos both of them are gonna take a while and We're allowing both of them to be modifying the same bit of memory, right? The same memory that's stored inside the cell at the same time So we have no guarantees that these threads aren't gonna be stepping on each other Right imagine like this thread runs for a while then goes to sleep Then this thread runs for a while and then it goes to sleep We'll see an interleaving of ones and twos in this particular case that the test when we run it in practice doesn't fail And the reason it doesn't fail is because of the underlying memory system being fast enough that these interleavings Don't actually show up, but this is something that can happen and then if it happens You basically can think of this as we're gonna end up with a corrupted array We're gonna end up with an array that contains some ones and some twos even though nowhere Did we set that to be the value right? We expect at the end of this test for the entire array to be ones or the entire array to be twos We do not expect it to be interleaved, but the way we've set this up that could happen if the threads start yielding inappropriately Yeah, the so there's a there's another way for us to demonstrate this which is Which would be this imagine that this thread does X X equals X one dot get X one dot set X plus one It's like that the silly way to do this right X to We're gonna do this. This is just gonna be a zero We're gonna do this Hundred thousand times Maybe this is a better way to demonstrate it. You're probably right and we're gonna assert That when we get out the value it's gonna be two A hundred thousand right Because each thread is incrementing by one a hundred thousand times So hopefully by the end it should be two hundred thousand and we're gonna stick in here a thread yields It's too tight of a loop The the computer is too fast. I doubt this will actually pick up. We'll see try to run this great. It failed Fantastic it expected the value to but it will still fail great it failed It expected the value to be two million and instead it was this lower number and the reason here is because the threads are The threads get to race they're both modifying this value And so some of the modifications end up being lost because one thread writes its value and the other thread writes its value And they both read before they both write before they read again Hopefully I've sufficiently Convinced and confused you That this implementation is necessary So specifically because sell if we declare a cell is not being synced Then now this code won't compile because it'll recognize that we're trying to share the cell across threads and that's not okay And so really if we if we want to do this sort of the proper way, we're gonna document why this is safe We know no other No one else is concurrently mutating self dot value because not sync and We know we're not invalidating any references because We never give any out similarly here safety We know no one else is modifying this value since This since only this thread can mutate Because not sync and it is executing This function instead right a given thread can only do one thing at a time and Because we know it's not shared between threads and we know that it's calling get because we're in get that means set is not being called And so therefore this value is not being modified Okay, so hopefully This should explain why cell is safe Actually, I guess I can leave those tests in in theory, but Okay, the cell makes sense We went back and forth under on a bunch. I apologize for that, but Sometimes happens What is the point of allowing T to be non-copy if we only have the get method for copy types? Um Or you're saying up here Why not do this and require it or why not do this and require it for the whole thing? We could totally do that Generally the only thing that requires it is the get method and So far the like idiomatic rust way is to only put the bounds where they're needed and This is usually you usually don't want it on the type because then Any type that contains the cell would also need the copy trait and it ends up just putting a bunch of extraneous bounds all over the place Putting it only in the most constrained space place means that Callers only have to put it where they are actually using the cell themselves Will you be able to give a quick explanation of what's under the hood in unsafe cell and can you explain why we need Unsafe cell and cannot just unsafely cast the rest the shared reference to an exclusive reference Yes, this is an important point the only way in rust to Correctly go from a shared reference to an exclusive reference is with unsafe cell You are not allowed to cast a shared reference into an exclusive reference You it's just not allowed the only way is through unsafe cell the reason why that's true is a little complicated and comes down to the way that the rust compiler optimizes your code and how it interacts with LVM for example But you are never allowed to cast a shared reference To an exclusive reference just ever except by going through unsafe cell If you do the compiler might optimize your code in such a way that it breaks All right, so that was cell now. Let's move on to ref cell So ref cell is a little different Unsafe cell is really just a it's just a t but with the compiler has special knowledge of unsafe cell Cell does not have any special component instructions. No unsafe cell is a special type But cell is not All right, so ref cell let's go back to our Documentation here. So ref cell is a little different You know if you look at the documentation, you'll see it says a mutable memory location with dynamically checked borrow rules So normally in rust The all of your borrow checking is done at compile time, right? The either you have a shared reference in which case you cannot mutate or you have an exclusive reference in which you Can mutate but those are all determined at compile time what ref cell lets you do is basically It it lets you check at runtime whether anyone else is mutating This is really handy if you have a value that appears like you're traversing a tree or something or or you're traversing a graph where there might be cycles and you are Like it might be that earlier in your recursion You already got a mutable reference to this thing but later down you're trying to take a mutable reference to the same thing And ref cell will catch these cases But imagine that you have a graph that you know has no cycles. So you know that You have like checked your graph that it has no cycles But this is a runtime check and so you know that you can always get a mutable reference to any given node But the compiler doesn't know that because the graph is not known at compile time So ref cell is a way for you to get safe dynamic borrowing dynamically check borrowing Yes, exactly. So ref cell is a good use case for for things like graphs and trees And in fact ref cell is a fairly straightforward. So ref cell is a type that is basically also just a Use a cell it is basically also just an unsafe cell of tea But it also has this special value that keeps track of How the thing is currently borrowed we're gonna call this I guess flag References and we're let's make it an ice ice And what we're gonna do for ref cell Is that ref cell is gonna have a new shake a value Tea and gives you a cell friends like standard setup stuff, right? And The reference is basically gonna be a reference count it's gonna be how many references have we given out to this thing and of what type and Think of it as a positive number. It's gonna be how many shared references are there and of course that number can be any value Like there can be zero to a million references or to however many It's basically infinite, right? And there can only be ever be one exclusive reference because this is what the rust ownership system requires that we guarantee and so we're gonna have a method called borrow Which is gonna take a shared reference to self and it's gonna give you an option Let's for now go with an option reference to tea and then we're gonna have a borrow mute Which is gonna do this Illusionally, let's just make these be done. So this is the basic API We're going for where if you try to mutably or exclusively borrow a Ref cell that has already been borrowed whether exclusively or not then you'll get a none back because the compiler is not We're not willing to give you another exclusive reference because that would violate the reference rules and rust If you try to borrow then you will get a sum unless an exclusive borrow has already been given out Because we want to guarantee this contract rust has of if you have a shared reference Then there are no exclusive references and if you haven't if you get an exclusive reference There are no shared references and so that's why this has to be an option This is API roughly makes sense We're gonna make it be eye size because it needs to also handle the case when there are exclusive references It can be an enum instead if you'd rather have it be an enum Like in fact if we want to be a little bit more express explicit about this we can say Ref state which is gonna be either It's gonna be either Unshared or shared with some count or exclusive And this is gonna be a rough state Maybe that's easier Why can't we use the borrow and borrow mutrate here? The borrow and borrow mutrates are for something very different Also, you'll see why in a moment Okay, so it looks like the rough API makes sense. So let's try to actually write one of these, right? So what is borrow going to do? Well, yeah, it's gonna depend on the state of self Basically, if the if self dot state Is ref state unshared Then we're gonna give out And then we're gonna give out and otherwise We're gonna give out nothing, right? So if it's currently hasn't been shared then we're willing to give out the value otherwise, we're not and Borrow is gonna be a little bit similar in that it's going to be if it is currently state If it's currently unshared Then we're fine to give it out, right? Because no one else has a reference If it's currently shared with some number, then it's also fine to give out and if it's currently Exclusively borrowed out then it's not fine to give out. So if we've given out an exclusive reference If we've given out an exclusive reference then it's not okay to give out a shared reference Similarly, if we have given out any reference then it's not okay to give out an exclusive reference Right, so this is really just us typing up the rust rules for references I don't think this is deviating from the ref cell API. I've simplified it Because I want to explain why we can't simplify it this way as you'll see shortly So there's one thing that's Yeah, so there's one thing that's obviously missing here and that that is that we're not doing any ref counting Right, like we're never changing self dot state. And so this is clearly not okay, right here if we give out an exclusive reference that we need to set self dot state to be exclusive right and and similarly if If it was unshared, but we give out a shared reference to it Then we need to set that it is now shared And if it was shared and we give out another shared reference Then we need to update the reference count Right, so that that's certainly one thing that was missing so but of course this won't actually work because we have a shared reference to self and We're trying to mutate something that's inside of that shared reference and so this won't work Well, one thing you'll notice here is that we're modifying ref state here in a way That's not thread safe right if you had multiple this is basically another instance of the problem We saw before if multiple threads were allowed to borrow at the same time They might both read the old n both set the new end to be n plus one But you would end up losing one of the increments, right? So this can't this just this type just cannot be thread safe So we're this also just like sell is not sync but if this type is not sync and we need some way to mutate state In a place where it's not thread safe anyway. Well, we can just use sell So in fact what we can do here is we can make this a sell Because if you think about what we talked about for sell sell doesn't Like the restrictions for sell is that it's not thread safe, which is fine because ref cell is also not thread safe It doesn't allow us to get references to the thing that's inside and that's fine ref state can easily be copy right It's nothing really preventing us from that Great. So why don't we just use sell? right because sell gives us the ability to Mutate something through a shared reference. So it gives us exactly the thing that we need So someone's asking could you use an atomic ice ice to make it thread safe? We'll talk about that when we get to mutex mutex is basically a well RW lock and mutex are basically thread safe versions of ref cell. So we'll get to that later Okay, so let's use create sell use the cell type we just made because Why not use the thing we made ourselves? All right, so sell new this is going to be a dot get and This is going to be a dot set Sell seption. Yeah, that's right Okay, so now we can write the safety argument here right so No exclusive references have been given out since state would be exclusive and similarly here, this is also safe for the same reason and No exclusive references Actually, that's not even necessary and down here for this no Other references have been given out since then state would be shared or exclusive When using something like rayon with ref cell cell make no sense Yeah, with rayon you need something that's thread safe and this is not thread safe So you need a mutex or RW lock or some other sync primitive Right. So as chat just observed the problem here is that we're increasing these but we're never decreasing them So if you wrote this code the moment you exclusively borrow something you can never borrow it again Which seems kind of useless, right like The moment I stop having this exclusive reference I want to be able to get shared references again Otherwise this whole data structure is kind of useless and so this is why we can't really have these be just shared references and exclusive References because we have no way to track when they go away And so really we need some other type here as we're gonna do is we're gonna use a ref type And a ref mute type and let me define those down here So a ref is gonna have a lifetime that points to the ref cell Right because when the ref cell goes away We certainly need to make sure that all of the references have gone away Otherwise those references would have dangling pointers. So that's not okay And we're gonna have to figure out what actually goes in here and same for ref mute and What really are these well? These are really really they just contain a reference to the ref cell That's all they really need to hold And now what we can do is we can implement Cell Implement T Drop for ref Right, so we want to implement drop for it And when you drop a ref then we want to decrement the reference count So what we're gonna do is Let We're gonna do a match on self ref cell get and If it is Actually dot state Dot get and if it's a ref state Exclusive then that should be impossible right The ref is a shared reference and so the state must be shared because otherwise how do we get here in the first place? So that shouldn't be possible If it's a ref state Unshared that should also be impossible Because we have a shared reference. So clearly it's not unshared Right, so these two aren't possible But what about shared well if it was marked as being shared with one reference well, then Now it is unshared when this thing goes away ref state unshared And if it's shared with some higher account and I guess Well, let's leave that for now then we're gonna share it with Account ref count one lower does that tracking make sense right so anytime we give out Every time someone shit borrows a shared version of our inner value then we increment the count and We return one of these refs and when that ref is eventually dropped Then we decrement the count and set it either to n minus one or to unshared if there are now no shared references All right questions about this Right, so this ref type. Let's just make this not be Return sum of ref Where the ref Cell is self and same thing here But now if the user gets a ref, how do they actually get to the tee? I'd like previously we gave out a shared reference to the tee, but now they're just given this weird ref type and Really if they borrow what they want to do is get to the tee And the way that we solve this problem is that we implement the D ref trait. So I'm gonna Stick that out for a second. So we're gonna implement standard ops D ref for ref of tee and The toy so D ref is basically the trait that gets invoked whenever you use the dot operator So if you have something of type T and you do like that value dot and then some method Then if T doesn't have that method, but D refs to something that does then the D ref trait gets called it's basically a way to get Sort of automatically follow deeper into a type And this will make a little bit more sense once you see the signature, right? So that for the D ref trait What you're saying what the D ref third requires you to give is the following signature Given a reference to self Give me a reference to this target type in this case the target type is tee What this allows you to do is if you have a ref of tee Then you can call any method that requires a ref of tee on it. It basically it D references into that inner type tee So basically in this point ref is a smart pointer, right? It's a pointer that D references that is really just a transparent pointer to some inner type But it has additional semantics when you drop it Which is basically what a smart pointer is and Then the question becomes how do we actually get the value? Well doing that is pretty straightforward We just get the value inside the ref cell. We have to update the safety argument a little though To say that The Ref is only created if no exclusive references have been given out It's good to get into the habit of like writing these out Since we Once it is given out State is set to shared so no exclusive references are given out so D referencing The D Referencing into a shared reference is fine is the argument here D ref automatically does the arrow operator from C. Yeah, you can think of it that way And we basically want to pull the same trick for ref mute, right? So let me go ahead and copy all of what we just did for ref We're gonna make this ref mute and ref mute works much the same way It will also implement D ref Implement D ref, but it also has to implement D ref mute. So D ref mute is a similar trait to D ref except that it Says that you can go from a mutable reference to the smart pointer type and Get a mutable reference to the inner the pointed to type And this is true for ref mute, but it's not true for ref Right, so the reason it's not true for ref is because it can be multiple refs to the same value And so us giving out a mutable reference to the thing inside would not be okay Because that way if I have a ref to a thing and you have a ref to a thing and we both try to do D ref mute We would now both have a an exclusive reference to the inner value Which of course is not correct that should never be legal and rust Whereas with ref mute here we can we can write the argument better, right? Which is a ref mute is only created if no other References have been given out Once it is given out state is set to exclusive So no future References are given out So we have an exclusive Lease on the inner value so dereferencing is fine Mutably or exclusive mutably dereferencing is fine and the safety here is See safety for D ref mute Is it common practice to write safety comments for every unsafe use? Yes, I highly recommend you do this People vary a little bit in how they do this You can look at the stream that I did a while back porting Java's concurrent hash map to rust We did a lot of this but in general you should make sure to document all your safety requirements Both where you write unsafe and for the module as a whole Okay, and then we need to implement drop for ref mute and this one is also pretty straightforward Here shared or unshared should not be possible right It should not be possible for us to get for the ref cell to be in a shared state because we have an exclusive reference And it shouldn't be possible for it to be unshared because we have an exclusive reference So it must be in the exclusive state And now we're gonna be when we drop our exclusive reference We now know that it's unshared and of course here. We now need to do ref mute of ref cell So and notice here that there's nothing stopping people from writing code here that will crash Right, they can unwrap this option all they want But there's no way for them to get two exclusive references to some inner value at the same time Okay Does ref cell roughly makes sense. Hopefully this was a little bit more cogent than the explanation of cell Should borrow mute return a ref mute. Yes. Does it not? It does Great. They both make sense. All right, perfect So now we're gonna get to RC and RC is a little bit trickier So let's do pub mod RC Okay, so what is RC? Well, let's go back to our documentation here So RC is a single threaded reference counted pointer and the type RCT provides shared ownership of a value of type T allocated in the heap Invoking clone on RC produces a new pointer to the same allocation in the heap and when the last RC pointer to a given allocation is destroyed The value stored in that allocation often referred to as the inner value is also dropped Shared references and rust disallow mutation by default and RC is no exception You cannot generally obtain a mutable reference to something inside an RC if you need mutability put a cell or ref cell inside the RC Which is what we've already talked about before So an RC is like in some sense similar to a ref cell in that it keeps count of references But it's dissimilar in the sense that it never provides mutability all it does is Allow you to have multiple shared references to a thing and only deallocated when the last one goes away This is also useful if you have things like Usually this useful in data structures where you might have one element to be present in multiple places so imagine a It's the best example of this Imagine that you Have some string in your program and that's there's a or some large like binary blob or a configuration or something You don't want to keep multiple copies of that blob around in your program You just want to keep like pointers to it But the problem becomes How do you know when to deallocate that big blob? Well, the answer is when all the pointers go away, but how do you know when all the pointers go away? And this is what RC gives you But crucially RC is also not sync It's also not send we'll get back to what that means in a second But basically RC is not thread safe So it will only do reference counting on a single thread But even on a single thread this is useful often again in the context of data structures or things like graphs How does RC handle cyclic references? It doesn't the if you have a cycle then the cycle just prevents it from being deallocated generally though Especially if you look at the standard library implementation of RC you have weak pointers and strong pointers We're not going to implement them because they're not that interesting But basically the difference is a weak pointer will not prevent the thing from being deleted Whereas a strong pointer will so if you have if the strong pointer account goes to zero then the thing is deallocated and weak smart pointers Will you basically need to upgrade them to a real pointer before you use them and that upgrade will fail? alright So what is RC well RC is basically a pointer to some type T Right that's stored on the heap. It needs to be stored on the heap because if I have multiple functions in my code are all Referencing this this type that stored somewhere It can't be on the stack of any given type of any given function Because when that stack frame goes away when that function returns the value would disappear for all the other places where I have it as Well, so you can sort of think of this as it has to be a box T Of course, it can't actually be a box T because if we clone the RC We're going to clone the box and cloning the box clones the T right, so really this is just going to be something like a value again and It's really going to be a pointer to a T And then what we're going to do is we're going to implement clone For T and notice that we don't actually require here that T is clone And the answer should be there the reason for that should be apparent, right? Because when we're cloning the RC what we're really doing is we're increasing the reference count We're increasing the reference count But we're not actually copying the inner value. There's only one of the inner value And so the question then becomes where do we keep the reference count? We can't keep it here, right? If we keep it here, then each clone of the RC would have its own reference count So how would we ever know when the count goes to zero? Instead the reference count has to be in the value that is shared amongst all the copies of the RC So we're going to do is we're going to define like an RC inner and The RC inner is the thing that's going to actually hold the value And in addition it's going to hold the ref count and when you clone what we're actually going to do is do a We're going to get a reference to the inner thing we're going to increment the ref count and then we're going to return an an RC of Just another RC Right, and then we're also going to implement D ref the same way that we did before for RCT And it's going to D ref into the inner type T and this is going to be unsafe right because there's a This is a raw pointer So you'll see we'll I'll explain why we need these unsafe blocks in a bit so The problem we run into here right is that if you have an RC the compiler doesn't know whether this pointer is still valid Right, so think of this as if we define new this might become a little bit clearer so new is really just going to do Inner is going to be box new RC inner Where the value is V and the ref count is one which is the current thing we have And then it's going to return an inner which is going to be box into raw of inner Right, so we're going to do a heap allocation where we're going to stick to this like shared state We call this shared instead of RC inner if we wanted to and then we're going to Box into raw sort of consumes the box and gives us a pointer to it And the reason we want to use box into raw rather than just De-referencing it is because otherwise if if I hear just did this Right, then when this scope ends then the box gets dropped and so the memory gets freed And so that wouldn't be okay. We've we needed to not drop the box even though we don't have a box anymore And so we do that by doing this But of course inside of the D ref Let's move the clone down here for a second inside of the D ref Here we're just saying take this random pointer that's inside of this RC and De-reference it. It's fine and then give back a pointer But the compiler doesn't know that this is still valid it doesn't know that the box that we initially allocated hasn't been freed since For example, if we had written this the way I wrote it earlier, right if I had written this as this This same code would have compiled but it would have been wrong because when this function returns the box is freed And so this pointer that we stored inside the RC is invalid and so this de-reference would be invalid And so this needs to be an unsafe unsafe block and what we're asserting here safety Is self.inner Is a box that is only de-allocated when the last RC goes away We have an RC therefore it has Therefore the box has not been de-allocated So D ref is Let's see RCs are quite useful when building GTK apps since passing a reference to a closure is somewhat icky wrap your value in an RC clone and send your closure Yeah, in anything that's like single threaded which is often GUI loops for example RC is great for this kind of stuff If you could explain the difference between reference types such as ref muti star muti and star consti So star mute and star const are not references. They are raw pointers So in rust there are a bunch of semantics you have to You have to follow when you're using references like if you use the ampersand symbol an ampersand Alone means a shared reference and you have a guaranteed that there are no exclusive references to that thing And similarly, if you have a an ampersand mute an exclusive reference, you know that there are no shared references The star versions of these like star const and star mute do not have these guarantees If you have a star mute, there may be other star mutes to the same thing There might be star const to the same thing You have no guarantees, but you also can't do much with a star If you have a raw pointer The only thing you can really do to it is use an unsafe block to de-reference it and turn it into a reference But that is unsafe and you need to document why it is safe The difference between star const and star mute is A little fuzzy But the basic semantics there are a star mute A star mute is something that you is usually something that you Might be able to mutate Something you might have an exclusive reference to Whereas a star const is intended to signify that it will never be okay for you to mutate this And so for example in general, you're not able to go from a const pointer to an exclusive reference But you can go from a Mutable pointer to an exclusive reference What does box provide for us? The box here provides us with heap allocation All right, that's what lets us go from this rc inner which would otherwise be on the stack to a pointer that is on the heap Which is what we store here Okay, so for for the clone here We're going to increase the reference count But here we have the same problem as we did for ref cell right, which is we have a shared reference to self But we need to mutate something inside of it And so here lo and behold the problem is the the answer is the same thing that we've done before It is our friend cell Right if this is a cell U size Then now What we can do is ref count C is inner dot ref count dot get And then we do inner dot ref count dot set C plus one Isn't unsafe a pretty weird keyword name It just means something the compiler cannot guarantee is safe not that it's actually unsafe Yeah, the unsafe keyword is a little weird because really what it means is I have checked that the stuff inside the brackets is safe. It's like I as the programmer certify that this is safe. So it's not really unsafe It's like in some sense saying that I acknowledge that this code seems unsafe, but it's actually safe So I agree with you. It's a little bit of a weird keyword name Um, so we have the same problem for Rc as we did for ref cell, which is when an rc goes away. This is the smart pointer part, right? We need to make sure that when the last rc goes away Then we actually deallocate otherwise it's going to be a memory leak, right? Otherwise you keep cloning your rc's eventually all of them go away But nothing here actually drops this box and so the value is just going to live on forever on the heap Which is obviously not okay So we need to implement drop for rc And so the question here becomes should we when you drop an rc should we or should we not drop the inner value? And so what we're going to do here, right is we are going to Check what the count is If the count is one We are the only reference We are the only Rc left and we are being dropped For after us There will be no rc's and no references to T Otherwise, uh, there are other references. There are other rc's So don't drop the box Right because other things are going to need it But up here, um, we are the last rc. And so at this point we need to actually drop the inner value And what are we going to do that? Well, there's box from raw Which takes lo and behold, uh, a Let's do a drop in here Which takes a raw pointer and gives you back the box And we're going to drop it immediately What is the relationship between box into raw and box leak? box into raw Gives you a raw pointer that you then can do whatever you want with including mutating through it box leak Gives you a static A static shared reference to the to the heap memory because when you leak a value, right? It's going to live on the heap until the end of the program And so giving a shared static reference to it is fine because that shared reference will indeed Always be valid as in what static implies But you can't then mutate through it, for example Because it's just completely shareable You'll see that this code doesn't actually compile and the reason here is in the difference between mutable or star mute and star const box from raw requires that we give it a pointer that Has that that is not star const that is star mute instead The contract here as I mentioned is a little like fuzzy But basically what they're trying to encourage here is that you don't take That that you don't take some raw pointer that might be shared and Uh and try to turn that into a box now There are some fairly subtle thing at things that work here And this ties into Something called coherence in rust Um Well, sorry not coherence, but variance in rust coherence is another beast But this ties into something called variance in rust Is one of the primary differences between star mute and star const It's not something you will usually run into so i'm not going to dive too much into it here But basically we need to give it a star mute and currently we're giving it a star const We could just make this a star mute but instead what we're going to do is The standard library has this really neat thing called non null So non null is so you'll notice here it mentions variance The primary reason to use no non null though is for optimization purposes Which is basically if the compiler knows that a pointer can't be null Right as a star mute can be point to can be a value of zero can point to nothing But a non null the compiler knows that the the pointer is not a null pointer Which means that it can use the null pointer as an extra value So for the example they give here is if you have an option non null Then the compiler can use the null pointer to represent none So there's no overhead to an option non null The other thing that's nice about non null Is that it's sort of like a star mute So we give it a star mute. This is where we get from box from raw And we can use it to get back this This star mute t which is what we need for into raw So let's go ahead and use that instead use standard pointer non null So this is going to be oops. No, this is going to be a non null Of these And then here we're going to do this is going to be a non null new unchecked And here of course the safety argument is box Does not Give us a null pointer Because the box actually does give us a heap allocation and now what we're going to do here is We can use the unsafe as ref method on non null instead of having this star ampersand star thing So as ref and same thing here and now down at the end here Uh We can now do self in our as pointer And this is obviously is unsafe right the compiler. This is another case where the compiler doesn't know The compiler doesn't know that We have the last pointer and therefore that it's safe to To turn this back into a box and drop it But we know because we know that we're keeping the reference count correctly Can't we leak a mutable reference to the value in ref cell by calling deref on ref mute Storing the return value somewhere and then dropping the ref mute No, so this is a good question. Let's go back to this briefly Um This ties into the way the deref and deref mute works So the observation was why don't I just get a ref mute call deref mute Take the pointer that I get the reference that I get back the mutable reference and save it somewhere And then drop the ref mute and then use the the mutable reference This won't work and the reason is because Because of how rust deals with lifetimes. There's an implicit lifetime here of this Which is the mutable reference that we return Live only as long as the mutable reference to self. So the mutable reference to the ref mute So if you tried to stick the mutable reference You got back from deref mute somewhere and then drop the ref mute and then try to use this mutable reference again The compiler would say no, that's not allowed. You're trying to use this mutable reference After the lifetime it's tied to has already expired because the ref mute has gone away As the compiler would not let you do that If we have a mutable pointer, why do we need a cell? We don't have a mutable pointer in rc So we have a mutable pointer, but it's not safe for us to mutate through it Is the difference a mutable? So this is why this is the difference between a mutable pointer and a mutable reference a mutable reference Guarantees that no one else is currently modifying it. It is an exclusive reference A mutable pointer has no such guarantee a mutable pointer is just this is a pointer with Certain semantics and we call it star mute It does not it does not carry the additional implication that it's exclusive Which is what allows you to mutate through things This is memory is size aligned in rust can the compiler fit other non null variants? In zero one two and three. This is something that's being discussed in the unsafe In the unsafe working group for rust. I don't think they've reached a verdict on it yet though ooh restream seems to be Duplicating a bunch of my things um someone asked why do I drop the inner before I do this? So this is me being paranoid At this line We're dropping the box and so any pointer into that box is invalid The moment we do this Inner is a pointer into that box And it doesn't get dropped until here or here And so technically this reference is no longer valid from this point forward And so if I didn't put this here Someone could later accidentally come along and write inner dot ref count minus equals one or dot set zero And the compiler wouldn't warn them that this isn't okay here They're accessing something through the pointer that we just deallocated, but they won't know that this is the case if I do this So that's why this this code compiles, but if I drop this this code no longer compiles or at least shouldn't Because the inner has gone away So that's the only reason all right So now we have uh cell ref cell and rc great Now let's talk about actually, no, there's one more thing we need to do um And this is going to make your head hurt. So I apologize for that in advance Uh, let me just type this out first and then I will explain why it's there Okay, so I think we need a test for me to demonstrate why this is a problem So this is Something in rust called the drop check And I don't want to get into it too much because it's fairly complicated Uh, but I'm going to try to cover it a little bit if you look at the nomenclan the unsafe nomenclan It goes into a lot more detail here Um, so here's what we're going to do We're going to how am I going to explain this? This is a good question. This is some really gnarly stuff that Um Usually you will not even need to know about Uh, but I'm going to cover it because we're implementing rc and we should do it properly um Imagine that I write the following code by an x and then I say x is And y is rc new This I'm going to try to explain What goes wrong if we don't add the stuff I added and then I'm going to explain why that is the case um Well, there are actually multiple things um If I remove this for a second that might make it easier to explain I don't know how to explain this Yeah, this is um Let me try to explain it without explaining it Uh, it's going to seem weird, but uh, I think it's Too detailed to be useful um When we write this Rust if we don't have this marker here Rust does not know that this type owns a t All it knows is that this type has a pointer to a t But when this rc goes away, it doesn't know that there might be a t that gets dropped Uh, this matters if t might contain lifetimes So rust has this thing called the drop check, which I'm going to explain in very basic terms Uh, because it's fairly complicated, but the intuition is there um Imagine that I have uh, I have some type that Contains a reference and when it gets dropped it's going to modify that reference So I have some code that looks a little bit like this Struck foo T and it has a It has a v that's a mutable reference to t, uh, and I implement drop for foo itself Uh And let's imagine that this does like, um, oh, what does this do? This does like a v dot Frobify Some mutable function on v Uh, and now imagine that someone writes a main function, uh, and they create a new foo they create a A t String from hello, uh, and they create a foo Uh, and then they drop And then they drop the t And then they drop the foo This code is problematic Right, so here we create a string Uh, we create a foo that has a pointer to that string a mutable pointer to that string Then we drop the string at this point We cannot we cannot touch that string again, right because when t is dropped the string goes away But then we drop foo and dropping foo Calls a method on the string through its drop implementation But drop is implicit, right? So if I write Let uh foo and t Here what's going to happen is the t is going to be dropped first and then the foo because rust drop things in in reverse order But I haven't written drop anywhere And so the compiler when functions when any type gets dropped It has to assume that every use of that type Sorry, every drop of that type is a use of the type and any fields that it contains So even though I haven't written drop foo here There's an implicit drop foo at the end of the scope of main And rust is going to treat that as accessing every single one of its fields And so this means that this code is going to be rejected I'm just putting something here so that it'll compile So rust is going to treat this as dropping the When you drop the foo it's considered a a test of A use of all the fields of foo which includes the string And so if these were dropped in the wrong order rust would would actually catch this as a problem Right, it would say that the when foo is dropped it tries to access t, but the t has already been dropped This is what's known as the drop check Um, but it can only do that because it knows that foo Holds a t in here It it can only do this because it knows that a foo is dropped at the end of the scope Imagine that we did this with rc instead if I wrote rc new Of foo And now that rc is dropped When that rc is dropped rust looks inside of rc and looks does rc contain any foos In the old case Where we just had this rc doesn't contain any foo and so the the compiler is going to assume that when we drop a foo Uh, when we drop the rc no foo is dropped and therefore we don't have to check foos implementation of drop And that's not okay. That means that this code would actually be allowed to compile when it shouldn't Even though t has gone away because the dropping of this rc Will not count as a use of this string because the compiler Thinks that rc doesn't contain a foo By adding this marker So phantom data is a way to say there's one of this type, but i'm storing none of them. It basically tells the compiler Treat this type as though we have one of these in here Even though we only have a pointer to it This makes the compiler that lets the compiler know that we own something of that type And so when you drop an rc you need to treat it as dropping one of these Okay, that is a Complicated simplified explanation of a very complicated topic in rust Uh, I highly recommend that you go look at the nomicon drop check Oh, that's very bright. Sorry about that Oh So the nomicon on the drop check has way more details about what this check is About an escape patch that technically we need an rc, but i'm not going to go through it Hopefully that made a little bit of sense But i'm going to take some questions because I Agree that this is complicated what I just explained But I felt like I couldn't I couldn't not put this in there because without it rc is broken And I couldn't put it in there without explaining a little bit about it all right Yeah, so the phantom data tells rust that when you drop an rc An rc inner t might be dropped and you need to check that If we didn't have the marker rust would not assume this to be the case because it's only a pointer to one Would it be sufficient to have phantom data t instead of phantom data inner rct? Yes, although It's a good question I think this actually was a pull request to the standard library to change it from being just a t to an to like the wrapper that internally in the standard library we have I forget why that was changed Uh, I think it's to guard against someone accidentally writing an impulse drop for rc inner for example If someone wrote that implementation then this if we just had rct here the drop for rc inner would not be checked That's just off the top of my head. That would be the guess why This only needed this is only needed when t is not static Uh, yes, but we want to allow any t here, right? There's no reason why rc shouldn't work for other types um, so there's One other thing I want to mention for Uh rc in particular, which is if you look at the real definition of rc Over here, you'll see that the rc type Allows the t to be unsized so question mark sized here means it's opting out of The so rust normally requires that every generic argument is sized We talked about this in a previous video and question mark size is the way to say I'm opting out of that requirement um and sized We're not actually going to go through how this works internally in the stream because it's a little complicated And because you need some unstable features to actually support this bully um but If you are curious about the kind of stuff that Question mark size will let you do We'll probably do a future stream on trait objects that we'll go into it in a little bit of detail And you'll want to look up the Coerce unsized trait Which is a Which deals with some of the restrictions of why it's hard for you to implement rc fully yourself If you want to support dynamically sized types Uh The real question for me is that should this problem exist in the first place I'm feeling that if I write the same thing in c it will be clearer and I won't be wasting time in all of these So I want to stress here You will very rarely be writing the kind of convoluted stuff that I wrote right now in your own code What we're writing is a very low level primitive and rust right the rc type is a very low level smart pointer type And usually these concerns won't come up for you And it's important that you can write these restrictions in rust in c if you wrote this code The c compiler would never check Right and see the way this would manifest is you run your code and it randomly crashes at runtime In rust you have to think a bit a little bit more about the code to make the the types correct But if you do these problems are caught at compile time rather than runtime And so it's true in c you don't have to do this reasoning But in c You also don't get the benefit that this gets checked at compile time So I don't think this is wasting time. Uh, what is the difference between exclamation mark sized and question mark sized? exclamation mark sized means not sized and question mark sized means It does not have to be sized This is because the default is that everything has a sized bound. It's a way to opt out of that bound All right So the next thing we're going to cover in the last bit of time is the synchronous versions of these so If you have multiple threads Then the strategies we've written so far don't quite work, right? Um in the In the cell case if you have multiple threads that can mutate at the same time There just is no equivalent of cell because even though you're not giving out references to things Having two threads modify the same type at the same value at the same time is just not okay Um, so actually is no thread safe version of cell Uh ref cell is a little interesting. So In the ref cell we wrote Right, you have uh borrow and borrow mute and they return options You could totally implement a thread safe version of ref cell Um one that uses an atomic counter instead of cell for these numbers So it turns out that the cpu has built in instructions that can uh in a thread safe way increment and decrement counters Um, so you could do that in practice. This is usually not what people want Because if you borrow and get a none, but you need the sum Just because some other thread has it You would just have to like spin in a loop to get the sum that you wanted, which isn't great. Uh, and so the multi-threaded or the The synchronized version of ref cell is usually uh rw lock So if we look in sync, you'll see that it has a bunch of different types And rw lock is one of them and what a reader writer lock is is basically a ref cell where The counters are kept using atomics. So they're thread safe, but also borrow and borrow mute which in The reader writer lock are called read and write They don't return an option Instead they always return the ref or the ref mute But what they do is they block the current thread If the borrow can't succeed yet, so they block the current thread until The conditions are met. So for example, if you call borrow or the equivalent in ref in reader writer locks called read Then if there's an if a thread has exclusive reference to it It will block the current thread until that exclusive reference is given up And at that point that thread will resume and you'll have the shared reference Similarly, if you try to take the right side of the lock the exclusive part of the lock Then it will block if there are any shared references that are giving out And it will only stop blocking once there are no more shared references Mutex is sort of a simplified version of ref cell if you will where there's only borrow mute As you don't need to keep all these like extra counts for how many readers or how many shared references there are It's just either it some other thread has a reference to it or some other threads is not And it similarly has the blocking behavior where when you call lock on a mutex It will block until there are no other references to the inner value and at that point you're given that reference And it similarly has a a guard the same way ref cell does where you get back a ref mute And that ref mute when you drop it is going to decrement the count and let someone else go Okay, does the translation Oh, and um rc the synchronous the thread safe version of rc is arc or atomic reference count So if we go back here To sync you'll see that's an arc and this is a thread safe reference counting pointer And arc is pretty much exactly the same as rc Except that it uses these thread safe operations These atomic cpu atomics for managing the reference count rather than a cell the way that we did And this is why an arc is indeed Send and sync actually that's a good point our rc needs to not be sent I don't know whether this is already not send But this needs to be Not send Which i'm gonna mark as a Ooh, what types are not sent pointers are not sent Actually, I don't know if non null is send which means we may be fine So with a with an rc it's not safe to send it to different threads because the count is not Is not thread safe right so If I sent an rc to some other thread and that other thread Dropped the rc and I dropped an rc at the same time Both of us would try to use the cell to decrement the count, but that's obviously not okay Because cell is not thread safe And so the rc cannot be sent And non null is indeed not sent by default Can you guarantee thread safety across as a f of i boundaries or is that just The c code calls into you need to uphold some guarantees So across f of i boundaries, there are no guarantees. That's why everything across an f of i boundary boundaries inherently unsafe And any guarantees you want to give you have to establish by wrapping it in an unsafe block Why would you ever prefer rc over arc? Um rc is much cheaper. So the there's a cost to using these atomics they're much more expensive in terms of number of cpu cycles and Coordination overhead between cores. So in general you want to prefer the non thread safe versions if you can Because they are cheaper to to use they have lower overhead Is there an async rw lock? Yeah, I think both async stood and tokyo and The futures crate and the futures intrusive crate have Asynchronous mutexes all right so Now we've covered the oh, sorry. So Rc is indeed not sent because non null is not sent. So we're fine there Um, we have a couple of minutes left Uh, so in the last couple of minutes Let's look at The borrow module Um, so the borrow module is a little weird Because it's not it's sort of a smart pointer. We're going to look specifically at the cow type So the cow type is an enum that is either owned or borrowed So if you have a cow of tea then the cow either contains a reference to a tea or it contains a tea itself roughly Um, so think of this as either it contains a reference to a string or it contains the string itself The name comes from copy on right and the idea here is that Um Is that when you have let me pull up the here So cow implements deref so you can get a shared reference into the cow And if the cow is itself just if it just holds a reference to something else It just passes access through there But if it if it owns the thing it contains that gives you a reference to that But crucially if you want to modify the value Inside of a inside of a copy on right then if it's a reference you can't modify it because it's a shared reference So what cow will do the magic of cow in some in some sense is to if you require a right access You call like get mute and it's currently borrowed Then it will clone the value and turn it into the owned version So if you had a reference to a string it will clone the string and store a copy of the string instead Hence the copy on right and then it will give you a mutable reference into this thing Uh, the reason you often want cow and the the place it usually shows up is if you have some operation where Most of the time you don't need a copy because you're only going to read Um, but sometimes you need to modify it Right, uh, this comes up pretty often in um string operations. So imagine that you have a um What's the best example of this imagine that you have an escaping function? So you have something let's go back to lib Imagine you have something like an escape that takes a string And it returns a string Um, now sometimes when you're given a string there are so this is going to be like It's going to turn every like single quote into a backslash single quote It's going to turn every double quote into a backslash double quote Etc. But imagine that you're given the string food Right the string food just turns into the string food. There's no escaping. You don't need to modify the string at all And so it would be kind of sad if we returned a string here Because then when you're given a food you would still need to clone it even though you didn't need to modify it at all Because the signature says you're returning a string And this is where cow comes in really handy. Um, you can here return a cow Um, and of course the lifetime here is the same as the input And now what we can do is if we don't have to modify it if sort of already escaped if you will Then we can return a cow borrowed of s and we don't need to do any allocation And only in the other case do we do like a let String is s like two string This is mutable and we like do something to string Like add backslashes and then we return a cow owned of string So the benefit here is if we don't have to write if we don't have to change anything We just pass it through and only if we do have to change something do we do the cloning and do the mutation And if we do have to do the copy we also pass that ownership to The caller right they're going to get a cow owned that they can then operate on Including they then have ownership of it So why does from utf 8 lossy return cow, but the other utf variants don't yeah, so that's a good example um so in the string type the string type in the standard library has a A function called from utf 8 lossy And what this does is it takes a It takes like a bunch of bytes And it turns you a cow stir And the reason it does this Is that if the given byte string is completely valid utf 8 then you can just pass it straight through You can just cast it to a string reference and just pass it on through Right this should be borrowed Of like bytes s stir It's not quite what it does but like if valid utf 8 bytes And this um, and if it's not valid then what they do is they like walk Uh, they do like a bits It's going to be vek from bytes Because this is a slice right so this create a vector of bytes and then they're going to walk through bits Uh, and replace with uh this like Un invalid character Uh utf 8 symbol If not valid utf 8 And then they're going to return cow owned um sort of bits astring This is a little simplified little pseudocode right but The idea here is basically the same as what we saw for the escaping where if you don't need to modify then don't allocate And the cow type lets you do that The other from utf a types um usually allocate regardless and if they allocate regardless then Then there's no reason for the cow type. Uh, some of them are like Checked from utf 8 so they return a result and then there you never need the mutation so you can always return a reference um I think we've now covered most of the smart pointer types in rust right so we talked about Um, go back here. We talked about cell first, which is for uh, non thread safe Uh, non reference Uh interior mutability then we talked about ref cell, which is dynamic interior mutability Then we talked about rc, which is for Dynamically shared references So this is something where you don't know how many references that are going to be you don't know when the inner value is going to be dropped And that you only know that at runtime then we looked at the Thread safe versions the synchronized versions of those types And then we looked at cow, which is not really a smart pointer, but kind of a smart pointer, right? It's like a copy on right pointer That upgrades when you need it All right, I think that's all I wanted to cover Um There's a lot to digest here. So consider going back and watching the stream again When the recording goes up and see if you can tease apart some of the explanations, which are a little convoluted at times And apart from that Thanks for watching and hopefully I think next thing will probably be like a Bit of like a potpourri like a mix of different things. Maybe trait objects Maybe things like the borrowed trait um Oh, I forgot to talk about trait delegation. I'll probably make it into a subsequent stream Great. All right. Thanks for watching. Uh, I'll see you next time by If I can make this work nice