 Hi folks, welcome back. This one's gonna be a sort of a switch of pace from the previous video in that we're not gonna be talking about the sort of weight-free simulation stuff that we've been doing for the past few videos. I'll link those above if you're coming here for the on-demand video. And instead, what we're gonna do is try to implement a sort of a safe memory reclamation scheme that's designed for concurrency. I'll talk a bunch about that over the next five hours. But before we dive into that, I wanted to mention in case anyone has missed that I am writing a book. It's called Rust for Rustations. It is as its title implies, a book that's written for people who know Rust who want to know more about Rust. And so you can get that early access now, you'll get all the chapters ahead of time so that I wanted to mention that before we actually dive in. So the scheme we're gonna be implementing today is hazard pointers. Actually, let me go to this one. So hazard pointers are this really neat structure or algorithm rather for safe memory reclamation and concurrent programs. And in order to understand why that's important, you probably want to have watched the previous streams. Like it should be very obvious there why we need something like this. But I'll try to give a sort of very high level overview of why this is necessary in the first place. If you come at this from like, I know some data structure stuff but hazard pointers or concurrency looked cool so I just jumped right into this video. So when you write concurrent data structures that allow multiple threads to read and write a given data structure at the same time, you run into this problem where, let's say we have a, it's a good example of this, a linked list, a concurrent linked list. And one thread is walking the linked list instead of reading every element. Another thread is removing an element from the list. And now imagine that the thread that, or you don't even have to imagine, the thread that removes from the linked list wants to deallocate the thing that it took out of the list. Because otherwise you're never gonna deallocate memory and that's bad. And now you run to this really awkward thing where the thread that removed the item doesn't know whether the thread that's reading the list or indeed any thread is currently looking at the thing that it just removed. It knows that once it's been removed from the linked list, no thread that sort of comes along after that point is gonna find that node because it's been removed from the linked list. Like there's no way to actually get to it anymore. But readers that were around before it did the remove might still have a pointer to that thing. And you can't deallocate it until you know that no one else has pointers to it. And that's where we get into this question of concurrent memory reclamation. There are a bunch of different ways to deal with it. You can do things like reference counting or there's a structure called RCU that's a little bit more involved maybe. And reference counting is sort of the standard one that people reach for where anytime you grab a pointer you like increment the reference count for it and you only dereference it when the reference count reaches zero because then you know that no one has references anymore. That works really well. It does have some downsides though. The primary one among them being that in a concurrent program if everyone who wants to read or write any element in the data structure that you've constructed has to increment and decrement this one reference count number. Then as more and more threads try to access this data structure you end up with contention on that counter. So that counter is initially one which is whoever allocated the thing in the first place but it keeps going up and down, up and down, up and down as people read and write the data structure. And this can be really costly because now these threads have to sort of synchronize their access to that one counter value. And crucially readers have to as well if all you're trying to do is read every element then for each element you have to like increment the count read the thing decrement the count increment the count, read the thing decrement the count. And so you read operations become fairly expensive and they don't scale very well as a number of threads go up because more threads, more contention, lower throughput. And so that brings us then to these concurrent memory reclamation schemes that are designed specifically for being efficient in the face of this kind of operation where you want your reads to be fast and you're okay with your writes being a little bit slower because you expect them to be less frequent. And hazard pointers is one of these. There are a couple of others. The primary sort of other contender is Epic-based reclamation which is what is implemented by Crossbeam Epic for example. I won't go too much into Epic-based reclamation at least in this stream. We might do that some other time. We're just going to focus on hazard pointers here. But the basic scheme that they both use is that you have sort of a mechanism guarding access to a given pointer that doesn't necessarily require that you do like a shared synchronized counter like the way you do in reference candidate schemes. It was originally proposed in this paper from 2004 and we're going to look through this paper a little bit. It talks a bunch about the way you would implement it and some of the trade-offs that you're making. We're also going to be looking at Folly which is a Facebook open source library that has all sorts of interesting like data structures and algorithms. And one of them is this implementation of hazard pointers that has a bunch of other optimizations and features beyond what's in the paper. And this Facebook implementation is the sort of example implementation for a proposal to add hazard pointers to the C++ standard library which is this proposal over here. I think that's sort of stagnated a little bit but it does have some really good explanation of what's going on and sort of the trade-offs involved. And so we'll talk a little bit through that as well. It has this handy table too which is a comparison of deferred reclamation methods. And you see there's reference counting, split reference counting which is really you have multiple counts so that the increment and decrement of the different counters can be done in parallel rather than just having one counter but it fundamentally has the same kind of problem. RCU which we're not going to talk too much about and then hazard pointers. And so I'll leave this table up while I sort of pause here to see if there are questions before we continue. Let's see. Oh, I'm glad people like the book. That makes me very happy. Sounds like the flurry hash map. So when we did flurry which is another video which is a port of the Java concurrent hash map to Rust when we did that, you run into some of the same problems, right? Of how do you know when it's safe to deallocate? In flurry we used cross beam epic which has this epic based memory reclamation where you have like these guard objects that you hand out with any reference to a given thing. We're probably going to end up with something somewhat similar here with hazard pointers. We'll see which path we end up taking. So the problem is very much similar. Like we could have used hazard pointers in flurry as well rather than using epic based reclamation. Let's see. There's a reason we can't use cross beam epic for the weight free simulator. So there isn't really one difference is that with epics you have to use these guards which can get a little bit annoying to use. Like they tend to complicate the API a little bit whereas with hazard pointers you might be able to get away with not having that guard more often. So we'll see it how that plays out. In practice like you can see here as well, actually this isn't talk about epic based reclamation but there is some trade off where with hazard pointers you get slightly more eager memory reclamation than you get with epic based reclamation. So there are trade offs between the two that we might go into later too but in some sense the reason I'm doing this is because it's like an interesting implementation. It's an interesting problem and an interesting algorithm and hopefully we're gonna learn a bunch while going through it. And the other reason is that they're just like isn't really, well there are some hazard pointer implementations in Rust but they're all not really maintained at least as far as I can tell and I feel like it would be nice to just have like a solid implementation of it that we can make use of. There's a third reason too which is that the hazard pointer implementation that we're gonna be looking at is written in C++. Most of the other porting streams we've done have been porting from Java. I mean there's a port of Flame Graph which was Perl but it doesn't really count. But in this case like we're porting a C++ code base instead of a Java code base and I think that's valuable so that we sort of have more different things that people can look at when they're trying to learn Rust through these videos. Okay so now that hopefully the sort of problem area at least is clear like why we need something like this. Let's go through, actually let me, how do I wanna do this? Do I wanna do this by text or by graphics? So this paper from 2004 is actually really nice to read. It's written in a very informal style that makes it pretty straightforward to follow and it sort of makes all of the problems you run into pretty obvious. But it's also written with like C++ code inlined in the paper which makes it a little bit hard to follow. So I think what I'm gonna do is actually draw this. I think that's gonna be the easiest way to try to convey what's actually going on. So this is a good question. Are hazard pointers weight-free? No, hazard pointers are not necessarily weight-free. It depends on the underlying algorithm. In our case, I think the implementation we have will be weight-free, is that true? No, it'll be lock-free but not weight-free because we're gonna end up using a concurrent linked list that isn't weight-free. But if you use the weight-free list there then it would be. Okay, so here's the basic premise that we're gonna be dealing with. Let's say there's some area in memory, there's some value in memory over here, X. All right, ooh, I'm using, that's fine. So there's some value in memory X over here that we want to guard the access to. We want to make sure that if some threat, like there are a bunch of threads that are all trying to read X, right? So these are all readers. And then we have some thread over here that's trying to write X. It's trying to sort of set X equals Y, I guess, or change this value to Y. And the question is how it's gonna do that. In practice, the memory image doesn't really look like this for concurrent access, right? The way that it more commonly looks is something more like this, where you have an atomic pointer here that points to X. I don't know. I've ended up with a pressure-sensitive marker, which is real weird. That's why these end up with blobs. And then what happens in practice is that all of the readers have access to this atomic pointer over here, and they sort of, they load the atomic pointer, and then follow it, and then dereference whatever they get to. That's actually how this pattern ends up looking in memory. And similarly, if you're a writer, what you actually do when you try to like, if you're a writer over here and you're trying to set this to Y, what you would actually do is you would do a compare and swap of I guess this yellow thing from X to Y. And so effectively what you're doing is you're changing this so that it instead points to your Y value, right? And so that way any reader that comes along after you've done this swap is gonna see Y instead of X when they follow the pointer. Um, so the question then becomes, as a writer, how do you, what do you do here, right? Like when you do the swap and now any subsequent reader is gonna see your Y instead, then what do you do with the X? Because there's still like this reader, for example, that followed the pointer before you did the swap and is presumably still hanging onto X. Like how do you know when that thread goes away? And the way that hazard pointers solve this problem is that every reader, every reader is gonna have a hazard pointer. Okay, so this is gonna be a hazard pointer. And there's gonna be one for each thread like this. And what the hazard pointer is, is it is either empty or it is a pointer that the thread is currently using. So let's take the progress here of like, let's call these, these are one, two and three. So this is the hazard pointer for threads one, two and three. And now let's say this is thread one's hazard pointer. So initially it's empty because one doesn't hold a handle to anything. Then one reads this pointer, it reads the atomic pointer and it gets back X. Okay, so at that point, what it's gonna do is it's gonna set its hazard pointer to be the address of X. So notice the address of X is like this. It's not the pointer that led us to X. It's the value we loaded out of the shared atomic pointer. So the address of X is what we write into here. And then what the writer is gonna do is after doing the swap, it's gonna look through each of these. Let's see if I can draw an I here, right? It's gonna look at each of these and see if any of them contain the value that it's trying to deallocate, right? So it's gonna look for the address of X in all of these. In this case, because thread one is still holding onto this, it finds that there's already someone, sort of there's some hazard pointer is guarding the address of X. And that means we're not allowed to deallocate it yet. And so what that means is the writer has to wait to deallocate the allocate X. Can't do so yet. It can either sort of go do other things in the meantime and sort of remember X for later, or it can go in a loop, but we'll look into how you might choose one or the other of these. But ultimately the fact that some hazard pointers guarding this address means that this writer isn't allowed to deallocate X at the moment. So let's say that at some later point in time, either by spinning or by going doing something else coming back, like some time passes and thread one finishes with its, let's just go back a little bit here, finishes with its read of X. So it then wipes out, it has a hazard pointer again, sets it to null because it no longer needs that pointer to remain valid. Then the writer comes along again and does this sort of search again. This time it doesn't find any occurrence of the address of X. Like the address of X is not guarded by any of the hazard pointers. And therefore it's now okay to deallocate X because no one is claiming that they need X to stay alive and therefore we wipe away X because we now know every reader is either, every reader from the past must no longer be holding X or any reader to come in the future is gonna read the atomic pointer after we did the swap and therefore we'll be seeing Y instead of X. Okay, does that make sense so far? This is sort of the basic scheme for hazard pointers. There's some more implementation details but this is sort of the overall design that we're going for. If this is lock-free but not wait-free, could you use the wait-free algorithm to make this wait-free? No, there's like a recursive dependency here where the thing that turns a lock-free thing into a wait-free thing, depends on memory reclamation in order to do that. And so the memory reclamation can't depend on the wait-free thing. It's a circular dependency. Once everything has an error. Yeah, one thing you could do here, right, is like implement hazard pointers with like cross-beam epic. No, implement the wait-free simulation with cross-beam epic and then release that as one major version of the library and then release another version of the library that uses the wait-free simulation on top of the epic-based reclamation but you still end up like somewhere down at the bottom there, there's an implementation that's not wait-free. This really doesn't seem that different than epics. So it's a little different than epics because there's no explicit epic annotation. Like there's no epic counter here. Every value that gets deallocated isn't allocated to a particular epic. Rather, this is like set of sort of guarded pointers that the writers have to check. Rather than every time you deallocate, you sort of mark it with the current epic and stick it somewhere. And then at some point later on, you deallocate everything in that epic after all the readers have moved along. You can sort of think of hazard pointers as being a little bit more targeted than epic-based reclamation is because it's looking at which exact pointer are you using rather than in what epic did you read? This is also the reason why epic-based reclamation can be a little bit more efficient than hazard pointers is because it tends to do more like batch deallocations. But it's also the reason why epic-based reclamation tends to use more memory because you can't collect an epic until, you can't collect anything in the epic until the entire epic is passed. Whereas with hazard pointers, you can deallocate the moment there are no readers for that thing. So thread two and thread three can already see why. Yeah, the idea here is that two and three aren't currently reading it, haven't seen X, have seen Y. If either of them were holding onto X, then the writer would be blocked on those as well. Won't this cause the wait-free simulator to only be lock-free? It will. But it'll still be better than something that is purely lock-free because we're still wait-free for more cases. There are now a much smaller sort of set of cases where we actually end up being only lock-free, if you will. How can one thread know if another hazard pointer is holding X? So this is where, this is sort of the first implementation detail, which is, as you correctly observe, the writer needs to know about all of the reader hazard pointers, right? Otherwise it doesn't know how to do this, like look up check in the first place. So the way we're gonna do that is that all of these hazard pointers are basically gonna constitute one long linked list. So there's gonna be some pointer over here. This is gonna be the sort of hazard pointer list. And this is gonna be a lock-free linked list that just links together all the hazard pointers. And it's shared. So the idea is that if you want to do a read, then you grab a hazard pointer from the list, there you rather, you grab an unused hazard pointer from the list or if there isn't one, you allocate a new one and add it to the list. If you're a writer, you walk the entire hazard pointer list, right? And check all of them. So that does have to be shared between the readers and the writers and basically between every thread. Like this has to be accessible from anyone who wants to interact with this data structure. This of course also raises another thing that's kind of neat here, which is you can keep multiple hazard pointer lists. Like imagine that, what's a good example of this? Oh, just imagine that you have, let's say we expose this as a library, right? And there's like, the user creates two different hash maps, like concurrent hash maps. Each of which uses hazard pointers internally. Well, you don't really want a writer in one hash map to have to check all the hazard pointers in the other, right? There's no need to do that because a reader in this map just does not affect this map at all. And so they can have different hazard pointer lists. And this is sort of a nice thing that we want to expose. So it doesn't actually have to be a global pointer list. It just has to be a hazard pointer list for anyone who might read this thing or this set of guarded objects. The other thing that gets complicated here with hazard pointers, how do we ensure thread safety on the pointer list? Oh, this is just gonna be a standard like atomic link list. Like you just do a compare and swap with the head. And you never have to deallocate the things in the hazard pointer list. The, I think the trick, well, we'll look at this when we look at the implementation, but the basic trick is gonna be, when you're done with your hazard pointer in the list, you just like mark it as inactive and stick it at the head. And that way, if some other thread comes along and needs a hazard pointer, it just like picks up the one that you, like it reuses the one that you used. So we don't actually have to solve the reclamation problem for the hazard pointer list itself because it's basically gonna be a sort of, there's gonna be a reclamation, sort of a reuse thing where we avoid allocating and deallocating. Technically you can, right? Like so you can, I think there are ways you can safely reclaim from it, but in general, we won't want to. This hazard pointer list is gonna grow probably to the number of threads. Like that's the size it's gonna be. And at that point, it won't really grow that much more because you're not gonna be allocating another one if you already have one. And so there isn't really a reason to reclaim these. These are small anyway. Won't readers have to create linkless nodes on each read? So each read still needs to do an atomic operation. So as a reader, it is true that you have to have a hazard pointer in order to do a read. But you can A, you can reuse one across multiple reads. B, in general, grabbing one doesn't contend with other readers, right? So grabbing one is just gonna be look for one in the list that isn't used by anyone currently and then try to grab it. Or if all of them are used to allocate a new one. And unlike something like reference counting you're not generally gonna be like compare and swapping or fetch adding a value that lots of threads are all trying to access because you're already only gonna be doing it on something that's marked as inactive. It is true that there's a little bit of contention there on grabbing a hazard pointer but they won't have to like necessarily allocate a new hazard pointer each time or compare and swap it into the list. Okay, so there's one more thing that is necessary here which is a sort of race condition that I skimmed over if you will, which is this. Let me draw back our atomic pointer here. So it's currently pointing to X. And let's just take the case of a single hazard pointer. So there's only gonna be one reader and one writer in this case. And the writer wants to swap this to instead point to Y, right? In fact, it doesn't even, the one thing that's nice with hazard pointers too is that the writer can just do the swap first and then it just needs to decide what to do with X after the fact. It doesn't have to wait to do the swap. So it just does the swap and now once it's done the swap, like it's sort of done with the right and now the only thing that's left is the reclamation of X, right? So what do we do with this? And here a race condition happens. So let's say that the order of operations is as follows. Let me do this with green, why not? So number one, the reader reads the pointer. And it needs to read the pointer first because otherwise it doesn't know what to write in this box, right? It needs to read the pointer in order to know what value to put in there. Number two, the writer swaps. Number three, the writer checks the hazard pointer and it finds that it doesn't contain anything because the reader has not a chance to put the value at red into the box yet, into the hazard pointer. So it drops X and then for the reader uses pointer, which is two X. So this is a problem, right? We can't allow this to happen because if we do then now at four, right? The reader is accessing a pointer that's already been deallocated in step three. And so for this reason, hazard pointers require a second read. So the way that hazard pointers actually work is like one, load to store. And then three is load again. And if not equal, then go back to one. That was really ugly drawing. But the idea is that you load the atomic pointer, you store it in your hazard pointer and the moment it's been stored in the hazard pointer, you sort of go, oh, now it's safe for me to use it, but that's not actually true because the writer might have checked it in the meantime. So you go to check that the pointer is still the same as it was and only if it is is your hazard pointer safe because then you know that if a writer came along, a writer must have come along after you did the store in order to do the swap because you read it after you did the store, right? So if we go back to this case, right? So one, R reads the pointer, two, W swaps, three, it checks it and drops. Then at four here, R is not gonna use the pointer just blindly, it's gonna load again. It's gonna see that the pointer is now different because of the swap over here, right? And so therefore it realizes that its current pointer is not safe to use and it's gonna go back to storing the new pointer it got and then loading it a second time again. In the other sort of world where we go first, so let's say one is R reads the pointer, two is R reads pointer again, right? So I guess actually this is important, R updates the hazard and then three R reads pointer again and you can even imagine that like, now let's go with this one. Four, W swaps, my writing on this tablet is terrible today. Five, the writer checks all the boxes and at this point when it checks the box it realizes it's not safe to drop X and therefore it backs off. So you'll see that this handles both the normal case but also the challenging case. And the idea is that by doing these two reads and doing the store in between you know that if the read is the same before and after the store that must mean that either there was no writer until the second read or actually it must mean that there's no writer before the second read. There might be a writer coming right after but that writer must already see your store and so you're safe. Nice. So that's the basic algorithm that we're gonna be implementing here is this sort of guarded pointer bit and that includes like the linked list of hazard pointers, the guarding that the readers have to do, the writers having to walk that whole linked list. If we switch back now to, let's go back to the diagram from the standardization spec. This actually, I looked at this diagram and it's really unhelpful but I think there was another one. Let me see if I can dig that up. No, this is just a bunch of pseudo code. All right, that's fine. So if you go back to the Facebook hazard pointer implementation, actually I think I have that in my editor. It's gonna be a little bit nicer to just read. So it has a pretty good description of what hazard pointers are, why they're useful for, I think they're, when not to use is not as helpful as it could be. Like I think there are other cases where you don't wanna use it. It goes through the basic interface and here you'll see that they've made some design decisions for how this library should be structured that maybe we wanna do the same, maybe we wanna do something a little bit different. We might see if we can encode a little bit more in the type system than they were able to in C++. We'll find out about that. But they say in the hazard pointer library, raw hazard pointers are not exposed to users. Instead, each instance of the class hazard pointer owns and manages at most one hazard pointer. So I guess hazard pointer holder here is sort of like a guard that lets you read anything that supports hazard pointers. There's like some, rather than having users like right into the hazard pointers themselves. Classes of objects protected by hazard pointers are derived from a class template, hazard pointer object base that provides a member function retire. So what this is saying is you can only use hazard pointers to read from things that support hazard pointers. This is a little bit of a weird restriction that they've added, which is they want to say that you can only use hazard pointers on data structures that have chosen to be guarded by hazard pointers. Well, we'll see how this gets into the actual API design and whether we want to do the same thing. But this retire function is kind of interesting. So the idea here is that, remember in the linked list example we've had, right? Where the writer wants to swap X for Y. After it's removed X, and then it's trying to figure out whether it can drop X, right? It's removed it from the data structure, but it wants to know whether it can actually call like deallocate X. The retire function is sort of the in-between, which is I have now removed X, now I'm gonna retire X, and at some point in the future, I'm gonna like drop every retired object that is safe to reclaim. And we'll see this terminology a bunch throughout this code as well that you have things that are retired, which just means that they're no longer accessible, but may still be under guard if you will. They might, there might still be hazard pointers pointing to them. And then we have this notion of reclaiming, which is you take things that are retired and then you check all the hazard pointers and you verify that no one is using it anymore. And at that point you can reclaim that retired object. And indeed that's what it says here too, when an object A is removed, retire is called to pass responsibility for reclaiming A to the hazard pointer library. A will be reclaimed only after it is not protected by hazard pointers. And then we get a brief sort of overview of their API. The essential components of the hazard pointer API are the holder, which is this thing that manages a hazard pointer for you, right? There's get protected, which is a, ooh, spelling error. Someone wanted to do a PR to folly, which is a member function of the hazard pointer holder that protects an object pointed to by an atomic source. Right, so the function signature here is useful, which is imagine you have a hazard pointer, you have a, what do they call them? The hazard pointer holder, right? You have a guard or a, I guess holder is the word they're using. It has a method on it, get protected that takes like a reference to self, right? To the holder itself, a reference to my hazard pointer. And it takes a reference to an atomic pointer. So note that this is not a pointer to the final object. This is a pointer to the atomic pointer that we need to read. And the reason why we need to give that into the library is because the library needs to do this double read. So it needs to know which atomic pointer we're reading from. So that's why this takes a pointer to an atomic rather than the thing that's being pointed to. And it gives you back a reference to the T that's behind that pointer. But it will, that pointer will only be valid for as long as this hazard pointer exists or the hazard pointer holder. And this is sort of important, right? The idea here is that if we wanted to write this in sort of rust terms, right? We're gonna have a impulse, I guess, hazard pointer holder. We might choose better names for this. But it's gonna be something like get protected. Actually, I guess it doesn't even need to be this. It can be this. And it's gonna take a mute self and a reference to an atomic pointer to a T. And it's gonna return a immutable reference to T. And I think the lifetime annotations here are important, right? So we're gonna say this, not this and this. These are the lifetimes involved, right? So the idea is you have to, the reference you get back, you're allowed to continue using for as long as you still have the mutable reference to the holder. If the atomic pointer goes away, that's not actually a problem. We don't need to borrow that for the reference because we've already read the thing that the atomic pointer pointed to. So the lifetime of it is not important. But it is important that the holder still remains because otherwise the hazard pointer itself would go out of scope and the thing would no longer be protected. It's also important that this is a mutable borrow of self even though we get back an immutable borrow of the T. This mutable borrow of self is necessary because otherwise you could call get protected A and then get protected B, which like is gonna, the get protected B is gonna overwrite to the hazard pointer that's stored in the hazard pointer holder to point to the B, like it's gonna guard the B at which point the guard of A is no longer active anymore. So we really need to say that this has to be an exclusive reference. And I think that the rust type signature here really helps over the C one because here there's nothing that really enforces that you don't use this pointer that's returned after the hazard pointer holder goes away. It just sort of an informal contract of get protected. My guess is if we look at the definition of get protected probably says get protected. Oh, maybe it's not holder. Well, that doesn't really have any. Well, so they have a usage example, right? Which is you call get protected pointer is protected and then you call H dot reset and now the pointer is not protected. But in C land, there's nothing that prevents you from using the pointer here. Even though it's not protected anymore. Similarly, if you called get protected a second time would be the same kind of problem. Nice. Yeah, and here we can sort of see how this is gonna be used as well. And you'll notice the one thing that's nice about this method signature is that there are no guard objects. So in like the Epic Base Reclamation Scheme, you sort of have to like pass in a guard here and then the thing you get back is tied to the lifetime of the guard, which I guess is sort of what the holder is. We'll see whether this actually ends up being a nicer API than the Epic Base. We'll see, we'll see. We'll see every pointer as a hazard pointer. Yeah, that's sort of true. All right. And then this hazard pointer object base, which is this base class for protected objects. So this one I thought was kind of interesting. This is the T that is pointed to has to implement sort of this trait, if you will, and sort of see nomenclature it's a base class. But the idea is that anything that you want to be able to get at needs to implement this. And the primary reason why you have this is so that you can expose the retire function. The way we might do this is it's unclear whether we even need retire here. I think the idea is that, okay, so in order to call retire, you need to have access to the hazard pointer linked list. So something has to store that pointer to the head of the linked list. And what is going to store that? And I think the answer is, I think this is why they have the base class is so that they can store an additional field that sort of wraps the T that contains that linked list. So this would be something like, well, you can't have associated state with traits, which is a little unfortunate in this case, but we would have something like hazard pointer, I guess, target T, which is really just a sort of a real T and a hazard pointer, which is like a linked list of hazard pointer holders, right? Or something like it. And then we're going to sort of implement all the D rough stuff for hazard pointer target. So that's all fine. And then we're going to implement, I guess it's a little bit of a weird, I kind of feel like we don't want to do it this way, which is, I assume is why they structured it the other way, which is have a pub function retire on, which actually consumes self. And it is going to use self.has pointers and do the whole like retiring reclamation business on there. But if you think about it, like this is a little bit of a weird API maybe, because if you're implementing a data structure, it means that now you need to make all your atomic pointers, like anything that used to be an atomic pointer T, or like my type, right? In your data structure, now is to become an atomic pointer, has pointer target my type, which is a little weird. It's sort of unfortunate that you have to make that change. And so what they did in the C++ library with this has pointer object base is that instead of doing this business, you can sort of imagine we have a trait has pointer target that has a retire function that takes self, but it also has like a has pointers, right? Like it sort of declares that there needs to be, there needs to be a field in whoever implements the trait that we can then use from inside retire, because retire will be like a provided method, right? This isn't something you can write in Rust today, like sort of an extension trait, if you will, that also adds fields. So I think we're gonna have to go with the wrapper type approach, which is a little sad, because it requires these changes to the, ooh. Well, we can sort of do it, right? So one way you could do this is require that it provides one of these. Right, this is sort of the equivalent. And then we could have like a macro on, a macro that people can use on their type that automatically implements this trait for them by adding the necessary fields to the type. Why not do it in draw? That's a good question. So we could, using this wrapper type, we couldn't do that with the extension trait, but with the wrapper type, maybe we could do like implement draw for has pointer target T, where this is self.retire. The reason why I think you might want retire to be explicit is that if someone loads an atomic pointer and then gets a hazard point target, it's not clear that they want to retire it. Like retire is a very explicit operation. I guess the moment they get an owned one, that sort of implies that they wanna drop it. But, but like, okay. So remember, if you do a, let's say we have an X of this type, right? If you do X dot load, or maybe more clearly if you do X dot swap with Y, old. This is maybe the right way to write it. So like at this point, let's say you wanna deallocate or you wanna retire old. Then old here really sort of has the type of a, oops, it has this type, right? And so the question is, how are you gonna retire it? Because if you need to drop it, that means that you have to like dereference that. You have to do like this, which is real sketchy in Rustland. Like dereferencing a raw pointer in this way, you can do it, but it's, you can do like standard pointer read, but it's like wildly unsafe. I sort of feel better about something like hazard pointer target retire old. That feels nicer to me. And in fact, this maybe even takes a self that is actually a mute self, like a raw pointer to indicate that you're expected to do this with the sort of output of the raw pointer rather than having to dereference it. Do you think traits that require members on implementers is a good idea? I think it could be really helpful. It's sort of similar to just saying you need to provide accessors like a setter and a getter, but without having to do a setter and a getter. So I do like it as a, like it fits really well in certain very specific scenarios. I don't think it's necessary. Like you can get away with using accessors instead, but fields feel like a nice way to do it. Yeah, we're gonna have to think a little bit through what the right abstraction here is. Like this is the path they chose for C plus plus. And, you know, I don't hate it. Right? Like the equivalent here would be this takes this, right? I don't hate that. This is rapper type, like it does maybe let us do something around drop, but it feels nicer for this to just be a thing that you can do directly on the target. And it avoids this extra intermediate type. All right, well, we'll see when we get to it. Yeah, and you'll notice retire automatically reclaims the object when safe, right? So the idea here is that it's gonna walk the hazard pointer is if none of them contain the address of the current thing, then we can just de-allocate it straight away. Otherwise, we need to like stick it somewhere for later consideration. And the question is also like, what is this later consideration? Like how do we decide when to collect later? And the, in fact, if I switch back to the paper, where is? Yeah, it says if a removed map is not matched by any of the hazard pointers, then it's safe for this map to be de-allocated. Otherwise the writer thread keeps the node until this next scan of the hazard pointers. I see, so the idea here is that every time you de-allocate, or sorry, every time someone tells you to retire something, you compare all of the things in your reclamation, in your list of retired things with the set of hazard pointers, right? So if we go back to the drawing here for a second, let's see, so we do this. I really need this to be a little smaller and I need it to not be this funky marker, maybe. How's that? Okay, that's better, maybe. So let's say that I'm a writer and I call like, whoa, that's bad too. Pen, pen sounds nice and simple. So I call retire of X. And then I go look at all the hazard pointers and I find some hazard pointers that has the pointer to X, so I'm not allowed to reclaim X. So what I'm gonna do is just keep on the side a retired list that initially is empty. But when I notice that I can't reclaim here, then what I do is I stick X onto this list and then I just move on. And then at some point later when I call, say, retire Y for some other value that I care about, then when I go check the lists, let's say this is now ampersand Y, then I don't consider, like here, I considered X for reclamation, but here I'm actually gonna consider X and Y, right? So I'm gonna consider Y because it was the thing that was retired, but I'm also gonna consider X because it was the thing that is sort of left over in my retire list. And so even though this one I can't retire, so Y has to go onto the list, X can be retired because this one is not equal to X, right? Like X is not equal to Y. And therefore here we can reclaim X and then we stick Y onto the list for later consideration. So that's sort of the basic scheme here. And then you can imagine that you have also a method that's like cleanup, which is, okay, I really, even though I don't have a thing to remove right now, like I don't want to retire Y or anything, just like go through and make sure that everything that was retired, like go block on everything that's in my retired list, go clean that up. So that's the sort of reclamation idea for what do you do if you can't reclaim right now. All right. Let's keep walking through the Facebook documentation here. So there's a default domain and default to leaders. Okay, so my guess here is that a domain is the thing that holds the hazard pointer list, right? So the idea is that you can have multiple domains if you want, say, two different instances of a map to have separate hazard pointer lists. And then you also have a default one, which is like globally shared across all data structures. It's interesting though, here they say most users do not need to specify custom domains or custom to leaders. And by default use the default domain and the default to leaders. We sort of suggest that generally, like even though you might have multiple different data structures and instances of them, you should just use one hazard pointer link list to across all of them. It's an interesting recommendation. I'm guessing it's based on evidence. But this certainly seems like something we would want to provide too, that there's a default sort of global static hazard pointer link list. And then you can also supply your own if you really want to. Although that does raise the concern of you need to make sure that you use a hazard pointer holder that's tied to the correct hazard pointer link list. And when you delete, you have to use the right hazard pointer link list to check, like you have to use the same one that all your readers are using. So allowing that to be swappable does come with its own concerns. And I'm guessing we're gonna see that in the like contracts for different methods too, that like the, it's like a contract probably on something like retire that the domain of, the domain of the hazard object is the same as the domain used by all the readers. I might not be documented right here, but actually while on the topic of documentation, the official sort of standard proposal here is pretty good about this. So if we scroll back, I did unlike what I usually do, I did actually read a bit of the documentation this time before I started. So if we look at, where is retire? Class template hazard pointer domain retire. Right. So this is the sort of spec definition for retire. You see it takes a deleter this D thing. We don't know what that is yet. And a hazard pointer domain, and it says the requirements are, it mandates that T is a hazard protectable type. That's fine. This is a base class sub-object of an object X of type T. Boom. X is not retired. That's fine. That's fine. That's fine. Invoking the retire function may reclaim possibly reclaimable objects retired to domain. Yeah. So it will, interesting. So it doesn't actually say anything about the requirements on the domain here. Like the domain matching, matching what the readers use, which is a little interesting. I wonder why that is. Cause like, if you think about it, it makes no sense to try to retire an object. And then if you give like the wrong domain, then you might just like reclaim it, even though there are readers that still have it, they just use the different linked list. Like that seems like a big problem. I wonder if there's a way we can enforce that using just the types. Interesting. All right, back to the code. Simple usage example. Okay. So you have a class config, and I'm guessing config here is the thing that you want to guard with hazard pointers. It extends the object base. So this would be the equivalent for us of, it implements the hazard pointer, like target trait. Yeah, so one thing that's a little unfortunate with us saying that, for this hazard pointer target, right? Saying that we need you to provide this method is that the implementer has to like add this field manually and add this method manually. If this instead was, if Rust had support for like declaring a field on the trait, right? Then the implementer wouldn't need to do anything. They would just implement the trait with an empty like square bracket, and it would just do the right thing. Whereas here, it can't be empty. They actually need to like fill in this. They have to add a field of this type to their struct. This is why I'm proposing that maybe we want like a procedural macro or just a macro that can wrap the user's type and add those fields and add this implementation for them. That might be nice. Yeah, so in our case, this would be like a struct config and an impulse of hazard pointer target for config. And then this is just an implementation on config itself. I guess if this is keys and values inside the, yeah. So the implementation of get config here is it creates a hazard pointer holder. It calls get protected of config. So config here is an atomic pointer, right? So it passes in a reference to the atomic pointer to get protected just the way we saw earlier. And that's gonna be guarded by that holder. So the holder here is the hazard pointer. And that gives you back a pointer to the actual config, the thing that's behind the atomic pointer that is safe to access because it's stored in the hazard pointer. And then this returns, I see, so this is gonna call get config on the thing that's pointed to with the appropriate argument. And then the moment this returns, pointer, the destructor of the holder is gonna make the hazard pointer sort of blank so that it no longer protects anything so that the writer can now swap it out. So the moment that we return from here, pointer is no longer guarded, which means that the writer can swap out the config. One interesting point here is this is not quite safe the way the C code is written. Because imagine that this is instead of just VV, let's say that this is like K key and this is V for value. But imagine that V here is actually like a pointer, like a string pointer or something to into the config. Well, when you call get config with the key, what you're gonna get back from this is a pointer into the config. But the moment we return, the holder is dropped so the hazard pointer is zeroed out. So the config can be deallocated so the return pointer is invalid. There's nothing that guards you against that and the C interface is currently specified here. If this is a copy type, like if this is like, I don't know, U32 or whatever, then it's not a problem because you copy out you no longer have references to it. But in this kind of situation the C code would actually not be safe. You would need to keep holding the hazard pointer holder for as long as the user wants to use this V or you would have to like clone it out into an own string or something. In Rust, I think this would be caught by the compiler because the value returned here will be tied to the lifetime of the hazard pointer holder. So if this was a reference into the config, the compiler would tell you that this lifetime needs to out... No, this lifetime needs to outlive this lifetime. And it doesn't because you're trying to return it when this has been dropped and therefore the code wouldn't compile. This is a nice case where like the Rust type system and the bar check will actually catch this concurrency problem as long as we get the lifetimes right for the hazard pointer holder. So this is the example of a read and for write, it takes in a new config. It does an exchange, which is like a swap, right? It gets back the old pointer and then it calls retire on that. And this is similarly for us, right? Like this is a star mute config and calling retire on it is gonna stick that. It's gonna first try to eagerly reclaim anything that's in the retired list, right? And if it can, it'll reclaim them. And if it couldn't claim, if it couldn't reclaim this particular config, it'll stick it in the retired list so that on the next retire, that config will hopefully be reclaimed. So far, this matches the sort of mental model that we drew out earlier of how this works. Optimized holders. The template has pointer array provides most of the functionality of M hazard pointer holders both faster construction and deconstruction at the cost of restrictions. Interesting. So this is if you want to guard multiple axes. Like imagine that you want to read two pointers out of a data structure and you need to actually access both of them. Like imagine a doubly linked list where you need to read like the head and the tail or something, right? If you only had one hazard pointer holder, then you would read the head, but you couldn't read the tail because that requires immutable access to the hazard pointer holder, which is already tied to the head. And so you would have two hazard pointer holders use one to load the head and one to hold the tail. And this generalizes to M, right? If there are M pointers you want to read, you would need M hazard pointer holders. And so hazard pointer array, then it says is a more efficient way to have multiple hazard pointer holders for this kind of use case with certain restrictions, which I'm guessing we'll see when we actually do the implementation. I think what we'll probably do is ignore this for now. This seems like a good future optimization, but not something we need for the core implementation. The template hazard pointer local provides greater speed even when M equals one, but is unsafe for the current thread to construct any holder type objects while the current instance exists. Oh, this is interesting. So this is like rather than grabbing a hazard pointer holder from the linked list, we're gonna, actually, I wonder how they implemented this. It's probably at like a thread local hazard pointer holder. So you don't have to allocate one each time. Or maybe it's a, actually, maybe it's a hazard pointer holder that is not allocated on the heap. It's instead allocated on the stack. And you just like put its pointer on the list or something. It's like some, I don't know what they implemented. This also seems like an optimization that we'll want to look at later. Right. So they're saying like this can be more efficient than just using a standard hazard pointer holder. And my guess is because when you call get protected here, what that's gonna do is like hazard pointer holder is sort of an empty hazard pointer holder, right? It's not associated with an actual hazard pointer yet. Because for that, it needs to go talk to the linked list and fetch out a hazard pointer or allocate one if there are no free ones. So like the structure here is actually a little bit weird. Let me try to draw this out. This might be helpful where we have, yeah. So there is, what color should we use? Should use, let's go with like a bright orange. So there is a holder. I guess it's a hazard pointer holder over here. And then there's a hazard pointer over here. And then there's sort of a, I guess, a hazard object over here. And I guess technically there's also like an atomic in the middle here, right? So we really have all of these different components and it's important to understand that they're different. So the atomic points to a hazard object. A hazard pointers points either to nothing or it points to a hazard object. A hazard holder points either to nothing when you first create it or it points to a hazard pointer. I think that's the structure here. And a, I guess to switch colors here, a writer is gonna look at all of these and it's gonna also access this, right? So it's gonna swap this and it's gonna check these. And a reader is going to allocate one of these which is gonna have like, initially when you create a hazard pointer holder in a reader, it doesn't point to any actual hazard pointers. You might, it might then point to a hazard pointer the moment you try to access something. So it's also gonna, I guess, access all the transitive arrows here, right? But it might actually hold, I guess, one plus hazard pointers. To imagine that you're a reader, you create a hazard pointer holder, then you do a read, like over the course of a given hazard pointer holder, you could do multiple different reads. I guess you probably wouldn't give away your hazard pointer. You might stick with the same hazard pointer over time. But the idea is that you don't, when you create a hazard pointer holder, it doesn't immediately give you a hazard pointer. Well, maybe it could, maybe it eagerly should. Not quite sure. We'll have to look at whether this distinction is actually necessary or whether the reader can just grab a hazard pointer straight away. Interesting, we'll have to think about this. But the idea is that the reader can actually reuse a hazard pointer holder and a hazard pointer pointer too, over time over the course of multiple reads. But the writer will only ever check the list of hazard pointers. It doesn't actually know anything about the holders. Great, so let's go back here. Memory usage, the size of the metadata for the hazard pointer library is linear in the number of threads using hazard pointers. Assuming a constant number of hazard pointers per thread, which is typical. Right, so this is saying, we might not deallocate objects straight away, but we're not gonna allocate more than, or I see, so this is actually for the hazard pointers themselves, so for the metadata of the library. We're gonna have this linked list of hazard pointers, right? And we're never gonna deallocate from that list. It's just gonna grow until at some point you can always reuse one that's in there no longer being used. And the observation here is that the number of things in that linked list of hazard pointers is gonna be proportional to the number of threads, assuming that every thread uses only a constant number of hazard pointers over time, which seems fairly reasonable. Like, I sort of buy that this is typical. But if you didn't have a use case where a thread would just use, what's an example of this? Like, imagine that it has to create, it has to walk like a super deep tree, and it has to create a, it has to keep one hazard pointer for every level. Then like now every thread needs to have like log N of the size of the data hazard pointers. So now the size of the metadata is gonna grow as number of threads times log N where N is the size of your data, which might be much larger, but that is probably not typical. Like in general, a given thread can probably get away with just using like a constant number of hazard pointers for any given algorithmic operation. This isn't quite test num threads because we don't need to know the number of threads in advance. In some sense, this is a nicer design, right? Where we're gonna have this linked list where if there are no things that are free, then I'll get a new one. If there is a free one, then reuse it. And what that means is over time, what you would expect to happen is that the linked list will grow to the sort of number of hazard pointers that exist concurrently in the program at any given point in time, which is probably linear in the number of threads. So we don't need to know how many threads there are, but the linked list is probably gonna end up being that length or something fairly close to it. But not having to know it in advance is a big benefit. The typical number of reclaimable, so retired, but not yet reclaimed objects, of should not be there, but not yet reclaimed objects is linear in the number of hazard pointers, which typically is linear in the number of threads using hazard pointers. Right, so the observation here is that when you retire something, it might not be reclaimed immediately, but imagine that, okay, so let's say there are N hazard pointers. They can guard at most N different pointers, but by definition, each hazard pointer can only guard one address, and N objects will have N different addresses, so N hazard pointers are needed. So this means that if you have more retired objects than there are hazard pointers, some of them must be reclaimable, right? So the number of reclaimable, so retired, but not yet reclaimed objects is linear in the number of hazard pointers, which again is typically linear, as per the point above, typically linear in the number of threads. So the idea is that the amount of stuff that hasn't been deallocated yet is gonna be linear in the number of threads, which is often better than what you get with something like epic-based reclamation where the amount of stuff that hasn't been reclaimed yet might be much larger, because any thread that doesn't move on to the next epic prevents reclamation of everything in the past epic. Protecting linked structures in automatic retirement, hazard pointers provide link counting API to protect linked structures. It is capable of automatic retirement of objects, even when the removal of objects is uncertain. It also supports optimizations when links are known to be immutable, although link counting features incur no extra overhead for readers. Interesting, so this seems like an extra feature they've added for supporting something like a linked list or I guess the idea here is like, imagine that you have a linked list implementation and you remove 10 objects at once, then you want them all to be allocated at the same time, so maybe you can have some magic in the hazard pointer implementation that knows how to chase the pointers and deallocate them or retire them. But this also seems like an extra feature that we'll look at subsequently. This is nice, so they provide an alternative safe reclamation methods, like things you might wanna use instead of hazard pointers. Locking is the obvious one. If you don't have concurrent access because you use the lock, which provides exclusive access, then this is all just trivial. If you wanna delete something from the hash map, you take the lock, you remove it from the map, and you drop it immediately because you have exclusive access and then you release the lock, so great. But of course the problem is you now lose out of the concurrency because it's exclusive. Reference counting, as we mentioned, it's nice, it provides automatic reclamation, you don't need to do very much special, but the downside is high reader and writer overhead and contention because everything is operating on this shared counter. So RCU is a little weird because, so RCUs read copy update. The idea is that if you want to update a thing, you like prevent other threads from running, then you read the value, copy it, update it and map it back in place. I forget the exact implementation of use of space RCU, actually, but I think the problem is that you have to, there's like a section of execution where it's not okay for the thread to be interrupted, so you need to like put some safeguards in place, and that means that you can't like, if you, what's the best example of this? In RCU, I don't think you can read a value, like read out a pointer and then like issue a blocking system call and then continue to use that pointer. Like that doesn't work in RCU because you end up with deadlocks. I'm sort of, I'm watching the explanation of RCU, but it's because use of space RCU is actually kind of complicated. Simple seems incorrect to me here. But regardless, like read copy update has its uses, but I don't think it's a particularly important comparison right here. Right, this is a comparison that's similar to Epic Base Reclamation that RCU or Epic Base Reclamation also protects everything rather than a specific object. That's fine, I'm just gonna skip over this. Oh, this is kind of interesting. Differences from the standard proposal. Right, so this link here is a link to the C++ standardization proposal that I've been looking at as well. And I see the substances of the proposal was frozen around October, 2017, but there were subsequent API changes based on committee feedback. Oh, interesting. So I'm guessing the implementation here is more up to date than the spec. I think in general, it's useful to have both because the spec is probably a little simpler to follow and a little better sort of enumerated exactly what the conditions are and the implementations are because as a spec it has to. But the implementation is clearly one that's being used. So that gives us some confidence. Main differences are extra atomic template parameter for testing and debugging. Okay, that seems like something we can ignore. This library does not support it's a custom polymorphic allocator. That seems fine, we can just ignore that. I mean, this is relevant to Rust too, right? With maybe you want the user to be able to provide their own allocator for use with the hazard pointer holders. But I think we can ignore that for at least for the initial implementation. The construction of empty and non-empty hazard pointer holders are reversed. Interesting, I don't know what they mean here. I mean, so empty and non-empty hazard pointer holders, this is if you construct a hazard pointer holder, initially it doesn't point to an actual hazard pointer. And I'm guessing maybe in the standard, like the expectation that you always construct a hazard pointer holder for a particular hazard pointer when you make it because it's rare that you want a hazard pointer holder but not a hazard pointer. We'll look at this when we are at the actual implementation. Hazard pointer holder member functions get protected and reset are called protect and reset protected in the latest proposal. That's fine, we can choose our own names here. Array and local are not part of the standardized proposal, right? So this sort of supports the idea that these seems to be extensions that we can do later on. Link counting support and protection of linked structures is not part of the standard proposal, great. So this is also an additional feature. The standard proposal does not include cohorts and associated synchronous reclamation capabilities. This, so I skimmed a little bit of the Facebook code and I think the idea here is that you can say like these objects are related and should be allocated and deallocated or retired and reclaimed together as a unit which can sort of help with if you have a large batch of things you wanna claim, it sort of amortizes the overhead of retiring and reclaiming them. I don't think we're gonna implement the cohort part at least initially, but it seems like something that's also a cool extension we could eventually do. Nice, okay, I think we're now at a point where we can start doing the actual Rust implementation. I think we now have a decent idea of the structure we're gonna build. I think what we'll do is actually, one question is whether we want to follow the Facebook code here or that we just wanna write our own at least high level structure and then just look at their implementation. I think maybe that's the way I wanna go. I wanna write the Rust interface first and then the implementation is probably gonna be very similar to what the Facebook one is because clearly it's actually correct. Yeah, all right, now that we're all the way through the documentation about to sort of actually start the Rust code, let's do sort of questions on this structure, the hazard pointer library, the sort of data structure and algorithms involved before we dive deep into the code and also while people are thinking of names. Like what should we call this library? So we wanna know what are synonyms of hazard? Obstacle, is that too lame? What does it do? This library, it ensures that, it assures that concurrent accesses are safe. It guards the accesses, char hazard, it could just be zard. I mean, okay, so one option here is to go with something really stupid. Peril is pretty good. Peril is pretty good. Although for some reason, I feel like there is a project called Peril. Yep, and in fact, this is a hazard pointer implementation, interesting. Yep, fast and safe hazard pointer implementation for Rust. Interesting. This seems like a slightly different approach. Nice. Yeah, this would be a good thing to benchmark against actually. Danger Rust, warning. Warning is also kind of funny. The other thing I was thinking about is we could go really silly and go like, or really not silly, very straightforward and be like safe atomic pointer. Because in some sense, that's what we're providing here. The goal should be that using this, we can provide something that looks like atomic pointer, but it operates like on references instead. Hap hazard is also very funny. I mean, it's not really haphazard. Okay, haphazard is too funny to skip. Hap hazard. Nice. I can't not go with haphazard. That's too great. Danger RS, also pretty good. Can't drop this, that's funny. Okay, so I think the API that we have in mind, right, is let's use the names from the spec first and then translate them once we have the structure in place. So we have has pointer holder. We have a hazard pointer. We have a, I guess we have like standard sync atomic, atomic pointer is gonna be relevant here. But I think we might wanna provide a wrapper type around it. And we have a hazard pointer like object. And we're gonna have a private hazard pointer. I guess they call it domain, right? Which holds the linked list of hazard pointers. And it's gonna have to implement default. Yeah, and then this will probably be something like, actually, I think what we'll want is actually like a pub struct. This is also gonna be a pub struct shared hazard pointer domain where anything that uses this as the hazard pointer domain is going to like share it across the entire execution of the program. And what else do we need? So we're gonna need a sort of linked list for things that have been retired but not reclaimed. So it's like a link list for things that have been retired reclaimed lists. Or I guess it's just retired really, right? And in fact, here's what I think these should be called. So actually, let me try to write like what I would want this code to actually feel like when writing. Which is something like feels good. So let's say that we have a atomic pointer, new box, new box into raw, I don't know, 42. It doesn't really matter. Okay, so I think what I would expect to be able to do is as a reader, okay, so the C code does this, right? So you create a hazard pointer holder default, then you do let X or my X is H dot, I guess load of X where this is now a reference to the I32. And then if I drop H, then my X here should be invalid to invalid like this should not be okay, right? Because the holder has gone away. I think that's sort of the, that's roughly the C API. I wonder if we can make this nicer somehow, right? Like one option would be to have our own atomic pointer type that actually let's write out the writer as well. As a writer, I want to be able to write hold this X dot swap over 9,000 and I guess, sync atomic ordering, whatever, it's not terribly important. And here I want to be able to call old dot retire somehow. And all of this I guess is with the default domain. So I think what we'll do is sort of not worry too much about exposing like genericism over domains in the first instance of the implementation. And then we can add to that as sort of the second part. So let's then say that this is gonna be private and there isn't gonna be a shared and hazard pointer holder is gonna use the sort of standard shared domain. Old dot retire. So this will only work, I see. So this is where we have the trait, right? So this is gonna be a trait instead that's gonna hold retire where self is a star mute of self. I guess it doesn't need a T actually, fine. I thought this would already stay bliced, but I guess not. And we're gonna also need a, I guess in our case actually if we're just using a shared domain, then this should be able to use like shared domain dot whatever, right? So we're gonna have a static shared domain. This is probably gonna end up being like a, maybe this has to be a lazy static, but let's just say for now it's gonna be one of these. That's fine. And then the expectation is that we sort of wanna implement hazard pointer object for any type. Like is there really a restriction on what you can point to with this? Like why would we not just have impulse has pointer object for any T, right? And then this is gonna be I guess has pointer object retire. And has pointer holder is gonna implement default. That's fine. And has pointer domain is gonna have a list of hazard pointers, a list of hazard pointers, right? Which I guess, let's just say for now that it's gonna be a has pointer. Like in reality, there's gonna be a linked list of those. So let's just make a struct has pointers. So this is gonna be one of those and this is gonna be one of those. And then the h.load on hazard pointer holder has pointer holder. This is gonna have a pubfn load which takes a mute self and a reference to a atomic pointer of T. And it's gonna return a reference to T. So something here about this probably needs to be an option reference to T in case it's null or it needs to be unsafe, right? Like there's a, what's awkward about this, right? Is that we don't actually know that what we get back from atomic pointer is a valid pointer. One thing is it could be null but it could be like an unaligned pointer or something like that. So in some sense, I see, unless we define our own atomic pointer type, like because this is just the atomic pointer type from the standard library, nothing prevents a caller from like creating one and storing like the pointer one in there which is just like not a valid pointer but is also not the null pointer. And then calling load from our poor library which then goes, and tries to dereference whatever it loads, which is not a valid reference and you get undefined behavior. Which sort of means that this has to be unsafe which is unfortunate. If on the other hand, we made this, if we provided our own atomic pointer type we could just disallow setting raw pointers. Although that would mean that we would only allow passing in like a box, like something that's owned in order to store. Maybe that's worthwhile. I'm not entirely sure. It would mean that you couldn't store like stack pointers and stuff. I think this actually does need to be unsafe. But it's like unsafe in a different way than just a normal load is unsafe. And we'll have to document why that is. And this is just gonna do, I guess, so has pointer holder is gonna hold an option, in fact, option star mu task pointer. So this is gonna be if self.zero is none. Actually, we can write this a little bit nicer if I'll do that in a second. Then we need to use the shared domain. Acquire, I guess, right? So if we don't already have a hazard pointer for this holder, then we need to get one from the domain. Otherwise, we can just use the one we have. So this can be rewritten a little bit nicer as self.zero.asmue. Actually, let's just say if let sum then has pointer otherwise pointers this and give back has pointer. So we're gonna get out the has pointer, then we're going to do the first read. Right, remember, we have to do multiple reads, right? So we're gonna read the real pointer. Which memory ordering we use here is probably gonna be dictated by whatever the, whatever the Facebook library does because presumably they've already thought hard about this. In this case, I'm just gonna go it's sequentially consistent for now because we're just sort of writing out the code anyway. So we're gonna load the pointer one, then we're gonna, I guess, set the pointer, or I guess we could call this like protect, I guess, pointer one, then we're gonna load pointer two, like we're gonna load the pointer again. And if pointer one is not equal to pointer two, then we have to try again, which really means we probably wanna do something like a loop here. So we're gonna load it once. We're gonna try to protect that. We're gonna load it again. And if the two are equal, then all good protected. Otherwise, what we're gonna end up doing is set pointer one equals pointer two, and then go around the loop. This just saves us an extra load. And this is going to break with pointer one dot pointer non-null from new of pointer one dot map. So I use non-null new here because it's a way to check that the pointer is not null and get an option. It doesn't have to be this way. And now we can say this is gonna be, this is now it's fine to dereference this pointer. And this bit here, right? Safety, this is safe because one, target of pointer one will not be deallocated while since our hazard pointer is active, deallocated for the lifetime, for the returned lifetime, since our hazard pointer is active and pointing at pointer one. And two, pointer address is valid by the safety contract of the pointer. Load, right? So there's gonna be here, we're gonna have to say something like safety, caller must guarantee, guarantee that the address in atomic pointer address in atomic pointer is valid, aligned is valid. Yeah, caller was guaranteed that the address in atomic pointer is valid as a reference or null. Caller must also guarantee that atomic pointer, that the value behind the atomic pointer will only be deallocated through calls to, and I guess this is, what did we call it? Hazard pointer object retire, hazard pointer object retire. So as long as a caller fulfills this contract, then it's fine for us to dereference the pointer down here. Ooh, dot as ref. Then it's fine to dereference the raw pointer as a reference down there, because we know it's valid as a reference or null, and the null is taken care of by the non-null new, which returns an option. And it's fine for us to turn it into a reference with the lifetime we end up returning here, which is the lifetime of the mutable borrow of self, because we know that hazard pointer object retire will sort of respect all of the hazard pointers of which we are one. Great, so now we need a choir, and a choir on hazard pointer domain. It's gonna take a self and returned a hazard pointer, a star mute hazard pointer. In fact, I'm not gonna make it a static because it's not necessarily a static reference, even though the hazard pointers never get deallocated, remember, once we add them to the list. It might be that we give out a mutable, that we give out a pointer to a hazard pointer to other threads in the future once it's returned to us. So giving it a static lifetime isn't quite right. There is someone safety there that needs to be upheld, and this one we haven't implemented yet. And hazard pointer protect. So here we're gonna have hazard pointer. We're gonna have a FN protect, which takes, I guess, a star mute to hazard pointer. Now I think this is gonna take a mute self and a pointer address. What should that even be? This is sort of like a, it doesn't really matter what it is. Like a hazard pointer can guard any pointer, right? So this is really just gonna, a hazard pointer here is really just a pointer, like the thing that's being guarded, which might be, it might be null. Or it is a, well, there's gonna be something like active here so that we can reuse them. But I think for now we'll just have that be pointer. And all that has to do is self dot pointer equals pointer. That's all protect needs to do. Right. And this one, right. So in order to call protect we need to, we need to get a mutable reference to the hazard pointer, which in some sense is what we get back from shared domain acquire. Like whenever we get a hazard pointer out, it's ours until we give it back. But we don't have the appropriate lifetime to stick here because the lifetime is sort of, it's sort of an owned hazard pointer. That's really what it is, right? Like we own it until we return it. But it can't be one of these because it's also owned by the, by the domain, by the linked list because writers need to be able to look at it. So I guess actually, I lied a little bit here, protect is gonna take a shared reference to self. And this has to be an atomic pointer itself, actually. Because it has to be storable by the thread that owns the hazard pointer. And it has to be readable by any writer. So this actually does need to be, has to be an atomic pointer. It does need to be a shared reference though. We need to make sure that it's still valid. And this is why like it's a little weird, like in some sense it's like tick static, but it's not. Well, tick static is not, even a shared tick static is not quite wrong because it is true that we never deallocate them. And every axis is a shared axis. So maybe that is what we do. I think that is what we do. I think this returns a tick static has pointer at which point this is fine. And this is just saying, you're trying to protect the pointer of type T and this takes a pointer of type nothing. So we do this, nice. Yeah, I just saw the comment and chat that the API feels like it was designed to accommodate the algorithm rather than being ergonomic. And I agree with that. That's part of the reason why I wanted to write this code first to sort of see what it feels like and see whether we can do something better. But I wanted to at least write it out the way it is designed to then see if we can adapt it. How many hazard pointers would you expect to use on the list as a VEC unsuitable? A VEC is unsuitable because it needs concurrent access, right? Like multiple threads are gonna be concurrently accessing this hazard pointers list, including adding to it. And so it can't be a VEC. Do I recall retire after reading? No, after you read, you don't actually have to do anything except like, at some point you have to say, I don't care about this reference anymore, which you can do just by dropping the holder or you can call like h.reset. So at this point, this should be invalid. And so I guess to sort of expand this a little bit, right? Like here it should be valid to dereference this. Here it should be invalid because you've reset the hazard pointer. Here I load it. So here it should be valid to read this again. Then I drop the holder and now it's invalid. So as you can imagine that like either of these patterns are ways that you would write as a reader. And this is unsafe, which is a little unfortunate, but the safety here is atomic pointer points to a box. So it is valid. Safety one, atomic pointer points to a box. So it's always valid. And two, writers to atomic pointer use hazard pointer object retire, which are the two safety requirements. I expected I32 find option integer. That's weird. I wonder why it complains about that. Load of T gives you a reference T and the type of this is atomic pointer I32. So that's what it gets in there. So T is I32. So this should give an I32. I guess I can do this. Oh, it's because it's an option. That's why I lied a little. This is an option. So we're gonna hear I guess expect, not null. And same thing here, expect. So retire is only called as a writer. Is it possible that swap can return retireable object? So the challenge with that, I mean it could, but then we need our own atomic pointer type, which it could very well be that that's what we wanna provide because that way we could sort of have the wrapper return things from swap. But really like if this just worked, that would be nice and that will work actually if we do rust up override set nightly and then we do feature arbitrary self types and then we do retire. So this is gonna be self. Now we can just be old as retire. So then we don't actually need to wrap atomic pointer ourselves because we can just say there's a blanket implementation of retire for any type. Now, whether this is actually reasonable, I'm not quite sure. Like maybe we do want people to opt into this. I'm not, I don't actually know. I think the reason why the C++ library doesn't do this is because it can't, like I don't think it has a meaningful way to give this kind of blanket implementation. Actually, no, it's because it needs to add the field. We don't have the domain here. Once you need the domain, you need to have a field which means you need to have an impulse. But I think with a blanket implementation you should be able to do this. But right, like imagine this had a linked has pointers or whatever, right? That takes a self and gives a like list or I guess domain, right? Then here you need to write that implementation. And the question is like, what do you write here? You don't have anything you can write here. If we had fields and traits, this wouldn't be a problem because, well, assuming that the way fields and traits worked was that you could basically inject the field in the type then this would be fine. And practice has probably known what it's gonna do. And so people will need to add this implementation themselves because they need to add the domain. Like if you think about this, right? Like, how would you write this implementation? You can't really. And this is why you can't actually call retire on something that's just an I-32 because you don't have a reference to the domain which you need in order to call retire. So it does actually kind of make sense that this won't really work. So maybe we do need a wrapper type. It's a little awkward. This is why I was thinking like a procedural macro around your own custom type because like we could have like a pub struct. Actually, we could do both. We could say has pointer object wrapper T, right? Which is a real of a T or I guess inner and a domain which is a has pointer domain. And then we can implement has pointer object for has pointer object wrapper for any T which returns self.domain, right? And then this would also like implement, I guess, DRF. So use standard ops DRF and DRF mute for has pointer object wrapper. So this is self.inner. And then DRF mute, this does not take a T. Mute, mute. And so now callers can either choose to use our wrapper or they can use their own thing. In fact, we just make both of these fields be pub if we wanted to. But so then we would have like Imple, I guess, T from T for has pointer object wrapper from T self is very standard. Oh, actually that wouldn't work. It would be this new T and it would take a domain. But in our case, we'd say with default domain because we don't provide anything else yet. We're inner is gonna be a T we're given and this will eventually hold domain but for now, that's just gonna be the standard domain. So this is just gonna be shared domain for now. And then retire is gonna do, it's just gonna be using self.domain and call retire on that. Self. And retire probably has to be unsafe because you have to, the safety requirement on retire is that the caller must guarantee that self is no longer accessible to readers. It is okay for existing readers to, it's okay for exist, caller must guarantee that pointer is a valid reference, obviously. Callers guarantee that self is no longer accessible to readers, it's okay for existing readers to still refer to self. And at that point, we can do this and retire, yep. So that given these guarantees, then it's okay for us to take a, to turn this reference into a shared reference to ourselves, get the domain out of that using the accessor and then retire and pass in the pointer that we're trying to retire. And so now for the hazard pointer domain, we're gonna have retire, which is gonna be, I guess here, a pointer, which can really be a pointer to anything, I guess this is probably gonna be something like a didn't drop because otherwise how would we retire anything? And who knows what that's gonna do yet? Yeah, this can't be, this cannot be used as a trait object. The method might not be found because of this self type. This seems wrong. I'm pretty sure that trait that should not be required because everything implements drop. But I guess, I guess maybe not. Why does this need to, that's really awkward. It has to implement drop, why? All right, fine. We'll just make this this for now. That's fine. No, it needs, so okay, the challenge here is in order to reclaim an object in Rust, we need to call this destructor, which is in drop, but not every type implements drop. So the sort of a, well, what do we, like it sort of should be didn't drop, but then the type needs to implement drop, which like, maybe that's fine. Maybe we just require that there's an implementation of like, impolt drop for past pointer object wrapper T. This seems like a really stupid requirement because it doesn't, the drop doesn't do anything to do get rid of this requirement. But like, we need to have a drop because otherwise you can't construct a V table for it. Why does this may not live long enough? That's fine. There are all sorts of bounds we're gonna have to add here. I don't really want to have to add all of them right now because we're still at a fairly high level of the code and that's probably not gonna work. That's fine. Right, so now what you're gonna do down here is this is actually gonna be a, I guess, has pointer object wrapper with default domain of 42. That's what we're gonna store in there. So this is not gonna be an I32 really. This is gonna be a reference to a has pointer object wrapper. But that's fine. So this is gonna read out the I32 but we have to double dereference now because it's a wrapper. Although this might work. This is no longer necessary. We're gonna say that this is an I32 though. Same thing here. And this is gonna swap in a has pointer object wrapper with default domain of this. And this is now unsafe. And safety here is old value is no longer accessible. One, the pointer came from box. So it's valid. And two, the old value is no longer accessible. All right, I've been ignoring chat for a little bit. So I'm gonna go to chat and see all the questions that you have about this stuff. It's gonna be, couldn't you make an aligned atomic pointer T wrapper? So we could have a wrapper around atomic pointer. The challenge is that we would end up re-implementing basically all of the atomic pointer things anyway, including things like storing raw pointers because it's really annoying to work with atomic pointer if you can't use raw pointers there because you don't always have a box, right? Like we could make, I guess like store unsafe. That would be one way to make this wrapper that when you store you have to assert that you're storing a valid pointer. But I feel like it's probably nicer to just require it on consume rather than on produce. What happens to the writer always swaps between two reads. I mean, the reader would not make progress. But the question is why is the writer doing that? Like the writer's swapping means the writer is making progress as someone is still making progress. Don't we need ordering protection around hazard protect on the pointer DRF? Yeah, so this is, I've made all the ordering sequentially consistent for now. What we'll do is we'll walk through and make all of the match what's in the Facebook library in a second. Why is an explicit retire needed? Couldn't it just do the retirement when dropped? When would one retire but not drop? So there are some examples of that where you might want to like, what's an example of this? Like imagine you're shuffling a linked list, right? You might wanna remove a value like swap it out but then you're gonna stick it back in and you don't want it to, when you drop the thing you got back from swap for that to be deallocated because you're gonna put it somewhere else after. Okay. We could have, we could sort of require that all of this is like passed around owned. That feels like it would be fairly restricting when you actually implement a data structure on top of this. Right, okay, so let's think this through. So if swap returned a, like owned reference, right? First of all, it's not clear that it's owned. Actually, this is a great example. Think of a doubly linked list. So you have the next pointer of the previous node and the previous pointer of the next node both point to the node that you're about to remove. You do a swap from one and a swap from the other. Both of them get back one of these owned references. Dropping one is fine but dropping the, dropping or using the other is now undefined behavior. So just because you did a swap does not mean that the thing is no longer accessible. And this is why I think it's probably better to just say that retire is unsafe because it's up to the implementing data structure whether it's sufficiently inaccessible, right? Like the safety requirement we said for retire said caller must guarantee that self is no longer accessible to readers. And that doesn't necessarily mean just one swap. Yeah, it's a good point to came up from chat. It's because the drop thing, copy types don't have limit drop because they're copy. There's no way for you to drop one. So like maybe didn't drop is the right thing to do here. It just feels wonky but it might be the way we have to go. Yeah, so the other option, right? Is to have a wrapper that we return from swap that you can choose whether or not to retire but it defaults to not retiring it. But at that point, calling retire seems like the easier thing to do. Yeah, okay, so we need the double dereference. That's sad, but all right. We don't have a reset on hazard pointer holder. That's fine. So reset is actually kind of neat. It's really just gonna say, I guess we'll probably have to factor out like FNHaz pointer which takes a mute self and gives you back a tick static hazard pointer. And just stick that in here. This is gonna be then the self dot hazard pointer. So we're gonna do the same thing in reset. Actually, we can do better in reset because reset is a no off if we don't have a hazard pointer in the first place. So it's like if let some hazard pointer is self dot zero then hazard pointer dot store standard pointer null and ordering sec, right? So reset is just saying, make my hazard pointer point to nothing again because I'm willing to give up my reference. And crucially here, the borrower checker is gonna take care of making sure that you don't use any pointer you got back from load after calling reset because reset requires an exclusive reference to sell. And load also takes an exclusive reference to self and ties that to the lifetime of the team. Maybe let's make this like explicit here of this is tied to this. So in order to give an exclusive reference to self to reset, you need to not be holding this anymore anyway. You need to not be holding this anymore anyway. All right, fine, I'll just to make this stop yelling at me. I'll just, I'll just do this and do this, oops. Great, so that stopped yelling at me, that's good. Uh, bounce on drop are useless, but it's not useless, I need it. This is a bad link from the compiler. Oh, why? Oh, so this is something that's arguably an erroneous lint which is, so when you have an unsafe function, the body of the function is also a giant unsafe block, which means that you don't need unsafe blocks for individual things, but it feels a lot nicer to explicitly say when things are unsafe or not down here. There is a unsafe block, this one. Unsafe blocks and unsafe offense. So there's actually been an RFC on this that we should just not, we should require unsafe blocks even inside of unsafe functions because otherwise you can't really tell which part of the function is using the unsafe guarantees. And so I want to enable this one. Unsafe op and unsafe fn. Unsafe op and unsafe fn. I want to block, deny. And now you see this is no longer linting me because now if I remove this, now this is gonna yell at me saying you're trying to do something unsafe inside of an unsafe fn, you need an unsafe block. This is arguably a lint that hopefully will someday be turned on by default, but of course it breaks any existing unsafe code because people rely on not having to re-specify unsafe. But I definitely like the explicitness here better. Hazard pointer domain in public interface, that's fine. We'll just make that pub for now. That's okay. Field is never read, that's fine. Where are compiler errors? They're down here. This needs to be declared as mute, h.reset. Ah, okay, so here, if I do cargo T, you see it's, sort of want to get rid of this warnings, but that's fine. It says, cannot borrow h as mutable more than once at a time. This is the line that we said should be invalid, right? Which is, if I call reset, I then can't use the thing that I got back from load. And indeed, that's exactly what this error is telling me. The borrower checker saying, this is not okay. Same thing down here, if I drop the holder, then trying to use the pointer after that is not acceptable. And so indeed, if we comment out this one, comment out the invalid uses, then now that code compiles. So the borrower checker here enforced that contract for us, rather than us having to sort of document it and rely on people using it correctly. Great. So now we have the sort of high level interface down to match what the C++ code does. So now the question is like, do we want to, before we start implementing all of this, do we want to change it or do we want to implement it and then see if we can tidy up the interface? Because the underlying mechanisms, like the sort of the actual checking of the hazard pointer and stuff, that we're going to have to implement regardless. Like we don't really have a choice in that. That's just how the algorithm works. And so we can always implement the inner things and then go through and tidy up the interface and try to make it more ergonomic on top later. So I think that's what I want to do, is like basically implement the to-dos we have. Because if you look about it, like there aren't really other to-dos here at the moment. Right? Like the implementation isn't that complicated. The complicated bits are going to come here with the actual hazard pointer domain, which has to manage this list of objects. Which is really just, it has to implement a linked list. All right. So I think these two to-dos are going to be next on our list here. What's that? There's a third warning. Right. I mean, one option here is to like, it's dumb, right? Because really it should be okay to cast any type to this. If I make this like me, is this a bug with arbitrary self-types? This feels like it's at the very least a bug in the lint. And there's certainly here, if what's the standard mem needs drop of self, if it doesn't need a drop, then retire is a no-op, right? Like it's been removed from the data structure. There's nothing more to do. So that's fine. And there's sort of an argument that like, if it doesn't need drop, then it doesn't need to have domain either because we're not going to use it. So it's all like a little silly for types that don't need dropping. This is like in practice never really going to happen. But this cast, like this would have been fine if it didn't apply to like copy types, but if you have a wrapper type that contains the type that implements drop, then this still wouldn't let me do this. So I think this is just going to have to require drop, which means I guess I need to allow, what was it called? Drop bounce. But this very much feels like just a bug in the compiler. Like it should not give me that warning because it's not useless. It's usually useless. It just isn't useless. Great. And I think this domain here probably needs to be static or something. Like this probably needs to be like a star mute, star const. And then like this has to be I guess unsafe because yeah, there's like something, there's something here that's going to need to be an unsafe pointer dereference because the thing that gets pointed to has to have a pointer to the domain, but it can't be a reference because the domain might be tied to the data structure. So it's like some kind of weird reference, not sort of not cycle here necessarily, but like the data structure holds the domain but also holds pointers to things that all pointers to the domain. So I guess it is a cycle. So this probably needs to be a pointer, but that means that dereferencing it is unsafe because you need to guarantee that the domain still lives when you try to retire an object, which I think is going to tie back to probably domain being unsafe or something. Yeah, I think the way that it's gonna work in practice is like if you make a new, like new is gonna be unsafe because you need to promise that the domain matches, that the domain lives for as long as any of the objects that use it. So I think for now we're just gonna not, not worry about that. Yeah, yeah. So let's go implement acquire and retire because they are sort of the keystones that are gonna make this actually work. Why do you need the cast at all? I need the cast, so that this cast, this cast is needed because hazard pointers can guard objects of many different types. It doesn't care. All it cares about is what the pointer addresses or the pointer, what the pointer is, the address of the pointer is. And so you might be retiring a bunch of different types of objects that you might be retiring, like some boxes over here, some vectors over here or whatever. And all of them are using the same hazard pointer domain. The hazard pointer domain is not generic over T. So it just keeps a collection of things to reclaim. And that's why this has to be this. One thing we're gonna have to be careful about is when we compare the pointer down in retire, no, I guess let's use, when comparing pointer only compare data not vtable. So when we're looking through the hazard pointers, where are the hazard pointers? The hazard pointers all store like star mute this, right? For the pointer that they're protecting. I guess this could be a U8 or something instead if we wanted to, doesn't really matter what we use here. But this just stores a star mute U8. But the pointers we get into retire are star mute din drop so that we can drop them. And a star mute U8 and a star mute din drop, like the star mute din drop is a fat pointer and the star mute U8 is a thin pointer. And comparing them might do weird things, but we want to make sure that we only compare the data portion of the fat pointer. Otherwise we might get these two are not the same when in fact they're pointing to the same data and the other one just has the additional metadata of where the drop method is. So you have to be careful about that. What about types that need drop but aren't dropped themselves? Yeah, so an example of this is the has pointer object wrapper type we did, right? Its drop is empty because it doesn't need to do anything on drop, but we want, we needed to implement drop because of this requirement, which is like kind of stupid but what are you gonna do? Can't you use mute din any and call mem? I don't think, if this took a mute din any, I don't think you can drop star of that. Like what's the drop in place? It's actually a little weird because I'm just like thinking this through a little bit more and it's not necessarily just the target type that should be dropped because imagine that someone, imagine that this pointer originally came from box into raw. The way to drop it is to call box from raw on the pointer itself. We don't just wanna drop the target of the pointer because again in the case of box, if we called like drop in place on the target of the pointer, we would call the destructor of the target but we wouldn't, we would never actually free the memory because we never called the deallocator of box itself which is kind of awkward. You know, maybe we just need to enforce that all of this is box. That's a little unfortunate. I'm guessing this is why they have this like deleter trait, right? So there's a, they have like a generic D where there's a pub trait deleter which takes, I guess delete, takes a immutable reference to self or maybe it's just a, ooh, that's not what I meant to do. Think it was, no, that's too large there. It takes like a star mute, I guess, din drop and then the idea would be that your deleter would be the thing that knows whether this is really a box and you need to call something else, right? And we can even have a default implementation of deleter which is just calling drop in place but not dropping the actual holder or something, right? Or we could specialize all of this to box and see how they're alternative, right? So now, if we say that this doesn't even get this, it just needs to be a deleter. So where D is the leader or maybe this is even just a generic argument here. So retire. There's like something sort of wonky here of maybe the domain has, actually, let's go look at what the Facebook library did here because I feel like that might be just instructive. So for a hazard pointer, make hazard pointer, specialized algorithms, hazard pointer object base, this is the equivalent to our hazard pointer object trait. Retire is generic over D. Yeah, right. So this is exactly the observation that the deleter is the thing that's responsible for taking a pointer of that type and calling the appropriate destructor on it as is appropriate for that deleter. I think one question here is retire needs to pass the deleter into the domain. So where's the domain factor into this? So the domain doesn't have a deleter, I see. So really what this is saying is that retire takes a D, that's a deleter. It passes that to this on the domain, retire. And then this deleter needs to also be a part of what gets put in in fact, this probably then needs to take like an instance of that deleter or something. So what we need to stick in retired is actually something like the pointer and like a D default or something. This is probably a, it can't be a zero size type. It's a function pointer basically saying in order to delete this pointer, use this function. Yeah, so this is gonna be a function pointer type. Maybe the deleter doesn't even need to be a trait. Maybe it really just is a function pointer. That is certainly one option. Like to say that this is gonna take a fn of star and you didn't drop, which is a function pointer type. But it doesn't need to be, right? Like this could totally be just a D, which is gonna be a star mute deleter. Or maybe we require the deleter is a copy or something, but we're still gonna need, well, what gets awkward, right, is that the list of things to retire does not have, it's not generic over T or over D. So we need to stick in things here that are like all of the same size. So maybe actually this is gonna be a star mute din deleter and we stick the deleter in there with the pointer. Oh, I think I know what we do here actually. Maybe what we do is actually just take a closure and then we stick the closure in the pointer to compare and the closure to call to dispose of it in there. Yeah, we're gonna have to think a little bit about this interface, but it is gonna be necessary to take the deleter and somehow stick it into the linked list so that we know what to call later. All right, I'm gonna be back in a second and then let's implement these linked lists. Ooh, and I can put up J. I have returned, I think. Seems like this interface would be simpler if domain was generic over T and D. It would be, but you don't really want it to be because that would require that you have a, you couldn't have one domain that's shared across all users of hash pointers. It would also mean, well, because the domain would then be specialized to a particular object type and a particular deleter, and that's unfortunate. Like if we can avoid that, that'd be better because we really want to amortize the cost of managing the domain. Having multiple domains is unfortunate. Actually, I wonder what the Facebook library does for hash pointer domain retire. The retired list is a head-only list, that's fine. The holds object and atom. Wonder what atom is here. Hash pointer object seems to be what it does. So retire allocates a new node to stick on the list. To stick on the list. Where to reclaim is the D. I see, so it creates a private type struct hash pointer retire node. Yeah, I think what it's doing here is is it's just, it's creating a, for each T and D, you have a different node type, but they all implement hash pointer object. And the linked list is of these hash pointer objects. The question is, how does it end up invoking the particular implementation? It basically ends up using sort of dynamic dispatch, right? So there's, I think reclaim is a method on hash pointer object. So if you get like a din of this, then you just call the reclaim method, which is implemented for this particular implementation as the de-allocator of this particular instance of that type, which calls the appropriate reclaim. Yeah, so it's really just dynamic dispatch. And we could do something similar here too. In fact, we could probably do it in almost the same way. Basically the idea is you stick a bunch of trade objects on, in fact, we could probably use din drop here. And just stick a bunch of trade objects that do dynamic dispatch to the actual deleter onto this linked list. Or not much is what we sort of wanted to do. All right, so, actually if we go back here for a second and look at the Facebook code, this is a definition of domain, right? So domain has a bunch of things in it. It has has pointers and retired, which we've already looked at. These other things are, I think these are all related to the extra features. So we can probably ignore them for now. And these I think are just the heads of lists, right? So retired and has pointers, retired list. Yeah, this is just a head and has pointer wreck. We're gonna figure out what that is. I guess that is an actual has pointer. That's the type that we've just called has pointer. And they're both just linked lists. And push retired just compare exchanges the head of the retired list with the new node, adds to the count, if check. Right, so when you're trying to retire something, you stick it at the head of the retired list. Add to the count and then if check, then we call check, cleanup and reclaim. And I'm guessing this is, yeah, so this is the thing that decides to actually try to reclaim objects that have been retired. Like I'm guessing try time, clean it up is if it's been a while since the last time we did clean up, then try to reclaim things. But also if we have enough objects on the retired list, then try to reclaim a bunch of them. And what does this do? Bulk reclaim just steals the whole list, right? It exchanges the retired list with nothing. Then it loads the head of the list of hazard pointers. Then it records all the pointers that are guarded by hazard pointers, right? So it gets a set of pointer values. It loops through all of the hazard pointers, reads all of their hazard pointer values. And then here for each retired object, like for each thing in the retired list, if it's not in any of the hazard pointers, if it's not in the set that we collected, then reclaim it. Why is this a while true? Bulk, lookup and reclaim is, yeah. So this walks through all of the objects. And if it's in the set, if it's not in the set, then reclaim it. Otherwise just keep it because we're gonna have to put it back. And it puts it back. Okay. So this I think we can write ourselves. Like this is not a fairly, is not a particularly complex linked list. Great. Although something has to make sure that multiple writers don't try to reclaim at the same time. Which like should be in here somewhere. But, oh, the thing that guarantees that is because we exchanged the head of the list. So there's two writers come along. One tries to reclaim, it steals the head. The other tries to reclaim, it tries to take the head. And the head is then none. So it doesn't try to reclaim anything. Great. Okay, that seems simple enough. There's like all these optimizations for timed cleaning and stuff that we're probably gonna want to implement. But the basic implementation seems pretty straightforward. Now, here, this is a pretty standard linked list, really. Question is, is there one that we can just reuse? I think we can just implement it ourselves here. I guess let's do a list. Actually, I can probably do something even easier than that. Which is, has pointers is just a head pointer, which is an atomic pointer to a has pointer. And a has pointer has a atomic pointer to the next has pointer. And it's also gonna have to have an active so that we can know whether to reuse them. Great. So I guess let's do a choir first. Oh, and we're also gonna have to write drop for holder. Which is gonna be the thing that returns the hazard pointer. Return the has, return self.zero to domain if some. So this is gonna be it to do. And actually, this one we can write pretty easily first because this one is just self. I guess, if let some has pointer equals self.zero, then has pointer.active.store false. And head is, and I guess this is gonna be sort of while loop or a loop, it's gonna be, head is, oh, the has pointer holder needs to know which domain it came from. Which I guess can be stored just in the has pointer, but really it kind of feels like it should be in the holder. So this should hold a domain, which in our case is just gonna be shared domain.head. Or I guess.haspointers.head.load. So we're gonna load the head and then we're gonna actually, we're gonna keep that one around because we're gonna wanna reference it again. And we're gonna do something like let head is head pointer.load. Then we're gonna do has pointer.next.store head. And then we're gonna do head pointer.compare and swap. And we're gonna compare and swap the, what we read as the head with ourselves has pointer. And again, all of these currently like this. So we're gonna do a match on this. So this is just, taking the, actually we may not even need to swap it back. I don't think we need to swap it back because it's already in the list. So all we need to do here is just, yeah, we never remove it from the list in the first place. The hazard pointer linked list is never unlinked. So we don't even need to do that. What we do need to do though is make sure that if it's currently guarding something, it stops guarding that thing. So we're gonna do self.reset if necessary, right? And then we're gonna set it to be not active so that other threads can now pick up and reuse this thing. So that's dirt easy. And then acquire, I guess is really just gonna be while, oh, this static is only true because the domain itself is static. We're gonna have to deal with that at some point. So here, I guess what we're gonna do is the head is gonna be self has pointers head. And then I guess what we'll do is let me node is head pointer dot load while node dot, not node is null. And, and the nodes active, then node is node dot, sorry, let me write this out and then I'll explain what it means. Node dot is null. Okay, so what we're doing here is in order to acquire a hazard pointer to give out, what we're gonna do is just walk the list of hazard pointers, the linked list of hazard pointers. And if we get to a null pointer, then there are no hazard pointers we can reuse and we need to allocate one. And if we're gonna look at like, we're gonna look at the list in order. And if the next one has the active flag set, then we can't use it because it's already being used by someone. It's already active. So we need to also keep looping. And the loop is just gonna load the next pointer. Like we're just gonna walk the linked list in sequence. And then at the end, if the node is null, then we need to allocate. Otherwise, we now just sort of own this node. But only kind of. I guess maybe we want this to be, I think what we want is this. Here, what we need to do is try to acquire this node. Which is node because it might be the two threads both try to like take the same hazard pointer at the same time. We need to guard against that. So we need to do a compare exchange week of, it currently was active and we're gonna make it active because we're gonna take it. So if is okay, it's ours. Ooh, it's complaining about something. That's fine. This is gonna break. Then it's ours and we can actually just break with that node. In this case, we haven't written yet. And in this case, it's sort of like someone else grabbed this node right before us. Keep walking. All right, so in this case, we just keep looping through and we continue from the node that we were just looking at. Like there's no reason for us to restart at the top of the list. We can just keep walking towards the end. And this may be an optimization here where like, maybe this means you end up allocating slightly too many nodes if there's too much churn. But I think this is okay, at least for now. So for allocating, allocating here is super easy, right? Which is, we just do has pointer is has pointer box new. Has pointer of, is this just implement default? I forget. It kind of should, but even if it doesn't, we can just do this box into raw of that. That pointer here is going to start out being null. Next is going to be, we're going to stick this, might as well stick it at the head, right? Just atomic bool. This one is active. And the next pointer here is going to be the head pointer. Actually, it's going to be, we're going to fill this in in a second actually, is what it's going to be. This is going to be an atomic pointer new of that. And this is going to be the same for now. I'll show you one a second. So what do we have to do here, right? We allocate a new has pointer. No free has pointers need to allocate a new one and stick it at the head of the linked list. So to stick it at the head, we do, this is going to be a loop again. We're going to read the head, which is going to, I guess, let mute head. It's going to be head pointer.load. And what we're going to do is we're going to keep looping and doing has pointer. And this is going to have to be unsafe. We're going to set the next pointer. I think there's a get mute. So we're going to set that to be head. And then we're going to compare and swap the head pointer from what we read the head to be to our new has pointer node. If that doesn't quite make sense, let me explain it in a second. Then we can, if the comparison succeeded, then we can just return that has pointer. And if it failed head now, then the head is going to be head now and we go around and try again. Okay. So let me try to explain what's going on here. So we're trying to stick it at the head of the linked list. This is a standard linked list operation really. It's just concurrent, right? So we take the head pointer, read where it points. We can take our node and we set its next pointer to be the current head. And then we compare and swap in our has pointer that we just allocated with its next pointer to the old head. Try to stick it in there, but it has to be a compare and swap because someone else could have slotted in a new head and we then need to use that new head as our next pointer. And so hence the loop. When eventually we succeed, when eventually we do set the next, then we can just break and it's safe for us to use the has pointer we just allocated because we allocated it with active true. So we can sort of here add in a safety, active is true, so no one else is... Actually, this doesn't even have to do with active. Safety is will never be deallocated. And safety here is has pointer was never shared. So mute is okay. Does that loop make sense? Doesn't the head pointer load need to be in the loop? No, it doesn't actually. So the idea here is that we first load whatever the head pointer is and then we try to the compare exchange and when compare exchange fails, it tells you what the value was when it failed. So we update head here. Let me add a comment that's probably helpful. Head has changed. Try again with that as our next pointer. Is that clearer now? So this basically ends up, the compare exchange doubles as our extra load rather than having to do an additional load at the top of the loop as well. And great, and here I guess the up here, safety here is also will never be deallocated. I guess the safety has pointers are never deallocated. This is maybe a better phrasing. Now this will no longer be true once we move to a hazard pointer domain because then they can no longer really be static. They actually have to be raw pointers because the moment the domain goes away, all of the hazard pointers associated with it should no longer be used either. Yeah, so all the link, once we move to a domain that's not static, right? Like maybe one way to phrase this is currently hazard pointer domain, there is only one and that is the static one we declared at the top, right? Once we go beyond that, this will no longer be true. Although even now, we're never deallocating any of these nodes. Like even if you drop the domain, we're not currently deallocating those. So this still remains true for other domains as well at the moment. It's just once we add the ability for a domain to deallocate its nodes, then we need to deal with that here. But for now, this is okay. So that's a choir, not too bad. And then it's a retire, which is gonna be a little bit more annoying. So retire is also gonna be a link list. Where is my... And we can obviously split this up more, right? Like there's nothing that requires that all of this implementation is in this one file. In reality, what we'll probably do is move the link list implementation into its own file. And we might even be able to reuse the same link list implementation for both, for both has pointers and retired. In practice, I think there's a... There's an argument for keeping them separate, which is you can optimize much better. You can optimize the implementation of the link list much better if you know that it'll only be used in one particular way. And I think in the Facebook library, let's see if I can dig this up. They have... They have has pointer detail link list. They have has pointer detail shared head only list. I'm pretty sure they have another one too. So there's like different implementations for different use cases because you can optimize them appropriately. And so I think it's fine for us to just inline the implementation now and then split it out later. So retire, one thing we need to figure out is what should they even hold? I think there's gonna have to be a retired list, I guess. There's gonna be a retired list. And this is gonna be, oops, nope. This one is gonna be a retired. But what does retired actually hold? I think what retired is gonna hold is a pointer which is gonna be the actual like pointer to the data. The thing that we're gonna compare with all the hazard pointers. So maybe data, no, let's go pointer. And then we're gonna have reclaim, which it's a little unclear what reclaim should be. It is kind of tempting to say that reclaim should be a tick static din deleter. Well, then this has to be mute drop, mute din drop. Or if you think about it like an FN from din drop. What do I do? It's complaining somewhere. Oh, I'm gonna have, it's up here that is complaining. This is gonna have to be head, it's gonna be atomic, pointer new, default. Nice, can't be default because default is not const null. And hopefully maybe this too. Tired list. It's a really not a const null constructor for atomic pointer. Nope, this default, but default is not const. There's from, but from is not const. But I think the new is const. So we're gonna have to do new, oops, new pointer and null mute, and then same thing here. Great. Okay, I wanted to stop complaining about my dead code because I'm not done writing the code yet. And those warnings get in the way and highlight things when I don't want them to be highlighted. Yeah, so the question is what should retire do? Well, my thinking here was that each of the things in the list of things that are retired should certainly have the pointer. It needs to have the Vtable entry for the drop implementation of the target type. And then a reclaim should really just be a function pointer, which I guess it doesn't even need to be this. It could just be FN, which is a function pointer. It doesn't get to be a closure even. And then I guess what retire is gonna do is let's first stick pointer onto the list of retired objects, right? So this is gonna be retired is gonna be box into raw, box new of retired of pointer. And I guess reclaim is gonna be d delete. Delete. And then it's the same procedure as we did up here where we want to stick it at the head of the list. And this code being the same is like a little bit silly, but we do actually need retired then to have a next pointer which has to be the atomic pointer to a retired. Right, so this is how they're gonna chain. And head pointer is here self.retired.head. And next initially here should be a null pointer. Null pointer, retired was never shared, retired, retired. And if it succeeds, then we're done. We don't actually want to return anything. And otherwise we have to try again. So this is the standard like push onto the head of a concurrent link list. Now check if we need to retire. I guess this is where it's gonna be handy to have a count of I guess atomic use size for like how many elements which are here, we're gonna do self.retired.count fetch add one. Notice that we have to do this before we put the thing at the head of the list because otherwise someone else might decrement the count, including our own before we add which makes it go negative, which is a wraparound, which would be bad, increment the count before we give anyone a chance to reclaim it. And now check if we need to retire. So if self.retired.count, like this is where there's like some heuristic in the Facebook code base where maybe we could be smarter about exactly when should be load, when we should reclaim and when we shouldn't. But here what we can do is just like if it's not equal to zero, then we're gonna go do some work. We're gonna try to reclaim some objects. We might fail, but we're gonna try to do better heuristics. And just to make this a little bit nicer, we're gonna do bulk reclaim, bulk claim, great. And so in bulk reclaim, what we do is we sort of steal the entire retired list. Do just a compare exchange week. Or actually we just do a swap of standard pointer null mute. If steal.isNull then return nothing to reclaim. And then what we need to do is get all of the hazard pointers, right? So node is self.hazardPointers, find all guarded addresses, .head, .load, ordering. And then we're just gonna do while node, while not node is null, node. Here, this is again, the safety is still the same, which is hazard pointers are never deallocated. So node is gonna be end.next.load. And I guess here what we're really gonna do is like pointers is gonna be a hash set. So use collections, hash set. And we're just gonna go ahead and pointers.insert, end.pointer.load, right? So we're just walking all the hazard pointers and collecting all of their pointer addresses into this hash set. Oh yeah, it's if once in a while. That's right. Once in a while. So we're just gonna collect all of the guarded pointers. Let's do guarded pointers. We're gonna insert all the ones we find. And then now we're gonna reclaim any retired, reclaim any retired objects that aren't guarded. So here too, we're gonna have basically the same thing, except this is gonna be the steel. And here, I guess the safety is we own these objects. So in fact, this can be box from raw of node, which is totally fine because we made them inaccessible and we know that no one else can be walking them because the only reason you walk them, the only reason you ever walk the retired objects is because you are swapping the head. It is because you have swapped the head. We own these objects. So really the safety is all accessors only access the head and the head is now is no longer pointing here. And then what we wanna do is if the guarded pointers contains end pointer, and I think we can just use get here now because we own the end. Oh, I'm being stupid. And this should be, what did we call this? Just end dot pointer. This not safe to reclaim, still guarded. In which case we sort of need to, we need to put it back. So to do, put it back, otherwise it is safe to reclaim. And in this case, what we really need to do is just call, call, call, call, call. End dot reclaim on end dot pointer. No longer guarded the reclaim using the leader. Right, so this is where we need to be careful. We need to do this as const u8. And then we need to do a reference to this. And this putting back is sort of important because really what we're doing is like, we're walking this linked list and some of the elements in the linked list we're gonna reclaim and others we're not. So I think really what we're gonna do is like, let mute remaining is gonna be standard point normal. And then we're gonna do, if we have to put it back, then we're gonna set remaining, right? We're actually, we're just gonna reconstruct a linked list. We're gonna say end dot next is remaining. We're gonna set, actually how are we gonna do? Well, so what I was thinking was we said end dot next to be remaining, we said remaining to be box into raw of n, yeah, into raw of n. I'm gonna go ahead and load that here. And this can just be get mute because we already own it so there's no need for an atomic read. End dot next is, I guess this can be dot get mute. I mean, whenever we have like mutable access to a thing, I don't wanna use the load and store methods because we should take advantage of the fact that we know that they're not needed to not specify an ordering, for example. So here really what I'm doing is just like, anything that I don't find, I just like put back onto this linked list and then what we actually need to do here is at the end here to do put remaining back. This is, so we need to swap this back into the head of the linked list but if you think about it, it's a little bit awkward because we need to set the head to remaining but we also need to set the next pointer of the last thing in remaining to be the head of the current list which makes it all a little bit awkward. We can, I mean, we can sort of do this by, maybe we could put them back one by one but that seems unfortunate. How can we atomically do this? It's a good question. I'm wondering if there's some efficient way for us to put all of them back without having to put them one by one. There's certainly like, okay, if remaining is null, then there's nothing to put back so we can return. If there's only one element, then it's trivial because it's a standard like compare and swap loop like we did above. But what should we do if we have more than one? I'm wondering if we can like reverse the linked list. That doesn't help either. I mean, one option here is steal the new linked list added to remaining and then just keep going until the head is null but that seems problematic too. So if we walk the current list and then set, so if we set the last thing to put ahead and then do a cast, oh, you're right. No, you're totally right. You're totally right. Yeah, you're totally right. Ignore me. So we're gonna say, we're gonna have the same loop again. This is a loop we're familiar with now. Where's our loop here? Should really just make this a macro or something. So we're first gonna say that the tail of remaining is gonna be new tail is none. If tail is none, then tail is equal to remaining just so we don't actually need to walk this list. Tail is, if let, and we're just trying to avoid the unwrap here. If some tail is tail, then tail. Else return, and here we can also assert remaining is empty, remaining is null, I mean. And if we want to, we can also assert it's not null here. And then what we do here is say, we're gonna read the head pointer. We're gonna set the tail. The tail's next is gonna be head and the head pointer is gonna be remaining. Yeah, that should be right. Okay, so this is a little silly. Oh, and this is, we still have exclusive access to remaining, which includes tail. Okay, so the plan here is we're gonna, we're gonna construct this like segment of a linked list, right, which is everything that didn't, everything that we couldn't reclaim. The head of that is the thing called remaining. The tail value here is a pointer to the last element in that linked list. And what we're gonna do is we're gonna load the head of the retired list pointer, set the next of the tail to be that head, and then compare and swap the head to the remaining. Right, so the start of this linked list. And that compare and swap is sufficient because if the head has changed, then the compare and swap will fail and we'll update the next pointer of the tail again and do it all over. And if the head pointer hasn't changed, then the next pointer of the tail is still correct and the whole linked list remains intact. Great. Great stuff. So now we have put remaining back. All right, what's this complaining about? Count, that's fine. That starts out as new zero. And I guess what we also need to do in retire is, all right, this needs to be mute, is we need to ret-reclaimed is zero. Reclaimed plus equals one, self.retired.count fetch sub reclaimed like so. This complains because we cannot borrow and dot next is mutable. Only one error left. Old retire. Right, so here the problem is we don't know how to retire this because really this has to be like, maybe this can just be Fox from RAW. Right, so the deleter here is a little awkward. Like how do you specify the deleter? It arguably doesn't even need to be a trait. The advantage of it being a trait is that it doesn't have to be a, it doesn't have to be just like a function pointer. We could associate state with the deleter two, which is valuable if someone wants to bring their own allocator, for example. And so the delete can't really just be a freestanding function. Right, so if we said it's gonna be this, then this is now gonna be a star mute deleter, din deleter, and then retire is gonna take a deleter of type D or reference to one, just a D. Maybe we're gonna require that the deleter is moved. Or that it's like static or something. Right, like, okay, the deleter we get in here, I guess we're claim is gonna be deleter as star mute, din deleter, which is why this has to be, yeah, so the question is what do we want retired to actually take? Like do we want someone to have to pass in an instance of a deleter? Or should we just say you just have to be generic over it and then it just has to be a sort of stateless closure? We could make this interface a little simpler, right, by saying we also have a retire with drop in place that just takes a pointer and all it's gonna do is call drop in place on the thing. But it is tempting to sort of allow the user to pass in a deleter, but the interface is a little bit weird here, because the deleter they give us has to be, it like has to be behind a pointer because otherwise we can't stick it in the list with other things that might use other deleters, right? So this is why we can't just stick the deleter here because then we'd have a D and retired is not generic over D. It could be a pointer, but then this has to be a pointer and if this is a pointer, then what is it even a pointer to? Like maybe it's a static, but if it's static, then why not just have it be a static method instead? Like why not just have it be a function pointer if it can't hold state anyway? There's like a, it could be a star mute din deleter that gets passed in here and it's up to the caller to make sure that this is like valid for long enough, but long enough is really poorly defined here, right? It's like the lifetime of the domain because you don't know when it's gonna be reclaimed. That might be, it might not be until the domain actually gets destroyed, which might be a super long time from here. So it's hard to assign a lifetime to this. You know, I think we're just gonna go with what we had here for now. I think this is just gonna be a function pointer. We're not gonna have a deleter trait and this is gonna take a D which is gonna be an FN. Oh, actually no, I do wanna deleter trait just because it documents the interface better, but I don't want it to do this. And then the question now becomes what do I pass to retire? So one thing we certainly could do is have a, there are too many things in this file now, but we could have a pub struct drop in place. So maybe this is a pub mod deletors and impold deleter for drop in place, delete, mute and drop, which does the standard mem standard pointer, drop in place of pointer. And similarly we'll have a drop box which does a just box from raw of pointer. This is unsafe and this is unsafe and these are safe by the contract on retire. Which is has pointer object retire. And then we're gonna have to go back to retire and add here column is guaranteed that pointer is a valid reference, column is guaranteed that self is no longer accessible to reader and caller must guarantee that D is a valid deleter for self. Right, so if you're using the box deleter or drop as box or whatever we ended up calling it, then the part of the unsafe promise you need to make in order to call retire is that the star mute you pass in used to be a box. And I guess we could document this on here, right? Which is sort of safety can only be used on values that were originally derived from box. And similarly for drop in place can only be used, I guess drop in place is just like always safe. So this one is actually always safe to use given requirements on retire on pass pointer object retire, but may lead to memory leaks if the pointer type itself needs drop. Can you interact with the leader behind a local generic function? You behind a local generic function? I mean, yes, it can be any function pointer. So I think what you're suggesting if I understand you correctly is that here we do FN drop box which takes a pointer which is a star mute didn't drop and it does unsafe box from raw pointer this and then we pass old retire, oops, drop box. This drop box is a function item, not a type. Function item types cannot be named directly. Yeah, so this is what we're going to do. I mean, it is true that we could have, we can maybe do this and then say that retire, the generic over the leader and takes an instance, or maybe it's not generic over the leader, maybe it just take, well, I guess it would have to be, it takes a deleter and I guess it would have to be. This and then the leader is the D and then this is the deleter, right? Like D delete, like the argument is unused which is a little weird, but maybe it makes it nicer to work with the interface. It is true that now trade deleter is not implemented. Right, so this is where maybe we don't actually want the deleter trait, is that nicer? Like I'm not quite sure, right? So if we comment this out like this and then say that D is going to be an FN star mute in drop, right? Then we do the same here and then we say this is just D and then I guess actually this then is D. This has to be in, this is this, it's not generic anymore and then this same thing is not generic anymore, it just takes a function pointer that the deleter, like that, is that nicer? And then let's actually call this deleter to have it sort of match up, so this is going to be deleter and this is going to be deleter. Maybe that is nicer, maybe you're right. It doesn't mean that like people need to write these closures themselves, but maybe that's the nicer way to do things anyway and then here safety, safety because by the safety guarantees of retire and because retired object and because it's only used when retiring box objects or something like that. Why not pointer read instead of box from raw in drop? Because pointer read would just give us back the T like the din drop, but we want box from raw because we want to run the destructor of box so that we free the memory as well instead of just running the drop for the backing type, right? So imagine this is a box file, right? Like imagine this is really a box file then if we just call standard, well, if we just called standard pointer read for this we would get back a file, we would drop the file but we wouldn't drop the box. So the file will be closed but the memory used by the box would never be freed. That's why we need to turn it back into a box first. Could you drop and place a string? You could, but actually what would drop and place string even do? I mean, if you mean if you have a star mute string can you drop and place it? You absolutely can drop and place a star mute string and it would drop the string correctly but it wouldn't drop whatever this pointer type was. Like if this originally was a box string then you would end up dropping the string but not the box. As you would call the destructor for string which deallocates the heap memory used by the string but you wouldn't call the drop method for box which frees the memory associated with the box itself. So that would be a memory leak. Oh, yeah. So one thing we could do actually in order to keep the deleter as a trait is impold deleter for fn star mute din drop. Delete pointer star mute din drop. No, that won't work. We would need a self and we don't get a self. Maybe we do get a self. Yeah, the problem is we don't get to store anything except a single function pointer. It's like awkward because it's really just because of the lifetimes that this gets awkward. Like you need to make sure that the deleter is valid for is valid for the entire duration of the domain. So I guess we could just require that it's static, right? Like, okay, here's one way to get at this which is delete of, see, maybe it just takes self. Yeah, so we do this, we do this, we do, so this is just gonna be this of pointer and then I guess this is like a little bit weird, but let's go with it for now. And then what we do is on retire, we take a deleter and that's gonna be a tick static din de, din de, din deleter actually. So it doesn't even need to be generic over D and same thing here, this is gonna take a tick static din deleter. This is still, this is more general than what we had before, not by much because of the static bound but it is maybe a little nicer. And then what we do is we go here and we say this is gonna be a tick static din deleter. And now I think, actually, I wonder, I think closures that don't capture from their environments are considered function pointers. Like if I do this, yeah, I was afraid of that. That's fine. I mean, it can just be this instead. I think expected one argument found zero. That sounds like a lie. I mean, I feel like I did something silly, like the syntax is like not valid. The trait bound fn star mute din drop plus static drop box implements deleter is not satisfied. Let me, I want to, I don't believe that. I want to see the output. Oh, something else failed. Deleter dot delete. That's fine. Now there are two v tables but that's sort of what we would need this to be anyway to allow the deleter to be somewhat stateful. Like in the future, this might not need to be tick static, right? It's just going to be tick of the domain which might be something we can find a way to express. At least this way, we're not building our way into a corner where we won't be able to add it in the future. Yeah, the C++ library uses this like, it basically constructs a function that calls the deleter which is basically equivalent to what we're doing here. Except that you would have to, in that case, you would have to allocate. Could the reclaim API be like, reclaim T? No, because you no longer know the T once it's in the retire list. The retire list contains all sorts of different types. We can maybe do something here with like type IDs but it gets real wonky and I don't know that it gives us much of a benefit here. I wonder why did I mess up my trade implementation here? Is that what it wants? Could a delete closure be saved along with a pointer in the linked list? That's basically what we're doing with the vTable, right? Restoring a closure is really just a function pointer. A function pointer with some associated state. Ooh, I hate too many dry notes. So storing the vTable is sort of equivalent because the vTable is the function pointer for a closure. So the reason I didn't write, the reason I did not write this for f where f is fn, the reason I didn't write that is because it's a blanket implementation which means that no one will be able to implement this trait themselves for their own type. So that's why I didn't write that. I'm a little confused why this isn't allowed because like this function signature looks an awful lot like this. This question is what does this mean? I'm guessing that's the name used for it to just help me figure out which one, but are you planning on making a fuzzing harness for this? That'd be really cool. One thing that's challenging about fuzzing stuff like this is that because it's so concurrent, you sort of need to fuzz the concurrency. That's not always trivial. One way that it might be doable is to like, you generate a, you basically fuzz a command sequence and a schedule and then you execute that. That could be really neat, right? So you generate like, here's a sequence of 100 commands and you generate a schedule among like three threads, a schedule among those commands and then you run them. But it's just really hard to fuzz concurrency like this because you don't really have control over the relative ordering of events across the different threads. Maybe we could do it with something like, like Lume here might work really well. Like testing those with Lume seems like a really cool idea. I think we still have a little bit to go before we get to that, but I think that's probably something we wanna do. Yeah, Lume would give you assurance that you're actually handling every schedule, which is basically exactly designed for something like this. But why, oh, why, oh, why does this not implement deleter? Like if I do let X is static din deleter is equal to drop box, like does that complain? That complains. If I here do underscore, does that complain? Okay, so it is static. If I do fn star mute din drop, oh. Like am I just being dense here? Like why, these are the same. Hey compiler, right? Like if I just do this, these are the same and we've already confirmed that it's static because I can do this and it doesn't complain. So it must be tic static. So this part is not a problem. I think it's because it's an inner function for some reason. Like if I'd clear it out here and then do out drop box. Huh, no, not that either. So it's not because it's an inner function either. That's weird. This feels like a bug. All din pointers have drop in their V table. Yeah, I know that, but I can't, but look. If I don't take drop as a bound here, then I'm not allowed to cast to a din drop. So I mean, I guess I can just do like, what else would it be than din drop, right? Like din, there's no other trait here. And I can't drop this because if I drop this, I'm not allowed to cast into a din drop in the first place. But see, this is trying to cast it to the function type and I'm not allowed to do that. Okay, what if I do this? Like, can I do this? Okay, and now can I do, why is, this is bizarre. Why is tick static of that? Can that be and X? What? So if I give Y here, X here, and get rid of this. Okay, this is a more helpful error message. So this is just saying, you can't take the, you can't take the address of X because X is on the stack. That's fine, but the address of drop box, I guess maybe it's just like, it's just drop box itself is the function pointer. I feel like this is really what it should be. Oh, what did I do? Right, I should not have deleted that. And now the problem, right. I think what's going on here is, this is not a meaningful type because this is saying, like drop box is the address of the drop box function already. So what even is the address of the address of drop box? Like that's not a meaningful sentiment. And I think that's what it's trying to tell me. It's telling it very poorly, but I think that's what it's trying to say. But if I try to actually just pass in the address, then it doesn't know that this is already a, it doesn't know that this is already a pointer type that can just be turned into a DIN. Like I'm saying that I need to take a reference and this isn't the reference. So maybe this really just needs to take a, and we need to cast it to a V-table pointer. It's just really not nice. Basically, this isn't a fat pointer. I think that's what the compiler is trying to get at here, that this isn't a fat pointer and you don't have a way to give me a static fat pointer to it. But maybe if I do like, okay, static foo is an FN of star mu DIN drop is equal to drop box. Can I now give and foo? It's so dumb. It's so dumb. Yeah, because this is a pointer to a pointer. It wants a pointer to a thing that implements the trait and that means in our case, a pointer to a function pointer. But this is clearly not an ergonomic interface. But we can't have that be the interface. I mean, another option here is the like retire just takes, but the reason we have to do this, I think, is because we need to construct a fat pointer to pass the retire and the raw function pointer doesn't have any data associated with it. So there isn't really a translation, but by constructing a reference here, it sort of, I guess, understands which input to use. I really feel like this is a bug that the compiler should be able to handle this for you. Okay, so maybe what we do here is like, so silly, it's so silly. But maybe what we do is for, at least for the time being, we say that this is a pub static. And then we do the same thing for drop in place, right? And then we document the pub static. Can I say that a static is unsafe? I don't think so. Right, and then now in the test, I can have this be, I guess the leader's drop box. In fact, I can go even further than that and say, oh, this is even worse. What I'm actually gonna do here is do pubfn drop in place, takes nothing and returns a tick static dindaliter. And it does this. How do you like them apples as it goes, as the saying goes, and then drop box is gonna be static dindaliter. And it does this and it returns drop box. And this one is unsafe. Nice. Now we actually do get to document all of our oddities. And then down here, this is now gonna be the leader's drop box. It's real stupid, but it's the best way I know it for us to work through this weirdity. It does not work with a cast. I already tried it for the cast. What cast are you proposing to use? Oh, you're saying, all right, let me, so your proposal is down here that we can use instead of this, all right, drop in place, but whatever, as fn star mute dindrop reference. I think you're lying. Oh, like this. But why? Why? This should not be necessary. This cast should not be necessary. I don't know if this is better, actually. The reason I don't know whether I think it's better, like, okay, first of all, why on earth is this work? Or rather, why is the explicit coercion necessary here? Because that's already the signature. This shouldn't be necessary. But at the same time, I kind of like this because it lets me document these separately. Like, I could provide drop in place in Dropbox as just function pointers into liters, but there wouldn't be a way to annotate that one was unsafe and the other was not. Although I guess retire is unsafe anyway. So maybe, okay, so maybe, maybe you're right. Maybe what we can do is where is my deletors? Like, I don't want that cast to have to be there in user code. Can I make it a static, oops, drop in place? Like, if I do this, I guess that is another way to do it. And then specifically say, allow uppercase static. What's it called? Non-uppercase globals. Non-uppercase globals. And then can I do the same thing here probably? Dropbox, Dropbox2, Pub, Static, Dropbox. And then this, I don't need this anymore, that's fine. And now this can be this. Okay, now I'm happy. Great. Yeah, it's just like, the ergonomics would be too bad if the caller had to actually give this cast. The pointer came from box, so it's valid. Two, the deleter is valid for box type. And three, the old value is no longer accessible. And I guess let's number these so that it's easier for people to document their safety. Nice. Do we even have any more to-dos? Get rid of the sub-quirement, better heuristics. Okay, so I guess maybe it's time for us to run the test. Okay, feel good passes. So if I do assert equals this, 42. And I guess down here, assert equals x, 42 still. And I guess I'm going to do the swap down here. Oh, so here's what I'll do. If I create an H, oh, it's confused. And then I load this H here. I assert that I can still read 42 there. I can still read 42 here. And what else can I do? And then if I now do, I guess old retire all, we're going to need some like eager reclaim. So that's fine. And at this point, after calling whatever that is, I should still be able to read that. And then if I then drop H at that and then run old eager reclaim, like to do check, actually reclaimed. So I loaded before the swap, so I should still see 42. I should still see 42 all the way up there. And I guess after the swap, if I do an H two and load that, then X two should be a 9001. And here, if I drop H, not drop H two. Actually, I guess there's also an argument here for, I create a thing that loaded, and then I guess this is H temp. Like I'm going to have one that loads the old value, but then goes away. And I want to make sure that doesn't hold up the reclamation either. So at this point, eagerly reclaiming shouldn't work because I still have this holder. Same here. Here, I should be able to eagerly reclaim. And then here, I should check that it actually was reclaimed. Here we can do something around like the magical drop implementations or something, right? And eager reclaim is something we can add on, actually, what do we even, it's not entirely clear what we call eager reclaim on because we've already given a well retire. So this I guess should be shared domain. I think this has to be on the domain. So there's a pubfn eager reclaim on self, which just does a bulk reclaim. And I messed something up. It's borrowed. That's fine. Does that all pass? Okay, that all passes. That doesn't tell us very much. We don't actually assert that things aren't dropped because it could be that it like, this retire actually drops the 42, but the memory isn't reused for anything. So it doesn't end up being a problem. So what we're going to do is add a little sneaky type that's going to be like count drops. So we're going to have a struct count drops, which is going to hold an arc of atomic U size. We're going to impl count drops, and that's going to be self.zero.fetch add. We could probably use like a mocking library here to make this a little nicer, but I just want something that's simple in this case. So 42 and count drops drops is an arc new of atomic U size new zero. This is going to take drops assert equals my x dot zero. This is just going to nice. This is going to be count drops, I guess count drops of drops over here. Let drops 9001, and this is going to be, I guess drops 42, just so we can keep them distinct, that's all fine. And then here, we're going to assert equal count, no, drops 42 load, is still equal to zero. Here it should still be zero. Here it should probably still be zero. Here though it should be one, and the 9001 drop should still be zero, and this can be zero. And borrow of moved value, that's right, because this has to be arc clone of this, and this has to be arc clone of this. Oh my, I can't type anymore. All right, let's see if that does the right thing. That does the right thing, wow. Did we just write error free code the first time around? I don't quite believe it. You can detect that with zero wise being implemented on the type. Yeah, there are a bunch of libraries that aim to help with this kind of stuff, but it's so simple to just set up this type for ourselves here that I'm not too concerned. I guess we could like, Imple count drops fn new, which returns to self, which is a self arc new, but I don't really wanna do that. It seems more worth that it's. Yeah, I don't believe this yet. There is, so we're also gonna want some compile tests here, some compile failure tests for things like this being invalid. And for this, oops, where's the other, this one being invalid? That's neat. I mean, that seems like it works. That's a little bit scary. I mean, remember, we haven't done anything about the, like we only have this shared domain business, which is something that we would want to fix up. I guess actually, like if I now do say, drop h2, and then this, then this should still be zero. So silly test, but there are almost certainly things missing. Like for example, there are memory barriers that are not documented in the paper, but the implementation has. There's also all of these memory orderings that need to be corrected because currently they're all sequentially consistent. And in reality, that's not what we want them to be. What does eager reclaim do? Oh, eager reclaim is like, if you specifically want to make sure that you free memory before you do something else, eager reclaim is what you would call, right? So it's, I don't have a thing to, I don't have a thing to retire right now, but I still want you to reclaim things that have been retired in the past, right? Because remember, a call to retire might not succeed. Like it might not be able to reclaim the item yet. So eager reclaim is a way for you to say, I now want to reclaim things that I retired in the past, even if you have to block in order to do so. So arguably actually, eager reclaim should say, this should be true, this should be false, and this should be like block as a bool. And then like, there should be some kind of here, like a, where's the best place to do this? I'm gonna go with down here. Actually, it's really sort of around here somewhere, but I'm gonna for now just say, if block, then return self dot bulk reclaim. If, if tail dot is some, and block. We, caller wants to reclaim everything, but somewhere left, so try again. And here we can do like thread, yield now. All right, we can be a little bit nice about retrying. Oh, we can do like thread, yield now. All right, we can be a little bit nice about retrying. Oh, fine. I guess if remaining is not null, then do it again. And I guess this could actually be a parameter here, right? Like, do you want to actually block or do you want to just try? And so here we could say, in all these cases, we don't actually want to block, because we sort of know that blocking shouldn't be necessary. And maybe eager reclaim returns how many things were reclaimed. Maybe that's nice. So this is gonna be return zero, and this is gonna be return reclaimed, and this is gonna be reclaimed. So now our test can actually be like assert eek and zero, and we can do the same thing. Here it should be one, and here it should be zero. Great. And then the one thing, okay, so we are getting pretty close to time. One thing I do want to run though is cargo Miri test. I want to see if Miri spots something silly here. Like Miri is particularly good at spotting incorrect things in this kind of unsafe code, and it might spot something silly we did. Like ultimately I want loom test here too. The following memory was leaked. That's fine. So we are expecting this to leak memory at the moment. There's certainly like a info drop for has pointer domain to do. But that's fine. I'm not too concerned about that because it's sort of a it's a known memory leak. And that doesn't bother me. Arguably, yeah, that's fine. Like we are gonna have to implement drop for has point domain. I think, can I even do this? Like if I do this, will it panic? No, great. Yeah, cause destructors for domain, destructors for statics are never run. Great. The recursion of bulk reclaim should be added to reclaim results. You are completely right. Which actually means that it can't be tail recursive, which is well said. We could make it be tail recursive actually. Reclaimed. Did I even use a claim for anything? I don't think I did. So this way it can be tail recursive because this is still just a recursive call. If this was instead reclaimed plus, then it couldn't be tail recursive because you need to run something after the function returns. It won't necessarily be turned tail recursive anyway. Allow tail recursion by passing down reclaimed. Yeah, Rust doesn't guarantee tail recursion, but it is nice to write things that can be tail recursive once we get something like become, which is a keyword that's reserved. But even so, like, if we didn't do this, it couldn't be tail recursive, even if the compiler sometimes made it. Yeah, there are a bunch of sanitizers we might wanna run here, like the thread sanitizer and stuff. Miri will work perfectly fine just here. The challenge with Miri and with sanitizers in general is that they will only check the code that you wrote, like the code that actually ran, whereas very often the concurrency bugs are in particular interleavings that you haven't run, right? And that's where a library like loom comes into play where loom will ensure that every possible interleaving gets executed or approximately, which means you're much more likely to find these kinds of problems. And then you can run Miri on top of loom, but then things run really slow. I could make it a loop, but making this whole thing as loop is actually really painful because the whole thing ends up being very long. That's why I didn't wanna do that. So I intentionally wanted to do it this way instead. Great. I think now that we actually have a test that passes and code with no to-dos except features, I think that's a good place for us to end today. I guess I will push this code to GitHub. It's not super interesting yet. Like it's a pretty simple implementation of hazard pointers. And there are obviously things that we want to add. Actually, I just realized one thing that we missed, I think. Yeah, spotted a thing that we missed. Oh, no, nevermind, nevermind. There are a bunch of things we wanna add. Like I'm guessing for the next stream, what we'll probably do is implement domains so that they actually work so that there's not just the one static domain. Maybe we'll take a look at like the other features. We'll certainly fix up all the memory orderings and the atomic barriers and stuff that are necessary. And then probably write a little bit of a test suite with Loom because I'm just like not convinced that this actually is correct. And that should give us some more confidence in the correctness of the implementation. Let's see. Yeah, it's a haphazard implementation of hazard pointers. That's right. That's very funny. And of course, this is like better heuristics for this once in a while. Like it seems like the Facebook library has a much smarter implementation here that we'll want to investigate now that we have the basic structure up and running. But I think that's all some things we're gonna save for next time. So I guess, are there any sort of tail end questions from chat on everything that we did today and sort of the path we're gonna go next? And then after that, I think we're gonna call it quits for the day. Ibrahim, I know you were very much screaming at the screen earlier to get me to realize the cast for the function border. I'm glad I caught it eventually. I do think we ended up at a good compromise though of having the, there's too much code on screen now. I really need to split this up. Of having, where even is it? Having it so that the caller doesn't need to do the cast. It's a little bit of like an ugly trick here, but I think it's the right thing to do. All right, if there are no further questions I think we'll end it there. Thanks everyone for watching and I'll see you in two to three weeks when we do another one. So long for well.