 So we're going to talk about, so today we're going to talk about unraveling RC usage mysteries with additional use cases. This is a follow on to the December 7th presentation. And so this is what we're going to be doing. We're going to be a really quick review of last time's presentation. We're going to look at the use cases and we're going to go through some of them. So a quick review and the URL at the bottom of the page is the URL for the previous presentation in case you want to go back and look at that again. But to start off with the big issue, the thing leads to RCU and also hazard pointers is noted there is the global agreement is expensive getting all the CPUs all the threads everything to come to agreement on something takes time. And this is due to the finite leaders speed of light and also the fact that items though very small are not zero sized. And as I said last December, if you had told me 50 years ago that I would be saying that the speed of light is too slow and that items are too big, not for space travel but for computing. I don't know what I would have said but I guarantee you I would not have believed you, but here I am. So the way around this is we use both spatial and temporal synchronization read or write locking strictly uses temporal. We're going to add spatial to that RCU is one way to do that and again hazard pointers is another way. So we'll do a real quick run through of how that works here. So here's the core RCU API and we have the temporal. Yes. I'm so sorry to interrupt. I think perhaps the zoom menu bar might be blocking your titles on your. Oh really? Yes. Okay. Other people can see this too. That's interesting. Let's try clicking here and making it go away. Let's try and all that doesn't make you away. No, it didn't really seem to do anything there yet just looks like there might be a black box above the titles. So we're like right now seeing RCU some man. Okay. So right now you see RCU core a bunch of garbage and then spatial. That's right. Wonderful. Wonderful. I love zoom. Yeah. Okay, let's see if I can do something to make this go away. Or perhaps if you just read the title out and then we'll make sure to catch this in the slides. Okay, what I'm going to do is I'm going to stop sharing and start again and see if that causes something to explode. Okay. Okay, so I stopped. And then I started again over here. And then I do that and I hit share. And there's that better. So yes, and yes, the title you we see it. Yeah, okay, we see the majority. The top of the tea is probably chopped off. But the rest of it is there. Yeah. Okay, well I'll read the titles out but now people can see them better so thank you so much. Okay, thanks for calling my attention to that. As I said, I love zoom. Yeah. Okay, although you gotta admit, if you'd have told me that I'd be presenting to people scattered all over the place in the comfort of my own office. 50 years ago, I wouldn't believe that either. So yeah, I would go that's a salad goes I guess. All right, so Corey RCU API temporal versus spatial. So the blue stuff is temporal. So we mark readers starting with RC read lock and RCU read unlock, giving the time of reader starts and ends synchronize RCU and call RCU or the synchronous and asynchronous respectively interfaces to wait for all pre existing readers. And then those so that's the stuff dealing with time dealing with space we have our CD reference to load an RCU protected pointer. And then the reference protected for the update side, if there's a lock involved RCU assigned pointer to update a pointer, and those could, in theory, just be assignment statements, but CPUs and especially compilers can be a little bit obnoxious at times if you're trying to write concurrent code, and those keep both the CPU and the compiler from doing silly things. And then elements of the API there's quite a few more. And at the bottom is a URL of an element article talking about those as of January 2019 there's been a few added since but not that many. Okay, going on to RCU semantics graphical. So we've got that core API and we're just talking about three of them now RCU read lock and RCU unlock as you can see delimiting at a reader, and we'll start in the upper left hand corner there. So in this upper left hand corner we have a synchronized RCU that started after reader did. And that means that reader might see the data that was removed, it might have a reference that data, and that means synchronize RCU has to refrain from returning until that reader gets done, at which point you can free the old memory without worrying about messing up the reader and let me tell you, user after free is a bad idea, and having RCU and conjunction with user after free doesn't help. So that's important. However, on the other hand, if the synchronize RCU started first. Well that means that this is where in the upper right hand corner now. So in that upper right hand corner synchronize RCU started first that means that the reader can't possibly see the data that was removed. So that reader can't be inconvenienced by that memory being free so it's okay for synchronize RCU to return. But despite the fact there might be later readers are still ongoing, because those later readers can't see the stuff that was removed. And so that's the key benefit here we've taken the update and split in two pieces we have a removal part that's visible to the readers, and then we have a free part. And so we put enough time between those two, so that the we wait for the pre existing readers don't have to wait for the later ones. We can do both we can have both belt and suspenders that's the lower left here. If we have a reader that showed up after the removal so it can't see the data that's being freed it's send it gets done before the memory is freed well that's doubly good because it didn't see the memory and it wasn't freed until it was done anyway. So the bad case the you have a bug in RCU case is in the lower right hand side. We cannot have a reader that starts before a synchronize RCU starts and ends afterwards, because you can see right there it could get a reference that old data, and it was might still be using it when it's freed and that's again a bad idea. So that kind of gives you a graphical idea of what the constraints are to puts on time ordering. Okay, the next slide is RC max again but looking at restrictions on RCU gets its performance by being a specialized tool. Alright, it doesn't try to be everything to everybody. And that allows it to work very well where it's where it's designed to work. And so as a general rule of thumb, the more you're reading and the less you're updating the better RCU is going to work for you. This may sound strange but the speed of light applies to things outside the computer. And what that means is that if we get a change in the computer. By the time that changes in the computer it's already old news. And so if we have an algorithm is check tracking external state. And they should know state doesn't have to be all that external, even like devices connected to the CPU is external from this viewpoint. Then we're already dealing with staling consistent state anyway. And so a little bit of additional timing consistency is often okay. In the lower left hand corner that red triangle. That's the place where RCU is not going to be working very well for you. And 20 years ago I would have said just don't use RCU there. But one of the really cool things about working with Linux kernel community is they're able to come up with new things that I hadn't considered and teach me new things and they taught me two reasons why you do want to use it there. We'll be talking about the first one will be showing a case where RCU even though you're updating all the time. RCU is providing some protection to simplify non blocking synchronization. The other case is if you're not reading that often when you're reading their severe latency constraints. And in that case you might want to use RCU as well. An additional thing to consider that RCU is frequently used for linked data structures now in December we saw a case the face state change where there was no no pointers or links anywhere in sight. So it can be used in other cases but the majority of the uses the most common uses involved link data structures. All right, so that's the restrictions. Here we're just kind of showing the cost of global agreement title there. And this showing reader right locking first. And so what's going to happen is we got a bunch of readers, let's say we've got something where we have readers all the time. And that means if an updater shows up. We have to wait for all the readers to get done. And then we have to wait for the agreement that the readers are done to propagate throughout the machine. So no new readers start up. Then we can update. And once the updater is done we have to wait for agreement that the updater is done to propagate through the machine and then the readers can start again. And this big red area there those five red bars are that's lost time they can't be made up. And of course as you add CPUs that right area gets bigger gets bigger in two ways first you have more bars obviously. And secondly, the agreement latency for larger machines tends to be larger than for smaller machines. And that can be a problem. I mean, obviously this is a problem for in terms of latency even for aggressive real time if you want your readers to show up and you want it to be sub millisecond this can be a problem. But even for more normal workloads data data centers for example, where you have web based latencies if your updater takes any particular length of time, which often they do. This can be unacceptable in data centers as well. So we want to do this with RCU on the bottom of the slide. So here we have readers all over the place. And we split the updater and to like we had before on the previous slide we had a removal and a free. So we got the removal and free their updater one and updater two in the middle bar the kind of greenish blue. Sorry if it looks different to you I'm red green color blind so I'll give you approximations approximate names of the colors. What's happening here is we have very little overhead. Now, there's going to be cash misses the updater removes something that's going to cause a cash miss so the readers are running right at that time may see an extra little bit of time for a cash miss but there isn't the big agreement latency, no big red areas. So we can get much better latencies and much better efficiency as well. And so let's go back and look at a particular slide from the seventh, yet December 7. And this is the one where we show the spatial and temporal synchronization so time is advancing from top to bottom. And because we aren't don't have that big red thing if we had the big red thing then readers either see the old value of the new value and they'd be at separate times and ever to be very very understandable, and very organized and very control friendly, but we'd have that big gap in the middle we'd be paying for that privilege. So we do is use the address space in conjunction with the time to straighten things out. So above the horizontal dashed blue line. Everybody agrees that we're dealing with the old data. There's only the old data the new data isn't there yet. If we look at below the second blue dashed line, everybody agrees that there is new data and no old data anymore. Between the lines, different readers may get different either the old data the new data but a given reader. Any given reader will get one or the other it will get both. And so it turns out now there's some cases where that's not sufficient and we'll talk about a way of working around that later in this presentation, a particular use case of the quasi multi value multi multi version. Consistency control slides will talk about that. But for the meantime, if we don't need that degree of consistency we just need a given reader to be seeing the same consistent results this will guarantee the consistency using time on the one hand, and address space on the other. Okay, so we can map that back onto the previous slide and show the readers in blue that are guaranteed just the old value. The green or guarantee that T to see the new value and the kind of greenish bluish color in between there are ones that might see one or might see the other. But once a given CPU or thread start seeing the new values that CPU threat thread will see the new values forever after. So we don't know ahead of time on the top thread which of those two readers the third and the fourth counting from the left. Once it starts seeing the new values one of the other will or or maybe maybe they'll both see the old values but once it starts seeing the new values it'll see the new values forever. So that kind of gives you a pictorial representation of what will be happening to the readers as the update is going on. Okay, so that was our review. Again, we had the URL of the previous presentation people want to dig deeper into a review. So we'll look at the, the diagram of the use cases. Now face state change we talked about last December and we're not going to talk about it again, quasi reader right a lot and we spent a lot of time on last December, and we're going to just add a couple of twists to it this time, if we have time. I'm going to go through all these if we have time if we run short of time I'll skip one of the ones that I believe are less important but of course Murphy says those will be the ones you need next right. Yeah, life's like that sometimes. Let's start with the add only list. So the add only list is over in the lower left corner there. It's connected just to the link publish subscribe and that's intentional. It is very basic. It doesn't add much to just raw RCU. All right. So, let's illustrate the with code but first we're going to start with a typical add delete list because this is going to be more familiar to those of you who have been through the Linux kernel. This is an updater, and these be running in separate threads. The reader is in started by our street lock and our end with our street unlock as you'd expect lists for each entry RCU just as an iterator that goes through a list. The list is going through is this RL thing, and the iteration variable is P. Okay, NXT is just the field in the structure that has the list head pointers that is uses to do the iteration. So what it's going to do is going to go through the list and for each element is going to call do something. And it may be that that element gets removed while we're doing this but it won't be freed so to do something will have real memory to work on it's not going to have anything yanked out from under it. Of course that means the updater has to take care. And so let's take a look at the updater. There's a global spin lock ml. It's going to acquire that and it releases it at the third to the bottom statement, and that avoids any interference from other updaters we have multiple updaters trying to run at once that locks and protect us. That lock has no effect whatsoever on the readers, the readers just still piling through while we're doing this as we saw in the previous diagrams. So the next line does list first entry to pick up the first element of the list, whatever it is. And then the next line does list lrcu that removes that element, the underbar rcu causes it to leave the next pointer in place. That means a reader that happens to be looking at that element just as remove it will still be able to get out of that element to the next element in the list. Okay, so we list del by itself would poison both the previous and the next pointers list lrcu poison just the previous pointer leaves the next pointer there so the readers can keep going through the list. Then we, we've got some pointer q that we somehow allocated initialize somewhere earlier, don't worry about how that happened. I wanted to fit on one slide and it does this way. So we do list at rcu to add that to the beginning of the list. So we take that q pointer and add it to our list, we release the lock. So now another updater can can come in and do things. So this is a list of rcu to wait for all the pre existing readers, and those pre existing readers are the only ones that could possibly have a reference to P, because we use list l rcu to remove it. So once we get done waiting for all readers once that synchronize rcu returns, we can safely do a cave free of P, and get rid of that thing. So this is an add delete list the standard thing sort of thing you'd see in the Linux kernel, usually either add or delete rather than doing both at once as we do here but again one slide. So what we'd have to do this thing to make it be. So we remove code to make an add only list, with one exception we have to add a true to keep locked happy on the list for each entry rcu. That true doesn't affect the operation of it just tells lock depth hey, I understand I'm not in a reside critical section and it's okay don't complain. So because we're never removing anything we don't have to mark our street lock our street unlock. So those can go away on the updated side we're not removing we're just adding so we can obviously get rid of the thing to grab the first element and delete it. The list first entry list l rcu those go away. And we didn't move anything so we don't have to do a synchronize rcu we don't have anything free. So those go away. So the red crossed out lines are going to go away. The next slide will show what we have left over. So the reader just iterates the list and does something each one. We're only adding we're not moving so the elements can't go away. We don't have to worry about them going away they don't ever. And then the updater is only adding something. There actually is code and links kernel it does this approach it may seem kind of silly why would you and that's memory leak right you're only adding you're never deleting, but there are some situations where you occasionally have changes, you need to keep track of the old state that's happened since this boot. And so they the list just grows, but for various reasons they can't go very far perhaps you have devices that are inserted. They can't be removed, or they might have devices that we brought online. Once they're brought online you add them. If they're moved you see keep track of the same list. And when you add them again you use the entry for example. That way the list can only grow as big as there are the number of devices. And you never have to worry about deleting anything because it may be a movie still keep track of the fact that it used to be there. So that's one use for that. Okay, here's how the thing operates and this time I'm showing the allocation installation that I left out of the example. So we have our list RL the red box in the top and time is going to advance from left to right. So we have this list, we have the arrow showing the operation that changed it or color coding it. The red boxes are things that readers can get to the green boxes are things that readers have no way to access. I mean, aside from just going and taking through memory which is a bad idea anyway. All right, so we allocate the thing. It's uninitialized. And we end up with the second state there with the RL pointer still being null, but having a block of memory to play with. We initialize it and now it's no longer defined we've got a equals one big two C equals three and so on. We use list at RCU and suddenly readers can get at it, and they use list for each entry RCU to iterate across it, and life is good. The reason this works is the list at RCU is careful to write make the compiler write the pointer in one shot. It doesn't write invited time or something. And let's reach entry RCU forces the compiler to load it in one shot. And if an architecture like that alpha requires special instructions to make the ordering work out on the read side. It provides it. And if an architecture do require list at RCU to add appropriate ordering. So it does so when when needed. So that's kind of what is happening as we run the update, and the readers are going through all the time. And so we have a way to add stuff to list without blocking the readers to wait for an addition, and still having things come out straight or readers can either see the null list or we'll see the new element but it's not going to see some something strange. Okay. So if we've added three elements of the list over time, it looks like this. We have a previous and next pointer. That's the NXT field we saw earlier. And we have our backwards and forward pointers. The pointers loop around the back or read because I'm going to leave them out of all of the next slides that have this list because they get in the way. All right, they're still there. You can put them in your imagination, but we need to decorate the list and and we've got to make room for it and so those pointers go. We might have a lock will have some cases that use that and we have other data. For example in the previous slide we had a B and C being in the other data. So the way RC is working is that we're spreading out synchronization responsibility among different mechanisms. So, in this case we're only adding, we have the red dotted line indicating the synchronization responsibility of the ML lock. The updater did a spin lock on ML, and it was doing that to protect the previous and next pointers of the elements. It's inserting a new element and it's going to have to mess with the previous next pointer of the adjacent elements that are being inserted between. And then RCU published subscribe in blue there. And this is embodied with a list for each entry RCU is making sure that the initialization is visible to the readers they aren't going to get a pointer to something and see what was there before the initialization happened. So that's how we split things out. In some cases, the other data might be mutable. And so in that case, the lock would come into play. And that's the yellow boxes there and typically what would happen is the lock would be responsible for guarding the other data. So that means using RCU why are we throwing a lock in the key thing is that ML lock is a goal lock. If we have much of any contention on that life gets hard. These locks in the data structures themselves. Those are per data structure they're separate. And so the contention gets spread out overall the elements. So as long as you don't have some kind of a hotspot in your data. That means your contingent stays recently low or can stay recently low. Okay. So, going kind of summing up RCU to add only list, we take publish subscribe for link structure and that was that white box in the lower left corner in the, in the, you are here diagram sometime back, and we don't add anything to it that's all we have. It's very simple. It's very basic a very primal if you will use of RCU, which is why it comes first. Okay, let's advance the lead only list, we have an add only list we have delete only list as well. It may sound equally silly, but we'll talk about a possible use case so we're over here it's based on existence guarantee so it's a little less primal that that only list was. So if we start with the add delete list I'm going to go through it again this is exactly the same as some slides back. We'll look at what we change to get from an add delete list to a delete list, remove code for delete only list here. And not much happens the reader. I mean, we could I mean there's a embedded in that list for each entry RCU as an RCU do reference and it could be read once instead but that's not going to make much difference, especially not in recent kernels. But we're not adding anything so we get rid of the addition in the updater. So when I said that delete only list is less primal than an add only list I met it, you can see that there's much less change in complexity from the add delete list. So why would we have a delete only list. You maybe you have something is keeping track of memory devices and maybe you have to stop using them with they're showing failure rates if they're showing ecc errors for example. And in that case, you might have something keep track of each memory but once it's become unreliable you may just delete it from the list and never pay attention to it again. And you may not have a way to actually plug a new memory into this system it's that's not that common a capability these days. And so in that case, the memory can only go away it couldn't come back and you'd have a delete only list. I believe I've seen one or two is the latest kernel but I couldn't tell you where they are, or if they still exist. Okay. So if we just show the code, you can see that it's the same pretty much we've gotten rid of the ad and here we are. So this is how the delete only list works we're adding a another color yellow. And that's a case where readers can't get it at it, but old readers might still be there. So we start over a list and it has a cat tucks on it. It's all read there could be readers there anywhere. We do list lrcu on the cat. Okay. At this point, our L points to tucks now and new readers have no way of getting to the cat. So there might still be readers that happen to load the pointer from our L before we did the list lrcu and they might be there for quite some time, they might be who knows what they're doing to the cat. Especially the name shrugger. But in any case is an old readers. That means we do synchronize rcu those old readers have to have been completed before synchronize rcu returns. So the cat now becomes green it's safe. No readers have access to it and we can safely cave for it. Okay, so this is just going through how the lead only list is working and how it's keeping the readers safe. We have very similar looking synchronization responsibilities. We're getting from the list, and that's guarded by a global lock ml we saw the spin lock and spin unlock of that thing a few slides ago. And we still have to prevent the compiler from interfering with readers. And now we don't have to worry about initialization, because we're not adding anything however, we do need the compiler to store the changed pointers in a single instruction store so that the reader, which has to use a single instruction, doesn't see some mashup of a couple pointers that is bad for the actual statistics of your kernel, as you may already have experienced. And again just like before, we could have a per element lock in case the immutable data, where we just scan the list to find what we want we do that locklessly. When we get to the element, we may find out that we need to modify something and we may need to do that under the protection of a lock. So we have that option to delete only list as well. The lead only list is interesting in that we're taking the existence guarantee and removing something from it. We're no longer relying on the publish describe piece but we are relying on all the other things that make up the existence guarantee. Okay, so let's talk about the existence guarantee. It's right below the lead only list. It looks like the lead only list adds to it, but that was a head fake it's actually removing something for it. So the existence guarantee is actually less primal than the lead only list. So what we're doing here in this code is we're going to do a reader then updater sort of a thing. So we have a reader and we start with the R3 lock there and the reader answer the R3 unlock about a little about two thirds of the way down. And we're going to look for an element and we're going to put the point of that element in Q. So we start off with Q being null. We go through the list using list French each entry R so you just like we have for all the previous examples that's going to discuss step through the list, each in turn. And if we have the right key, we set Q equals P to remember that we found the thing, and then we acquire the spin lock. And so what's happening here is RCU is making sure the element stays around long enough for us to acquire spin lock. If we didn't have those R3 lock and the R3 done lock, we could end up with somebody removing the element freeing it just as we're kind of spin lock and that's not good for your kernel either. All right. But because we're in the RCU readside critical section we got that RCU lock protecting us at the top. We can do that spin lock. And we can then break to get out of the loop. If we found an element Q will be non null. We check for being deleted and we'll see how that works on the next slide but if it's already been deleted, we don't want to do anything. We only want to do some update if it's been deleted in that case. Now we don't know the RCU protection anymore but we do hold the lock and we'll see on the next slide that prevents the updater from removing the element. Then we release lock and we're good. Okay, on the next slide. We're going to acquire the lock, we get the first entry. Now we add a spin lock. So we're acquiring the relevant lock and only now are we setting deleted true and deleting it. Okay, so a reader might come in and it might get us locked before we do in which case our spin lock will wait and the reader will go and it'll do its thing and it will call that function. Okay. If it shows up just afterwards it's going to see the deleted equals true and it's not going to do anything. Okay, we do our list at RCU for the addition. We release the global lock. And then we do synchronize RCU to wait for the readers and then we can safely free it. Okay. Now, here we are relying on the lock and we're relying on the lock to cover not just the other data as we did before but the entire element. And that's because the deleter has to hold the per element lock to do the removal. So it's protecting the next and previous pointers of the element being removed as well. Okay. Otherwise we have the same sort of thing. The global lock is also protecting the deletions also protecting any additions that might be happening. And it also also the list for each entry RCU is ensuring that the readers see valid pointers and also see initialization of added elements. Okay, so again we have a delegation of different responsibilities to different synchronization mechanisms, making it all work together. Okay, so again the lock is doing more than it was before. It's protecting the other data and it's also preventing the corresponding load from being removed. The lock is either already been removed or it won't be removed one of the two. Okay, so we're starting with RCU RCU to a distance guarantee transformation, if you will. We start with both wait for readers in the public subscribe both of the white boxes that were at the corners of the bottom corners of the diagram earlier. And we're using a heap allocator and we're using deferred reclamation deferred reclamation being either synchronize RCU or K free RC or some of the newer fire and forget tile cell primitives. All right. The distance guarantee is fairly powerful. If you're in a reader you get the thing is guaranteed to exist. But sometimes it can be expensive. And so, for those times we have type safe memory. And so type safe memory is a similar to existence guarantee is a little bit weaker. It's a little bit harder to use but you get in some cases you get more efficiency. So let's take a look here. The way type safe memory is is implemented Linux kernel is you have a slab allocator. Okay, you do came came in cash create and you pass of the slab type safe by RCU flag. Okay, and this is actually an approximation of real academic or type safe memory but it seems to work fairly well for us and sometimes approximations are better than the original. And the, the advantage of this is you get better cash locality. If you have a type type a slab type safe by RCU slab allocator. If you free something, it can be immediately reallocated don't have to wait for a grace period to free it you just free it right now it'll be reallocated maybe immediately. And that means that the memory being reallocated is still hot in the cash. On the other hand if you wait for grace period as you do for the existence guarantee that memory may be colder in the cash and you may take a bit of a performance hit when you reallocate it. But there's no free lunch, especially since this is RCU and synchronization in general doesn't have much free lunch if you may have noticed. So that means the readers need a validation step, because you didn't wait for grace period so you may be a reader going and grabbing something and it may be freed and reallocated off from under you. So you have to make sure you've you're dealing with the thing you think you are. And so take a look at this we have a TSM state diagram this is specific to the Linux kernel. So we have the slap type safe by RCU slab cash came in cash thing. And we can allocate from it came in cash alloc there. Once we've done that over on the far left it's in use. At some point we may decide to free it, in which case it goes back to the slab. That element happened to the last thing in that slab we have an empty slab, which we free but we free that back to the page allocator, we wait for an RCU grace period at that point. All right. So, any readers that might have access to it are known to be done and have left drop the references before we go to the far right hand free pages cool. Of course, if came in cash alloc says hey I need something and there's no nothing to be had, then the slab cash is going to grab a new slab grab some pages make a new slab out of it. And so that'll come around the top on the right hand side so we've got to kind of figure eight here that the cash is run through. But the interesting part is the figure eight on the on the left, because a given piece of memory go around around that circle really really fast, there are no grace periods here. You can free it, it can be immediately reallocating be used for a little bit be free to admit and just whirl around really quickly. And most readers, perhaps not all of them, but the ones I'm aware of, they need to stop this, they need a way of stopping that churn. And that is what the validation is about. All right, so one way to stop the churn. And we're going to look at some example code here that's been kind of brutally chopped down and there's the full stuff on a particular tag in the dash arched tree at the bottom of the slide there. I don't intend to put that upstream but there's a tag keeping it for posterity. So what you can do is use a reference counter. And the way to avoid freed items is you use something called atomic add unless. So atomic add unless doesn't atomic add unless you have a specified value in this case we'll use zero. All right. And so what will happen is we'll do an atomic add unless and so what will happen is we'll do an atomic increment, unless the value was already zero which cases sorry, I couldn't do it. What that does is it allows us to reject items they're currently being freed. So it might have been freed then reallocated before we figured out what was going on. So even if we succeed in getting a reference, we also have to recheck the key and make sure it's the thing still the thing we want. So we have to conditionally acquire a reference. If we acquire the reference we have to recheck the key. So let's take a look and see how we do that. So we're showing a struct food we're going to use. It's got a list head, which is used in the example code and RCU we're not going to pay attention to it we're just going to assume that we know how to make RCU protected lists since we've gone through that several times as presentation already. We have an atomic reference counter ref, and we have a key, which is an integer. We like to have our own came in cash. That's our food cash there in the middle, and we created just came in crash create as you always would, but you have that red slab type safe by RCU flag you pass in to tell it hey wait for a grace period before freeing up an entire slab before handing an entire slab back to the page pool. You came in cash destroy just like you always would. One nice thing about that is it finds your memory leaks for you, which I had a few as I was debugging the code. Okay. You have to allocate initialize specially and we have a full Alec is going to do that for us on this page. We do a came in cash Alec. And if we didn't get anything we return null telling the sad story. Otherwise we set the key to the desired value that was passed in. And then we do atomic set release. Okay, so we have an initial reference of one. And that's kind of an implicit reference for the data structures the callers responsibility to add it to that data structure we've given the reference to begin with. And that the atomic set release means that if somebody does an atomic add unless, if they get a reference they'll see the new value of the key. Okay, so that I'm ordering is important. All right with that in place. So let's look at getting key so the reader after it finds the element is going to have to do a food get key to make sure that it's got a reference and the thing has the same identity. So it's going to do this inside an RC read locks we got RC read lock at the stop top we've got an RC read unlock at the bottom just before the return. Now we do a full look up which we don't showing here but does a look up and we give it the key and it gives us back a pointer. So back a null pointer which case we're just going to return null. And that's the empty then clause there on that if statement. But if we did get a pointer we're going to try to atomic add unless it. So, if it is zero that's the last argument, we atomically add one. We consider not atomic income less but we have an atomic add less. If that returns false. That means that the reference was zero means somebody freed it well after while we're looking it up or just after we came back. And in that case, we didn't get the reference, and we set P equals null and then clause of the LCF. Now, if we get to the last LCF there. That means we got a reference. So we check to see if the key matches. If it doesn't, then we have to call foo put undo our reference count and my failing foot there there was in fact one of my bugs that got me memory leaks. So we foo put undo our reference, and we set P equals null. And then we return whatever we had for P. So return P pointing to the element if we got a reference and the key still matched after getting the reference, or return null otherwise either we didn't find the thing in the search structure to begin with. We weren't able to get the reference, or we got a reference but as the identity did changed in the meantime. Okay, so this is example validation code that we used to deal with the fact that we aren't waiting for grace period. Before we free the element, we just free it. We might reallocate it immediately. And this per overhead on the readers is the price we're paying for not needing the grace period on the hot free allocation pass path. Okay. So any questions about that that's a little bit of stuff really quickly. So I'll give people a chance to ask. Going once. Okay, so we'll go to the next slide. This probably looks familiar if you've been through the latest kernel. We do an atomic destined that can test that takes our reference count and decrements it. If the result was zero, it returns true otherwise turns false. If it returns true. Then we've got a zero reference now, and there are any readers referencing it and if a reader tries to reference it, the, it'll fail in the on the previous slide, the atomic add unless we'll fail. So we can just came in cash free it. All right. Now that means that we may have readers with pointing to the free list, but it's a special free list because we have a slab type safe by rcu. We're in the free list we won't go back to the page allocator without a grace period beforehand. So we can be recycled as the same type by another came in cash alloc, but we won't get actually free and use for something else. All right, so that means that the layout of the structure remains the same, and we can count on it, not morphing into something else it may morph into a different instance of the type. But that's why we have the reference allocation and the recheck of the key in the food get code. All right. So one question you might have is this is complicated why not she's locking I mean we got this thing it stays around why not just acquire a lock. And the problem is that came in cash alloc might return uninitialized memory, and we'll we'll see how that happens and other slide. The problem is that the initialization I mean right as it stood before we just set the reference to zero. And if it was the first round for this particular element, it just it set it from garbage to zero. If it had come back through came in cash free back through came in cash alloc, the atomic that can test it set it to zero and we were setting to zero again. Okay, so we were setting zero without changing its value. The problem is spin locks are a little more complicated than that. If we do a spin lock in it, and somebody's trying to acquire that spin lock at the same time that's not going to be a good thing that's going to get bad really fast and we can't allow that. And the other thing is, I mean if you use came in cash alloc way before that's the problem we have. If we use came in cash is Alec that'd be another option. The problem there is that's going to clobber the lock as well. You could imagine locks were setting a zero releases it but the problem is that just because it was free doesn't mean there aren't readers paying attention to it. So you get the reader trying to acquire the lock just as just as we zero the thing and that would be bad as well, because it would sort of automatically release the locks of all those readers. So, that stands the reason this happens. And here's what you might try to do right you might do came in cash Alec you might do a conditional initialization that's the red box over there we just used to go straight to use. The problem is that when we get a new slab it's not zeroed. That means if we've, if we did a came in cash Alec and the memory we got was just newly allocated hadn't been came in cash free to just come out of a free page. It's got arbitrary stuff in it. And so we have no way to determine how to initialize it. Now it may well be that we should change that it could be the latest current really needs to have a way of saying hey this is a slab type, save my rcu thing so zero the new pages into it please. That might be worth looking into, because that would if you did that, then you could use as just a spin lock. The spin lock would stay a spin lock as you went around you would check to see if there was a flag set zero. If it was. Excuse me, you could, and the flag was zero you knew that it was a new slab just coming in, you would set it to one and that would and the flag being one would tell you that it been already initialized you wouldn't have to initialize it but that's something for another time. Do readers need atomics. It turns out they don't. I thought they did. John Cara explained that to me a little while ago there's the lower link if you want to look up the nail. And the trick is that in a x before uses this, you can have another object you have a reference to this and you could know that the type safe item has a longer life time than the stabilized item. In that case, just the fact you have a reference to this other object, and the type safe item is tied to it has a has a reference to it, you can just check that non atomically and be good. Okay, so that was kind of cute that was a surprise to me another fun thing about to work in the links channel community. And to move from RCU to type safe memory. You, it's kind of like moving to existence guarantee except for a heap allocator you had a slab allocator, and instead of having deferred reclamation all the time, you have to defer reclamation on the slab only. So as you hand pages back you deferred reclamation at that point, and not on every free. So, let's take a look at lightweight garbage collector. What I'm going to do we're, I'm going to go through this quickly. It's something is kind of easy and concept but the details me a little ugly. One thing can happen is that there's non blocking algorithms are subject to something called ADA. And what happens is that reallocated memory can cause failure. And so an example is a FIFO stack an atomic FIFO stack where you push and pop single elements. All right. If you push a single element and pop the whole list all the time then life is good but that's not always what you want. And so, I'm not going to go through this in the code in detail. But I'll just say that. So this this was push. And this is pop, but it's buggy. And why is it buggy well that's the next slides. So let's say we have a top and we've got our list like we had before with with cat and tux on it. All right, so we now try to pop the upper element and the pop gets preempted. So the pop, this is the first pop list pop one, it has the old pointer the pointer next and it's going to just now take those and feed them to clear swap. The idea being that if the old pointer is the same as top, then it's okay to make the top be the next pointer and that would kind of automatically pull the cat out of the list and have the list have tux at the top. Except this got preempted or interrupted or something has been delayed. And in the meantime, somebody has a list pop. Well, they pop the top off the list. And now the list just has tux on it. And the preempted list pop still has the cat but it's, it's on the free list. So it's kind of a ghost of the cat at this point. And we can have a second, a third list pop and tux is gone now and so both our pointers are pointing off into the free list of these things that used to be real things. And then somebody else might do a list push a dog and happen to reallocate the cat. So now old P points to this thing that's now a dog that used to be cat will pretend that the compiler can get upset about that yell at you and cause all sorts of ugly things to happen. But let's pretend we have a simple compiler, or we've carefully used a Tom X or something like that to keep the compiler out of the picture. So old P points to dog but old P next still points to tux. So let's suppose that the original list pop resumes. Okay, so it's in this state it's got a pointer to the dog and it's got a pointer tux. So it does is compare and swap. Hey, guess what old P is equal to top so the compare and swap succeeds. And once we do that we have a list and pointing to the free list, which is a really bad idea. And that is the dreaded ABA problem. We had kind of a thing where where the, where the, a given a chunk of memory, change roles and causes problems. And we can prevent this by preventing reallocation of cat and the way we can do that. And I'm not going through this in detail, but we just put a RCI critical section around the atomic operation. And this is the thing I was referring to where we're always doing updates but RCU is still useful. What's happening here is that if somebody freeze the cat out from under us, it can't be reallocated until we get done with the compare and swap in fact it can't be reallocated until we get completely done with the compare exchange loop. And so that means that either the compare exchange succeeds and it's the same element ever was, or if it's a different element if it's if that element has removed from the stack, then popped off the stack, then the compare exchange is guaranteed to fail. So we're using RCU readers and obviously also have to defer free the nodes. This is a case of not type safe memory but of, of existence guarantee to make this work. We have to, we have to wait for grace period before each free. And by doing this, we're making it so that the simple nbs code doesn't need to have a whole bunch of other hair that you have to put on it to make it work without deferred reclamation. So this is a situation for that lower red triangle we saw earlier in the slide where we're almost always doing updates but RCU still has a play a role to play. Which again would have been a surprise to me 20 years ago, live and learn. So what we're adding type safe memory here is non blocking synchronization. With this we can go to a quasi reader writer lock, but I'm going to skip this because we already covered and it just covers a couple of a couple of things here. And what we'll do instead is look at the multi version consistent control. Okay. Now, what this does is, what we're going to do with RCU is giving you versions is kind of publish you publish a new version, you do an RCU reference you subscribe to it. And they may publish a new version and some other reader may get that other versions you may have as we saw you can have multiple readers having different versions at the same time. And that's the spatial synchronization, helping us out so we don't have to enforce strict and expensive to portal synchronization exclusively. So this is something is actually use links kernel, the decash the entry cash uses it decash, and it helps with path name look up. The way that works is you're given a path name, and you want to find the corresponding I node. And what this does is it traverses in memory directory entry cash. It does it locklessly. And if something bad happens for rather surprising large number of values of something bad. We fall back to a more heavily synchronized reversal or we may just repeat the lock was traversal depending on what's going on. And one bad thing that happened is that you might be doing a path and you come to a segment is not in the directory entry cash, because it just hasn't loaded off of disk. Yeah. Now this code is very complicated. Neil Brown did an excellent LLN series shown on the URLs below what we're going to do is go through a really amazingly abbreviated version of that code. Just to show the workings involved with the with dealing with the version. Okay, so first let's start with what happens if you just, if you if you don't have the multi version concern is to control in place. Let's say we've got a file system that has these two path names this path name does exist is one path name and that thing might not exist as the other path name. And we're looking up this path name does not exist which the truth in advertising, in fact does not exist. What's going to happen is we're going to locklessly go down the path where are you protected so we don't have to worry about things disappearing out from under us. So the this finds that this path name goes ahead, the does finds this thing as well. And then somebody doesn't move this path name to that. Okay, so we're going to take the on the left hand side, the path name box over the second or the top. And move it to have it be under the that the top box on the right. In other words, like this. Okay, so bang, it's moved over I'll, I'll do that transformation again, we started here. We did that move command and we end up like that. Now, the entry cash, when you do that the entry elements don't change identity. So the pointers we have still work so the traversal is still looking at that does box does box down there. So this wasn't enough let's do another one. Let's say we do an MV that thing might not. So that is the not on the right hand side the second box from the bottom, and remove it under the that path name does so the middle box in the middle the does box there and do that move. And we're going to do that and there was a move that the not box over under does and I'm going to repeat that I'm going to back up and repeat that again so you see it. Now, the reader continues. Well hey there's a not now great we've got it. And guess what there's also exists. And so what's happened is we've matched this path name. We found an exist thing that goes with that. But that path name ever actually existed at any point in time. Okay, there never ever was in this whole process of slash this slash path name slash does slash not slash exist. So there have been people have argued that that behavior is just fine but a lot of applications really don't like that sort of thing, and the Linux kernel prevents it. Okay, so the language kernel does not look up path names and never existed, at least if it does that's a bug. So how do we avoid this race condition. So what we do is we use rcu but we also use sequence locking. And these again have different roles rcu makes the lockless traversal save to prevent stuff from freed out from under us. And the sequence locking detects rename so we can determine if we might have traversed a path that never really existed. And we'll just do a quick take a quick look at the sequence locking core API. So, read seek begin starts a reader read seek retry checks to see if there has been an interfering operation. Okay. And right seek locks starts a writer seek unlock ends a writer and so what happens is that each rename is enclosed in a right seek lock and a right seek unlock. So what that means is if you do right read seek begin, and by the time you do, and you do re seek retry. And there's any kind of overlap of any type with that with a rename the read seek retry will say hey you got to retry something bad happened. So what that means is we can do something like this this again is brutally simplified if you want more introduction there's the element articles again. Of course the source code is the final authority. So we take a variable seek and assign it read seek begin of and we got this rename lock we're using, and the renames again are going to do a right seek lock and right seek unlock and that same thing over the rename. We do an RC read lock, and we do a whole pile of complicated stuff to reverse the directory to cash which may involve going across mount points it may be involved symbolic links directories hard links the whole show everything, all the stuff that can happen. And eventually we do a read seek retry, and we giving the rename lock again and we give it a seek we passed in. And if that returns true something bad happened, and we do something to retry I haven't given the label for the retry but we go somewhere and say hey, something bad happened we have to deal with it. If we seek try retry returns false that means nothing bad happened we can just do our RC read unlock and declare success and be happy and know that this path name we walked locklessly really did exist during the traversal. Okay, so if we go back to our look up. We could still happen we could still go down there and that exists, but we did two renames during the passing look up, and that means the sequence, the sec variable would be invalid anymore. And so read seek retry would fail at that point say hey, you know yeah you look this up but it's wrong. Get out of the pool and do something different because this didn't work. So what's happened is that we've used our suit to make the traversal safe again seek lock rejects incandescent reverse traversals, and it rejects traversals that have raised with a rename. And what this is doing is just identifying a version is just verifying that the entire read happened within the context of a single version. There, we're not going to cover them in this talk. There are more complex schemes that can allow multi concurrent traversals concurrent consistent traversals a different word. I'm doing really well right now let's try this again. There are more complex games that can allow concurrent traversals of different versions. This code has been written by there was an initial shot of it by many notes Sony back about 15 years ago and Nick Pagan sometime later did an upgrade now bro has made it what it is today. Christoph Hellwig has been involved in a lot of reviews and upgrades of it of course Neil Brown as we know has been has documented it and possibly down their contributions. So it's a really impressive piece of code it's a cool piece of code and kudos these guys for making it work keeping it working. And it really does some serious improvement on pathing look up. Okay, and if we're taking our suit and making a quasi multi version concurrency control. We start with the existence guarantee, we have to have some sort of snapshot operation the readers and that was that read seek begin call which gave us back the seek number. And there has to be constraints on readers and writers, you can use a single object. We use sequence lock as example. There's a whole bunch of schemes use version numbers and those allow multiple concurrent readers doing different versions of the same time. And then there's a quick challenge which has a weaker concurrency consistency criteria. Okay. We can go through quasi reference count. That's up in the upper corner there. And RCU is kind of a strange thing and then it can be kind of a implicit reference count. You can think of our CD references obtaining a reference to the object. This limited to the enclosing our series critical section see the RCU, the reference you get an object you got a sort of funny implicit reference on it. And that reference lives until you do the RCE done lock. You can also think of RCU is providing a bulk reference count. You can see the RCU read lock containing a reference on each and every RCU protected object in the system. Okay, all of them at near zero cost. But again, you're going to be limited to the enclosing our series critical section we get to the matching RCU read it unlock those references disappear. Now, what's the code look like well actually you've already seen it. What's the code samples that we've seen before and interpret it as quasi reference counting. I mean you had RCU read lock you had RCU reference you could argue that those were taking references as opposed to how we're going to think about it. And that can seem strange I mean how can the same code because it's locked and quasi read right to locking and quasi reference count what what's going on here. And the answer is that that's life. Okay. So this is, if you have an atomic ink what's it doing. Well, that atomic ink might be doing any number of things right. It might be acquiring reference count if you know you've already got a reference on an object you can do an atomic ink to get another one. So you got to reference the object maybe you want to pass one off to some other thread or something. It might be doing a statistic you might be counting statistical atomic ink hopefully that's a statistic that's being very rarely counted because atomic ink gets kind of expensive if you do it a lot on with lots of CPUs. But that might be what it is. It might be advancing a concurrent state machine you did atomic ink to advance in the next state. Okay. There's a number of things you might be doing with that. The fact using atomic ink doesn't necessarily tell you a whole lot about what the intention is. And the same is true of RCU. All right. RCU has primitives you can use, and you can use the RCU read lock the RCU do reference and you can think of it as a read or write a lock sort of. You can think of it as a reference count sort of or as a multi version of currency control sort of right. And that will depend. A lot of times you're using RCU you're taking existing code and applying RCU to it, and what what will control really is what the old code was doing. So if your old code was using a read or write a lock you're probably and are and probably should think that you're using RCU as a quasi read or write a lock. If it started off using reference counts using RCU to replace those references and you probably need to think of it as a quasi reference count. If you had some kind of explicit version control and you're using it using RCU as a lighter weight version control, you probably want to think of it as quasi multi version currency control. So that's the trick. If you have some primitive, you can use a number of different ways, atomic ink and RCU the same sort of thing. So RCU to quasi existence, excuse the quasi reference count we start with existence guarantee. And we have RCU readers as the individual or bulk unconditional rest accounts acquisitions. And in some cases you'll want to bridge to a per object lock or reference. For example, you may be using RCU as a reference but need to do something that sleeps, in which case you, if you sometimes needed that you might want to acquire a physical reference on the object in the object itself, and then get out of the RCU production, networking does things like that to deal with with round trip delays. Alright, so we're here we've gone through all of these things. In December and now. So we're through the whole list. We want to reemphasize the area of accountability. RCU is easiest to use and works best when you are in a situation where staling system data is okay, and you are mostly reads and this is why it first started showing up networking. Networking is kind of a poster boy for it you've got a routing table and networking, but for various reasons. The routing information propagates across the internet fairly slowly. If you try to propagate the routing information too quickly, you end up having all sorts of strange failure modes, where you can have sprouts flip back and forth it's pretty ugly you have packets going around the circles forever. So they artificially slow down the routing updates to promote stability. So that what that means is by the time the routing update reaches the system. And sending back is the wrong way for quite some time. And so saying the wrong way for for a little while longer while the readers get done is not a really a problem. In some cases you need consistent data remodeling consistent data, and we saw an example where that can happen. And that is the case where we have a lock on the object. So we use RCU to go through some kind of a search structure locklessly. We have a tree for example avoiding avoiding a global lock bottleneck on the river the tree we just go use RCU to go through that without a bottleneck. We get locks on the leaves of the tree, which means we can spread the contention across all the leaves of the tree and have fairly low contention. And, and that way, we get the consistency at the leaf or we need it. And don't have massive attention on the search structure itself. So another situation where we have read write and need consistent data RCU might be okay and the D entry cash could be considered an example of that. There are a lot of workflows that add and remove files and directories quite frequently. But the fact that we're avoiding a massive locking bottleneck on the root. I know it means that even though we're changing the tree things are working fairly well plus we're mutating some part of the tree and there's a lot of traffic going through the tree that's unaffected by the change. So, the D entry cash and example the yellow line we saw an example with the non blocking data structure with the single element push and pop in the red zone there where it actually does work. Still, yeah, I like RCU I was calling better of it along with jacksling wine. I had a lot of fun with it but the important thing is to use the right tool for the job. I mean, sometimes RCU is the right tool and that's wonderful and I love that. But if RCU is not the right tool you should use something else. So, you could argue the RCU is a specialized thing it is specialized in fact I would argue is specialized to give an idea of roughly how specialized it is. One useful thing is to compare it to read a writer lock. If you look at the usage of reader writer lock in the Linux kernel compare that RCU. It indicates that you RCU is well RCU is used is used about four or five times more intensively the reader writer lock. So, you could argue that reader writer lock is four or five times more specialized than RCU. On the other hand, normal locking exclusive locking is used about 10 times more often than RCU. So you can argue that there is a spectrum from reader writer locking maybe being the hammer. Maybe being the screwdriver at about a tenth the usage of the locks and reader writer locking being I don't know the spanner or something like that, at about 40th or 50th exclusive lock or a fourth or a fifth of RCU. Just kind of gives you an idea of four things are overall. And the actual ability of RCU you could use RCU you could use our reader writer lock in most of the places to use RCU and theory, but we saw the global agreement this implied by reader writer lock is a real problem for multi processors. Okay, so let's get to the summary. As we saw before, RCU synchronizes in time in space as well as time, and that it's intertwined so we have the time and space kind of working together and very closely. And that allows us the near zero cost reads and synchronization. We went through several example RCU use cases we've seen them there. Again, as in December RCU is dirt little secret is that it's dead simple. The semantics are quite straightforward we saw a slide with those four quadrants with the blue boxes. But in order to make good use of it you have to change the way you think about your problem. My hope is that these two presentations the one in December and the one today, help you understand how to think differently about your problem. So you can make use of RCU where it makes sense to do so. Again, hopefully this this helps understand RCU, but there's the old saying just like in December. I hear and I forget. I see and I remember I do and I understand. And being able to see and remember is a useful state to be in. If all you do is see this, see this presentation it might help you review code or kind of see what code is doing. If you really want to get to the bottom of RCU really seriously understand it, not everybody does but it would be better if more people did in my opinion. You need to really play with it. It's available in the Linux kernel of course there's also the user space RCU library that you can use if you like using user space debuggers. You play with it and see what it can do do measurements on it that's a way to really get down and take a look at it. We're here we're done they're all blue now because we've covered them all. And we've got this slide which is the same in December except it shows you where part one is. So with that, open for questions. Thank you for your time and intention thus far and for your interest in RCU. Paul, I have one question. I was wondering if you, you mentioned a couple of slides ago that RCU, like any other tool. It needs to be applied as it's good for some and good for not good for everything right. So, can you go over maybe some examples in the kernel that you consider RCU is apt and RCU the other side of the coin being RCU is not good. Okay. Yeah, that's a good question. So this is the slide you're talking about right. Correct. Okay, great. I'll give a couple examples of one of the early wins was security. In the 90s, Linux was just something that was forwarding mail it was printing it was doing serving disk it was being used for general things. And security wasn't a big deal. For a couple reasons one was that Linux wasn't being used for really important data. And the second was that this that the internet was a much more friendly place back then. Okay. But as there was enterprise interest in the early 2000 security became more important. And so things like security policies got added. And so there were, I think it was Kaga Kohai whose name I just mangled and I think James Morris I probably got the wrong name they decided that they were going to add what became SA Linux. And what that meant initially was there's a global lock acquired on each system call. And the problem they had is it's working great on four buys but you get a 32 by and, you know, you're just stuck in contention on that lock. And they found out about RCU and applied it and got just massive improvement improvements and performance. And the reason is of course you got a security policy but was set, you know by some executive at some point. And then it trickles down and people actually implemented so you've been doing the wrong thing on security for weeks now probably. And so, you know, what's another millisecond or two. And that and that got them linear scalability. In fact, the performance penalty for Jesse Linux it was there you can see it, it was it but it was, you know, a few percent as opposed to orders of magnitude. So that was one that worked really well. At the other end the the entry cash is something that has worked out much better than I would have expected. I expected that that we'd have to kind of move and we do move slowly it took a good 10 years to get it to where it is today. But my pride story on that was at a Linux company you the first one I attended in 2004. And a bunch of us had dinner. I was out at a and I'm flaming out of the guy's house he's a he was a BSD hacker he's now retired. And I'm sorry I've lost his name I really feel bad about that. In any case he had a soul for dinner up at his house near Adelaide. And he happened to have a BSD system and then he figured Linux system side by side and his in his office. And of course, it didn't take us long to come up with a benchmark. Which was which was to do a find across Linux source tree so we put a Linux took a terrible put a link source tree on both machines the BSD machine Linux machine. And did a find on both of them with some other stuff going on I don't remember the exact benchmark. But it completed in 10 seconds on Linux. And many minutes later it was still running on the BSD system. Okay. And Lina's happened to be there and he looked at me and said I'm very proud of our decatch system and that I can't tell you how good that felt here and say that. All I did was fried RCU at that point the decatch itself is many Sony's work and Nick and Al of course picked it up later. Okay, so what's not so good about RCU give an example where it doesn't work. One we've had problems with so far and we're working on it is a map some. Okay, that's a reader writer semaphore. And there's some work going on to try to help things long. Peter Zolstra in 2010 started the SPF the speculative page fault. So I'll do for took it over a few years ago, and Michelle left weapon asses taken over recently. So it's you know this has been going on for like 12 years. There was also a MIT research effort that did something that made it use RCU but had performance degradations in some workloads and unfortunately important workloads. That was in 2003 as plus 2013 as plus excuse me. The speculative page fault path attempts to do this page fault without acquiring a map some and uses various techniques to figure out that that didn't work, and doesn't the hardware acquiring re acquiring a map some of it has to. Another, another piece is, and this is Matthew Wilcox and Liam Howlett. And we're working on the maple tree variant, which allows part of the search to be done under RCU protection. And I believe it avoids a fallback path. And, and those two patch sets are are something that is moving towards that goal. SPF turns out to be really helpful for Android it has, you know, 20% speed ups and application launch time which the so much so that many of the Android vendors apply SPF as an out of tree patch. If you're running an Android you probably have that you're probably running that patch it was a recent Android. So, you know, it's being used a production even though it's having a hard time making a mainline. Let's see how things go with that. Another one is something that's applied partially. And this is a power PC has RCU protection on the pay shables, the pay shable structures. And doing that would allow lockless slash proc entries. But these are, and it. So these are things where, where RCU has been difficult it hasn't been used in the past, we may be able to partially cover MF sound but we had, but we, we don't have any ambitions to completely I'm eliminated at this point in time. Maybe we can get there sometime. So the MM system with the MF same as one example where RCU has had some difficulty. There are probably some other ones that's the one that comes to mind immediately. That's a good question though, and I don't expect. I actually had a question I was giving an academic presentation in the in 2012. One of the guys said okay so when's RCU taking over the world and I said no it's not going to I mean, we use exclusive locks for updates for example. But I think it's, it's a very useful tool it's gotten a lot of use and I'm very happy I'm happy for it to get more use but I'm also happy to learn about other synchronization primitives and how they can make things work. So good question, other questions. Thank you for explaining that that's that's great because I know we talk about RCA a lot, we use it a lot in the kernel. This gives me a good picture of when it's useful and another way to find more of them would be just to look up read lock and write lock and, you know, the various semaphore and mutex, read side primitives, take a look and see what's there. Okay, great. Well if there aren't any other questions. I'm happy to wrap this up. Thank you all so much for your time today. Thank you Paul for presenting on this topic and thank you for being here and addressing questions for us. And I hope to see everyone on a future mentorship session and this recording will be up on our YouTube later today. Thanks again everyone. Thank you all. Have a great day.