 All right, we're going to go ahead and get started. For those of you who just walked in, we have a microphone now, so if you guys want to talk, just raise your hand. We'll shuttle it around and go ahead, Paul. Turn my mic on. Yeah. The people on Zoom don't care who I am anyway, so we're good. Or they shouldn't. Anyway, yes, as Steve said, it is RCU's fault and it's under active development. This came as somewhat a surprise to an academic guy I was talking at at a conference a few years ago. Quite a surprise, actually. He had a rather interesting view of what synchronization primitives were. I didn't have the heart to tell him that locking in atomics were also under active development. I'm not sure how that would have affected him. But here we are. So what I'm going to do is I'm going to do a very quick review of RCU. There's some little URLs at the bottom of the slide that you can't read. And those are each like a 90 minute presentation that will cover this material. This is just a reminder or a taster, depending on whether you've seen those or not. OK, so the big thing is just that top two bullets. Global agreement is expensive. My 50 years ago self would have been shocked to hear me say that, but the speed of light is just too freaking slow. And atoms are too freaking big and that's causing us massive problems in concurrent software right now. It's just the way it is. Laws of physics can be brutal. Now the way we work around that with RCU is we don't just synchronize in time. We do so in space as well. And we split the core RCU API up into the temporal on the top and the spatial on the bottom. And to take a look more at the temporal piece, this is four different scenarios. Time increases in each scenario from the top of the screen to the bottom. The upper left there is a situation where a reader started on some CPU before an updater did. That means the reader might have reference to something that the updater removed. And the updater is going to remove it. It's going to invoke synchronized RCU. It could also use call RCU, but let's use synchronized RCU for now. And that synchronized RCU had better not return until that reader is done. Because freeing that object out from under the reader would be a very bad idea. And that's the key thing. Synchronized RCU needs to wait till all the pre-existing readers are done. Now on the upper right over here, this is the other way around. The reader started later. It can't possibly have a reference to that object because it's already been removed. And so it's OK for it to be hanging out still after the free, because that free can't affect it. It doesn't have reference to the thing being freed. If you think of the upper left as the belt and the upper right as the suspenders, then the lower left is the belt and the suspenders. The reader started after the removal and finished before the free, so we're doubly protected. The lower right's a bad situation if this happens, which it does occasionally. That's a bug in RCU. If the reader's there and gets a reference to the thing being freed and is still hanging around after the free, that's a user after free. That's just as bad with RCU as anywhere else, maybe worse. All right. This is the global agreement cost graphically for a reader right to lock. The updater has to wait for the readers to get done. It takes time for that to propagate across the machine. And once the updater gets done, it takes time to propagate that information across the machine to allow the readers to continue. You know, put that big red blob in the middle that's lost time. So the wait, go ahead. Okay. The way this works is we have time going from top to bottom. And this is a summary of like a 20-slide animation in one of the presentations. But we have the red dotted line when the change happened. We have the beginning of the grace period and we wait at the blue dotted line. So we've made the change, we're waiting for all the readers. And at the lower blue dotted line, the readers are done and we can proceed free. And the reason that works, we've got this kind of zone of what might be confusion between those two blue lines when you might have processes having both the old state on one hand, other processes having the new state on the other hand. And the spatial address space synchronization takes care of that. A given reader will either have the blue structure or the green structure. You won't have some mush of both of them, at least assuming you've programmed it correctly. And so both time and space allows us to get rid of that red zone where bad things, nothing is happening. We end up with something like this on the bottom where we can actually be having useful work happening at all times. Okay. So why are we putting up with that extra complexity of both time and space? Well, this graph shows why. This is log log. So that a nice slow ramp of RWLock is going up orders of magnitude. As we add CPUs, reader-writer log gets a lot slower. RCU does not. It gets fuzzier because we have more interference. If we have lots of CPUs, we've got different cases of a given core having two test instances or just one. And so you get the variation throughput because of that. The hyperthreading essentially is why the variation is there. Now this is the best possible case. This is the RCU marketing slide showcase because the critical sections are of zero length. We acquire the lock and immediately release it. We enter the RCU critical section. Don't do anything, just go away. If we add some critical section, then we get something like this. And as you can see, the shorter the critical section and the more the CPUs, the better RCU looks. The more help it's giving you compared to reader-writer lock. And of course, no matter what your situation is, RCU was getting rid of some deadlocks. Not all of them, but it's getting rid of some of them. Okay, so that's why we're doing it. This is a list of the use cases that I've thought through and written up, which are in the presentations. And that's our full speed review of RCU. Okay, take a deep breath. Let's go into the recent changes. This is a list of them. I'm not gonna go through all of these in great depth. I'm going to guess at which ones are most important and look at those a little more heavily. I'm around the rest of this conference for if you want to look at either the RCU in general or the particular changes in more depth. Flavor consolidation. So I was mining my own business and I got an email looking like this. Those of you who received this kind of email know that this is a sign you're not gonna be reminding your own business very much longer. And I wasn't. This was an exploit in a use case of RCU. And the problem was they did something like this. Their readers use RCU-relocked SCED to protect the data and the updaters use synchronized RCU. And that's a problem because back before 4.19 and earlier you had to keep it straight. So if you're using RCU-relocked on the readers you couldn't use synchronized SCED like they did. Excuse me, I got it backwards. If you're using RCU-relocked SCED on the readers you couldn't use a synchronized RCU you instead had to use synchronized SCED. You couldn't mix and match. You had to keep the readers and the updaters straight for these particular three flavors of RCU. And the problem was somebody didn't keep it straight. That meant we had to use after free. Somehow or another, I don't know how. I wasn't in the security loop and I'm glad not to be actually. Somebody figured out a way to exploit this. So, Linus to his credit said, look can we do something about this? And the way you do something about this is you just make it so synchronized RCU will wait for all of them, any of them. All right? That way you can just use synchronized RCU no matter which type of reader you're using and it'll just work. The just work part took about a year of my life to make happen but it's there now and that's good. And that's wonderful because you can just do whatever reader you want. You can synchronize RCU and wait for it unless you've got some changes you need to backport to 4.19 or earlier. Now, if you're not backporting that far, if it's 4.20 then your synchronized RCU can stay a synchronized RCU. Your call RCU can stay a call RCU, it's great. If you go back earlier than that and you look at your use case really carefully and see oh yeah, the readers are using RCU read lock and only RCU read lock, then life is still good. You can do that. Hey Paul, so do we have a booby trap now for backporting patches? Exactly, we got a booby trap. And that's exactly the point. Here's your get out of jail free card right here. Okay, for synchronized RCU. Instead of you turn your synchronized RCU into a synchronized RCU mult, you pass it as parameters call RCU, call RCU VH and call RCU SCED. It takes variable numbers or arguments, you can pass whatever else you want as long as it's type matches call RCU. What this is gonna do is it's gonna invoke these three guys and wait for all three callbacks to fire. And that emulates post 419 synchronized RCU. So if you're using more than one or you can't be bothered to look and I'd probably be in that letter camp myself actually, you can just use that and be okay. If you know what's going on, you're familiar with the code then maybe you only need to use one or two of them or something like that and that's fine. And you can even add other functions if you want to. You're limited only by your stack space on how many parameters you pass there. Now call RCU, what you have to do is chain them. Okay, I mean you can if you want to set up something like this where you make your own thing and you have your callbacks have a datomic that they decontest and go with that. But it's simpler if you can stand the latency to just have do call RCU, have its callback to the call RCU VH, have its callback to the call RCU SCED and then have that callback do whatever you really wanted to do in the first place. Okay, so those are your two get out of jail free cards when you're going to 419 in case you want to loo the booby trap. Of course if you like booby traps I suppose that's another story but please let me know which stable release you're using and I'll try to avoid that one. Okay, so that's this guy, Joel Fernandez put in addition to list for entry RCU and H list for entry RCU but there's only so much space into 5.4. The idea is that what happens, these guys have an interesting relation with LockDep and rather than blow up the API like we've done elsewhere, what he did is he made it so you can have an optional LockDep expression shown here. So you can say list for each entry RCU, give it the position variable, give it the head of the list, give it the struct member of that the list is and then you can have something like LockDep is held ampersand event mutex and that allows you to have some code that can be called by the updater held by the lock or by the reader called by RCU or however you want it and not have to have 55 million different names for list for each entry RCU. So that's in 5.4 and it's something that might be useful. There's also single element KV3RCU, KV3RCU, this is Vlad Redsky, it's in 5.9 and later. The classic way you use KV3RCU, KV3 and KV3RCU are synonyms now. There was some thought of, I'm not sure what the plan is going forward, but here we are. With KV3RCU, you give it a pointer to the thing you want freed, you have to have an RCU struct in that structure and you give the field that gives you that RCU struct. So you pass the pointer and the field into it and the nice thing about it is it doesn't sleep. You know, it's just gonna cue some stuff and get on with life and it's gonna stay around. There's need for it, it'll be here, but one of the problems with it is if you have a really small structure and you got a lot of them, throwing in 16 bytes for an RCU head may not be your best strategy, okay? And what this guy allows you to do is not have that. You just pass the pointer, okay? So the field is an optional parameter now and that means you don't have an RCU head, you don't have that space overhead, but if the system is out of memory, it will sleep. What it does is it tries to allocate a page of pointers and it just kind of fills things in those pages of pointers, has a whole mess to RCU and then the cool thing is is that when it would come time to invoke callback, it just scans down through that list and K-free's them, which means you aren't getting massive piles of cache misses or at least not as many, okay? And Willie likes that, maybe the rest of you guys do too, I don't know. But you might be out of memory and you try to allocate the page and it isn't there. And you may be in a context where it has to be dainty about trying to allocate, like you have interrupts disabled or something like that. You can't just say, give me the page, you know? So what it does then is just call synchronizeRCU. And when synchronizeRCU returns, it's great, well, just free this thing right now, we're good, let's go. And so that gives you a trade-off. You have smaller structures, but you can sleep, okay? And that's why the tool to parameter variant has to stay around because it really can be called with interrupts disabled. Okay, and that's been around since 5.9. The bit about the page of pointers and the reduced cache misses has been around since 5.7. Both of those are Vlad's work. Okay, I'm not gonna talk much about this except to say they exist because you might see them on a stack trace and wonder what the heck's going on. These are used mostly for tracing, for handling trampolines. They're very specialized. So if you feel the need to use them, please talk to me and also talk to the people using them because they're very specialized and you can mess things up badly. Now, this is just their names. If you happen to see this on a stack, that's what's going on. The RCU tasks has been around since 3.18. It's been around for a long time. The other two showed up in 5.8. And Niraj Upadjie helped me out a lot with this, finding corner cases in other places where I wasn't doing things quite right. So that was really cool. Nice to find them that way as opposed to blow-ups in the field, you know what I mean? Pole grace period APIs have been around for a long time as well. Get state synchronized RCU and con synchronized RCU around since 3.14, but there have been some other variants that added more recently. And I'll make these slides available. I'm gonna go through these in vast detail. But this is the classic 3.14 APIs. This is something Andrea Archangeli asked for. So the idea is you call get state synchronized RCU and it hands you back a cookie unsigned long, as it turns out now. And then you go off and do something. When you come back, you know, has the grace period ended? Well, what you do is you pass the cookie to con synchronized RCU or upcoming things. You can put an expedited there. What it'll do is look at the cookie. If the grace period has expired, it says great. Do enough memory barriers to get things taken care of and then return. Otherwise it invokes either synchronized RCU in the one case or synchronized RCU expedited in the other. Either way, guaranteeing that the grace period has elapsed when it returns. Okay. And that's actually works quite nicely except that sometimes you can't sleep. Okay. And that last bit about sleeping can be a problem which is why there's been more recent things. In fact, I think the requesters of all this stuff are in this room. So here we are. So this is talking about how you might use the non-sleeping interfaces. The idea is we've got some kind of thing in the far left there. We've got some kind of a list that has structures on it that's accessed by RCU. The other three lists are not touched by RCU readers. And maybe it's a cache of things or something like that, but occasionally you yank something out of it and say, you know, this thing's been here long enough it hasn't been used for a while. So we don't want to get rid of it right away because it might be used again. We'll put it in the second list. When we put it in that second list, we call GetStateSynchronizeRCU to get a cookie. And that'll tell us later on whether Grace Periods expired since we removed it. Now GetStateSynchronizeRCU only gives you a cookie. The good thing about that is you can call it from an NMI handler if you want to. Okay. The bad thing about that is it does not guarantee that a Grace Period starts. And if you have some embedded device that goes really, really quiet, that Grace Period might never start and that might be a problem. So the thing underneath, the StartPoleSynchronizeRCU, maybe with an expedited on it, will give you the cookie, but it'll also make sure that the specified type of Grace Period actually starts. All right. And then you can put the cookie in there and then you're in the state shown in the third box there. You can call PollStateSynchronizeRCU passing of the cookie. You'll either say yes to the Grace Period expired, or no it hasn't. You can call that from NMI handler too if you want to. Perhaps a reader goes and looks for the item and doesn't find it because you yanked it out of the list. So maybe you find it here and put it back into the list using ListAtRCU, in which case, things fine. It's back into active duty and who cares. But if PollStateSynchronizeRCU says, yeah, that Grace Period is done, then you can call GetCompletedSynchronizeRCU if you want to. What that does, it gives you a cookie that's kind of pre-expired. No matter what happens, the cookie returned by GetCompletedSynchronizeRCU is gonna look like an expired Grace Period. And that allows you to just leave in the list with the marking this guy has done. So later on, if you looked at it again, you could just throw it away or whatever you're going to do. All right. And the reason that's important is because, you know, a cookie is a big thing. It's got at least 32 bits, but counters do wrap. And there are configurations in the kernel. We can go through Grace Periods really, really fast. I mean, more than one per microsecond, okay? And that's specialized ones that status center people never see, unfortunately, but, or maybe fortunately, I'm not sure, but it can happen. So counter wrap, and this is real, at least for 32-bit systems. All right. So there's a caveat though. What you might think you could do is say GetStateSynchronizeRCU, grab the cookie, do SynchronizeRCU, and then do warm-on-once PollStateSynchronizeRCU cookie. And there's two reasons why that warm-on-once can trigger. The first, of course, is counter wrap. If you waited long enough between there, well, you've come back around, and it says, no, it hasn't been used. It goes, look, I just handed this cookie out. The other one is that because I'm taking and I'm sort of consolidating both the expedited and the normal Grace Period counters into one counter, something has to give, all right? And what gives right now in DashRCUs, this is not yet in mainline, is that if you happen to have partially overlapping Grace Periods, expedited and normal or vice versa, one of the other of them won't be recorded, okay? Which I think in production is just fine. I mean, in RCU torture, I can make that happen pretty quickly, but I don't think the production workloads are gonna have a problem with it, famous last words. If production, well, the next thing I guess is if you do two Grace Periods, two things on this Grace Periods, then either one of these two guys, one and it's reflected in the counter and you're good, or somebody else, one that started after the first one and they're reflected in the counter. So if you do two, the only way you can fail is counter wrap in case that matters. But in any case, there are some ways to avoid this, but you pay a price one way or another. Either the cookie grows from 64 bits to 128 bits, which seemed like a bad idea, or the counter wraps a lot faster, which also seemed like a bad idea. But if it becomes a problem, we can make some adjustment or another. All right, callback offloading. This is something that's mostly of interest to real time people and HPC people. The idea is normally what happens is your callbacks are invoked on the CPU and the CPUs accuse them. But that can be a problem for some kinds of HPC workloads and also real time workloads because you got this callback who cares when it executes and bang, it shows up in software queue priority, preempts your heavy duty real time guys and causes trouble. So you can tell the kernel to offload them to put them into a K thread and then you can put the K thread where you want it. Or you can rely on the fact that the K thread is a K thread and the scheduler can put it where it needs to go, one way or another. What Frederick has done is now, traditionally what happens is you assign those at boot. You say which CPUs are offloaded or not at boot time and they can't be changed without rebooting. What Frederick has done is put in the infrastructure to allow them to be changed at runtime. So as long as the CPU is marked at boot time as possibly offloaded, you can offload it and de-offload it back and forth as you want. And there's some in kernel functions you call do that right now. Hopefully at some point those get wired up but that's where we are at the moment. I think that he's going, this is part of a cunning scheme on Frederick's part to make know where it's full also work this way. Okay SRC, remember that I'm not gonna look at this in great detail but some of you may have noticed that SRCU struct can be quite large especially if you're in a distro kernel that has NR CPUs set to like 4096, which happens. And this isn't that much of a problem. The SRC struct isn't that big but it can mean that the compiler gets to use ugly instructions for accessing fields beyond the SRC struct and whatever structure you're using. So we have something that is going in to put this on a memory diet. The idea is one to not allocate the array until runtime which means we know exactly how many CPUs there are and we size the array based on the CPUs on the machine instead of how much the distro thought it might ever have someday. So that saves a lot of memory right there. In addition, there's a lot of SRC use cases where you don't need the combining tree. You could have a single global lock and it'd be just fine. And so we have a mode where it runs that way. And if so, if you have a problem where large SRC structs are causing you trouble, let me know and we can talk through the options and what we can do about it. I also have an option to leave it without a SRC node array and then expand it up if there's contention but that has not yet earned enough trust to be default. Okay, it's there but you have to ask for it or it will be there next merge window. Real-time expedited grace periods. Back in the 1990s in 9HPTX, this is a proprietary UNIX system. There was something like RCU and it had RCU CPU stall warning timeout of 1.5 seconds. I'll admit that I was kind of disappointed that in Linux it had to be 60 seconds but to be honest, Dynix PTX was a much, much more controlled environment. We didn't have random device drivers, somebody somewhere coded up that were being used that didn't care about code paths, the real-time or anything. We did decrease that down to 21 seconds and you can imagine my surprise when I got a patch decreasing the expedited grace period timeout, just the expedited, not the normal one, to 20 milliseconds. And yes, that is through order of magnitude faster than the current timeout. This is something that's looking good for the next merge window and it's enabled by default if you have Android set up. All right. Of course, if you know something about readers and grace periods, you might have some questions. There's another patch that goes with it from McAllish Singh, the previous one from Vlad Retsky again. And the problem is that expedited grace periods are driven by work queues which are scared other K-threads behind them which can run slowly if the system's overloaded. What they did was they made K-thread create worker and they put it at schedule. This is strong medicine. So it's enabled only if you have less than 32 CPUs. You aren't running the RT patch set which doesn't use expedited grace periods in the first place, so I bother. And if you have priority boosting enabled. The test results were impressive. He got the latencies down from 200 and some odd milliseconds down to less than two milliseconds. Which gets them down, so they're having sort of a wimpy but a very real, real time that has an expedited grace period on its fast path. If you'd told me that last year, I would have lapped in your face, okay? But here we are. Okay, so this looks like a Megan's 519. There are a few remaining issues. We probably have to have more granular setting of priorities for the various RCU K-threads. The callback invocation K-threads get in their way. For example, so we'll probably have more patches coming along that make that work better. There's some miscellaneous stuff. I'm not gonna go through it in particular detail. We have some real painful thing that was in the schedule between the schedule and RCU for a long time that Lai Zhang Shan got rid of. And RCU now watches most of the idle loop getting rid of some weird stuff and tracing that used to be there. And Peter and Thomas took care of that for me, which is cool. Okay, looking to the future. So this is kind of a banner year for me. This year, I turn 100. For you guys that like these new fangled 8-bit bytes, 40, okay? 64. Now, I still expect to be around for a while. My grandfather and my father's side was doing technical work, mechanical engineering stuff, to age 89. My other grandfather was servicing radiation therapy of chains to age 90. My father celebrated his 90th birthday a couple of months ago, and he cut back a few years ago to four days a week. Although the way he'd put it is he works two days a week, but it takes him four days to get the work done. So I expect to be around for a while still, but the fact is that any of us could find ourselves forced into Mother Nature's Retirement Program, as I put it at any time. So it's good to be prepared. So here's some things that might happen in the first. Along there, we've got things, ways of dealing with these that I'm working with people on. The common case, of course, is stuff I don't see coming, which means we need to have people with an understanding of the basic infrastructure. All right, so if I look back, if I look at 2017, five years ago, at the two years preceding that, that's kind of a distribution of the first 12 contributors, not worrying about keeping email addresses taken care of. I let John and Greg worry about that. And I was worth 74% of the commits, and there were 46 contributors, although the vast majority of the contributors gave one commit over those two years. If I look at the previous five years from the present, I'm at 63%, which is down quite a bit, all right? There are more commits, and a lot of that is because after coming to Facebook, I've done a lot of work to make it so that RCU can use distributed testing, which is taking all the work that's been very helpful. And the other thing is, of course, what's a commit? If a guy gives 10 commits, is that good or that bad? I can tell you that a bunch of guys in the top few there have done work that is actually quite deep and serious. So I take this as a good thing. We have 79 total contributors, again, most of them have just one commit. Okay, longer-term trends, the blue line on top is the total commits, the green line on the bottom are the ones that are from somebody other than myself, and the trend is generally up on that bottom line, so we are getting this information submitted out. We got a lot more work to do. Okay, we're early days on getting it across there, but at least in contrast to five years ago, we're making a start. Okay, now there were people that did serious contributions to RCU, but what tended to happen in the past was somebody would get to where they demonstrate the ability to do that, and then somebody would pay them a lot of money to do something else, which is not a bad thing because you have these people scattered over the place that have at least some context at some period of time in the past, which is better than nothing, but the cool thing about having recently is the people have been staying focused on it, which is very good. Now they may have looked at my age and drawn the natural conclusion, which is not a bad thing either. Okay, summarize, mostly I'm just gonna say that RCU's underactive development is driven by its needs of its users, and that's actually been a lot of fun. It's been a great experience working on this over the last 30 years. There's places you can get some more information. The two presentations I'll call out again, those are 90 minute presentations that talk about use of RCU. We might have time for questions or not, so I'll hand it back over. Oh, you got a mic coming at you. So two obvious questions, and one is when I think about file systems, sometimes the most helpful contributors are those contributors who come up with test cases, user space tools, and things like that. And when you think about a three letter word that's hard to spell like RCU, a lot of people don't even know what it is. So one of the things I was wondering about is, with the BPF guys over there doing wonderful things, are there RCU contributors who have done a great job with tracing and instrumenting for your example, and also for finding the evil people who haven't embraced RCU, and think about the guys who sit there scanning 30 million lines of code and finding those annoying commits, thousands of them, the style changes and the adding the global changes one by one. So are there developers outside the RCU tree who have been really good at helping with those two tasks? There have been. In networking, Eric Dumasay is near as I can tell Mr. RCU for networking for years and years and years, and he does a great job of it. The BPF guys you mentioned, they're on my team, so we talk a lot. And in fact, RCU task trace was from their needs directly. All right? But I had a conversation in the break with somebody who was saying they were having problems because they used RCU, they'd get some really cool feature in, and they would tell the guy, this is great, but you're on a fast path, you need to RCU here and bang, the guy's gone and they never hear from him again. So if your point is that we need better propagation of the use of RCU, I can't argue. Now the two presentations I called out are my first attempt to make that happen, but I'm sure we need more. And I'm also, it would also be great, it'd be great for people looking not just for RCU, RCUs, I mean use the right tool for the job. I mean I'm happy when RCU is the right tool for the job, but I'm not so happy if somebody takes RCU and cranks it around and does something stupid with it, right? I think we need, I think it's a more general problem. If you've got some kind of synchronization problem, how do you decide what to use? I've written a book trying to get a start on that, but I'm sure that a lot more work needs to be done in that area. So one great example, and I just keep having trouble forgetting it, thinking of the BPF people, Hany Namati gave a talk, I don't know, two or three years ago at the storage developer conference. And he basically talked about BPF and how you analyze performance problems with files. So he's looking at Linux files and he's like going into the various trace points in the various ways BPF tools can do, these amazing things that detect problems you have. And everybody, every single file system developer is nodding their heads saying, this is amazing. But one of the things that's interesting is, okay, go one step farther, okay? Can RCU help, what are the red flags we're looking for that those, because he's looking at what's inhibiting serialization. So it's an interesting question to like how you spot the problem first. And I keep coming back to that presentation he gave two years ago as a great example of things that help, but I just don't know. So I'll give you the canonical one. I'll give you the canonical one's been around for a long time. And I showed that slide with all the boxes on it with the different use cases. This is for the use case on top of that chart. If you have a read lock, a read write lock, and the reader uses the value after dropping the read lock, then as soon as you drop the read lock, everything can change. And so that's one case where you might, I mean there's a bunch of other conditions that have to happen. That's one place where you might usefully apply RCU. The same thing with an exclusive lock. If you have something where you calculate a bunch of stuff and then drop the lock, you might, if things are set up right otherwise, be able to use RCU. You're already paying the price in terms of stale data. Because as soon as you drop that lock, that it is stale, or it might be. So that's one piece, but there's a lot that goes with that. You may be asking, I should make a presentation along those lines, I'd be happy to do so. Micro, my friend. Especially for files and blocks, because we're boring, right? Okay. So just examples. Because I think that there's a lot of common places where we screw up, where we don't take advantage of stuff like this. And another example is X array, right? Where we don't take advantage of X array, or we don't want to take advantage of some new feature. I think you should talk, for X array there, a guy over there you might talk to. Yeah, I'm trying to hide, I think. Oh, the thing that, one thing we could do would be if there are places where you're having scalability trouble. If you can summarize, I mean, I've been looking at code for a long time, it's still difficult for me to get it in my head around 10,000 lines of code in one whack immediately, right? So if you can help me out by summarizing kind of what you're trying to do, we could discuss it and see what might work for the different places where you're having problems, right? So you have a place where there's contention problem, or where it doesn't scale as many CPUs, or it needs to be faster, and the overhead of acquiring blocks is causing trouble. You know, summarize what you're trying to do, maybe we can chat or you could also have a chat with the other people who use RCU a lot. I suspect most people in this room are familiar with two examples that I get all the time. One is you got this async IO, DIO kind of workload, or IOU ring workload, and I've got a thousand threads writing tiny ridiculously small amounts, and why aren't I seeing hundreds of writes at the back end? The back end. Typical problem number one. Typical problem two is I've got a file copy workload, and I'm expecting to see these big chunks of writes scaling up eventually to whatever the R-size is, and I'm not seeing much on the back end. I'm not seeing much parallelism on the back end. Those are problems that come up all the time, right? And there are usually plenty of file systems specific reasons for screwing up, but those are problems that we analyze all the time, in every file system. Okay, I think that I'm being given a sign that we're done with this one, and ready to start the next one, so thank you all very much, oh, well. The difference between an ArrayWalk and a LinkListWalk is a factor of 12 on my laptop. That is how much, this is how little you should be using doubly linked lists. Cashmiss is a real, real expensive.