 Thank you, Sivan. It's good to be back at a BSD conference after about 10 years. So glad to be here and meet familiar faces and new faces in person. This talk is more or less identical to the talk that my co-programmer, Santosh Raju presented at Asia BSDCon earlier this year. And we've done a little bit more work on it, but nothing to show for it that we've published. So I'll just stick to the baseline of the talk. And essentially what I'm hoping to do with this talk is not really to present the guts of the development cycle or the experience, but rather to kind of share the opportunity that I realized after 10 years of kernel programming that a certain methodology or approach to coding can actually bring to kernel programming. And I feel like because this code was quite intrusive and at the deepest part of the VM subsystem, typically you'd expect a lot of chaos because NABSD's VM powers about I think it's about 11 CPU architectures and about 70 platforms. So it's really like a huge amount of dependencies and a lot of things can go wrong. So really most of the technical details are in the man pages. We won't cover that, but I just wanted to kind of go over the broad sort of storyline with the hope that I'm not preaching to the choir, and actually there are people from other BSDs and also diversity of port maintainers in here. So it's a little bit of evangelism as well if you bear with me. So this is the background. The background is that all the current BSDs have a very simple implementation for keeping track of RAM on the machine. Obviously it maps to a very simple static array. And this is the famous VM FISSEGMAX hash defined that has come up for discussion when large amounts of RAM show up or in the X86 world the EA20 map is very fragmented and then things happen. And this is obviously from a really long time ago when RAM was not so much and the layouts were not as complicated as they are on modern machines. And so the discussion about this or the idea for this project came up from the need for virtual machines to have memory added and removed dynamically at runtime. And right now we have a balloon driver which is a hack to kind of remove allocated memory from a virtual machine. But there's no way to put more in for net BSD at least and for the other BSDs I believe. So we said okay how do we take on this problem this is really you know the tentacles go really all over the place it's really hard to you know pull things out and separate them how where do we start you know I mean there's there's VM FISMEM is a global variable and it's strewn all over the place and it's used you know by every bit of the VM code everywhere. So the first job obviously was not to try and do any hot logging but to clean up. And so we said okay we're you know we want to use something more dynamic than an array but we want to clean up so that we have a hold on sort of you know the complexity otherwise it just explodes. So we said okay we'll clean it up and so that's when you know the idealist you know shows up and okay so now we've got a chance for a brand new API so I went away and designed this massive API with like tags and cache attributes and everything and then I had it all spec'd out we wrote ATF tests for it and then we ran it by people who actually had a lot of experience designing and building these things and they basically came back and said sorry I mean this looks interesting but so and we hadn't you know it hadn't got in the kernel and so Chuck Silver was really helpful with a lot of feedback he helped with the initial integration of UVM the virtual memory manager for NetBSD into the kernel and so you know idealism gave way to sort of pragmatism and we decided to kind of work the other way so we thought okay we'll try and put what exists and what we've cleaned up to kind of interface with the rest of the kernel through a simpler API and so that would mean that we'd separate out the existing API and expose it via header files and hide everything else in file scope and then most importantly have a way to test unit test all these functions. This is the first time in I think maybe 30 years since the BSD projects have begun that actually these functions were tested actually using unit tests so so so it was you know it's kind of interesting and and we yeah so so a lot of the a lot of the detail was basically trying to get this situation going so like clean up the code you know sort them into separate files and then try and organize getting the specific functions unit tested in a disciplined manner and we made the choice to kind of do this first so like this was in the live current tree so you know because you can you can go away on a limb and try and kind of you know create APIs and then everything breaks when you try and integrate so we use the approach that before we try and unit test everything we'll try and do the reorganization in the live source in a way that doesn't change functionality but then we can also sort of put the tests on top of them and so we initially got UVM 5 second dot c and dot h the code reorg into current before the tests were in place and now it was in a position where we could start looking at the unit tests and Suntouch actually did this entire project on a windows laptop with Sigwin so that's testament the FESD code cleanliness so so yeah I mean so the reorganization didn't really affect the ability for NBSD to boot and you know life went on as usual in current we didn't we didn't mess with that but then this is where sort of the modifications now started getting interesting so we essentially figured out how to now now everything everything to do with the array management the segment array management went into that file UVM 5 sec dot c and so now you've got like a little module that you can actually test and so we literally did a hash include UVM 5 sec dot c inside of a userland you know test rig and so now we had enough to compile a separate userland binary which then we could run unit tests on and the UVM 5 sec dot c that we're using was not a copy it was a live copy that was in the kernel so you know everything that we were testing was already in the kernel but we were testing it completely in userland which was pretty neat in that process obviously you know you have to export a subset of the kernel API to the to userland romp does this by namespace rehashing and so on and so forth if people are familiar with romp on NBSD but this is much simpler you know this this code doesn't really use you know a lot of API infrastructure so you know essentially it was came in malloc being stubbed by malloc things like that so it's fairly straightforward and so so so now we could focus on the guts of the of the move from you know static array to RB tree and so we said okay well you know what are the other options and we went through the iterations of that and and essentially boiled down the decision about RB trees boiled down to you know what was available that was easiest to use and gave us the best promise performance and literally you know NBSD's RB tree implementation was right there waiting for us to use it and it made a technical sense because the previous code had I mean an array if you want to insert you know things into an array you have to copy over you know bits of data and then insert and so there's a lot of overhead and so there were a couple of algorithms using hash defines so you could you know there was a hash define for binary searching for specific items in the array or for random search and so on and so there were three options and we said well you know all of that goes away and the RB tree takes care of that and so we just have one there's no compile time option now for searching in this array of segments of memory segments so there are no multiple strategies for maintaining segments and obviously less code clutter and yeah there were a couple of other options Q is anyway not a not an option because it was the worst of both worlds but yeah RB tree was there and we thought we'd go with it and we also thought we'll do some performance analysis from user space because hey that's cool so we'll come to that at the end some of the details so essentially the thing that's interesting for the rest of the VM system is you know what does this bit of what is this page offset in RAM point to what the VM page structure basically so I want to look given a given a physical address I want to have the pointer to the data structure that represents that page of memory that's essentially what the what this array gives you and so we essentially abstracted the idea of the index into the segment with a sort of abstract sort of data structure called UVM physique underscore T but initially we before we did that we ensure that this abstraction was sitting on top of the array itself and so so yeah I mean I went over this again it was quite interesting it was a huge number of architectures to port to so yeah so some of the challenges here and then obviously the performance implications so we look at that later so we introduced you know certain sets of abstraction so as you can see previously you had global variables being you know used outside of the module and then you would have sort of iteration through a macro and then we kind of hit everything behind explicit function calls and it can be argued that you know this is more less performance but then the setup and many of these loops are actually run only at boot time or during hot plug itself but when it comes to actually searching for things in when the system is running you know so this is not really so much of a hot path so we thought we'll you know we'll run with it and then we'll see how it goes in terms of performance this function came up because it was interesting to understand what a valid segment was a lot of systems code sort of ad hoc puts in a placeholder value like minus one or you know null or something to signify that you know an entry is invalid and it got interesting because we had to make this explicit and say okay what does this index point to is this you know is this a valid address or is it a valid index and why is it a valid index or not so we wanted to pull out that assumption in terms of tests so literally the approach was if there's anything that we had in doubt in terms of semantics or state we'd write a test for it and make it explicit and this was this was really useful because I think this kind of took care of the idea of the hero that we got that we got introduced to this morning because the hero can be reduced to ATF tests I think this is my discovery so you know a lot of the heroism is holding the state in your head and you know being able to process that data and make a call on design but I feel like if you can make it explicit in these tests and you can automate the open source hero into into tests by and large I'm not making this claim but at least from my own ego it was interesting to you know I became less relevant basically so so we you know it goes to doesn't need to be mentioned that we did quite a bit of test coverage and all the tests you know they're live tested using Anita in the NPSD build system so these are part of the formal test rig for NPSD and essentially what we did was we got the tests running so that the the objective was for the semantics and the state information to be exported explicitly in these tests and we made sure that the current implementation could be tested rigorously first before we touched it with arbitrary we said we want the current implementation to be you know running under these tests and and and so yeah and so sorry yeah we had some interesting situation so the original idea for VMFISSEG the array was explicitly meant for boot time so you boot up you populate your array after machine dependent code passes through your memory hardware dependent memory tables and then you populate the array and then that's it you don't touch it afterwards but we were interested obviously in you know being able to plug in new memory and pull out bits of existing memory and so immediately that brings a lot of interesting issues especially fragmentation and how to manage stuff and in the process of this integration you know there were certain assumptions like for example page first load was just expected to succeed because if you couldn't you know if you couldn't register a physical page during boot time then that's like something serious is wrong so you panic basically and so that was the assumption in the boot code it was avoid uvm page first load well that's not great for testing because we need to know if this function actually you know did the right thing or not and the function we should have some way to return that information so we essentially edited the function you know to return what happened to that load and so there were similar kinds of things that were you know modifications that we that we did so yeah so sorry i'm speaking i'm going ahead of myself here but essentially again just you know repeat that reiterate that point it was to make you know the test framework pull out all these assumptions that were inside of programmers heads you know and just you know put it down in writing and make it as explicit as possible um so um so so here's here's a like a slightly subtle one um we we kept getting this failure for get previous segment so given uh an array offset in that fizz sag array uh can you give the the function is asked can you give me the previous uh you know um entry uh and obviously it would be in an array it would be just you know minus index minus one uh but this was uh failing for us and this was kind of the situation in which that was happening and for a minute it had me going wondering what the heck is going on um and at this point we had the rb3 uh implementation as well so we were testing both together side by side we had the test running on the static array implementation and the rb3 implementation um and what we discovered was that um the reason that get prev was failing was because the assumption of what um of the of the property of the of the handle uh that we were passing through that to that function so for the array it would be the index value but for the rb3 it would be a pointer and there were certain implications for that um which i'll talk about in a second so this is the static array implementation so b is a loaded segment the system knows about it it's already been buvm fizz sag loaded and therefore the index if you were inserting segments would be in increasing monotonically increasing order so zero and then if a new one were a was inserted what would happen would be that that would get copied over and then a would go in and zero so you can you can see here that uh to refer to b initially you would want to refer to the index value as zero but then once the insert has been done zero refers to a so this means that the handle was not immutable we were making the implicit assumption that the handle was immutable but in this case the handle was not immutable in this implementation and so you know this brought it out um whereas for the rb3 there's no problem because the pointer remains the same so the you know the the handle is immutable um and so essentially again we exported that uh you know assumption into a test and said okay well we'll slap a test onto that so that we know exactly what's going on um so um yeah so now the the proper i mean the the behavior uh of both implementations is explicit and is explicitly tested for um right at that point of time we were also supporting conditional uh compiles of this static in fact we still do have conditional compiles of the static so you can use both in that bsd right now uh so you don't need to actually use rb3 you can also use the old static implementation um right so guess what happened yeah so um so at this point i was on my own sometimes um i wasn't very familiar with the net bsd system itself and it was um i was pretty pleased that you know i was able to break down the problem in um in a language and in a context that he could understand as a user space programmer and he was happy that he could actually get some you know uh work done on on kernel code uh but at this point i was on my own basically and i must give him a lot of credit he's an amazing programmer one of the best i've met actually um and um well that happened and normally at this point i'd be panicking because like if i didn't have all the state information out there you know it's the vm system there's like a whole load of things that could be going wrong uh but that wasn't the case i mean we had the tests and we knew exactly what was going on so we could actually flip assumptions inside the tests to figure out what was going on and we quickly realized that um the tests themselves assumed that malloc would always be available obviously because the tests run in user space uh but obviously the kernel uh doesn't have malloc running when it starts off you have to figure out how to present malloc uh you know to the kernel that's booting so that's the bootstrap problem that's very peculiar to uh you know boot up situation um i think freebsd has a uh a vm api kind of which does in a very clergy way i forget the name of the api now but um um anyway but that's in a different context um this um anyway so we essentially what we did was to kind of provide um so we tweaked the the the uh the code a little bit so that we could um have back end functions to do the allocation a little bit like the pool allocator i don't know if people have used the pool allocator but um pool cache allocator but basically your slab allocators i think have this backing um function thing where you can you know allocate either device uh addressable memory or actual you know system ram memory uh but anyway we use the same kind of approach so we had like a callback function that would get initialized at boot time which did the static uh allocation for uh boot time ram and then once we figured out that page in it done was you know done and kmem alloc could work then we'd switch the allocators to the regular kernel memory allocator um so that that was a little bit of a twist in the in the story but um um but yeah i mean again we got that uh so we again that assumption we put it out in tests so we explicitly put the you know the assumption of boot time allocation versus uh you know runtime allocation inside of tests um so um at this point uh more or less x8 is six was uh um working we hadn't integrated code yet mind uh because you can't do that i mean there's 70 architectures uh gonna get breaking if you start pushing code so um so um we started looking at okay so we're trying to you know uh fragment memory and uh sorry hard plug and uh and uh replug memory uh and we started looking at the at the fragmentation problem so um so you have a segment and then the um so how net bsd does this is that there is an array that essentially um holds the physical um data structures uh of all the system pages in memory now segments are not at page resolution segments are it can be like a one gig uh segment followed by uh one megabyte segment followed by so on and so forth but pages are 4k pages on net bsd i think are most architectures um so pgs is an array of vm page uh structures and not pointers these are actual structures that's where the actual data about the pages are stored so pgs is really important because again we we go back to the array problem where um as we saw in vm segments uh you know that was an array and then pgs is an array as well so now we've got the same problem coming back at us and uh we were like oh my gosh now are we going to have to use the arbitrary again have to go over the whole thing again but extents to the to the rescue so extents is a really cool um extent manager so any namespace you give it it'll manage the fragmentation insert remove uh operations on any any namespace it doesn't have to be memory so it doesn't make any assumptions about uh backing or you can tell it not to make assumptions about backing uh memory uh and so that was it we just plug that into the pgs array management so instead of manually going and trying to mess with the pgs array we just said okay well we'll oh sorry we'll uh we'll use extents we'll register the entire pgs array namespace into the extents and then just tell the extent manager to do it for us and it was brilliant just worked it was plug and play literally um and so at this point um we were ready um so yeah at this point things were things were going well and uh nick Hudson mostly and Maya helped uh me integrate most of the code and again like what could have been a real mess it was like two weeks and most ports were like done i mean 70 70 architectures i was pretty impressed i uh it could have been a nightmare but um you know the methodology really uh so the hero didn't do it the tests did it i think um performance is really interesting i mean uh it's really simple performance testing on for this uh codebase because all we all we are interested in is how fast can you find you know a segment given a physical offset that is literally the fast path operation in this whole api um and so it was really uh and there's a there's a essentially a macro that wraps over this function call so literally we just had to performance test one function and that was pretty straightforward um so what did we do well we had um you know these these these tests um had to factor in the fact that there were fragments memory um segments uh because this was not the case previously before unplug was available you know all the segments would be contiguous there's no big deal uh you know finding out the same algorithm could find them out but here if your segments were fragmented then your rb3 starts getting um you know filled up in not necessarily symmetric ways right so um so we wanted to see what impact that had um on performance and so what we did was we kind of faked the um uh so we used random to essentially uh fake the um the effect of a running system so we used the random system uh system called um essentially uh sort of spread out the the physical address offsets of also uh you could okay so you could test it saying okay from zero to the end of RAM you know run through a loop and see what page uh FISSEC find gives you but um in this case um you know that you know that's one way to do it or we could just sort of randomly look up offsets and that's that's what we did so um so we did a you know 10 you know 100 calls 10 million calls kind of uh kind of breadth of calls and and again you know random so it on a real running system uh physical addresses are not necessarily randomly looked up right i mean there's a lot of caching a lot of code that keeps running repetitively so this is not really um an indicator of the actual system performance and this is sort of work to do or maybe a comment about how this kind of testing can be done in a slightly different way load testing i think is not really um mapping straight on to atf testing um but um essentially um yeah we just you know we used what we had and all this stuff was done in user space um and so um yeah so we you know we applied different scenarios a fixed size segment fragmented segment just to see what effects it had on the performance and we'll get you some nice graphs but in a minute but uh yeah so this is what it would look like numbers will be pulled out like this and then um yeah and then we got tables and uh you're welcome to look at that later on but you can yeah i mean so the you can see the you know you can make a quick scan through the tables but the top one is rb3 and the bottom one is static area and you can immediately see that the rb3 implementation is taking more time uh you know to uh find uh segments so we wanted to see you know how bad it is um so yeah this is just a sort of uh you know testing between 10 million uh segments and 100 million segments kind of thing um um so yeah so this was the statistical kind of methodology we used because we were using random uh so we had to kind of compensate for that by using uh making a statistical approximation about how um you know how much confidence do we have in the actual number we're getting uh and so we kind of uh you know we stuck to a margin of error of 95 percent uh i didn't do this work so uh if you have any questions i'll redirect them to Santosh um but anyway um uh so that's the number 5.59 percent degradation and performance um which you know depending on how important it is for you to plug or unplug your memory live um is a design choice that you know somebody can a manager can make uh so um so there we are and um right um yeah so that's the details of uh the fragmentation and how we did the tests and colorful graphs but you can kind of get an idea um of yeah so those are graphs of the average maximum and minimum times for different um you know numbers of segments that were being tested and uh yeah just to go over some of the things that we were probably missing originally when i designed the um API the ideal API that had all the uh you know cash attributes and this and that uh we actually looked at romp testing uh testing on romp but the problem with romp was that there was a whole load of assumptions inside of romp and a whole load of dependencies that we'd have to bring in uh and so we abandoned that but i think for load testing uh romp is pretty good because it can uh interface with actual user space workloads and exercise your code parts so i think uh romp is interesting for load testing of this kind of behavior um yeah code well we could we could have a bit more code coverage i mean that's the question of how much time you have to um uh address but more importantly i think maybe if we could get some live numbers as well to make a comparison to um the numbers we got with user space um testing um with with detrace um i'm not sure detrace i haven't used detrace in net bsd but i don't know what the state of it is uh anyway maybe we'll get some feedback on that later so yeah this is the the big uh you know moment of enlightenment um you know uh existing techniques nothing fancy you know heroism required uh just you know apply interest you know standard techniques and you can do cool stuff um i definitely had a much less stressful experience than uh debugging zen intrapandlers so so so this is really cool um and uh obviously you know it did help that there were um apis inside of net bsd that made the least amount of assumptions about what those apis the context in which those apis could be used for example you know rb3 could be it's a you know it's a user space library but it can be used in the kernel or the extent um you know api does not assume anything about the nature of the backing ram for example so really cool to have that kind of tooling available inside of the kernel um so i mean literally this sort of pitch is for people who have access to or know about hardware that uh can do hotplug outside of virtual virtualized environment or even inside virtualized environment i think virtualbox um qmu has an acpi interface for hotplugging that i had to look at but it was way you know our acpi stuff is a bit shaky at the moment so a little bit more work required for that um zen is the easiest to do and i've done like a really um simple stub plug implementation uh what's missing is that if you want to uh unplug code your driver needs to uh kind of so there's a little bit more work required for unplug because obviously you can't just unplug memory that's being dma'd to or like you know memory that um you know is being used for something so there needs to be a way to uh figure out uh you know if if a page that you want to unplug is uh in user not um and that uh you know this can be this enables that but it it doesn't um do it for you so if you're writing a driver obviously you have to you know there's a little bit more work for that and also exporting this functionality to user space um you know uh user if you have a ram control kind of utility that talks to the kernel and says pull out this offset of ram or whatever so that api needs to be designed and so on uh but the basic uh interface with the virtual memory manager is in place and it's reasonably robust i think i haven't seen any complaints of weird page faults or panics on various architectures so far at least so i'm keeping an eye on it but um so yeah and the cool thing is that the vm system is probably the you know the the most one of the oldest in pedigree um because obviously all the bsd started off from a similar vm uh system so i feel like other bsds could actually benefit uh from uh this work so phibiasd especially i've had to look at their vm coding but uh you know they could benefit from uh this stuff uh open bsd i think uses uvm so um i think it should be fairly straightforward for them but phibiasd might need to put a little bit more work into it uh thank you all these people uh philips here and uh yeah the bsd foundation we coded this by the beach uh we spent four months it was brilliant highly recommended philips knows about it he's been to the venue um yeah and yeah that's uh that's the credits and yeah thank you everyone and yeah five minutes yeah we got five minutes for questions can we kill the balloon driver now uh well the balloon driver could use a hotplug uh so yeah right now the balloon driver uses the hack that it uses kmem alloc to reserve memory and then you know keep it away from the rest of uh uvm from reusing it uh but it also interfaces with the hypervisor uh you know so there's an assumption there about how that api works so we could reduce it to an api that's kind of named balloon but does something else should probably be called hotplug now but uh yeah uh but yeah the kmem alloc stuff would uh change for the benefit of the video can you explain what anita is and how it worked uh when you workflow um yeah so anita is a continuous integration um set up the net bsd has where um uh all the atf tests inside of the net bsd source three correct me if i'm wrong uh are a kind of so your source is built and then all the tests are run um i'm not sure for what the frequency is go on sorry anita is a python uh you know wrapper that actually uh takes a build and after that it uh spawns up either a uvm or another emulator runs a net bsd install you know from scratch and then starts running the test so it builds a new environment a new system make sure that everything works from you know the installer to actually running the full test suite and it completes and runs uh there is in the releng.net bsd.org you can go and follow the link and uh it has uh several architectures that run daily and others that run on emulations daily and others that run on real hardware you know weekly or whenever people get to run them the um e386 runs eight times a day um uh spark we aim to run twice a day and um others about it the same frequency all right i hope that answers your question now the question the real question is so first of all i think that we don't really unplug and plug memory all the time this is kind of a an event that doesn't happen too often so optimizing the whole thing to an rb3 seems to me like kind of an overkill right because you know you could just say stop the world and rebuild your array isn't that uh a feasible thing or what what what other thing does rb3 buy you uh that's the first part of the question and the second part is that the unplug is the most interesting part of it and you know the the reverse lookup is what you want to have really is to basically have the subsystem ask okay who's using those segments that i'm trying to unplug right now and if it's a device driver the device driver needs to relinquish them and if it's a process we have to wait for the process to finish or we decide to kill it or something right and that's have you given any thought of that yeah this is a trick question okay so i have fleetingly thought about it and and we when we did the work where we really had to have deliverables we wanted to you know put it at a point where we actually had some functionality and so initially i did actually go around and write a very rudimentary ram control tool just to think about how a user would interface with you know the kernel but it turned out that that is a house of cards that's going to fall very hard so my thought about this is that if we can incrementally start so the arbitrary question i'll come back to in a minute but my thought is that if we could have if we if we could have users so for example zen or any virtualized environment is the immediate one that i can think of and then you know there's heavy metal architectures out there that have a cpi based hot plug and i'm not sure if the other architectures spark or and so on but i'm sure that the server industry has these hardware mediated ram plug-on-plug things so if we can if we could have just the basic sort of plug-on-plug operation for specific cases that are maybe that's memory that is you know not dmaid or memory that is only allocated for a restricted set of things that can be tracked so for example things like maybe the some kind of cache maybe i'm not sure which ones but we'll have to you know do the analysis but a restricted set of these pages could then be looked at because the state tracking again it's you know it's that's a huge effort to figure out the reverse lookup of what page is used by what subsystem and then to keep track of all of that and then to make sure that this particular page is not being accessed by anybody that that is a significant search problem in terms of such space so yeah taylor is it often the case that you will want to use memory that you have hot plugged for say dma or kind of you know driver specific specialized purposes like that or could you prefer to allocate dma memory from what is provided initially at boot and prefer to provide hot plugged memory to say user processes where you could just you know you could unmap them and move the page data to another page and then remap them and let the user user space continue uh yeah that's that's a pretty uh that's a leading question actually so i think bus dma already has uh and this is a machine dependent question because perhaps one can imagine that you know there are architectures where hot plug memory can be used equivalent to boot time memory and then there are other architectures like x86 where the first 16 megs is you know only accessible to certain types of dma controllers and so on so i think that is a machine dependent question uh um sorry i missed the main part of your question so my question is um could you use this as a way to avoid having to do the search the reverse lookup problem and yeah yes make the problem simpler by pretending harder cases don't exist yeah more yeah so that's kind of what i was leading to with uh when i was saying that we should probably try and restrict where unplug is done inside of certain categories of memory or certain um memory management apis like bus dma or um i can't think of something else at the top you know at the top of my head but um yeah so restricted categories of of uh ram could be you know that's a start i feel like and then you know different subsystems can then decide whether they want to use this or not and then eventually we could kind of have a sort of global it's a complex problem i don't have all the answers right now but um it needs definitely it needs analysis but i would say i would like to actually when i get spare time this was a funded project thanks to the foundation but i would actually like to look at various use cases and start doing this you know bits of categories of ram and then see if this api can be exercised got two minutes to answer christos's original question which uh which was uh why rb tree um i think the simple answer to that was it was available it was easy to use um and uh yes we um yeah i mean we could use something that was already available so in in theory we could go back to the uh array implementation and make it do what the rb tree does now because we have the testing infrastructure in place to make sure that it doesn't fall apart so actually there's nothing stopping us from going back and reworking the array implementation to do exactly what the rb tree stuff does so yeah so it's possible it's just that that was convenient and we thought that it was it was not a very organized decision decision but we kind of asked around about the performance implications and we thought well we'll just find that number five point x percent and throw it out and see if people complain too much and if they don't then we're just gonna you know push it in but we're not forcing you to use it you can you know you can optionally use the uh array implementation without hop plug and then the rb tree implementation with hop plug and uh if someone's up for you know doing a hop plug implementation of uh you know using the array you're more than welcome i'm happy to to you know help with that so thank you very much