 Okay, I think we can start so Today we are gonna present the the work we've we've done for almost a year with Richard to support mlc-nandy in the in the canal So I am Boris Brezion and I'm working for three electrons and I'm a child of mine blogger and working for Sigma star game. They are and we are both doing consulting in the area of embedded Linux and This project we are working both on and had a lot of fun with Emerson and and Now we'll progress give you a short introduction to emancy and then I will take over and tell you More details on on ubi and other stuff Yes, so, you know SLC NAND is supporting for is supported for a long time now and it's started to Begin to to be bored we started to be bored and so NAND vendors decided to Invent and using to to to let us keep our job. So they invented mlc-nand So let's see a bit what the constraints are for on mlc-nands. So the first one is kind of well-known and expected You get reduced lifetime, which means you have fewer erase cycle program erase cycle You also have to program all the pages in the ascending order. We'll see later why We are that our retention issues, but we already are those issues with a substance the only difference is that we have a lot of them and Yeah, it tends to To Complicate a lot the process We also have the bird pages project problem, which I will detail a bit later and again, I don't know if NAND vendors try to be mean with us, but That's that's a bit hard to deal with and the unstable bits problem Which When you design your board your you'd better Make sure you have Some capacitors to to be able to finish the program or erase operation You are doing before the real poor cut happens. So we'll see if we can solve that So the first one is the reduced lifetime actually, this is not a big issue because Ubi has been designed to distribute the work over the wall device The only thing you have to do here is just tweak a bit the configuration That should work pretty well The second issue we have is the Data retention issue and before I describe that one. I just want to show you What's the difference between an SLC cell and an MLC cell so in SLC cell you have only two levels One is erased and one is P1 in which the bit is set to zero and In MLC cell you have four different levels This means you can only go from the bottom to the top and then when you want to go back to the Erase state you have to erase the wall block So Why is it a problem for a data retention? This is mainly because of Read and write disturbance so with MLC cells Since you have more levels this means the the distance between the levels are Smaller and you're more like more likely to switch from one state to another if you have some charge loss or if you gain some electrons So this is partly why MLC names tend to be more sensitive to to Read and write disturbance You also The fact that MLC cells are smaller and are closer than Our closer together means that each time you have to program or read a page the disturbance is even Stronger So here The only solution we have to deal with that is either to increase the ecc level ecc strengths But then again, you'll eat the problem at some point So no matter the ecc strength you put you'll have to detect that you reach a level of bit flips that you Count all the handle and just move the data somewhere else And this is what actually this is what ubi is doing ubi is When ubi reads a block it checks the number of bit flips and says oh this block is gonna be unreadable Soon, I'm gonna move the data somewhere else So yes, I just said that before Then comes the bird pages problem Actually, this is one of the biggest problem we have with MLC So let's go back to the MLC cell diagram You see that you have two bits inside the cell and That would be too easy for us if the two bits were assigned to the same page So then Vendors decided to assign them to two different pages And each time you want to program one of the bits So you have to first to program the low lower bit and then the upper bit and each time you want to program one of The bit you have to change from one state to another If you look at the different states we have here If the first bit you want to keep it to one and then you want to program the second bit to Zero then you have to go from erase state to p3 And if you have a pocket in the middle you not only corrupt the the second meet Which is kind of expected by the upper layers, but you also corrupt the first bit Which was already programmed before and assumed to be safe So this is kind of breaking a simple rule which a lot of people were relying on so That's that's the problem. We are we have with third pages So that's the reason NAND vendors ask you to program all the pages in ascending order And Actually, it's a bit worse than that Because if you look at the differing scheme we have in the wild So this is just an example. We have different period scheme, but this one is already Quite complex so You see that Lower pages and upper pages are kind of interleaved which means you can say, okay Even if my pages eight kilobyte Let's say that my page is actually 16 kilobyte and I program tries to page and at once And that's not possible because they decided to interleave the pages Actually, I think they have good reason to do that that It's not clearly and explain in the data sheet. I I guess they do that to improve performances and Also mitigate the right disturbance, but maybe someone in the audience can give some clue No Okay, so anyway, it's how it works and it's kind of painful to deal with And the last problem we have Actually, this is not something we experience during our test but some people reported that if you Almost finished the program or erase operation You might be able to read back the data after the poor cut But only once so you read back the valid data once and then when you read back again Really read it back again. It's invalid and Again, that's a problem because Ubi and Ubi FS they don't know Which block was programmed last and we don't have that information right now. So since We never faced that and it's not really described in Datasheets or Nen vendors tend to say no my name is immune to that Which I don't trust But since we never seen that we decided to kind of ignore the problem and My guess is that it's not just a black and white problem. It's a bit more subtle So my guess is that if you interrupt a program operation in the At the end of the operation, you might be able to read the data back but you'll probably see a lot of bit flips and If Ubi sees a lot of bit flips, it will it will decide to move the data somewhere else. So Hopefully we will we are kind of safe safe against the unstable bit problem But yes, if some Nen vendors are able to Give us some more clue about the this problem that that would be great Then there was some problems we were not We didn't knew before working on on the solution. So the main problem is that We we saw that if you don't write the block completely So if you only write partially the blocks, then you will see a lot of bit flips and the last pages you programmed and You have so much bit flips that if you read several times the same page the same last set of pages You will have the read disturb effects which will generate even more bit flips and soon the page will become Unreadable, so that's not really a problem because Ubi is able to detect that and move the data to somewhere else But this means that in this case Ubi skips moving data around and it just works the Nen much faster So yes, that's something We discovered and we that that would be better if we Knew that before so this is what we called the open block issue Well, we also discovered some good thing The main one and the one that we are relying for our solution is that you can program Blocks in SLC mode So in by SLC mode, I I mean it's a software software MLATS SLC mode Which means you only program the lower pages of each block And of course it works. It solves the page pair pages program It tends to solve It tends to show less bit flips in those pages and Those pages tend to be less sensitive to read read this write disturbance But the main drawback with that solution is that you lose out of the capacity and that's not really acceptable So just one last word about the name constraints Yeah, that's really good to have some more information from Nen vendors because right now we are just Trying things and failing and then trying to understand why it's not working and why the name is acting Weird so yes, please Nen vendors share the share some more information so now I'll let Richard talk a bit about UBI and then present the solution So now a few words on UBI basics because later we will dive into the details and We have to make sure that you know the the UBI basics first of all UBI nomenclature and that's something often gets wrong and people get horrible horrible confused by it As you know, UBI is a mapping between Logical and physical erase blocks all blocks UBI Presents to the user is actually only a logical block and is then mapped to a real physical block That's why we have denotation of the BEB. That's the physical erase block and LEB the logical erase block. That's the easy part Then What most people get wrong is the UBI image the UBI image is this File or or dump that you have on your empty partition For example the tool ubinize creates you a UBI image and then you flash this image under your empty T Can be one one empty partition or so the whole names Then at at the runtime we have the UBI device That means when we start UBI and UBI Scans the whole partition and it's ready to work then we have the UBI device. It is also being found under slash death is always UBI and and a number and Then we have also UBI volumes Just think of the logical volume manager. We have for block devices. That's something for flash so UBI offers you a way to partitions your UBI Device in multiple parts namely the UBI volumes That means you can have multiple UBI FS Instances on the same UBI device Please never ever create multiple UBI devices on your NAND unless you really know what you're doing It's a day. We had a nice discussion on that that at some point it makes sense to to have multiple UBI instances when you want to make sure that the view labeling won't hurt a specific part on your NAND, okay, but Actually, you should have the view labeling to domain as large as possible So and then finally we have the word attach That's basically the the process of floating and starting UBI because UBI has to scan Either the fast-map or the full flash image and this process called attach So when you when you report me about on a mailing list, then I ask you does UBI attach work? And then you know now what I mean so and now How does UBI look on your flash? Does this cursor work? It's really small No cursor, okay We have the the logical erase blocks on the top The logical erase block number starts always with a zero and start with zero for each volume that means when we have one volume then we have an erase blocks starting by By zero but when we have multiple volumes then the logical erase block number always starts with zero and The user above UBI for example UBI FS only sees the logical erase blocks That's it. That's important to know because UBI FS may report LEP number five is bad. Then I know that's the fifth logical block. It's not the physical block sometimes people get this wrong and then to a to a NAND dump and go in the hex fewer and Search the offset of the fifth block and get confused So then you see in the next row the physical erase blocks obviously the physical erase blocks start by by one and go to N, but they They have only one one instance for each number UBI just starts numbering then by by the by the offset zero and goes up to N Every logical erase block has a mapping to a physical one Of course, it can also have multiple mappings because UBI does copy on write This operation is called the atomic lip change. That means you can change the logical erase block atomically We need this to preserve power card safety for example when UBI FS Starts writing a new journal head Then it finds which logical erase block number has the journal and then it says, okay I update the data and then it just does an an atomic lip change and for a small time frame We have on our flash two copies of the same logical erase block data But since we have a sequence counter UBI can decide which one is the newer one and of course we have also A checksum when we do a copy that means when we face a power card and the data is not fully written we can use a checksum to find out that we have That that has a problem while writing the The new copy and can just go down to the old one so each Each physical erase block has two UBI headers Meaning the erase counter header also referred as easy header and the volume header also often called the volume ID header These are two have two headers As soon UBI finds a free physical erase block it immediately places the erase counter header on this block The purpose of this header is as the name says preserving the erase counter Because UBI wants to know how often each block has been erased for purposes of relabbing and As soon a physical erase block gets assigned to a logical erase block Then the volume header is placed onto the volume header Contains the logical erase block number and the volume ID and some other details, but for now we only Focus on the volume ID and the logical erase block. Is this clear so far? Do you have any questions? That's the basic concept how UBI works or nice a Room full of UBI experts. I have more questions So Boris said for before and we can write a physical erase block Just I'm too fast first. How does the how does the header and how structure look like on SLC names? as I said before we have the Erase counter header and the volume ID header So far so good now the Now the the first Special trick comes into place namely sub pages on SLC names You can write a single batch into multiple steps In data sheets you find this as the NOP not that number of programs is typically four That means you can program a single page in four steps And that's why Linux says when it finds an SLC name That we can have four sub pages for each page And that's why UBI is able to place the erase counter header and the volume ID header into a single page Otherwise we would have to waste two pages for the metadata Here in the graphic you see it as that both headers are in the in a page be zero that means we We have an overhead Being from you be only by one page. That's rather acceptable And the other pages are used for the payload of the of the logical erase block meaning the real data you can use For example used by UBI FS so Now a few words on the data retention issues mostly because they are really really visible on on MLC on on SLC you have a hard time to Reproduce them on MLC. You see them really fast so mmm on MLC you face mostly read disturb for example on an SLC NAND you have to read the same page a few hundred thousand times just to hit a single bit flip due to Read disturb on the MLC when you are unlucky 500 reads are enough So that's a real problem So as the typical solution to deal with read with read disturb is detecting bit flips Before we have more bit flips than we can incorrect and scrub the block scrubbing means we copy the data to other block and and after a Fresh programming then then the bit flips are are prepared to deal with that and Mostly to do more experiments with that we have the UBI health demon It's actually a very simple user space program that can fetch from UBI Statistics how often each page has been has been read and then UBI health demon decides, okay Let's trigger a scrub Now more on this slide Untypical SLC NAND You have a crone chop which reach from time to time your whole UBI volume and that's basically enough I write here on SLC without fast map for a reason because Fast map is a nice optimization for UBI to have a fast boot up because it doesn't have to scan all pages When we do a disk dump from UBI Zero underscore whatever then we read only a volume and we read only the payload not the metadata That means we can only address read disturb issues on the metadata and the the volume headers and also the internal UBI Volumes are only read doing boot up That means on SLC you are you're only safe and I do from kind of from time to time a reboot without fast map and do a disk dump On an MRC. This is not enough and that's why we had the UBI health demon The disk dump command is also not so nice because of course you have to you have to read The whole volume and it will introduce a large IO load and maybe your your real-time application will block whatever It's not nice and that's we have the UBI health demon. You can tell the demon, okay Please make sure that within time frame from one month The whole name chip will be read blocked by block and then the health demon will will start read commands After a few hours up. You can just make sure that within a given time frame the whole name is being read and Redisturb is being addressed We currently not sure whether it makes sense to keep UBI health demon as user space program When it turns out that it can be solved easily in the kernel, we will move into the kernel But for now we keep it in in user space just to do more more experiments with with specific writes and read patterns and Such things. We also use it as a small reporting tool to get known Which application introduces which read load because sometimes you have a rough Application with which reads a lot of data and maybe start rich Disturb more soon and you may have demon can tell you So and now I will hand over Where we have the read counter from and actually I've I've added a new interface to To to UBI which exports the read counters and UBI has an in-in memory counter That that exports them to the daemon and the daemon then has a configuration file where he can store the counters Of course, but we also say it doesn't matter when the counters are not precise for example when When we have a power cut before the daemon was able to to dump down the stats and we lose Two days of read counters. It doesn't really matter because we don't need exact numbers We only need a rough measurement. How often is this read roughly? It doesn't matter when it's read 5000 times or 6000 times it was often read Yes, of course the daemon writes the file more often than a single month And you can also Dump this counters into a UBI static volume Then you don't have to have it as file on UBI FS so and now I will hand over to to Boris and he will Talk about about the bird pages issue Yes, so So we saw that we need to program blocks in SLC mode and in order to do that the upper layers need to know Where are the lower pages? Where are the upper pages? Actually, this was the easy part We just needed to expose an API which is providing a way to retrieve Which space is a low page which pays is an upper page? And that's all So this part should be or at least Some part of it should be in the 4.9 kernel The rest should be In the 4.9 10 kernel So then you need to patch the upper layers and with UBI UBI FS you have two solutions as are you do that in UBI and UBI FS is Absolutely not aware of the spare pages problem or you try to handle that in UBI FS So the first Approach we tried is to solve that in UBI FS And inside UBI FS you have two approaches so either you Try to write everything in the SLC mode and then consolidate the data afterwards Or you try to write everything in the MLC mode and then when you want to secure the data you just skip the pages skip some pages and Try to start to write new pages but Far enough so that the previous pages are protected The pros for this approach is that UBI FS knows better than anyone Which data is valid and which data is dirty and so it can select which blocks Should be concealed solidated and which one should not And UBI FS knows when data needs to be protected so when the user space Does a sync on file then we need to protect the data So that's why we try to do that in UBI FS first Actually the solution to was not possible because of the pairing scheme we've seen so the interleaving pages the open block issue so we try that but it didn't work and I also tried to implement solution one that After day or two trying to modify the UBI FS implementation. I just realized it was Too invasive to to be acceptable So then we decided to go for the UBI UBI implementation I let Richard So Yeah, and then we came on a came on an idea how to implement a problem or how to solve the problem not implement in UBI so on the basic idea is we we write Every logical erase block by default in SLC mode meaning we use only half the pages and We of course waste half of the space but as soon when we run out of three blocks, then we combine or Consolidate to logical erase block into one block where both both page bars are used namely the lower page and the high page and that way we can gain one more three block That means That we can combine two Partitially written blocks into one fully written block and so all we have to do is is teaching UBI to to know when to combine blocks and then we can produce us a new three blocks that way we can use Over time all pages and pretty easy problem solved. Let's go home. Oh, no there are more slides. There are no more issues So now in detail On the left-hand side No, that's your right-hand side You see The the life cycle of a logical erase block without LEP Consolidation namely on an SLC NAND by default we start in the unmapped state and as soon we start writing It's in the open state. That means We can append more data or also unmapped it as soon as we write to that last page the the block is considered as full and then we can can only Unmap it again namely also erase it and then start writing again on the With LEP consolidation. We have a little bit more states Which I will explain you now first we start again with the state unmapped then again We can start writing being in the state open and Then as soon as we write the last page then UBI Has it as a candidate for LEP consolidation. That means you buy it now knows Okay, this one is full and when I find another full one I can combine them to a fully written one then it then it is in the state Consolidated that means when we have a physical erase block Which hosts two LEPs both LEPs are in state consolidated now consider the The unmapped we have the physical erase block which hosts two to LEPs and the unmapped one LEP Then we have a problem because We cannot change the data because we can only erase the whole physical erase block But when we when we erase the whole one then we would also kill the data from the other logical erase block We still need that means This block we would we we are willing to unmap is now in state invalid That means the data is still here, but UBI must not use it and As soon both LEPs are in state invalid then we can actually erase the block We will hear more detail that is on this later because it's it introduces some interesting corner cases So I think I have to speed up a little bit The slide here said everything one question why can we only consolidate LEPs with which are full The answer is When they are consolidated we are using high and low pages and then we can no longer change the data Because when we would change the data then up on a power cut we would kill already written data That's we only consolidate full LEPs So Now we extend our Existing UBI nomenclature for the following terms. We have an SLC LEP. That means it's an Logical erase block that sits only on the low pages This is also a candidate for Consultation when it is full The full LEP is when the when the last page is written We actually don't know whether it is completely written We only care about about the last page because when the last page is written You can no longer write a page in between Then we have the C LEP. It's the LEP which is consolidated meaning it sits on a physical erase block with more LEPs on it so and Here a short time frame. It's also also very important when should you be I start with consolidation Because when we would only start with consolidation when we run out of space Then the whole system may be slowed down. That's what that's why we have a trigger When you buy Decides that it is still has three blocks, but just does already consolidation and produces three blocks in advance here we see That we have two full LEPs LEP zero and one and one empty one and Then we start writing to the empty one being then LEP X X and then we We go over the threshold watermark and you guys start with consolidation and consolidates LEP zero and LEP one into the states See LEP zero and see LEP one and it produced one more empty block so When we have an MRC NAND and write a an LEP in in SSC mode Then we only use the low pages here For example, we have the erase counter head of being on the page zero Which is a low page and the volume ID head of being on the page One which is also a low page One interesting fact is MRC has no sub pages That means we have to waste two pages for the UBI headers That's not so nice because on MLC typically the pages are larger, but we have to use two pages and Then the payload is also used only is also only stored on the Low pages here. For example, we have page three and page n minus two and the other high pages are ignored on the other head when we have and consolidated LEP's then the layout looks like the following we have on the first page again the erase counter header and then on the next page we have both volume ID headers and then we have the payload for for for both logical erase blocks and Important we have to add at the bottom abetting that's the open block issue Boris mentioned before we have to fully write the block and Of course the stored data is is immutable. We cannot change it. So now some Problems we had or some challenges I typically SLC name. We have as many LEP's as we have PEP's because the mapping is one to one easy on MLC it is one to two that means we have twice as many LEP's as we have physical erase blocks Is it clear because we just using half a data, but we have in some the whole data We have to provide as far as many LEP's but when we combine them then we have to the full storage again That led to some interesting issues because within UBI on many places the number of LEP's is actually used as number of PEP's and nobody noticed and nobody cared to change and Yeah, we figured the hard way For example, that there's also an issue in the sysfs API where actually UBI exports the number of of LEP's but the user space tools think it's the other ones and get probably get probably confused But that's all only the minor issues One problem is also that you cannot assume that you can actually Use all all LEP's you have to you have to have to reserve a few for the internal housekeeping For example, when you have two full LEP's you you have to combine them into in the one New block to gain a new block. You need One more block as temporary copy because we do copy on right. We cannot change existing data Yeah, and we need some guarantees from the UBI user namely UBI FS UBI FS has to fill the LEP's completely namely it has to write the last page and it has to has to Has not to open and many LEP's in parallel for example when UBI FS would write One page into every LEP then it would run out of space But we could not combine them anymore because the last page is Never written But luckily UBI FS turned out to be really smart and and always tries to fill up a single LEP as much as possible before it claims a new one But we also have to know how many Open LEP's we can have for example for UBI FS we have one for the journal head One for the lab property tree. It's a number about around five. That means we have to hold back at least five blocks to be able to do LEP consolidation. Otherwise, we would run out of space while producing Producing space and now one. I really a fancy corner case Usually UBI can deal with the fact that you have and have an old LEP on your flash that is no longer used. It can happen when you have a power cut doing a copy and write operation Then you have two versions When we have now a physical erase block that contains the LEP X and Y but then unmap the The LEP X then it is in state invalid That means the data is still present But we would like to get rid of it so far so good When now the user above UBI FS Claims a new LEP with the same number for example X again, then we have two copies of LEP X One in SLC mode and one in MLC mode, which is invalid and now What UBI can do UBI FS can do is Unmapping and deleting the the newest copy of LEP X and we have the The the the old LEP X still present and when we now have a power cut We end up with the old copy as the newest one and UBI FS has a few corner cases where it is probing for LEP and then finds oh This one is still present. I will use it and then it's using all data and leads to interesting failures the solution is Either having a lock in UBI Implementing a journal that's impossible or really a lot of work what we do is what was fixing UBI FS namely Finding the Finding the places within UBI FS where it does an unmap erased by hand. Yes the question There is no flag. There's only Yes Because we cannot write in SLC mode or in MLC In SLC it's so it doesn't care because when you say UBI please unmap me this LEP And then it is queued for for erasure that are synchronous either erasure happened or it happened not Because UBI assumes that the user above it does the housekeeping on its own It knows UBI FS does only unmap a block when it does no longer need it so it's not a fully FDL it's only a kind of FDL and And says the user has to do the housekeeping so and now and we had to turn some Some unmap and evasion operations in the actually atomic lip change operations within UBI FS and that solved the problem because now We have the same LEP always reference it And the housekeeping of UBI works again So I'm not really late and hand over to Boris for the conclusion Because the zombie LEP problem is solved, but there are more problems. Yes, so basically we managed to Use MLC named with UBI and UBI FS And actually if you want to use the prototype all you'll have to do is Maybe create an UBI image using UB nice So when you create the image you'll have to pass a few more things like which pairing scheme is used on this name That's pretty much all you'll need the UB nice the new UB nice binary will be able to build the correct image for you As I told at the beginning of the talk we Already have most of the PCs for exposing the pairing scheme in the kernel, which is good we already have most ECC engine which are able to do strong strong ECC So the one we are using is the all-winner all-winner controller and it's doing up to 64 bits per Kilo byte, so that's already good Yeah, so keep in mind that you should use UBI else demon because otherwise you'll see Bitflips sooner, but It works Yeah, I Think I'll let Richard answer that one So the question was whether we can create a heat map of your NAND I have no tool to create me a heat map, but you can use the file that you by half demon creates It's basically a text file containing the counters and you can use the scripting language You like and create a bitmap But the UBI help them is currently heavy changed and we are not so sure what we should do with it Maybe we integrated into the kernel. Maybe we make it more advanced and keep it Currently we use it to to play with it So yeah, it as I said the solution works, but it's not perfect and For example lab consolidation is slow because you need to completely lock the the labs before consolidating Otherwise one could just unmap the lab the lab while you are consolidating it or or just update it. So That's kind of Yeah, that's expensive to consolidate labs together We saw that we have a problem with space reservation A few problems with the lab and pep concepts which have to be clarified And another thing that we think we should change is The fact that we only considered consolidate the full labs and actually this may not be the best decision because Imagining you have a lab which is updated a lot, but each time it is updated. It's completely filled and Even if it's full there is no It's it's useless to consolidate this lab because it will be just unmapped soon. So instead of doing the lab consolidation only on full labs, we will try to Maintain an LRU which is this time least recently updated database We'll all try to update you to consolidate the the ones which are Least recently at least updated. So that's that's the plan Yeah, it's still networks. It's it's currently been being worked on but it's not it's not finished yet So we are working on a version relaxing the consolidation thing Fixing the pep reservation and Reworking the consolidation stuff. So I didn't put any get up your URL because it's Not in a good shape. So but if you want some more information or want to access the Repository just ask on the entity manliness then I'll I'll provide the links Submitted We posted actually reached out position RFC. Yeah, actually since I'm the main dinner. I have to I have a preview of my own stuff No, actually, that's a big problem Currently there are not so many UBI experts out there and even less guys from From nand vendors who are willing to help. So I posted sometimes males with what we are doing and with links In patches, but there was not much feedback. So when you have an mlc nand device and want to test please Maybe we can solve a few issues, but I'm currently both and I are more or less on our own That's not a nice situation, but well That's it. Yes, you mean something like msc support for for Nancy Actually that could be done using Nancy And I have some batches for Nancy to to add the fancy stuff from emersy, but the the really odd things use your face only on the real hardware on my desk being now such a Pile of pricked nands That works best so we have a customer who is willing to give harder to us. That's good But we don't have a former language model. Yes So the question is what is the performance impact currently the impact is rather high because on emersy nand a Block could be multiple megabytes and consolidation means you have to copy around and read multiple megabytes That's one of the issues. We want to get rid before we remain like that so currently The impact is rather high Start when the UBI volume starts to be filled. So a lot of SLC labs in there and Consolidation takes place then you see a slowdown. Yeah We didn't do any measurement because the the main goal was to get reliable support for MLC and not Getting the best performances, but that's something we are taking into account for the the second version So you will probably do some measurements for the second version Yeah It was not all full. It's yeah Yeah, that that's why we start consolidating in background But the way the consolidation is done, which is led by our true labs by true labs, it's it's it's This is where the contention is so no because as soon as PEB is containing an invalid lab then it's a new candidate for a consolidation so you can Yeah, it could still be on the flash dirty data Yeah That's Yeah, but if you want to put the logic at the UBI level then you have to accept that Yeah, we thought about that Maybe we'll add something to say from UBI if it's okay this LEB is containing a lot of dirty data Please don't consolidate that one, but currently that that's not done. So The last question. Yeah, actually I'm working on an slc mod at the volume level which You will you will be able to create a wall volume which will always operate in slc mod Actually for read-only volume static volumes It's not so useful to write in slc mod So you can just write in mlc mod and pad the last block with zeros for example And then you you're safe because this is read-only volume and you won't write on it UBI is just Moving the data around and when it's when it's doing that then the wall block is compiled somewhere else So that's that should work. Thanks