 The story so far, I am attempting to port the Fusix operating system to the ESP8266. I have got the kernel up and running up to the point where it fails because it can't mount the file system. I spent the last session trying to get the ESP8266 board talking to SD card, because I want to use the SD card for storage, and failed miserably. Something is going on which I don't really understand, I am trying to get assistance on that. But in the meantime, I cannot proceed until I have a root file system of some description. So let's see if we can make the internal flash on the board act as our file system. Now, this is actually rather more interesting by which I mean complicated than it seems. The internal flash of the board is actually a raw flash device, and raw flash devices are awkward to deal with. Normally, this is all hidden from you by a microcontroller that lives inside a USB stick or SD card or similar. But we don't have one, so I am going to have to do it manually. So, let me first explain how flash works. So let's make a notes document. Normally, you have a set of physical blocks that live on a storage device. If you want to access data in block 2, you go look at block 2 of the device. Very simple. Flash is similar, however, it's limited while you can change one bit to zero anytime you like. You cannot change zero bits to one. Instead, you have to erase multiple blocks at the same time. So let's put block number down the side. On this device, each block is 4K, so let's say this contains like so. Reading is easy. If we want to read sector F, we simply divide the sector number by 4 to determine that it is in block 1 and read accordingly. But say we want to write. Well, the only way of doing that is a simple way is to copy the entire contents of block into temp storage. Then we erase the entire block and then we copy from the temp storage back into flash with the modified data. This isn't so great for two reasons. One, we have to be able to store an entire block in memory. On this device, a block is 4K and we don't really have that much spare RAM. But the other and more serious reason is that each block can be erased a certain number of times, typically about 10,000. So by writing that block, we've reduced the lifespan of that block by one. And say we want to write again to the same sector, now we've reduced it again. And the thing about file systems is that things like the directories get written to a lot. So just naively slapping a simple file system down onto the raw flash is both awkward because we need lots of temp storage in RAM. And also it's guaranteed to kill the flash very quickly. So what we do instead is we implement a ware levelling system so that each time you write to the flash, it actually goes for different physical location. So rather than writing again and again and again to the same block, we end up writing four times to four different blocks, which means we have only used up one available erasure across the board. The normal way of doing this is you have a lookup table between the logical sector, which in this case are letters, and the physical block where it lives. So that we can say that ABCD lives in 00, EFGH lives in 01, IJKL lives in 02, MNOP lives in 03. And I'm actually going to add a fourth one, which is empty. So one potential strategy for doing this is say we want to write, we want to replace sector F with the value X. Well, we write the updated, let me do that properly, the updated data to block four. We update our mapping and we erase the old block so that we can write something else to it. This works fine. So the next time we want to write, say we want to update MNOP, we write it to our spare, our empty block. We erase that and we update our mapping. The problem here is that we cannot make sense of the contents of the flash without the mapping table. Therefore, we have to store the mapping table somewhere in the flash. And of course, we can just store it in one of the blocks. We can have a directory block called like, you know, FF, which contains the mapping. But this means that every time we change the mapping, we have to update block FF. And of course, that will use up the lifespan of that block rapidly. So there's two ways to do this. Either the mapping table can move throughout the flash so that we never keep writing to the same block. Or we can spread it out throughout the entire flash. And that is the one we're going to be using. So let me come up with a slightly more concrete example. Our sectors in physics as terms are 512 bytes, which means that we get eight sectors per flash block. So let me try and come up with a decent way to represent this. So D, E, F, E, H, J, K, L, N, N, O, P. And we're going to leave the last one blank. We are going to be reserving one of these sectors for the actual metadata. So we can just eliminate the left-hand column entirely. This means that we're actually only storing seven sectors worth of data per block. As we only need about a few bytes of metadata per block, this is actually wasting an eighth of the storage space. But that's fine. We have lots of flash. And actually let's just put in the left-hand column here 512 bytes for the metadata. The metadata describes the logical block that is in this physical block. In fact, we can do that like this. So what this is doing is saying that physical block 0 contains logical block 0. Physical block 1 contains logical block 1, etc. Physical block 3 is blank. So let's say we want to update a value in logical block 0. We look up our mapping. We copy the entire block with our updated data into an empty block. We update the logical block of the new location of block 0. And we mark this block as being obsolete. In fact, sorry, no, we don't mark it as being obsolete. We erase it. So the next time we want to update something, like say we want to update logical block 1, we copy it into the empty block. We update the mapping and we erase the old block. We can see copy from one block to another without needing a complete temporary buffer. We basically just have to load a byte at a time. We're going to do it 512 bytes at a time because we will need a buffer that big, but this is much cheaper. Now, there is still a problem here. Let's say we update logical block 1 again. So that goes into our old block. We update the mapping. We erase the old one. And now we update it again. We copy it into the spare. We erase the old one. We update it again. You see that it is, in fact, alternating between the same two physical blocks. If you keep writing to the same sector of the file system, then it will not distribute the wear across the entire flash system. It will concentrate it in these two physical blocks and kill them off very quickly. So in fact, we are going to have to use a rather different approach. Let me just reset the state. And the last one is empty. The way we're going to do this is we keep the round robin pointer to a block. You're going to start at 1 for reasons. Now, we want to write the logical block 0. The first thing we do is we evict the contents of the round robin block. We just copy that into the spare. We erase the old one. We then copy our updated data into it. Erase the old and advance the round robin pointer. So we have in fact done two erasers here. We wish to update this again. So we do exactly the same thing. We copy the block pointer to by the round robin pointer to our empty slot. We erase the old one. We then copy our new data, erase the old location and advance the round robin pointer. So you can see how this is going. It ensures that every time we write a block, it will go into the next block along. We will end up cycling through the entire flash file system. The empty spare will move randomly around depending where the last block that we updated was. But the data will always be written to the contents of the round robin block. Unfortunately, it does involve two erasers per write, but it will allow us to cycle through the entire flash. This will distribute the load across the entire device. So let's do a few calculations to actually figure out how long this is going to last. Now, say that we are doing one write per second. This means that we are doing two erasers per second. We know that because of the way things work, the erasers will be distributed evenly. Now, I have an 8MB flash device. It turns out that I have a 4MB flash device. I'm sure it said 8 the last time I looked. I wonder if that might be unreliable. I have a 4MB flash device of which we are using 1MB for our code. That is actually going to be 3MB worth of physical blocks. Each physical block is 4K. That gives us 768 physical blocks. Each physical block can actually store 7 sectors worth of data because of the overhead in the logical block number. That means that we can store 2.5MB more or less. This system is absolutely fine. We can put a decent root file system on that. 768 physical blocks at 2 erasers per second means that it is going to take 384 seconds to erase every block. This flash device can cope with about 10,000 erasers. The total lifespan of the system is going to be 384 by 10,000 seconds, which is 64,000 minutes, 2,700-ish hours, days. That number of seconds, that many minutes, that many hours, that many days at one write per second, and then we will run out of flash lifespan and it will fail. Unfortunately, this means we need a new device because the flash is not replaceable. But that's one write per second and we're actually going to be doing far fewer than that. How many writes per second do we need to last a year? Well, the total number of writes we can support are, well, that number. So that many seconds in a year divided by the number of total writes is one write every eight seconds. Now, we are probably not going to achieve anything like that. Okay, if I sound surprised, it's because I did do this calculation ahead of time and got a different number. I think I may have forgotten to divide by 60 again that time. Let me double check things. 768 physical blocks, two writes per every update is 384, 10,000 writes per block means, yep, that seems like the total number of writes we can make. We're very unlikely to write eight times a second, but it is going to be extremely bursty. If you copy to a file, then it will write multiple sectors one after another. It is possible to optimize things by detecting that certain sectors do not contain data. So that, for example, if this was like so, but we know that DEF and G are empty, then if you write to this sector, we can just write it. We don't need to fiddle about with the mapping. However, in order to do that, we need information about which sectors of the file system are actually in use. And I do not believe Fusix has that. This is called trim support and I do not see anything. We could always add it, I suppose. I think that we're just going to go with this with the simple approach it will do for now. Can we keep a count of how many writes have been done? Probably not, because again, we need somewhere to put it. Actually, we can put it somewhere. We can put that in the metadata. This will work for now. We can keep track of the number of writes per logical block. That will give us an idea of the overall load on the system. That means that we can detect if we're writing too often and stop before the device actually gets destroyed. The root file system is actually going to be small enough that if you're going to do real work, you'll want to use an external SD card anyway. So, yeah, that all sounds fine. So, that sounds like a plan for our translation layer. Then we can just put an ordinary Fusix file system into the logical sectors. And it won't care that it's living on a flash device. The translation layer will seamlessly handle things. There are two things we need to do. Firstly, when the system boots up, we have to go and read every block on the file system and determine which logical block they correspond to. We will end up with a mapping in memory of a logical block to a physical block. This is going to be that way around. The mapping will need two bytes per block on the file system. We have 768 physical blocks, which will therefore correspond to one and a half kilobytes. That should be fine. We've got enough space for that and we can make more space if we need it. Now, the other thing we need to do is special for the ESP8266 and I will deal with that later. So, I have disabled the SD card stuff that I did last time, but I haven't actually committed it. So, let me just make sure that there aren't any stray files there do not seem to be. So, I don't recall adding that, apparently I did. Okay, so let's create a file for our flash translation layer and let's add that to the build. Okay, actually we don't want that, but we do want a header file describing what's in it. So, our FDL system is going to have as major entry points. So, this will read this sector 0 to 6 out of this logical block into the buffer. This will write in exactly the same way. We are going to have the mapping table, so we're going to define some constants to describe how big things are. So, this is the number of blocks in the file system. We have a four megabyte device, one megabyte is code. So, that will be, and we need one more entry point, which is the initialization routine. So, what the initialization routine does is it will walk through the physical blocks, looking at the metadata header of each one and it will populate this. So, the first thing this will want to do is to, every logical block must map to a physical block. But we also want one spare and we want our round robin pointer. So, that's actually going to be one smaller because we need the extra spare. So, for every block we want to read the header. Okay, I need to go and define some external entry points. These routines will actually access the underlying raw flash device. We will also need a buffer because we're going to want this to move things around with. The kernels already got buffers, but I don't think we can use any of them. The buffers are used for caching file data. And if we were to ask the kernel for a buffer and there weren't any buffers available, it would have to evict one and by evicting one it might have to write it to disk, which means it would need to call into the FDL code and that's never going to work. So, I think we actually have to define our own. So, what we're going to do is read the first sector of every block, which is sector 0, into the buffer. We read the...actually we can do better than that. These values need to be aligned. Actually, the problem is that because this is an array of U and 8, it may not be correctly aligned. We're going to have to do things a bit more complicatedly. So, that will read in the first sector. Sector 0 is of course not the... So, we've got 8 sectors per block. The sector numbering... Yeah, I think I want the raw flash to use sector numbers from 0 to 7, where 0 is the metadata sector. For these, these are actually going to be the logical sector, which is going to be from 0 to 6. Do I want to do it that way around? Or do I? No, I don't, because that will just lead to complications. So, minus 1 will read the metadata sector. We now have the logical sector number in there. So, if it is empty, then set the spare block to this. There should only be one of these. Otherwise, then we set up the mapping. And just for the purposes of being robust, we are going to ensure that we have a complete spanning set. That is, the file system contains mappings for every logical block. So, that should be all the code we need to initialize the FTL stuff. And it does seem to build. We haven't done the raw flash stuff, so it won't actually link. So, read a 512-byte logical sector. Now, this should be straightforward. We simply map to the appropriate physical block and read. This is why I wanted the sector numbers to be compatible. The idea is that the block device layer, which we haven't written yet, will do the division by 7 and ask for sector 0 to 6. OK, writing is going to be more complex. So, the first thing we're going to have to do is to evict the block being pointed to by the round robin pointer. Now, there are three potential cases here. Our round robin pointer can be pointing at a block that is in use. It can be pointing to the spare, or it can be pointing to the block that we are currently trying to write to. Now, we've already covered the logic of if it's pointing to an unused block. If it is pointing to, let's say we're writing to logical block 2, this one. If it is pointing to the spare, then we don't bother with the eviction, because we don't need it. If it is pointing to the current block, then we actually follow the same code path as we did for if it was pointing to the spare. We copy it to the spare and we erase the old one. So, we only need to do the eviction if it is not pointing at either the spare or the block that we are trying to write to. So, we look up our mapping to figure out what the physical block is. If the physical block is the spare or the round robin block, then we wish to copy the updated block into the spare. So, for every sector, if it's not the one we want to update, we want to copy from the current physical block to the spare physical block. Unless it's the, let's do that the other way around, unless it's the one that actually contains the data that we want to write to, at which point we do a raw flash write to spare physical block, sector, sector, and that data. So, that will copy our updated data. We then wish to, why did I write logical here that should be physical? We then want to erase the old, hang on a second, hang on a second, the physical block here cannot be the spare because the spare stores no data. So, this is the only case that matters. We are actually caring about our round robin block, aren't we? Our three cases are, if the round robin block is pointing at the block that we are updating or if the round robin block is pointing at the spare, there is only one spare, or if the round robin block is pointing at something else entirely. Okay, so if this is pointing at the spare or it's pointing at the sector that we are, the block that we are actually updating, then copy from the current block to the spare, erase the current block, the spare becomes the old physical block. We also need to update the metadata, which we do with that, and that should be done. So, if the case where the round robin block is actually pointing at something else, then we wish to evict the round robin block into the spare and then follow the same logic actually. Okay, so, if the, let's invert this conditional, if the round robin block is not the spare physical block and the round robin block is not the actual physical block, then we want to evict the contents of the round robin block, we move that into the spare. So, from the round robin block into the spare, that sector, and we want to copy the metadata as well, so we do that. We then erase the round robin block and the spare becomes the round robin block. Okay, and then we follow this logic, we can do this to copy our actual data from the physical block, which we know cannot be the round robin block. Well, we don't care actually. We copy data from the current block to the spare, we erase the old physical block, the spare becomes the old physical block, and we want to increment the round robin block. Okay, so we have an undefined reference to mod si3, that's in the ROM, which I forgot to pull up here in Github E268. Ah, that's why I couldn't find it. Mod si3, this is signed modulus, it's not there intriguing. Well, there's actually a simpler way around that, which is to just use unsigns for these, which we probably wanted to do anyway for those, but we don't want to do for these because we need the negative number. We don't want to do these because we need that, but it should work for... The round robin block is an integer. Right, now we need to actually implement our raw flash. These are the routines that actually talk to the real flash device. Add that to the build list. We've got raw flash arrays with 32 physical, write and read. Now, this is undefined reference to... Oh yeah, I haven't done that. Copy physical sector is a simple helper. All this does is read from source and write to the destination. Very simple. Now, the problem with touching the internal flash is that we are running code from it, and we cannot access it while we are running code. So what we have to do is we need to disable the flash memory mapping, do the work, then re-enable it again afterwards. That's relatively straightforward. The code to do it is hash, read, disable, turn off the flash. Afterwards, we are going to turn it back on again, and then here we need to call SPI... Hang on, just look at some code that someone else wrote that does this. Hang on, I want to look at this bit of code. SPI Unlock, SPI Arrays. We don't want to erase a block. So I've been talking about blocks and sectors, where a block is 4k and a sector is 512k, because that's what it looks like from inside physics. However, in actual flash terminology, a sector is a small erasable unit, which is what I'm calling a 4k block, and the block is a large erasable unit, which is 64k. So we actually want to erase, and we want to start at 1 megabyte. So that's 1024 by 1034 divided by 496, 256 4k sectors per megabyte. And there is actually one more bit of work we need to do, because we need to do the equivalent of this, or actually hierarchy flags, because this is what our code will look roughly like. And we actually need to add SPI Unlock is at 4878. SPI Arrays Sector is at 4800. We're also going to want SPI Write, which is at 484c. And SPI Read, which is at 4b1c. OK, so that is actually linked into a kernel. Unfortunately, this is where things start getting complicated. We can't put this routine in the flash, because as soon as we call cache read disable, this routine will become inaccessible. We have to put it into our precious instruction RAM. So let us see how big this is. Here is our arrays routine. Not particularly large. C64, C18, 76 bytes. Yeah, that's OK, I suppose. We can't use DI, because DI is in the main kernel, so we will actually want to inline assembly that. Or will we? Because it occurs to me that this would be a good opportunity to... It would have been a good opportunity to turn DI, IRQ Restoring, EI into macros, which can be inlined. Yeah, we can do that, but we have to modify the kernel a bit. There's our config. OK, so we should be seeing... Yep, these are not defined anywhere. So we are going to define macros for turning interrupt on and off in the CPU.h file. And we're going to steal these from the Arduino. Here are the modifications. OK, so this is the core of it, which is a little chunk of inline assembly users. This is dumb, I will do it like this. No, I can't do it like that. It has to be a macro. So what this does is, when you call the macro, it will inline an arsenal instruction, which turns interrupt on or off and returns the old state. So I can do a DI, that's in low level. A DI sets the interrupt status to 15 and returns the value. A DI sets the state to zero and returns the value. Now, IRQ restore is a bit more complicated because we need to write to the PS register. There will be a macro for doing this. WSRPS is the one we want. Yep, so this should inline everything we need, assuming it's done right. Let me just do a clean and build. OK, what's wrong here? EPS. Oh, we never defined Stringify. Stringify is a nasty macro, which turns its parameter into a constant string. It works due to weird quirks in the way the C macro preprocessor operates, which are weird. OK, that looked better. Don't see any warnings much. So let's take a look at our disassembly. So the advantage here is that this is going to actually make the code throughout smaller. And more importantly, calling DI is no longer calling out to anything. So you can see here, this has turned into a single instruction. We can actually probably hand tool these to make them smaller, but I think this will do. And here is our IRQ restore, which writes the value back to PS. So how big is our routine now? 65 bytes. OK, that'll do, I suppose. I do want to go here to the low level routine and just chop these out because you don't need them anymore. And we want to go back to raw flash and do the other routines, which are going to look extremely similar. So rather than array sector, we're going to be calling SPI right, which takes a byte offset to the address. So that is block number. So that gives us the address of the block plus the sector offset, the SPI flash buffer and count. I'm going to try to put some prototypes for these in because you don't get them by default. Buffer and the length. And it's exactly the same thing down here. Actually, OK, we don't get that there. Do we get it in the Arduino source? No. That makes me a little bit suspicious. Can I actually use these in ordinary code? I believe so. Don't see any reason why not. It's not doing anything particularly complicated. For two AC here is actually doing the work. Well, let's try it and see. So SPI read address buffer. I should really create a header for all these, which I should actually do now. These were all painstakingly reverse engineered so they don't really have standard APIs. Where was I as in boot.c or flash? We need SPI unlock and erase sector. Oh, and put these in here as well. SPI unlock. I don't know what that does. This sector, which takes a sector number. This is a 4K sector number. OK, we'll actually go look up SPI unlock in the disassembly. See if there are any useful comments. No. All right. So let's take a look at the disassembly. OK, so raw flash arrays, raw flash write, which there is kind of an annoying lot of raw flash read. We cannot put these here. We've got to put them in instruction RAM. So we're going to have to shave a bit off the top of our user code to put it in. And we're looking for this. No, we don't. So we already have a special block here for the code that gets loaded by the bootloader, the stuff that goes here in boot.c, which initializes the SPI flash. So we're just going to duplicate this, but put it into the other end of things. This goes into IRAM, nought, nought, SEG, which goes into... I need to understand these stupid P headers. I think we have zero and one. So bootloader code goes there. No, hang on. Yes, bootloader code goes here. IRAM-10 P header. See what IRAM-00. While this goes into IRAM-1. OK, let's see what this does when we try and compile it. Region IRAM-10 overflowed by 27 bytes. It needs to be bigger. OK. All right, now let's look at our disassembly. I need a new mouse. The left mouse button doesn't stay down all the time. So here is our FTL write routine, which is doing its work. Copy physical sector, raw flash read. Notice the address is here, 4021, indicating that it's in the SPI flash. Raw flash read is at 4010 FEBC. That's in the instruction RAM. That is correct. So this code's... This block of code starts at FE00 and goes on to the end. Now, I believe that the bootloader will initialize this for us. That is copy it out of the flash. But we're going to check that here in main, we're going to just do 40100 FE00. I believe that's the address. 4010 FE00. OK. ESP tool, which takes the L file and generates the stuff that we need to burn, appears to be happy. Let me just see how many output files it's made. Still two, which is good. And we're going to say... We're going to do that. And we also don't need this anymore. OK. So let's burn it and see what it does. We'll see what it says. Looked OK. I do wonder what is in this file. So this is what gets actually burned into the flash at zero. The header at the top. Then it's got the code, which lives here-ish. I'm a bit surprised by these strings. Why are they there? Oh, it's this bit of code here. I'm still surprised by the strings, which shouldn't be there. OK. Well, let's see what it does. It's the reset button. %p equals 0000. That seems wrong. Let's take a look at that disassembly. Yeah, it should be this. It's part of the literal table for the code here. So what this means is that the ESPTool command, I believe, has ignored that particular section and hasn't written it to the right place. Which is annoying. I wonder if I can get this to produce any additional tracing. Hmm. It seems not. But I am still curious about the contents of this. So let's take another look at- We are looking at the disassembly. So the actual information about what to write will appear after the headers. So if we search for 4081FF3F, we find that here indicating that this in the file is where this piece of code starts. So let's go down to this FOOD. There's not a lot of this, to be honest. O-D-F-O. Yeah, here we go. 1-2-C-1-1-0. This is disinstruction. F-O-O-D. Although this is actually a very common suffix in LX-106 code, so it will appear all over the place. Okay, so let's look for this. F-O-4-7-0040. It's here. 7-8-4-8-0-4-0. Yep, that actually looks like this bit of code. So that does kind of look like it's... This IRAM 1.txt is in the binary. I wonder, is this wrong? Let's try that and hit the reset button. This one's knocking stuff off the table. Those are all zeros. Has it not actually loaded it into the right place? Wait a minute, wait a minute. Let's actually try looking... You know, put it in the right place in memory. Give that another go. Okay, we've got stuff now. I'm going to update this assembly. Hit reset. And what are we seeing? 8-1-4-0. Wait, these are 16-bit values. That doesn't look right. Normally I like it when a mistake turns out to be me doing something really stupid, because that turns out to be very easy to fix. But it'd be nice if it happened less often. Okay, flash. Hit the reset button. What are we seeing? F-47-F-0. 47-F-0. 4-8-7-8. Okay, this is in the right place. For some reason, the Fusix K-print F is only printing 32-bit hex numbers. This is very much intended for 16-bit systems. But this does seem to imply that the code is now in the right place, which is good. We can call it now. So, we are going to just stick this in and see what happens. I mean, we haven't actually written a file system there, so it will probably go horribly wrong. It did something. And it failed. Let's try that again. Okay, this is actually now calling FTL in it. It's just taking a while. So, where is our FTL code? I'm just going to... As the contents of the buffer is going to be garbage, I'm not going to actually populate the map, because if this turns out to be out of bounds, it will screw all over memory. And I'm just going to do this, because we should be able to see it read stuff. Okay, that looks good. It ain't the quickest. Yeah, okay. So, that's read everything, has hit the end, spotted that it hasn't actually read anything, and has given up. Good. We do actually appear to have... We have successfully read stuff from the flash, and the fact that we've got different numbers here indicates it might be working. Our FTL logic looks more or less plausible. The fact that there's this stuff at the end suggests that there's data at the end of the flash that we might not want to overwrite. So, I think we should be good. All right, so let's have another look at the SP tool. What have we got here? We've got arrays... Ah, right. We can... We should be able to just say this... Okay, we can't raise everything. But I can do... Raise region from 5 megabytes for 3 megabytes. Doesn't support arrays region, really? How am I supposed to erase stuff? Let's assume that I just have to write stuff. So, I believe the next thing for me to do is to create a file system and write it. I think for the purposes of testing, let's make a 1 megabyte flash file system. And of course, we can only use 7 eighths of that. So, that's 256 physical blocks. 3.5k per physical block gives us 896k. So, 96k... Right, so this is our logical file system, which is currently empty. We're going to create a partition. This is going to be for the swap. We've got 16... I think 16 processors of 80k each. Let me actually look at my real calculator. Let me see. 64k of user plus 32k of code is 96k per process, which is 144k, 1.4 megabytes, which is bigger than our partition. So, let us just actually just make one big partition for now. There we go. And write it. And make a physics file system using one of the physics tools, which should be in standalone. Yeah, we've got mucfs. Standalone... mucfs... I don't actually know what numbers to give it, to be honest. Just looking to see if I can see any references. 16.512, that looks like a number. And in fact, this is going to erase the partition table I just put on. Yeah, that's fine. Okay, here is our file system. So let's write that to the flash. Address 1 megabyte file system.image. And this will probably take a while. Wow, that's slow. That's a long while. Okay, well, let's go up to our config. We wish to change the boot device here from partition two to the whole device, which is just a zero, because we're not using the partition table anymore. And clean, build. And next we want to... We don't want to call FTL in it from there. We actually want to create a block device that will represent our flash partition. So we're just going to copy the SD card code. Okay, take a lot of this stuff out. Oh yes, and we also want the initialization code, which is in the SD discard, which is this stuff. Again, most of which we're going to lose. So no drive. What do we actually need to do here? There's stuff here for getting the capacity. Okay, everything from here up goes. So this creates a block device. Give up if there's nothing there. The transfer function actually does the work. The flush command is used for calling sync. Don't know what driver data is. Okay, that's finally finished. This doesn't look like the code here. I think I copied the wrong thing, possibly the SCSI. Anyway, this at least tells us what this is. I think driver data is private for the block device. So we don't need that at all. We do need to give it a number for the number of logical blocks, which I believe are one kilobyte. It actually occurs to me that I'm doing things. In the FTL there, I'm doing things with 512 byte sectors, but I think I can probably use LBA blocks. I don't know how big they are. Anyway, this gives us the number of blocks on the device and scan looks for partitions. So that is all we need. Yeah, I copied the SCSI one. Silly me. So that's all we need to add the block device. We now need to add the transfer function. So what this does is it looks at various global variables. To determine what's going to do, that is to read or write a block. And it does it. It's not exactly complicated. So for what we are doing, we want to decide whether we're reading or writing. If we're reading, we are going to do FTL read. We need to get the logical block number. So if a byte address card, shift LBA to convert to byte address, 9. Right, okay. They're 512 byte blocks. So to get the logical block number, we want to divide by 7. To get the sector, we want to modulus by 7. So when reading, we are going to read logical sector. And the destination address is, where is the destination address? The destination address is in a different file. Do we have something simpler I can look at? Let's try IDE. Okay, here's the transfer function. Dev IDE read data. Also, I've just remembered that I wrote the logical file system onto the flash, which is just wrong. We need to turn it into a... We need to add the metadata. So I'm going to have to write a little program to do that. Is this... All right, block op address. Like so. If it's not a read, it's a write logical sector block op dot adder. And no error handling. So we just do that. For flush, uint, fast, ht, flush, cb, there's a void. Does... does nothing. Okay, what's this not liking? Ah, yes. I did not prototype my... I did not prototype my FTL functions. So that's, let's see, an int FTL init. Next, an int FTL read. Uint 82 t logical, int sector, uint 8 t buffer. FTL write const. Really, FTL init should return the number of blocks in the file system. Yes, actually, I'll make it return the number of blocks in the file system, which is going to be in the logical file system, the number of LBA logical sectors in the logical file system. So that's going to be flash blocks minus one, because we're not counting the spare times seven. There's a flash. So here in the initialization code, we want to do... that's entirely the wrong place. If std discard.c drive a data we're going to ignore, drive LBA count is the one we want. So that's going to be LBA divided by two gives us the size in kilobytes. This goes drive LBA count equals LBA. Okay, it doesn't like transfer CB. Transfer CB doesn't like... All right, this wants to be int. Okay, now what are we missing? We are missing a working file system. So let's create a tool to generate a working file system. This is not a kernel file. Okay, right, the make file is set up to build this for us. Okay, so input file. Okay, so first for each seven sector block, we need to write out the metadata. If we do that, we'll get a zero initialized array. So the metadata is going to be zero for the arrays count, but you haven't actually done anything with yet. And the block number goes in the next two bytes. In little in the order. So, okay. And now we write it. Now we read in the seven sectors and write them out again, as simple as you can possibly get. So that's from ESP. Look FTL filesystem.image to filesystem.ftl. Filesystem.ftl is one megabyte, which is exactly the number we wanted. We've actually forgotten one important bit. Anyway, here is FTL block zero. So we've got seven sectors. And then here we start the next block. As you can see, it is physical block one. Here is physical block two, et cetera, down to the end. Right, we actually need to do one extra bit, which is we need to write out one physical block worth of FFs. This will become the spare block. So down the bottom of the file, we have... Right, that is actually too big, because I created the filesystem the wrong size. So this did not include the spare. So that wants to be 892k. So we can recreate the filesystem, turn it into FTL. Okay, so here is our spare block right at the end. We want to write the FTL file, which will take a while. So let's go take a look at our FTL layer. We want to enable this. Let's put some tracing in... Okay, and we want some tracing here. So a printf read. Let's put that here. Okay, so we should now see a line of tracing or possibly even put semi-colon, a backslash there, where it's supposed to be for every read and write and also when the flash initializes. Okay, that's written. Now we burn it and run it and see what happens. Okay, it's reading the correct mapping. Spare is at FF. It has not done anything because I have not initialized it. We actually want to call flashdev init. Hmm, never got around to adding a global header file for the platform. It's here and in here. We don't want that anymore. We don't want that anymore. And that goes in globals. It really belongs in the kernel proper, but it doesn't seem to be there. And in main, instead of calling FTL init, we actually want to call devflash init flashdev init. Okay, we don't need to burn the file system. We just need to update the code. So there is our mapping. Wait, what's it done? It has... Okay, well, it successfully calculated the size of the file system at 892k. It has tried to read logical sector 0, which is in a logical block 0, sector 0, which is in physical block 0 and has failed. I suspect I am returning the wrong value from my transfer function. Yeah, if success return 1, otherwise we set an error. Let's see what this does. It still did not work. Do I need to do something with flush? Flush appears to not be being set by the SD stuff. So, okay, well, let's just not bother with that then. I do need to tell it the number of... I do need to tell MukFS how big the file system is, which is a bit annoying. And the block size... Do I need a partition table? It's possible I'm using the wrong boot device. So that was documented in start.c here. No, no, I am using the right device. 0000 is the entire HDA. Do I need to... In here, do I need to set something to tell it how many bytes have been read? It should just return the entire block. Do not see anything. No, that looks okay. Okay, well, let's take a look at dev nbr. Yeah, so this is the stuff that's parsing a... parsing the partition table. So I do not recall it saying that. So we can optimize that, actually. Okay, it's a said HDA. It's got here. It knows there's a block device. It's now trying to read the partition table, which is not there. So where is it actually? Oh, yeah, here it is. It's reading. Okay, I think that's actually working, to be honest. I think it is looking for partition table. Fail to find it. And it's proceeding to the root file system mount bit. It says it's tried to mount root dev zero read only. It has failed. Okay, so the mount code will be here in start. Here we go. Mount root file system. F mount has failed. This is our root device. So let's go look for F mount versus .c. So this will tell us how far it's getting down through the mount process. So let's see what this says. There's an optimization we can do in our raw flash code. If the sector is minus one, meaning it's accessing the metadata sector, we don't need to read all 512 bytes. So we can say, we can do that. Okay, so it got to here. It got to here. It then failed to read the buffer with the super block in it. It also looks like it didn't even bother trying. So why was that? Where is be read? And I think what I'm actually looking for is, hang on, be find, be find, be find looks for a particular block in the cache. Physics has a fairly small cache of red buffers. So if you try to access a buffer that's already there, then it doesn't bother hitting the disk. BD read, maybe. 97, 99, BD read failed. So BD read is, I believe, the low-level thing. All right, this calls the block device layer, which is here, which eventually hits here. The fact that it hasn't, we haven't seen any tracing from the flash device, so it hasn't even bothered to try and call it, which suggests that it thinks something's wrong and isn't actually doing it. Okay, so let's stick our tracing lines in here. Is our drive LBA count wrong? Does it think the device isn't big enough? Well, there's one way to find out. This mount process isn't much faster. Ooh, no tracing. No tracing at all. That means it's not getting this far. So let's remove this, in case it's the printing to the serial terminal does take time. So if it's not hitting block dev read, then something here is failing, but there isn't anything to fail. Okay, it's got there. Let's also take that debug tracing out of main, I think. So, oh. Remember what I said about stupid problems? I disabled config ID, which means there is, in fact, no block device there. So we want this. Okay, let's give this a try. Scanning flash. Fantastic. So we see all the tracing I put in. Mounting root file system. It has successfully mounted it. It says, okay, it has now tried to find the init executable and has failed with this line, panic-exec-ve, because it has hit here. It is actually trying to load a process of disk, load a binary of disk and run it. That's great. It means that we are actually genuinely getting somewhere, which after yesterday was a bit annoying. And I thought I might not be. So that is, I think I also need to, let's go into our, actually, I'm going to leave the tracing in the FTL there because I want it. Did I get that right? Yeah, it's apparently reading and writing one, reading and writing six bytes is not really much faster than the sector. Let's try that again without the tracing. Yep, okay. So this line, is it looking at the partition table? This line is it looking at the file system super block. The file system is currently empty, which is why, you know, there's nothing there. All right, that is some good progress for the day. I will commit to this. I can't call it an FTL layer. It's just not allowed. That FTL file can actually theoretically be moved into the dev directory as a generic FTL system for any other platforms that need it. That's why I did it in its own file. It would be nice to be a bit smarter. I mean, it is wasting one eighth of the flash and it also wants to be more configurable. I mean, you saw how long it took to scan the flash on startup, but that's just for a one megabyte partition. Imagine what it would be like bigger than that. Yeah, there are other ways to do this. You can stash the map in a bare flash block. That means it will have to move around whenever you update it, just like everything else. This will involve three erasers per operation, unfortunately. You can delay writing back the map until, you know, it's a little bit of time has passed or somebody calls flush. That's what flush is for. But you're not really terribly happy with any of that. This is simple and works. So I'm going to call it a day. Hope you enjoyed this video. Please let me know what you think in the comments.