 I'd like you to welcome Peter Chubb here, who's going to be a talk on SD cards. Good day, folks. This story started about two years ago when we got some Samsung TVs to stick on the kitchen wall to display stuff in our CI system so we could see what the bug rate was, who was winning in terms of the formal methods team and the kernel team as to who's got the most bugs and whether they've actually caught up proving all the code that we've done. Additionally, when we put up the Samsung TVs, the browser in those wasn't adequate. It didn't talk HTML5. So I happened to have one of these things sitting on my desk. We bought the Raspberry Pi some time before to try out for a different project and that project didn't actually get anywhere. So I had a couple on my desk. So I plugged one of these things in. The format was appalling. And I finally traced that to the SD card. And that's where this started. It turns out that there are a lot of embedded systems that I'm interested in where an SD card is either the only or the main storage device. They don't have SATA, they don't have embedded flash. All I've got is an SD card. And that means it's vitally important to get things right. And if you just buy the cheapest SD card from the local post office and take a Leonardo image and DD it on, you're not going to get anywhere. As I said, performance sucks. My first thought was, well, buy a faster card. Maybe we can get something like a, you know, a SanDisk Extreme Pro with GoFaster Stripes and my batteries died. And you might be able to get that 95 megabytes per second that they specify. You will certainly pay enough for the extra packaging. So I spent lots of dollars. Fortunately, my employer's dollars are not mine. So sometimes I remember I'm an engineer when I'm not doing workplace health and safety inspections. So the first thing you do when you're an engineer is you pull things apart. Here are two SD cards that I pulled apart. The left-hand one was a fake Lenx's chip SD card. It was purporting to be 16 gig. When I pulled it apart to find out why, it's got an 8 gig flash chip on it. Fortunately, they haven't filed off the number. You can't actually see it in the picture. But it's a Samsung standard flash chip. You can get the data sheet off the network and you can see how fast it's meant to go. We weren't making a tenth that speed. The one on the right is a SanDisk. They've layered everything inside layers of PC board and you can't really get at it. So I couldn't pull that one apart, which is annoying. I thought, okay, if I can get the flash chip off, I can desolder that, stick it in a flash reader, and maybe I can get the firmware out and have a look at it. And I thought about that and I thought about that and didn't quite get round to it because it's a bit more much work and my eyes are getting older and I'm not very good at desoldering anymore. But fortunately, in December last year, Bunny Huang did exactly that. And you can get his data there. His code was written for the Novena, so you can plug an SD card of the right sort into a Novena and get the firmware off and do whatever you want with it and even put your own firmware back on. And that's really nice and it opens it up and I'll come back to that later because what I wanted to do in the meantime was make my SD cards faster. To be able to do that, you really need to understand what flash is like inside. This is in all the flash data sheets so I'm going to go through it fairly fast. Flash memory, and I'm only talking about NAND flash now. There's another sort of flash called NOR flash. If you want to know about that, look it up on Wikipedia. NAND flash is divided into pages. Internally, those pages are accessed serially, which means that it looks a lot more like a disk than memory. The typical size of a page is somewhere between 512 bytes and 4K, although I've found some which have a page size up to 128K. Plus some ECC code. You have to have the ECC code there because flash is jolly unreliable. And most of the other things I'm going to be talking about are to try and fix up that unreliability. So your page size is maybe 2K plus some small number of bytes. How many did I say? 112 bytes. That 112 bytes is for ECC. Those pages, which you read and write at once, unfortunately, they're right once. If you want to write a new page, what you need to do is find an empty one, one that's been erased, and write there and mark the old one dead. You can't erase just a page at a time. You have to erase an entire block at a time, which is some number of pages. In this case, this particular one has a 4 meg plus ECC block, but I've seen them up to 8 meg and 16 meg even. And there's a few small ones that have got smaller ones as well. But typically, the class that you can buy off the shelf now have got a 4 meg erase block. What's more, inside that chip packaging, there can be more than one die. Or sometimes on the same die, you can have more than one plane. On a good SD card, you can read from one plane and write to the other without going off chip. And that becomes really nice for garbage collection, which I'll come to later. So this is what it looks like inside. Typical timings, and this is the timings for that Samsung flash chip I showed you just now, to read a, to write a page, that's a, take an erase page and write to it, is 200 microseconds. Typical. To read a page is about 3 microseconds. To transfer it to a form the SD card bus is about 3 nanoseconds per byte. So that's another 12 microseconds on top of the read time. And to erase a block is 3.5 milliseconds. So you really want to do that in the background. You don't want to do it while you're doing all this stuff. So what we do, what they do is inside that flash chip, they put an MCU. This one you can't get out unfortunately. There are about four or five things that MCUs got to do. The first thing is that flash is really, really cheap because they don't waste any silicon. They, when you manufacture silicon, there's almost always defects in it. So if they're manufacturing 32 gig flash chips, say, and 80% of those are bad, they can throw away all the bad blocks and market them as a 2 gig chip. And the MCU there gives you the illusion that it's a 2 gig chip with contiguous blocks in it. The MCU also has to run the state machine between the arrays and the transfers to and from the SD bus. So that's what that MCU has done. It's also got a van buffer because, again, this thing's got bitwise serial access to the various planes and you want to transfer it into bite-wise output. So you need to do some buffering. I don't know how big that van buffer is. One of the manuals, one of the data sheets I read suggested it was about two blocks worth, that's two arrays block size. Another one suggested it was about two pages worth, but they don't actually state it in the data sheets, which is a bit boring. So I've said that flash is unreliable. There are lots of different ways that flash is unreliable. First, you can get single bit read errors. They're fixed by the UCC. Lovely. Secondly, you can get bad blocks. Both ones that you get at the time that the chip is manufactured and ones that develop over time because the chip has a limited life. Typically 100,000 erase cycles before that erase block is dead. The other thing you've got to watch is redisturb. The way that the flash chip works is there's a floating gate in the FET that manages each cell and you transfer charge onto that by tunneling or some other quantum mechanism by raising voltages so that the tunneling is more likely to happen. Now it so happens that when you turn off everything else except for the cell that you want to read, you're also putting voltages on there. They're much lower voltages, but if you do it often enough, sometimes some electrons jump onto that floating gate. So if you're reading from this cell here, the cell next door might get an electron on the gate and start reading a zero instead of a one. That's called redisturb. You can fix that by erasing the block and starting again and you'll pick it up from the ECC being wrong, which means that your MCU you've got somehow to detect that, erase that block, sorry, copy the data doing the ECC correction so that you've got good data, copy it into a new page, erase that block and then start again. So you need another controller. This one is a block diagram from the Silicon Motion controller. The big manufacturers, Sandisk, Toshiba, Samsung and so forth, they manufacture their own controllers and it's totally proprietary and I've got no idea what's in there. Except that I've been told, and this is sort of unofficial, that some of the high-end Samsung ones, the ones that do Wi-Fi and Bluetooth as well, have actually got a high-end ARM chip-running Linux on it. But I've never actually seen one of those cards, so take that with a pinch of salt. What the SD card does is it's got a really, really slow, cheap MCU there, an 8051 typically. They cost maybe 17 cents per SD card. That kind of range. And because it's really, really slow and horrible, you don't want your data going through it, so it's got a completely separate data path down the bottom there. What the data path does is it's got the host controller that actually talks to the bus and that's quite complicated because there are three different modes of talking to the SD card bus. That talks to an SD interface which interprets the commands that you can send over that bus to decide what you're doing with the card. It's got a buffer. In this case, the buffer is 64K plus a bit for the ETC and other out-of-band data. And it's got a flash interface that talks whatever you need to do for the flash card, and there are two standards for flash chips. You just got one and everybody else has got a different one. So most of the third-party flash controllers you can buy will talk either, which is nice. It's also got a hardware ECC generator and check them in there because otherwise you couldn't do it fast enough. So that's the bottom part. Up the top, there's some program ROM. That's actually not ROM. It's usually Nor Flash to let you do executing in place, about 8051 cut things. Not only does it hold the program, it holds some metadata about where on the flash all your housekeeping data is. Because after all, we're moving stuff all around. We've got this great big flash chip there. We might as well use it to hold the mapping tables for where things go and where things are. And we've got a small amount of RAM. That RAM is not big enough to hold all the mapping tables for a large flash chip. So you typically hold those on flash and manage just bits of them in the RAM. So that's what the SD controller card looks like. What it's got to do is present the illusion of having a standard block device there. So it's got to do all the buffering, all the arrays, all the serial to parallel conversion. It's got to cope with a different block size because it presents a 512 byte block interface and the underlying flash could be 2K, 4K, 8K, whatever. It's got to give access to the different planes. It's got to do garbage collection in the background. It's got to do the erasing and the reallocation and the flash translation and the bad block management and blur. Good grief. Lots and lots of stuff. The problem is you go to the back of the SD card package and it's got all these lists of patents and the patent is meant to be an invention disclosure. So theoretically I should be able to go to a patent, read it and find out what it's actually doing. 90% of these ones that I looked up if they've got any documentation at all the US patent has got the claims which is what the people are wanting to protect against and then it says this patent includes in the entirety this other patent which is a Korean one written in Korean not just Korean but Korean patentees but not even my friend who speaks Korean could understand. So I don't know what they do. I'll just talk a little bit about where management because it's kind of interesting. There's two basic ways that you can do it if you're thinking sensibly and some of this has been documented in places like the IEEE conferences. What you want to do is you want to remap blocks when they begin to fail so when you start detecting the ECC going wrong you remap it and the other thing you want to do is spread all the rights across all your raised blocks so that you don't use any one of them excessively and start making it go bad. In practice you do both of those but with one caveat it turns out that for data longevity if you've got something which has been erased once at manufacturer time and write your data on there it will last longer on the flash chip than if you copy it around and erase it all the time. The data seems to last longer so if you want to have something that lasts 10 years you want it on a block that's only been erased one or two times. If you want to just keep it around then yeah. So for data that's going to last for a very long time like the flash program assuming that you've got an SD controller that has the bomb in it then you want to just keep it in one place. The other thing you want to do is in order to avoid that read disturb you count the number of reads and schedule a garbage collection after you've hit some read threshold that read threshold will depend on the chip because I think I've got one more there that's right yeah because manufacturers like ATEM and so forth tend to buy flash on the spot market they get whatever is cheapest and it could be Toshiba one day, Hynix the next Micron the next day after that which means they need to be able to reprogram those controllers so almost all of them have in-system programmability I think people should be fairly scared about that because any time you've got an MCU between you and your data that you don't control the program of it's possibility of a man in the middle attack and we'll come back to that later anyway a fat file system is what you normally get there fat file systems are actually a pretty good match to the standard usage of SD card because if you think about it, you stick it in your camera you fill it up with JPEGs transfer them somewhere else evade the card and start again so you're always writing on a clean card and a fat file system always writes in cluster size units and they're already 64K in these pre-formatted ones and you segment a line from the first partition which means your fat the fat table is in the first evade block and that's pretty nice and there's a conjecture that I have that some flash controls optimize for access to that that area because if you think about what you're doing you're writing to some evade block somewhere and you're writing contiguously and then at the end of the time you need to update the fat table use case will be really nice a fat file system looks something like this you've got a little bit of boot info a boot parameter at the beginning which has got info about where things are then you've got your file allocation table then you've got your root directory and then you've got cluster after cluster after cluster remember that graph because I'll come back to it I just wanted to mention interface speed a little bit you've got three modes of access to a typical card you can either access in a single line mode one bit of the time serial and fit up to about three bit per second or you can flip to a four bit wide system and talk across that or you can do double data rate and do it both on the lower bit sorry the rising edge and the falling edge of the clock which gives you up to 25 megabits per second there's also a standard UHS-1 and UHS-2 and UHS-3 but I haven't actually seen any of those cards which lets you get up to 104 megabytes per second at double data rate that's pretty nice you'll see these class numbers class 2 is 2 megabyte per second class 4 is 4 megabyte per second right class 6 is 6 megabyte per second UHS-1 is usually even higher but there's a caveat the standard says that the class 2, 4 and 6 give you that guaranteed right speed even when the disk is fragmented the flash is fragmented class 10 is only when it's unfragmented which means that for Linux use a class 4 or a class 6 card will usually give you better performance than a class 10 now the controller's got some van buffer the van buffer is smaller than the array size so a guy called Iron Bergman did a whole heap of work to characterize flashes and he discovered that the way that most of these flash controllers work is they keep some number of evades blocks that they're currently writing to and that number varies from 2 up to some fairly large number the size of that buffer seems to increase with the price of the card you can tell by reading something and then reading something adjacent to it after a little while and timing how long that second read takes if it takes a very short time it's probably in the buffer if it takes a long time then it's doing more reading off the flash because of that 12-microsecond delay Iron Bergman wrote a tool, flash bench which you can get off, get it's a really really nice tool all that's in the bin you can just out get and install it and you say flash bench dash A on a directly connected flash controller if you're using a USB adapter forget it this has got to be something that's directly attached otherwise you won't get the timings what it does is it reads a 2K block just before a power of 2 boundary a 2K block that's 1K before and 1K after a power of 2 boundary and a 2K block that's after a power of 2 boundary if you're crossing an arrays boundary then the one that comes before will be relatively cheap the one that comes after will be relatively cheap and the one that goes across is going to be relatively expensive because you've got to close one allocation unit and open another one this is for a Kingston card you'll notice the one of colour in red you've got 300 the diff at the end is the difference between the middle one and the average of the other two and it's 350 odds microseconds until you hit 4 meg boundary where it's 840 so where our conjecture is that the array size for this flash card is 4 meg unfortunately I lost the data from below 32K so for the small ones you can work out what the page size is by a very similar calculation you just look at the one where there's a jump in time and that's really nice because we can characterise the cards that normal file systems have optimised for use on spinning must or they're optimised for ways like XFS and ButterFS they do journals they do snapshots they do transactions but they're all really aiming to reduce the seek time on a single spindle flash doesn't work that way so let's take four SD cards and I've got a guy called Mark Johnson to do this as a summer project last Christmas on 2013 so we took four SD cards and tested them on all the file systems that I had available at that time here with the SD cards I've scaled them because they're all different number of gigabytes by the price per gigabyte the kingston was the cheapest at 80 cents the toshiba class 10x series was $1.20 per gigabyte the sandish extreme $5 and so on and we found the number of open allocation units for the cheaper cards there's two what's more for the kingston class class 10 card one of those allocation units was bound to the file system area so that was always open and then the other one went all over the place the others had nine which was quite nice now I tested these in a sabre light it doesn't have a UHS-1 interface so all the graphs I'm going to show you for these cards have a maximum transfer rate of 24 megabytes per second first thing we did was IO zone and the bottom line is price per gigabyte the line at the side is k per second for an IO zone read test you'll notice that the curve is pretty flat until you get to the kingston one which goes down fast that says that even at 80 cents per gigabyte kingston is not good value write patterns I mentioned before the one on the left is the block trace output for that file system creating a 4 meg file and it goes pretty pretty straight forward the time across the bottom block number across the right there's a glitch at about one second I don't know what that was caused by maybe the process got swapped out and swapped back in again I don't know but you'll notice that it writes pretty linearly and then it writes the fat area on the right hand side EXT4 writes a lot more the scale of this is much more expanded you'll see that the block number on the left goes up to 1.4 times 10 to the 7 instead of being 10,000 it writes the file then it writes the journal then it writes the directory entry and then it writes the superblock with a fairly big delay in between the last two writes for EXT4 the minimum number of writes to create a file is 4 for fat it's 2 enter another file system F2FS this is supposed to be the flash flash friendly file system it's written by Samsung so you'd expect it to work really well on Samsung chips we'll come to that the way it works theoretically it's supposed to use the on chip flash translation layer rather than working against it it's supposed to cooperate with the garbage collection and use those optimizations that were there for the FAT32 so the way it works is it divides the flash into two meg sections you could actually tune that and look for this time so two meg sections and each of those will be written continuously as a log with all of the data in the log before opening the next two meg section it always writes at the log head and those two meg even if you've got a four meg or an eight meg allocation unit would always be aligned with the allocation units and that's nice I'll just go through these, you can read them I'm not going to talk to them, my voice is running out all the metadata is written to the FAT area or the area that would be the FAT area on a FAT file system so if there's an optimization in the control of four FAT this will take advantage of it which is nice the other thing it does is it tries to split the data that's likely to be short term from data that's likely to be long term so that if you've got data that's going to be right once and then sit around forever like a JPEG or something it'll be allocated to something that's then not moved later on it does this by having more than one open two megabyte segment at once it can have up to six depending on the data that's coming in to be written to the place there's also NILFS2 which is an older file system it also is log structured it uses eight meg segments it defers garbage collection to a user process so the user process says what the garbage collection actually you don't say that directly you say what the protection period is for the checkpoints because it continually checkpoints and snapshots the data on the disk again all drivers in the log and there's only one right lockers so let's look at some benchmarks for that one here's postmark postmark reads and writes a lot of small files and as expected FAT32 performs really awful EXT4 performs reasonably well on the Sandish Xtreme Pro and F2FS forms pretty well on all of them the nice thing is that for the Kingston card F2FS performs adequately NILFS performs slightly better but that's a different story again anyway so that's that I've gone through that this year I got an Odroid XU3 it's got a micro SD card so let's try using that instead so I bought a whole heap of more things these are the characteristics I'm running out of time here you'll notice this time the Samsung EVO class 16 gig cards are really cheap at the moment the right hand column is a recommended retail price per gigabyte you can generally buy things a lot cheaper than the recommended retail price so we used Dbench this time you'll notice the Sandish Ultra performs worse than the Samsung Class 6 on every benchmark so on every file system the Kingston, by the way these are ordinal on the bottom in increasing order of price per gigabyte and the Samsung Xtreme Pro performs reasonably well but Nilifest performs really really badly in throughput terms by the way I had a protection period of 10 seconds which is similar to what F2FS provides for these typically you set it up to 2 minutes or something and here we look at the latency curve small is better for these Kingston's are really really slow and Nilifest gives you the best latency of any of the of any of the file systems so in this trade latency for throughput so depending on your workload if you've got a workload that's latency sensitive you might want to use Nilifest if you've got one that's throughput sensitive you might want to use F2FS Postmark write Postmark write performance Nilifest outperforms them that's lots of small files Postmark read throughput also Nilifest is beautiful that I care about like building the kernel this is a make-j8 or no configuration and what we see is that EXC4 apart from that stupid Kingston card performs at least as well as the others and slightly better than most turns out that when you do a make-j8 there's one thing it does at the beginning that's single process it calls git to get the hash of the current head of tree and that actually takes quite a long time it seems that whatever access pattern git does seems to work really well on the EXC4 and really poorly on the other file systems and I haven't managed to investigate that much yet you can also see that if you can spend a reasonable amount and get a Samsung Pro card it doesn't matter much what file system you use it's a equivalent and in fact at that point you're no longer IO bound just CPU bound and that 90 seconds or so is almost the same as if you're building from RAM disk so yeah, that's all my benchmarks there are a couple of other gotchas some cards and files have very high latencies there's another card that I bought that I haven't shown it was a Samsung Evo 24 megabytes per second card it took more than 30 seconds to complete some operations and the MMC driver kicked up its heels in despair and said argh, I can't do anything with this I worked out why it did that a UHS-1 spec says that at 1.8 volts a UHS-1 card can use up to 2.9 watts which means it's drawing just under 2 amps and my poor power supply couldn't cope with that and that's why it was taking more than 30 seconds so watch out for UHS-1 cards and measure their power consumption the other thing is the Nilifest 2 Cleaner can die I've been mudding that Raspberry Pi that I told you about at the beginning for now a year and a half on the Raspberry Pi and every now and then the Nilifest Cleaner D the user space thing that does the garbage collection dies the disk fills up with checkpoint and then everything stops it happens maybe three times in a year and I still haven't worked out why because when the thing hangs you can't get into it to look at any logs and when you pull the card out and look at it it hasn't written any logs because the card's full up so watch out for that one no faster system works well when the flash is full because then you've got to do lots of best fit allocations in what space is left from everything else you're doing F2FS is still fairly immature it hasn't yet been fully optimized for highly concurrent workloads and there is when I run it the benchmarks I showed is from an 8-core machine and when you run it like that you do see some logs being highly contended so there's some work there to do yet it's only been in tree for four or five months the other thing is that any time you've got a process of there man in the middle attacks are possible read the blog entry please I have and don't trust random cars you find in the street I forked the apotec code and ported it to the Saber Lite and to the XU3 for microSD cards so you can grab it and play with it there's also in there a debugger that you can download to the card and do whatever you want poke around, see what's there trace the the original firmware and see what it's doing unfortunately I didn't find any cars that I could actually hack the Samsung, Toshiba and Kingston ones that I had weren't hackable an interesting other fact I pulled those Kingston cards apart because they've obviously used this to me, that's just slow and when you sort of scrape off the epoxy that's there you can read the chip numbers and things and they've actually rebadged Toshiba ones but I reckon there must be second class ones because the Toshiba ones outperform them either that or they put their own firmware on so in conclusion buy a suitable car for your work look out for bargains, you can often get something from APAS auction really really cheaply align your partitions to evade blocks when you grab a standard distribution like Varsbian or the Leonardo distributions for XU3 or whatever they're typically aligned not on the evade block for your card but on a 1 meg or 2 meg boundary realign them and you'll get much better performance and thirdly use a suitable file system NLFS 2 EXT4 or FTFS depending on your card and your workload yeah I've said that and finally have fun try and attack some of these cards they're great and beware of fake cards okay wait for the mic there's one there so basically if what you're saying is that no matter what's printed on the label all cards vary and the performance is all over the place and you're never really sure what you're going to get what is your advice when purchasing cards to get the best performance if you don't know how it's going to perform it depends firstly on how many you're going to be buying if you're buying you know more than a thousand talk to the manufacturer if you're just buying one then find a reputable dealer and deal with them directly as far as you possible I found the cards I buy from Apersorption are reasonably good they are actually what they say they are they're not fakes and again on Apersorption nice apart from the Kingston cards which I found to be uniformly awful actually does more or less represent performance if you've got time and you're going to be deploying these fairly rapidly buy samples try them all out I think the gentleman behind you was next okay right in that case down here I was wondering if you could clarify a little bit the fat optimizations in the system it seems to me that with a fat table you're reading the same page again and again and again doesn't that blow the right cycles right out it would if that's what you were doing but what they do is you've got your your flash file system your fact's there and what it does is it puts that into the van and every time you write it just writes into there and then just before you pull the card out it writes it back how does it know when I'm going to pull the card out oh simple the power pins are longer than the others and it's fast enough that's only a couple of microseconds Murphy says it just isn't well I'm assuming you just lift it and it just disconnects and that's one of the reasons by the way that if you just power off the machine rather than properly unmounting it you're likely to get corrupt cards who's next Madam Chairman where are we next that gentleman how do you know how to realign the two arrays blocks easy first for most cards if you assume a 4 meg you're right so just make the partition on a 4 meg boundary if you want to be absolutely sure use the flash bench tool and measure it also if you get the if you look at the original partitioning on the card when you buy it it will come pre-partitions that will almost certainly be base block aligned before you start so just look at that and use that in all your other partitions a couple of kind of related questions at work we use industrial quality SLC USB sticks which are pretty expensive but so does this USB have the same sort of issues and is there any difference between the MLC in terms of what you've talked about today yes SLC has a longer life than the MLC in general in terms of number of array cycles but apart from that the rest is almost the same just that you've got a different interface at the front you've got the same problem of man in the middle attacks if you look at the bad USB papers from us next year you can see all that stuff yeah so very similar they typically have a higher power budget and more powerful posters just on getting samples and then buying the batch you do want we've brought we've taken samples found a good card brought hundreds of them found half of them worked half didn't packaging and externally identical not the same that's absolutely true if you read Bunny Huang's blog I'm happy that when they were building the Chumbly they bought a whole heap of Kingston ones because they were the cheapest supplier and found that they performed really really badly and they dissolved them in acid found that serial numbers were identical underneath the epoxy and it turned out to be a ghost grey run on the ghost ship run was there another one at the back there somewhere over there your speculation about some of the devices being fat file system are where it's correct more so for USB keys so what they do is they look up from underneath, recognise that fat is being used and change the allocation strategy based on the fact that you're writing fat so you'll get a not necessarily a drop in performance but suboptimal performance if you're not writing fat to one of those devices yeah that's quite probably true yeah any more questions yeah one down here is it sounds like the firmware is potential wait for the mic please so it's for internet land it sounds like the firmware is potentially an issue with some of these devices yes what's the possibility that you can get access to more directly lower layers within the device and like write the driver in the next land yeah that will be a really neat project first you've got to work out how to hack the card or have millions of dollars to talk to the manufacturer and get under NDA what they're doing if you happen to have a card with an apotec controller the hack's already done and that's the one that we I put the URL for out there if you don't have an apotec controller then try it use fuzzing on the programming pins see what you can do when you're talking to the SD card there's a defined standard you send down command bytes and then the data for the command bytes command 63 is allocated to the manufacturer and it's typically that that they use for in-system programming so try fuzzing it see what you can do oh yeah and go to Shenzhen and buy dozens of them anymore okay I think we're done are we having fun yet? thank you very much for the talk and it's a small present