 the hardware is full of features. Features. Random, yeah, it's a security feature. Because you know, if you have to reboot often, you know, you know, you patch and then you have to reboot to make sure that everything's clean, right? Yeah. So you've got a question back there. Oh, yes. So here's another really exciting thing about hardware. And in fact we ran into this on the middle board. So the comment was, you know, there's documentation for the hardware, and then there's the errata. And sometimes there's the errata to the errata. And sometimes there's the errata to the errata to the errata, and we just threw it out and started over. Case in point on the middle board, if you take a look at our documentation for the middle board max and the middle board turbo, there's a pin on our low speed header, pin 26, which is going to become the bane of my existence, unfortunately. When we first built the board, we had intended to use it as an M clock for audio signaling. So this is a reference clock between the SOC and the audio codec so that things supposedly line up. You don't have to have this, but it's really, really useful. Well, we picked a signal, it was supposed to work, then we found out in an errata, it did not. We also ran into it on HDMI, where, you know, we designed the board, we got it out. You know, the monitors we had all initially plugged into just worked, everything looked great, we shipped the board, and then we started getting reports of it not working. We're like, huh, that's kind of interesting. In the errata to the errata, we found a footnote, a hardware guy literally mashing his head. I wish you guys on the camera could see this. It's like, oh my God. That if you are on the process that our particular SOC is, you have to put a level shifter because the signaling's too low. If you're on the previous process, you didn't need this. And again, when you're, you know, we as software people, we assume the documentation's right. It's never right. Yeah, so do I, but I'm not mentioning who they are. Well, and that's a really good point. So the, for the recording, the gist and correct me if I'm wrong is that we all assume that this is going to work on the first or second try. You know, whatever hardware we're building. The reality is if you get it right on the first try, good Lord, you should get a bonus because I've only seen that happen twice, ever. Most of the time, you get it right on the fifth time. That's the time when it boots the first time, but nothing else works. Tenth time is when your Ethernet controller works. And the fifteenth time is when you realize that yet again everybody has screwed up how serial URs work. Because, you know, serial URs, we've only had these things for 30, 40 years and we're still getting them wrong. I really don't understand how everybody screws up serial URs because they're the most useful thing in the world, but apparently nobody can build serial URs correctly. But yeah, the, I think even on the turbo, which is just a minor set of fixes, we went through four, three or four revisions before everything got ironed out. So, yeah. I mean, oh, okay, yeah. The reworks, the white wires. Yeah. That's just a proving that your fix in the next board will work. The X USB maintainer. USB 3 maintainer, I'm sorry. Yeah. Yeah. So, for the people on the video stream, the short answer on that one is the internal documentation was wrong and you had to reverse engineer your own hardware to figure out how it worked. The statement is that's normal. Oh, and there's a working driver that you had to. So, yeah, I mean, I want to hear your guys' stories because, you know, I can stand up here and remember random things for the rest of the, I think I've got 20 minutes or so. But honestly, I want to hear what you guys are finding in the field because my stories are good, your stories will be better. So, yeah, a meta point. That's because they were, to be fair. Yeah, I mean, if you, so the, this is a call to action that the hardware simulators, we've got some reasonably good ones. QMU is pretty decent in that respect, just from a simulator perspective of a whole system. But just being able to simulate low level hardware is nearly impossible and it's really, really computationally expensive. If we can simulate more, we can do hardware better. So, if you, if you're looking for an open hardware project or something or an open source software project, go start taking a look at the simulator as they could use a lot of help. So, yep, there are people building their own CPUs and having reasonable success in that side of FTGA. You should do as much prototyping as is humanly possible before you actually try to build anything. Whether it be throw it into an FPGA and run it at one megahertz because at least you can still walk things very slowly, you know, it might take you three weeks to boot up. But, you know, you'll at least know that it's right. So, yeah, I could not even remotely give you any good suggestions on operations on shipping. That is a really tough problem and that is, that is almost worth its entire, its own entire conference because operations is form it up to someone else because you're going to screw it up to be honest. CalCen. Regulatory stuff, I mean, well, I mean, if you're actually going to do a product, regulatory stuff is a really serious and big hurdle and this is something you can't ignore. If you're, if you're intending your device to end up in, you know, normal people's hands inside of a house, you've got at the very least Class B FCC certification to deal with. You've got environmental stuff that's got to be done. This is a non-trivial cost and, you know, when you're building one board for yourself, these are things everybody ignores because nobody cares. I mean, to be honest, you shouldn't, you know, actively try and build a board that's radiating out into the stuff that shouldn't be but, you know, your one board probably isn't going to make a big difference. But, you know, if you're building a thousand of, you know, a thousand of these things, yeah, you have to, this is something that most people forget and needs to be dealt with. So, that's a mechanical problem and a lot of that particular, so the question is, you know, how do you deal with the mechanical issue or just the, the componentry of connecting the, the layers, the plastic layer. Some of that's just trial and error. Some of that's getting a good mechanical engineer from a, from a hardware perspective. Again, from, you know, me as a software guy, I hire people to do that because I suck at it. People have seen my 3D cases. So, the question is, you know, not trusting the documentation and doing a lot of testing up front, is that a step forward or backwards from the, to the software world for how all of this works? I think trying to even, I, I think at any point if we ever, you know, we as software people ever trusted the docs from a hardware or for a piece of hardware, we were idiots. So, I'm not sure that it's so much a step in any direction. It's just the status quo. It's the way it's always been. I mean, I, I can ask the heart, you know, the people who have done the low-level hardware, driver development, if we've, if you've ever seen a piece of hardware that didn't have 20 pages of a rata because it's all broken and I, I don't think any of them would raise their hands at this point. So, I, yeah, yeah, they're, they're usually adding to the rata because they try to bring something up and they find that it's still broken. Well, yeah, I mean, unless you can see the code for what you're working on and you can verify that, you know, the function that you're calling into from a software perspective is doing what you're expecting, you should be suspicious of it. If you, if you've got a black box, you know, you should be testing against that black box repeatedly because you don't know what's going on inside of it. And then, you know, a lot of hardware, even if you've got the documentation, it is still a black box. You know, you don't know what's actually happening in there. You just know that if you twiddle this GPIO in this way, this result on a different GPIO should happen. You know, from a software perspective, if you're calling into that function, you can't, you know, literally open it up and look at it. And in a lot of respects, how many people really want to go and read through every code of a library? I mean, we, we all just kind of take it as, you know, faith that, you know, GLib C is going to go and do, you know, what we're expecting in the general sense. Now, faith is, you know, generally well founded because we've been, you know, doing this for years. Smart people have been looking at this. And when we find bugs, we submit them. Or we go digging ourselves and we fix it. But, yeah, you should be suspicious of, you know, anything that is not your code or anything that you're interfacing that is not yours. Yep. I mean, the, the, the rise of, you know, I want to call this home manufacturing laser cutters, 3D printers, these kinds of things. If you're, if you're going to be doing mechanical design, get your hands on good ones of those. If you want some recommendations, ask me afterwards. And just, they'll be expensive, but it's worth, it's worth it just because of the, the faster prototyping for your own self. So, you know, getting back to the micro view and its casing, they did do a lot of 3D printing and testing, test fitting. Well, your turnaround time is so substantially higher that you can iterate faster, which means that your upfront costs are expensive, but your overtime costs are actually better. So the question is, is it actually better to use components that you can update after the fact in your designs versus, you know, just static components? It's going to be really subjective on what component you're talking about. The advantage of being able to update things after the fact is that, you know, likely that particular manufacturer recognizes that they're going to ship a bug in their hardware and that they're expecting to be able to fix those bugs, whatever they may be, because nobody really wants to, you know, upfront, I'm going to ship a bug that will be, this pin will be stupid in this way. I mean, I mean, how many people want to ship a piece of hardware that's fundamentally broken and you know it? Healer? Healer? Okay, nobody wants to ship that. So if you have a reasonably complex, you know, circuit or a system, having an update mechanism after the fact is almost essential, you know, I will pick on a, you know, take a look at any CPU these days. They all have microcode or, you know, or a reasonably complex piece of hardware. They all have firmware or a microcode or something that you have to upload into it. Probably a non trivial chunk of that is just working around bugs in the hardware that they, you know, the hardware folks have specifically gone in and built in more layers or more complex circuitry, because they know that, well, maybe this feature that they're doing is a little on the risky side. Maybe it will work, maybe it won't. And they can, you know, through software they can enable or disable it or even, you know, work around bugs. I mean, take a look at the news, there's been a couple of things even recently about, you know, effectively a software fix that's been pushed to fix very specific, potentially nasty hardware problems. Yeah, the statement is if you could ship everything as an FPGA and keep the cost down and keep the speed up, the world would be perfect because everything would be fixable. Unfortunately, FPGAs are not quite that awesome. They're awesome, don't get me wrong. They're expensive mostly and the power requirements on FPGAs is crazy because of how they work. So the question is in particularly out of Shenzhen, you've got folks who are turning around boards or prototypes in hours, not even days or weeks, they're talking about doing this in hours. And the correct answer is in the United States we probably can't even remotely compete with that. Well, we do, but it's... So, you know, at least there are some folks here in the US that are doing it. The real way that we kind of level that playing field is, you know, it may just be easier and cheaper to manufacture in China in the general sense, particularly from a volume perspective. I mean, I don't think there's anybody in the room who would say that it is not generally speaking cheaper to manufacture over in Shenzhen for a large run. Small runs, the math changes dramatically. But one of the things that we can do to level the playing field for everyone, whether they're here in the United States or in China or Japan, Korea or where not, is actually start taking a look at more open hardware. The minute that we democratize hardware more, the more that everybody gets, you know, faster turnarounds whether they're in Shenzhen or here, but it also means that we're taking advantage of everybody else's knowledge. So, you know, I've said a lot about how we as software people, I mean, we're really good at open source. I mean, I don't think there's anyone in this room that would not agree with me that we've got open source kind of figured out. You know, we've got the Linux internal and we've got, you know, conferences that have anywhere, you know, that have thousands of people who show up and want to talk about giving shit away. I mean, it's kind of hard to argue with that. The hardware world is still waking up to this. You know, we've got, you know, new initiatives with the open source hardware alliance and we've got, you know, some other things along those lines and we've got boards like the middle board of the Beaglebone or, you know, a lot of the Arduinos and those kinds of things. They're all open hardware. Everybody can go. They can look. They can compare. They can build from it. This gives everybody a really great chance to go, oh, well, you know, this platform doesn't quite meet what I need. I only need a few little changes. And a lot of what you're seeing in Shenzhen is, you know, they're not necessarily making a whole new board in that, you know, that day spent. What they're doing is they're taking something that they've already got some relative, you know, that they've already done and they're making some modifications, running it through quickly because they know, because they've done a lot of the work already, they only have to do the incremental work beyond that. And the open hardware stuff does the exact, you know, effectively does the exact same thing. By only having to deal with the incremental cost beyond what you're doing, or beyond what's already there, you shortened the time frame to get from idea to an actual prototype or even to mass production. And I mean, how many people in this room think that they could lay out GDR3 or GDR4 in memory? You, my good sir, are a very good person to have on the team. Because I couldn't. I can't even begin to have, it worked the second try. And that's actually very good. You know, working, you know, getting, I mean, these are things that literally have to be the exact same length. There's not, your margin of error is so small. For those of you on the video, he rolled his eyes very greatly when he said the CAD software does it for you mostly. And they try really, really hard, but CAD software makes a lot of assumptions that work for very specific sets, but it all, you know, even laying out a board at the end of the day is still a piece of art. It's not a mechanical thing because someone still has to go in, verify everything, and then we'll still have to tweak it because sometimes the software gets it wrong. Like, you know, taking your memory channel and routing it literally around the board twice. Yeah, that, that, that, it happens. Oh, God, yes, please. So the question, or the statement is, is should we involve more software in our hardware development processes and the answer is unequivocally yes. By and large, hardware is made by hardware people, for hardware people, and they don't understand software. I'm seeing a lot of people going, yes, yes, please. It's fascinating. I mean, if you take a look at any chip, particularly if you build anything, and it's made for hardware people, by hardware people, and you know, they go, oh, well, you know, the software guys can do anything. It doesn't matter what I do on software, or in hardware. And sometimes making very small tweak makes the software so much easier to deal with, or to, you know, oh, they don't need to be, there doesn't need to be a signal to let you know that the hardware is done. You'll just pull, pull it all the time, right? Yeah. Yeah, status registers are stupid. Just, yeah. I mean, oh, right, only registers. Yeah, you don't want to read them back? And why would you ever want to just write it over there? Don't, you don't need to read it back. You don't need to confirm that what you actually wrote is what the system thinks you wrote. Yeah, software is never buggy, compiles the first time. None of you are, you know, that guy in the XKCD on the chair playing, playing swords, because it's compiling. So, I guess in some respects, this is, you know, we as software people need to take a better look at how our hardware works, and we need to account for that more in our own processes. And if you're on the hardware side, please, for the love of God, start talking to your software people more, because we, one, we don't understand when we need to, and two, we need to share our software knowledge with you guys so that you can build better hardware so everything is less broken. So there you go. I can keep answering questions. I'm going to have to run over to the middle board booth here relatively shortly, so if you want to have a more detailed discussion, we should probably do that over there. Otherwise, thank you. And if you're really bored, I'm giving a talk on Friday. It's the last slot, I believe it's in this room, about how failure has had the fun of success. What? Is this Friday? What did we miss? Oh, I don't know. I miss everything. Testing. How's this? Good level? Yeah, I'm going to stick here, so that's fine. It's like a picture of employment at the DRAM faces. Looks like it's about time to get started, although I'm not sure where the embedded track moderators gotten to. So before we start, I'm curious as to whether there's anybody here who doesn't already know what U-Boot does who's not familiar with U-Boot. Is there anybody? Okay, good. We have a few people who are going to enjoy the little intro part then that I bothered to put in. So here we have our little rumor monger tux who's going to help us out with some rumors that we're going to explore later. But first we're going to start off with the basics. Basic facts. U-Boot was born in the year 2000, had a couple of parents called FADROM and 8XXROM. Those are both bootloaders for PowerPC. It was an old type of PowerPC by Motorola and it was for embedded stuff, not for Mac and Posh's. There were some, these PowerQuick chips and 1x was one of the things that ran really well on them back then because it was so dedicated to PowerPCs and its initial introduction. It used to be called PPC-Boot but nowadays it's called DOS U-Boot for its full formal name. It's almost nobody uses but that is what it's called. So let's take a look at what U-Boot actually does and I'm going to compare it to the PC scenario since if you're not yet familiar with U-Boot that's probably something that you're at least familiar with. So let's look at the PC boot. To start with the hardware you have your main storage normally a hard drive but inside the PC you've got this flash thing that your BIOS or your EFI lives on and of course you have your your main memory your RAM. So when you start the boot you have this BIOS in old days U-EFI and modern days that is the first thing that runs when you hit your power. The next thing that runs is the grub. Say that's pretty common for your Linux-based PC although there are a few other alternatives you could be using. Let's go for the normal choice. The BIOS or U-EFI whoops up the grub off of your hard drive and pops it into the RAM where it starts running. And the next step is that your bootloader goes back to the main storage and whoops up the kernel and starts running that out of RAM. So we've got something analogous happening on our embedded board and not quite exactly the same. To begin with the main storage can be a hard drive but it rarely is. It's more normal to have even the single chip type the the raw type storage in old days say back in the PPC boot days. We had the luxury of NOR chips which are very reliable. Nowadays you might have a NAND. You might have an SD card or it might be on an EMMC which is kind of like an SD card that's been soldered to your board. And in place of a whole chip to run your BIOS you have the CLEED SOC system on chip reviews that have ROM inside the chip from the manufacturer. And that's going to be the early boot. Plus they normally have some SRAM static RAM also inside the PPC. So static RAM is pretty expensive so this is just a teeny little blob of static RAM to get yourself bootstrapped and running. And then you normally for an embedded system that's big enough to run Linux you normally have the separate DDR memory for your RAM. So the first thing that happened when you hit the power on your embedded board is it goes into this boot ROM which is inside the CPU. Because this is so tightly coupled to the CPU it's like the old PC BIOS where every vendor does their own thing with the little boot ROM. So the specifics of how these work is you're going to have to get familiar with the data sheets for your part. There are some things that are in common but details are always you got to go with your vendor specifics. So somehow the boot ROM goes out to your main storage and it goes out and loads this SPL which stands for secondary program loader because the boot ROM is considered to be a first. So SPL once upon a time used to be a separate project separate software project. The goal here is to be really really tiny. So tiny that you fit into whatever the CPU designers gained to give you for that internal S-RAM memory. So we're talking like just a few kilobytes. In modern times you boot has actually learned how to pack itself into tight spaces shall we say and U-boot can actually build to a special SPL build that goes tiny tiny and fits inside the CPU. So the whole purpose in life of this SPL is to then go out and drag your U-boot in from wherever it's living to give yourself a proper full service boot loader. You get that loaded into the main memory and jump in and let it do its thing. So the embedded boot is actually tends to be a little more complicated than the old PC. You get you have to know your hardware a little more if you're you know doing a board support package. So then the U-boot is the thing that equivalent it's really equivalent to the grub job where it goes and pulls the kernel out of your storage and pops in the RAM and runs it. But it turns out that U-boot actually does a lot more. It is a little more diverse in its talents than grub or whatever your PC, your favorite boot loader is. So let's take a look at some of these things that U-boot knows how to do. So of course to qualify as being a boot loader has to know how to load and boot images that's pretty much a given. But it also is a lot more skilled in initializing and exercising hardware than your typical desktop or server BIOS because people tend to use U-boots more than just the end user final boot loader. But from the time that your custom board is born you know just the very first version back from the factory a lot of times people are using U-boots to exercise whatever the hardware guys change from the reference design and make sure it actually works. So instead of the classical menu system that you have in a BIOS you have the just the command line interface to U-boots. But the good news is that you have a lot of flexibility in managing environment variables so you can have a stateful system and save all kinds of settings any kinds of settings that you need to know in order to boot your system. And the final one that makes it really fun is that U-boot can actually run script. When you're sitting at the command line it has a command line parser which if you if you want to configure it actually can do a command line history you know like a shell. It has conditionals and you know if this and that it's actually pretty good. So suppose you had a you know that it gives you a lot more flexibility than say just the BIOS menus where a lot of times the vendor may not have given you the commands that you really wanted to be able to do. Like say you have just some desktop system and you wanted to remember to always power on after the power goes off right. That's a server thing. Well if you've got a desktop BIOS they might not have bothered to put that in. With U-boot pretty much you've got it just a ton of flexibility between the generic environment mechanism and running scripts based on your environment setting. You can actually have a collaborative thing going where when Linux is running it decides that the next boot is going to be a special one. Say you're going to do an upgrade and so you just set a variable and the script that you gave is the U-boot lets it know that you want to do something special that only happens on upgrade. So pretty cool. So now we'll get in after we've done the basics. We'll get into some of these rumors that were in the abstract. I've got my little helper the tux who's into rumors managed to dig up these leads. We're going to investigate to see how many of them are actually true. First one he claims he heard claims that U-boot is a network strike breaker for a small fee. He heard that U-boot likes to go around flashing bare hands without a permit. He heard that U-boot has been known to reprogram uncooperative boards for Jenkins and the last zany one that he didn't actually manage to fit into the abstract is that U-boot was heard to perform all chemical transformation. So started from the top. So let's talk about the normal case first. Under normal circumstances U-boot likes to team up with Ethernet to get pixie like booting going. So for those who don't know what pixie like booting is. In some of those biases for servers and so forth you had a PXE mechanism for doing net booting. You could have a whole cluster of diskless things that would boot off of the that would know how to boot off their Ethernet and then say either run out of a RAM disk or do an NFS mount. So you can do something similar with U-boot. You need some prep work unlike the real PXE. In the real PXE you have a special flavor of a DHCP server that not only tells the client what their IP addresses is but it also tells them where to go to find their boot images on the network. Tell them the server IP and stuff. So with this mere pixie like booting you have to go into the ENV and set up a server IP ahead of time and give it the boot art so that it's going to know to net boot. Then you use a couple of commands. You use DHCP to get at its own IP address and you use TFTP boot to grab the actual bootable image off the network. Okay so that's great. They're usually on good terms. Ethernet and U-boot. So what happens when the Ethernet devices go on strike? Say you have to pick a completely random example that may never have happened. Well yes it did. Okay flaky early board spins you find out that first version that comes back from the factory that you cannot get to a Ethernet and you're trying to do your rapid iteration with your development where you keep on building new kernels and booting NFS. That won't that won't cut it if your Ethernet fly got glitched in the first version. So there's another case at the opposite end of the spectrum. Sometimes these penny pinching hardware people end up taking off the Ethernet. So the final production hardware does not have an Ethernet. I may have seen this as well. So what happens when the local network is on strike? U-boot just teams up with the serial instead. You use the command load x, load y, load b and it's really old school because those things are x modem, y modem and Kermit like back in the modem days. You can use programs like Minicom is my favorite. Other people use other terminal emulators but still remember how to do these ancient protocols. Okay so on the other hand there are no fees involved. U-boot does work for free. We're going to call this one a partial truth. It does, it does not go with the net, it does not cooperate with network strikes but it doesn't charge a fee. Okay next rumor. Suppose that the theory is that U-boot goes around flashing bear nans without a permit. So for those of you who have never attempted to do it yet, raw nanships can be a major pain to program. Pain! You have to cope with bad block tracking. You know those lovely SSDs that make your desktop and servers so fast? Nowadays they have all kinds of firmware inside them to do the bad block tracking for those stupid little nanships inside them that are very, very flaky. In that it's not a question of if there will be bad blocks. It's just where will you have bad blocks? Like you know if you have monitors and they won't take it back if you have like you know a hundred bad pixels. Oh it's only a hundred. NANDs, the NAND manufacturers, they guarantee you that the very first one is good and beyond that who knows? Yeah, yeah. So not only do you have to keep track of bad blocks as if you without having the benefit of your fancy-dancy SSD from where to do it, you also have to cope with the ECC algorithms and the layout of the ECC bytes. So probably you've heard about the ECCs in reference to CD's and DVD's perhaps. You know they have the error corrections built in so that every little, every little scratch on your CD or DVD won't ruin the whole thing. It's a way of, it's a mathematical method for making use of redundancy. Okay I know this, I know that there's an error here and not only that there's enough redundancy that I can actually figure out what this was supposed to be even though we lost some bits. In the case of NAND it's bits that like to get flipped. And again NAND is so bad that it's not going to be, it's not unlikely that you have more than one bit flipped nowadays. In the earliest NANDs they use something called a Hamming Code and that was able to detect and correct one bit per block. Now you have to cope with some NANDs are rated for up to eight bit flips per block. So back in the old days when you had Norse you could, once you learned what the programming protocol was for a particular vendor, you could pretty much just say okay here's the image I'm going to throw on these exact bits and I'll verify it in the end and if the bits match we're good to go and so you have these little devices called flash programmers, right? So nowadays with the NAND you need to know exactly which ECC algorithm is the software is going to use because if you put in the redundancy using a different algorithm the software is going to think all the blocks are bad, not going to be able to read anything. So you have to have exact agreement in calculating these fancy fancy checksum and then you have to put them in exactly the right place because you have this OOB part out of band area in your NAND and the layout you know say you use bits by 0 through 52 and the programmer put them in you know shoved them at the end of the block instead you're also going to think all the bad blocks. Okay so that shows you why if you're getting small quantities of boards from a board house in early runs you're not likely to hear them volunteering. Oh yes let me program your NANDs for you. No, they're likely to come out bare blank. Oh yeah it was sometimes it's even worse than just knowing which ECC, sometimes it's knowing which two ECCs because the boot ROM insists on using one ECC and your kernel insists on using another. That's really a kill. Did I say it was a pain? Unfortunately, yes. You actually run into this problem with too many blank NAND chips. So it turns out that you boot does help you out in this situation. There's actually two differences general approaches. One is that you boot teams up with your boot ROM as a partner. You set your CPU boot jumpers for whatever your boot ROM knows how to do and you have actual hardware wired up to talk to it. Like for instance if you've got a board that has an SD card, a lot of embedded GPUs know how to load a block off the SD or you might have one that knows how to shove things in over Sira port using X modem, say, more of that old school stuff. You might have USB, you might have an ethernet boot actually programmed into your tiny little boot ROM. You have to read the docs to know. Sometimes setting the boot jumpers is not as trivial as you might wish. Say your hardware guy didn't think to put the jumpers on for the mode that's actually practical. You might need your soldering iron. So once you have it all set up, you've got the plan going. The boot ROM then loads your tiny little SPL. SPL loads UBoot just like we had in the earlier diagram. Then the UBoot is smart enough once it loads the image to RAM it will know how to write to NAND in the proper manner. In fact UBoot knows how to generally if you've got a platform where they annoyingly make use of more than one ECC it'll have a command that lets you switch back and forth. You say, okay I have just loaded my bootloader image which is basically another copy of UBoot itself. Now I'm going to burn it with the special bootloader ECC and now I'm going to load the kernel and now I'm going to switch over to programming the NAND with the other ECC. UBoot can also team up with the JTAG programmer assuming you've got a nice little JTAG port on your board which usually they which hopefully somebody thought of if this is an early revision of the board usually the hardware guys are going to want to use the JTAG too so they do remember that. So the JTAG then is able to set up the SPL and stuff it into SRAM it will you can put a breakpoint so that after the SPL is done it's usual dirty work of setting up your main RAM so they actually can remember things. Then you can hold it and let the JTAG go again. This time you load the UBoot into your woken up RAM and you run it and then everything works like before. So if you've got a JTAG port available and you have one of those JTAG programmers this is actually a pretty convenient method during early development because one thing the JTAG programmers easily run faster than they UART and you don't have to worry about some wacky custom protocol like you do with the USB sometimes. So we're going to say this is true. It does flash bare hands without a permit. There is no permit but if there was one you might actually get it. So moving right along to our next rumor we heard that it reprograms uncooperative boards for a guy called Jenkins. You've probably heard of him in other contexts because Jenkins is one of those popular continuous integration or CI framework that people use a lot on servers including just all software projects not necessarily embedded. When it comes to doing embedded projects though Jenkins sometimes sells for native server tests. Say you've got an application that's pretty generic and just happens to be running on an embedded board. Why not just test it on the server. Sometimes Jenkins sells for QEMU tests. For instance the open embedded you might have builds where it's pretty convenient to go ahead and run stuff in the QEMU platform and that would allow you to test that not only does the application work as designed does it work on the actual arm architecture without having some stupid compiler bugs getting in your way but sometimes there's just no way to get around needing to run on the real hardware because QEMU stimulates common arm architecture stuff. It doesn't stimulate your custom say you've got hardware inside the SOC that does take a random case video encoding or video decoding accelerators. QEMU knows nothing about that. You can't test your applications that are making use of those accelerators. So when it comes to that and you want your automated test without somebody sitting there and programming the latest greatest version into your board you boot is willing to step up to the plate and help out. This is kind of the overall strategy. Jenkins will when it's running one of these hardware specific tests will leave a you boot command script over in a TFTP area because you boot knows how to do TFTP. Then Jenkins will reboot your board using whatever mechanism you get it like you might have a software controlled power switch for the brute force mechanism or you might have SSH logins where Jenkins is able to reboot the thing. And then you boot comes up and you boot has been told ahead of time that it's supposed to go TFTP that script to Jenkins made it and just do what it says. So this one is also true mostly because uncooperative boards usually have the option to opt out. If you've got a really uncooperative board you usually need to bring back that soldering iron to talk some sense into it. Okay here we go this is the zaniest one of all. Does you boot perform all chemical transformations. Are we talking like lead to gold? No more like brick to board. So this one has actually a number of variants. This is a nice meaty rumor we've got here. The thing they all have in common is that the brick starts out with a serial console because you boot commands are run over the serial console. The first one is that you have a brick with a USB port in it and then you plug in your USB stick and manage to get your brick back into a board. Or it could be you have a brick with an ethernet port in it. You plug it in to your development laptop and you manage to turn it back into a board. Or maybe you have a brick with a JTAG pins on it and you plug in your JTAG programmer and turn it back into a board. There's a hint for you. Real bricks don't have serial console. So that's the trick. It is not really an alchemical transformation. It really was a board all along but UBoot can help you get the board to stop acting like a brick because it is really versatile and it'll work with pretty much whatever peripherals you've got. If you want a netboot that's great. If you want to shove things in via JTAG and then reflash your board that's great. We already talked about that. If you want to boot from an SD, you want to boot from a SATA, you basically however you want to get your recovery images in UBoot can pretty much handle it. But sadly it wasn't this since it wasn't really a brick. So I'm going to do some final conclusions and then I'm going to do some Q&A time and other people who like to work with UBoot are free to share their own war stories. It's the cool UBoot tricks. So first of all I argue that many books could be written about the exciting life of DOS UBoot. This barely scratched the surface of what UBoot is capable of. But unfortunately they probably won't get written. Too few of us embedded does really are inclined to write much so. Go to one more info. On UBoot might have to sell for some presentations. If you go to either of these lists you will find a few different presentations from the ELC Europe and it's both slides and videos on YouTube. Or you might try the UBoot Read Me which is really really long. Probably one of the longest Read Me you've ever seen and worthy of some light bedtime reading. I mean it's probably 70 pages printed out. We're going to roll the credits now. Thanks to Dan Malk for the fads rom back in the day. And Magnus Dan for 8XX rom. Wolfgang Dank was the one who did PPC boot which then got renamed to DOS UBoot because it no longer specialized. Tom Renne is the current lead maintainer and they have many friends helping them. Give credit to John Miller who is the guy who built the Motorola board. They ran PPC boot back in the day and he was one who recruited me to help get it on there. That's how come I'm familiar with UBoot even now. And credit goes to Tom King for starting a scale embedded track. Starting last year actually. I'm pretty happy with that. And my beloved other half gets credit for helping me come up with the topic when the deadline was looming. And Inkscape deserves credit for this whole presentation with all the PPC animations and stuff. It makes writing a presentation fun. And open clip art has graciously provided the traditional Yamas no wait. The credit Yamas yes. And the smelly chiefs and trainers of the Yamas used to get me to stroll in the right direction. And the brightly colored balloons and a space shuttle. And finally the horde of friendly penguins. They want questions on UBoot. We'll try to answer any questions you've got. I have a really hard question. I have a really hard question. I know there are a million versions of UBoot on the Internet right now. And recently you did some stuff using an arm chip with a signed boot. And that's a fun one. Yeah it is. That's an all winter chip because you can find the code on the Internet if you look. All winter is not known for being great at mainstreaming. Whereas if it's Qualcomm or TI you have to sign NDAs and you go to prison if you talked to anybody. What what advice can you give. Which UBoot is a boot to build to. Like when I do the next project. Do I use Denkz's version of it. Do I use one of the the GitHub versions. What do you think. I would go to the UBoot site and actually probably use one of their architecture specific trees. The Tom is the one who maintains the like the equivalent of the Linus tree that has all the architectures merged together. But he has the equivalent of lieutenants that are looking after different architectures different. You know like there's a TI tree. There's a TI tree if I were going to do TI. And there's quite a few of them. I forget if there was an all winter one. You might go with the main. Well I have the all winter one. Yeah. But it's stolen from all. Is it from UBoot or from the vendor. From the vendor and ended up on the Internet. So I can say that I have. This is a step of this is a step better than that. And that they're actually the changes coming from these trees that are listed as basically lieutenants. I forget what they what they term them. But these trees are like the equivalent of the ones that officially feed into the the mainstream one and they. Is there a sub tree that an official sub tree that deals with the sign boot. No there isn't. Unfortunately. There is. No. OK. So. That's in. That's in all the ones from the banks. Right. Like the the Chrome people Chromebook people did. Yes. I would. I would go look at what the Chromebook people did. If I were going to do one of the the fully signed trust. No there's the there's the Chrome for the Intel and the Chrome for the arm and there was a vintage of Chromebook that did what he's talking about. I'm pretty sure. There there the next version is not going to use UBoot in the in this process. But there was an error when they were using UBoot for the arm Chromebook to do the the trust is signed stuff. Doing it from UBoot. Oh you mean instead of doing a an upgrade from the Linux. Because we're we're using a normal version of you know the end user version of the software which may not be optimized for programming updates. So what the UBoot does is it will then I could have gone in a little more detail. So I'll do that now. So the UBoot can tell you to load a special programmer version where you have everything you need in the net ram at that. And that is where your M.T.D. UTILS gets installed. It's not necessarily installed in the normal. I'm just programmed to do that. To run this application kind of a build. Right. You. Yes. You booted into the special programmer. Yes. I'm not going to say a hundred percent because there's hardly ever anything that's a hundred percent. I'm going to say I have never I can only remember red boots being used once upon a time. And I can't remember the last time I've seen that in a board support. So board support package is pretty much come with UBoot and a kernel as a minimum. Nowadays there are actually usually an open embedded layer. So and those that open embedded layers if you're not using. OK. I thought of it. I thought of one. Minnow board doesn't by default do you do they do the EFI stuff instead. But if you're not doing the EFI then you're probably doing UBoot. People who are just doing their own thing are always free to run their their own coolest favorite. So I don't think there's any reason why you would literally make those things go extinct. But I think UBoot is going to be the standard for anybody. Well like the Android kernel I wish it weren't the case but if you've got a new chip nowadays it's probably going to be running an Android kernel if you're in something that could possibly be used for purposes. Now I mean you can't run other obscure kernels on the same hardware. But that's why I mentioned the SPL is knows how to do teensy timesy build. So if you if you can actually build it as a secondary program loader you can build UBoot really cut down. So it's all configurable. And in fact in the in the new mainstream the new mainline UBoot they've gone with the kernel they've done recently done a switchover to the kernel configuration style where you have the the menu config and all I want to turn this one on this one off this one on off off off off off. In old days you know that you and I work with mostly is it would go in and edit the header file. Right. And then you had to get familiar with that like bedtime reading of a readme to know what things to put in your header. So I think they're trying to make it more convenient to be able to turn things off so that you only have the things that your project specifically want. Yes. That one of those talks in the the Europe ELC that are online. The one by Simon Glass covers the device tree stuff way better than I could because he's the one that has been leading the effort there. He was the one who. So if you're interested in device tree I highly recommend watching his video. And they have made great progress with it last just in last year. A lot of things are going with the device tree and that that will in the future make new ports easier. New port. You know. Yes. It would be a you would use a different to you wouldn't use the same you both of you. That the software engineers use during development. I think they called it Falcon or something but there's a essentially what they did is they made it to where the secondary program motor mode where you compile it down with just a tiny minimum of stuff that it was smart enough to be able to load your kernel directly without going through. So you compile in just what you need into the tiny one. And it doesn't know how to do any of this net boot nonsense. It doesn't know how to do memory tests blah blah blah blah. It just knows. I know how to get the kernel from say FD or whatever you've got and pop it in there and run it. So I think that might actually have been Wolfgang who who did that project. But anyways you can you can look for you but Falcon pretty sure that was the main we've done. Thanks for coming. I guess it's time to hit the exhibition hall. One two three. One two three. There we go. Hi everybody. So it seems we've been having the wrap up session for about 12 minutes now. Hope you guys enjoyed it. Did everybody here get to see every single talk. Yeah. You guys liked it. Are you expecting were you expecting something different or does this kind of meet what you you know thought you wanted to see in a scale admitted talk. Yes. Go ahead. All right. That was great. It was great. Oh well. I'll let the I'll let the speaker know. So in this case. OK. So yeah you want more less less panel I suppose because that's probably the only way I can probably think about C plus plus and C because people are just going to want to battle it out. Right. Perfect. OK. So like something like a pitfall talk would probably fit in well. You know. Yeah. OK. And anybody else do you have any what else would you like to see here next year. Oh we could probably try. OK. Yeah. The micro view I think it's called. OK. OK. In fact I can I can probably see the minnow board guys go through that whole because they basically he basically started right. Gave you a little idea of what how hard it was to assign that board. But they didn't really go into the let's start. OK. So we had this idea and we thought about contacting these people to either make the board and such right. That that is actually a very good idea. So that that is like trying to find a manufacturer first of the birth of a feather. It's basically just round. You sit around with other people and talk about that same subject that might actually work better for this case because yeah trying to find a manufacturer for board is. Yeah. No. Well you can find way too many out there and you're just not going to know which one is going to be good for your typical your particular design and board and if they're going to deliver what kind of load they have right now and do you actually have to go to China to see the board in progress and build it and everything. OK. Yeah. Like really flush that out then that whole process. Yeah. That's what I'm thinking maybe maybe like an actual sit down for a couple of hours and try and see if you can you know get an idea of how long it would take to make something like an Arduino and you know from start design and concepts rather and you know out into production would would probably be a good idea to have. You had a question. OK. Yeah. I think the OK. All right. So the idea I'm getting is more hardware specific topics. Right. I actually get in in there because he brings up a very very interesting thing because sometimes even if you just connect your multimeter onto a circuit it'll just work magically. So that sort of stuff is kind of hard to debug. But I could see maybe a few sessions of people just going through different debug procedures and software wise like JTAG and you know all that stuff and somebody in hardware you know your oscilloscope best best way to you know maybe sniff out a bus in a system or something. OK. We had something like that last year. Right. I think David gave that talk. Yeah. Well we can we can just rehash that and add more. Yeah. So that one I think that's on the interwebs right now on YouTube. And they do touch on the what a JTAG is. You are. How do you look for you are some toys for example stuff like that which is something that you guys probably like a lot. So yeah you probably find that but I'll obviously let let everybody know that you are looking for more more talks like that. Anybody else have any other idea and anything you want to see in this track. Besides you know keep making it. You have a debugging kernel so it's fun isn't it. Yeah. Do you know why he didn't show up this time. I think we were all guilty of that. I had like two minutes left before the abstract thing closed down. Well there you go. OK. No of course so more hobbyist type project. Would that be it. That's that's actually a good idea. Yeah. That would probably bring in more people. Not just geeks I'm sorry. OK. I like that idea. Yeah I'm sure we'll probably keep the security talks up there because it's always a great topic and you know scaring people so we you know it brings in the crowd. Yeah. The only thing I in fact I wanted to bring up that ship whisper board they did went on Kickstarter. Didn't anybody. Fund that thing. The chip whisperer. You can do like site channel attack on board. Basically you know try and see if you can keep resetting a CPU and try and glitch it into doing different things stuff like that. Yeah. Yeah. Basically it does it does that but it tries to see patterns in for example the amount of current the CPUs are using a certain time. So you can try and break something like a lock in that case where it's or an encryption key and try and bypass that the actual finding bit if you reset it in a certain time. But it's it's like a really complex board but it I would probably like to see somebody talk about that. I don't think I have the skill to talk about it myself. That's also something that. Yeah. On Android I was yeah like and Android and Bluetooth low energy. Yeah. That's great. Well you can do it talk next year and that. Yeah. I haven't been able to play with any board that has a USB 3.1. I don't know how more power it's you know that USB like 2.0 has certain levels of power it can actually give you out to your device. This has more speed and more power basically. Yeah. Yeah. OK. Any more ideas. You guys happy with the embedded track then. You're going to come back next year. OK good. Don't forget tomorrow we're going to have more talks. Not totally. My talk is going to be embedded related but to game consoles which you guys are probably going to go to. You obviously do. You have to. And I guess that's it. OK so I hope you enjoy the rest of the scale event here and you can find me I'm going to be Tom for the rest of the day. You can find me and I'll be and I'll be I'll write down your ideas for next year. OK. OK.