 the hardware is full of features. Features. Random, yeah, it's a security feature. Because you know, if you have to reboot often, you know, you know, you patch and then you have to reboot to make sure that everything's clean, right? Yeah. So you've got a question back there. Oh, yes. So here's another really exciting thing about hardware. And in fact we ran into this on the middle board. So the comment was, you know, there's documentation for the hardware and then there's the errata. And sometimes there's the errata to the errata. And sometimes there's the errata to the errata to the errata and we just threw it out and started over. Case in point on the middle board, if you take a look at our documentation for the middle board max and the middle board turbo, there's a pin on our low speed header, pin 26, which is going to become the bane of my existence unfortunately. When we first built the board, we had intended to use it as an M clock for audio signaling. So this is a reference clock between the SOC and the audio codec so that thing was supposedly lined up. You don't have to have this but it's really, really useful. Well, we picked a signal, it was supposed to work, then we found out in an errata, it did not. We also ran into it on HDMI where, you know, we designed the board, we got it out. You know, the monitors we had all initially plugged it into just worked, everything looked great, we shipped the board and then we started getting reports of it not working. We were like, huh, that's kind of interesting. In the errata to the errata, we found a footnote, a hardware guy literally mashing his head. I wish you guys on the camera could see that he's like, oh my God. That if you are on, the process that our particular SSE is, you have to put a level shifter because the signaling is too low. If you are on the previous process, you didn't need this. And again, when you're, you know, we as software people, we assume the documentation is right. It's never right. Yeah, so do I, but I'm not mentioning who they are. Yeah. Well, and that's a really good point. So the, for the recording, the gist, and correct me if I'm wrong, is that we all assume that this is going to work on the first or second try. You know, whatever hardware we're building. The reality is, is if you get it right on the first try, good Lord, you should get a bonus because I've only seen that happen twice, ever. Most of the time, you get it right on the fifth time. That's the time when it boots the first time but nothing else works. Tenth time is when your Ethernet controller works. And the fifteenth time is when you realize that, yet again, everybody has screwed up how serial URs work. Because, you know, serial URs, we've only had these things for 30, 40 years. And we're still getting them wrong. I, I really don't understand how everybody screws up serial URs because they're the most useful thing in the world. But apparently nobody can build serial URs correctly. But yeah, the, the, I, I think even on the, the turbo, which is just a minor set of fixes, we went through four, three or four revisions before everything got ironed out. So, yeah. I mean, oh, okay, yeah. The reworks, the white works, or the white wires. Yeah. That, that's just a proving that your fix in the next board will work. That's the X USB maintainer. USB 3 maintainer, I'm sorry. Yeah. Yeah. So, for the people on the, the video stream, the short answer on that one is the internal documentation was wrong and you had to reverse engineer your own hardware to figure out how it worked. The statement is that's normal. Oh, and there's a working driver that you had to. So, yeah, I mean, I, I want to hear your guys' stories because, you know, I can, I can stand up here and remember random things for the rest of the, the, I think I've got 20 minutes or so. But honestly, I want to hear what you guys are, are, are finding in the field because my stories are good. Your stories will be better. So, yeah, a meta point. That's because they were, to be fair. Yeah. I mean, if you, so the, this is a call to action that the hardware simulators, we've got some reasonably good ones. QMU is pretty decent in that respect, just from a simulator perspective of a whole system. But just being able to simulate low level hardware is nearly impossible and it's really, really computationally expensive. If we can simulate more, we can do hardware better. So if you, if you're looking for an open hardware project or something or an open source software project, go start taking a look at the simulator as they could use a lot of help. So, yeah, there are people building their own CPUs and having reasonable success in that side of FPGA. You should do as much prototyping as is humanly possible before you actually try to build anything. Whether it be throw it into an FPGA and run it at one megahertz, because at least you can still walk things very slowly, you know, it might take you three weeks to boot up. But, you know, you'll at least know that it's right. So, yeah, I could not even remotely give you any good suggestions on operations, on shipping. That is a really tough problem and that is, that is almost worth its entire, its own entire conference. Because operations is, farm it out to someone else because you're going to screw it up, to be honest. CalCen. Regulatory stuff, I mean, well, I mean, if you're actually going to do a product, regulatory stuff is a really serious and big hurdle and this is something you can't ignore. If you're, if you're intending your device to end up in, you know, normal people's hands inside of a house, you've got, at the very least, Class B FCC certification to deal with. You've got environmental stuff that's got to be done. This is a non-trivial cost and, you know, when you're building one board for yourself, these are things everybody ignores because nobody cares. I mean, to be honest, you shouldn't, you know, actively try and build a board that's radiating out into the stuff that shouldn't be. But, you know, you know, your one board probably isn't going to make a big difference. But, you know, if you're building a thousand of, you know, a thousand of these things, yeah, you have to, this is something that most people forget needs to be built with. Mm-hmm. That's a mechanical problem. And a lot of that particular, so the question is, is, you know, how do you deal with the mechanical issue, or just the, the componentry of connecting the, the layers, the plastic layer. Some of that's just trial and error. Some of that's getting a good mechanical engineer from a, from a hardware perspective. Again, from, you know, me as a software guy, I hire people to do that because I suck at it. People see my 3D cases. So the question is, is, you know, not trusting the documentation and doing a lot of testing up front, is that a step forward or backwards from the, to the software world for how all of this works? I think trying to even, I, I think at any point if we ever, you know, we as software people ever trusted the docs from a hardware, or for a piece of hardware, we were idiots. So I'm not sure that it's so much a step in any direction. It's just the status quo. It's the way it's always been. I mean, I, I can ask the heart, you know, the people who have done the low-level hardware, driver development, if we've, if you've ever seen a piece of hardware that didn't have 20 pages of a data because it's all broken, and I, I don't think any of them would raise their hands at this point. So I, yeah, yeah, they're, they're usually adding to the errata because they try to bring something up and they find that it's still broken. Well, yeah, I mean, unless you can see the code for what you're working on, and you can verify that, you know, the function that you're calling into from a software perspective is doing what you're expecting, you should be suspicious of it. If you, if you've got a black box, you know, you should be testing against that black box repeatedly because you don't know what's going on inside of it. And then, you know, in a lot of hardware, even if you've got the documentation, it is still a black box. You know, you don't know what's actually happening in there. You just know that if you twiddle this GPIO in this way, this result on a different GPIO should happen. You know, from a software perspective, if you're calling into that function, you can't, you know, literally open it up and look at it. And in a lot of respects, how many people really want to go and read through every code of a library? I mean, we, we all just kind of take it as, you know, faith that, you know, GLibC is going to go and do, you know, what we're expecting in the general sense. Now, faith is, you know, generally well-founded because we've been, you know, doing this for years. Smart people have been looking at this and when we find bugs, we submit them or we go digging ourselves and we fix it. But yeah, you should be suspicious of, you know, anything that is not your code or anything that you're interfacing that is not yours. Yep. I mean, the, the, the rise of, you know, I want to call this home manufacturing laser cutters, 3D printers, these kinds of things. If you're, if you're going to be doing mechanical designs, get your hands on good ones of those. If you want some recommendations, ask me afterwards. And just, they'll be expensive but it's worth, it's worth it just because of the faster prototyping for your own cell. So, you know, getting back to the micro view in this case thing, they did do a lot of 3D printing and testing, test fitting. Well, your turnaround time is so substantially higher that you can iterate faster, which means that your upfront costs are expensive but your overtime costs are actually better. So, the question is, is it actually better to use components that you can update after the fact in your designs versus, you know, static components? It's going to be really subjective on what component you're talking about. The advantage of being able to update things after the fact is that, you know, likely that particular manufacturer recognizes that they're going to ship a bug in their hardware and that they're expecting to be able to fix those bugs, whatever they may be, because nobody really wants to, you know, upfront, I'm going to ship a bug that will be, this pen will be stupid in this way. I mean, I, I mean, how many people want to ship a piece of hardware that's fundamentally broken and you know it? Hewler? Hewler? Okay. Nobody wants to ship that. So, if you have a reasonably complex, you know, circuit or a system, having an update mechanism after the fact is almost essential. You know, I will pick on a, you know, take a look at any CPU these days. They all have microcode or, you know, or a reasonably complex piece of hardware. They all have firmware or microcode or something that you have to upload into it. Probably a non-trivial chunk of that is just working around bugs in the hardware that they, you know, the hardware folks have specifically gone in and built in more layers or more complex circuitry because they know that, well, maybe this, you know, maybe this feature that they're doing is a little on the risky side. Maybe it'll work, maybe it won't. And they can, you know, through software they can enable or disable it or even, you know, work around bugs. I mean, take a look at the news. There's been a couple of things even recently about, you know, effectively a software fix that's been pushed to fix very specific, potentially nasty hardware problems. Yeah, the statement is, is if you could ship everything as an FPGA and keep the cost down and keep the speed up, the world would be perfect because everything would be fixable. Unfortunately, FPGAs are not that, are, are not quite that awesome. They're awesome. Don't get me wrong. They're expensive mostly. And the, the power requirements on FPGAs is crazy because of how they work. So the question is, in particularly out of Shenzhen, you've got folks who are turning around boards or prototypes in hours, not even days or weeks. They're, they're talking about doing this in hours. And the, the correct answer is, is in the United States, we probably can't even remotely compete with that. Well, we do, but it's, you know, at least there are some folks here in the US that are doing it. The, the real way that we kind of level that playing field is, you know, it, it may just be easier and cheaper to manufacture in China, in the general sense. Particularly from a volume perspective. I mean, I don't think there's anybody in the room who would try and, or who can, or who would say that it is not generally speaking cheaper to manufacture over in Shenzhen for a large run. Small runs, that, the math changes dramatically. But one of the things that we can do to level the playing field for everyone, whether they're here in the United States or in China or Japan, Korea or where not, is actually start taking a look at more open hardware. The minute that we democratize hardware more, the more that everybody gets, you know, faster turnarounds whether they're in Shenzhen or here, but it also means that we're taking advantage of everybody else's knowledge. So I, you know, we, we, I've said a lot about how we as software people, I mean, we're really good at open source. I mean, I don't think there's anyone in this room that would not agree with me that we've got open source kind of figured out. You know, we've got the Linux kernel and we've got, you know, conferences that have anywhere, you know, that have thousands of people who show up and want to talk about giving shit away. I mean, it's kind of hard to argue with that. The hardware world is still waking up to this. You know, we've got, you know, new initiatives with the open source hardware alliance and we've got, you know, some other things along those lines. And we've got boards like the middle board of the Beaglebone or, you know, a lot of the Arduinos and those kinds of things. They're all open hardware. Everybody can go, they can look, they can compare, they can build from it. This gives everybody a really great chance to go, oh, well, you know, this platform doesn't quite meet what I need. I only need a few little changes. And a lot of what you're seeing in Shenzhen is, you know, they're not necessarily making a whole new board in that, you know, that day spent. What they're doing is they're taking something that they've already got some relative, you know, that they've already done. And they're making some modifications, running it through quickly because they know, because they've done a lot of the work already. They only have to do the incremental work beyond that. And the open hardware stuff does the exact, you know, effectively does the exact same thing. By only having to deal with the incremental cost beyond what you're doing, or beyond what's already there, you shortened the, you shortened the timeframe to, to get from idea to an actual prototype or even to mass production. I mean, how many people in this room think that they could lay out GDR3 or GDR4 memory? You, my good sir, are a very good person to have on the team. Because, because I couldn't. I can't even begin to, to have, it worked the second try. So, and that's actually very good. You know, working, you know, getting, I mean, these are things that literally have to be the exact same length. There's not, your margin of error is so small. For those of you on the, the, the video, he rolled his eyes very greatly when he said the CAD software does it for you mostly. And, and they try really, really hard, but CAD software makes a lot of assumptions that work for very specific sets, but it all, it's, you know, even laying out a board at the end of the day is still a, a, a piece of art. It's not a mechanical thing because someone still has to go in, verify everything, and then we'll still have to tweak it because sometimes the software gets it wrong. Like, you know, taking your memory channel and routing it literally around the board twice. Yeah, that, that, that, it happens. Yeah. So, yeah. Oh, God, yes, please. So, the, the, the question, or the statement is, is should we involve more software in our hardware development processes? And the answer is unequivocally yes. By and large, hardware is made by hardware people for hardware people, and they don't understand software. I'm seeing a lot of people going, yes, yes, please. It, it, it's fascinating. I mean, if you, take a look at any chip, particularly if you, you build anything, and it's made for hardware people, by hardware people, and, you know, they go, oh, well, you know, the, the software guys can do anything. It doesn't matter what I do on software, or in hardware. And, and sometimes making very small tweaks makes the software so much easier to deal with. Or to, you know, oh, they don't need to be, there doesn't need to be a signal to, to let you know that the hardware is done. You'll just pull, pull it all the time, right? Yeah. Yeah, status registers are stupid. Just, just, yeah. I mean, oh, right, only registers. Yeah, you, you don't want to read them back? And why, why would you ever want to, just write it over there. Don't, you don't need to read it back. You don't need to confirm that what you actually wrote is what the system thinks you wrote. Oh, yeah. Software's never buggy. Compiled the first time. None of you are, you know, that, that guy in the XKCD on the chair playing, playing swords, because it's compiling. So, I, I, I guess in some respects this is, you know, we as software people need to, to take a look, a better look at how software, our hardware works and we need to account for that more in our own processes. And if you're on the hardware side, please for the love of God, start talking to your software people more because we, one, we don't understand when we need to and two, we need to share our software knowledge with you guys so that you can build better hardware so everything's less broken. I can keep answering questions. I'm going to have to run over to the middleboard booth here relatively shortly. So, if you want to have a more detailed discussion, we should probably do that over there. Otherwise, thank you.