 I'm guessing you're all here because you have a project that you're working on or looking to start one that requires building custom hardware. You've already done this before and are looking to either do this again or maybe come as a rate with someone else who's gone through the growing pains. Or you saw the name Virgin Galactic and it piqued your interest. Well, regardless of why you're here, welcome. We're going to focus on the steps that you need to go from project conception to having hardware and operating system that you need to support whatever desired software application you're creating. So I hope you find something interesting to take away from this. So let's go ahead and get started. So first of all, let's start off with a very important question and I'm going to assume that you're one of the people who's looking to get a new project moving. And the question here is, what is it that you're really trying to build? After all, building a logic system isn't usually the end goal unless you're doing a reference setup. But there's something else that you really want to be creating. Are you building a consumer device? Some Internet of Things monstrosity? Wait, what conference am I at again? Some appliance, a test harness, a flight computer. Knowing this, what you're building is really what's going to define the sorts of tradeoffs you make. And it'll be good to keep this in mind as you're going. So regardless, you want to keep your focus and avoid getting mired in tool creation as you're working in a small team by making use of these sorts of existing tools. So how do we start? What do we even do at this point? It can be daunting to find this starting point, so let's at least start from the basics. Hardware takes probably the longest time to revise of anything. Software in comparison is easy. So you want to get the hardware right as early as possible. And if you're lucky, if you're in the development process, the hardware hasn't actually been set or determined yet. So you can take the time to figure out what is actually needed. As someone who works on software, you're going to want to make sure to get as involved in discussion as early as possible because hardware designers often get buried in data sheets and forget about the software development time that is affected by the decisions that they make. And the last thing that you want is an interface designed by the hardware designers that doesn't work well with existing software interfaces. And so at least to get moving in the right direction here, you'll end up starting off with what vendors usually provide different reference boards to get started from which are intended to show off a lot of the features of the hardware they sell. Regardless of whether it doesn't actually fit your specific requirements, it's at least someplace to start. And I just realized I forgot to advance all the slides, so sorry about that. So one of the things that's going to affect what you need is depending on what your team's current level software expertise is. If you have, for instance, a seasoned developer who's done a bunch of hardware and software bringups before, your requirements are going to be less than if you're working with a team of largely novices to this. But while that will affect your timeline, it doesn't really matter because if you're a small team and starting out with a new product and you're working with limited resources, you want to save time where you can, and that means starting with strong foundations. When you're a small team with limited resources, this is of paramount importance. Without building on the work of others, you'll be setting yourself up to fail. And unless you're buying a product from a completely new vendor, you'll realize that all of the projects that they have been created from what they've sold before have had other people using them and have had the chance to learn from past mistakes. So one of the ways you can make use of other people's errors and what's been learned from that is to make use of the vendor-provided software development kit and or more support package depending on the level of what it is that you're doing. And it's likely based on one of the following. Whether it's using Yachto embedded, OpenWrt, Buildroot, I'm probably going to annoy some people here by saying that it really doesn't matter because I know there's a lot of people in different camps. OpenWrt is obviously better for routers or Florian doesn't seem to be here right now or else I would bug him more about that. Whether you're using Buildroot because you like it because it's simpler, but if you're a small team starting out and you're working with new hardware, you want to use what the vendor gives you because they've tested it before and you can at least get started with something that will work. All of these share some pretty common things and this is why it doesn't matter too much. They have a large amount of commonly used software and you can get started with that, but in all of these cases they're probably going to be missing specific things for whatever niche you're designing for. So you'll probably be okay. We're getting some of the core utilities or some replacements like Busybox or other things that everyone needs like NTP, but if you want specialized math libraries because you're doing a certain flight, that's not necessarily going to be built into whatever you're trying to use. But in all of these cases, it's pretty easy to add in additional packages. In fact, the designers of these want you to do that frequently. So while some of the more elaborate setups will give you... Yachto contains seemingly everything including the kitchen sink, all kinds of different graphic libraries and some of them are more minimal. There's at least some components that are not going to be available. And all of these things you're going to want to make your additions in a way that is going to be easy to port forward later. Yachto, for instance, makes this particularly easy to do with its layer system, which is probably the reason that some of the vendors use it. It certainly saved us a lot of time at Virgin Galactic being able to have internally use our own layer and use that to port forward as new versions come out. But it's a way that you can save some effort there. And really, you don't want to ignore these tools. The vendor has provided them for a reason, and even though it might feel like a toy example when you're just setting up a reference board and saying, okay, great, I can get this reference board working, how does that really translate to what I'm actually doing? But these tools actually will grow with your needs, unlike this toy toolbox. An enormous amount of effort has gone into building all of the build systems you use, regardless of how clumsy it might feel at first. And as tempting as it can be to try and re-implement something because you like this other build system better, you're going to end up repeating the same mistakes as other people and really not creating the product that you are trying to do. And again, another stumbling block here is Yachto in particular has some very confusing terminology in case people end up throwing around terms like Yachto, open-abetted, pokey, bit big, et cetera. And it can be a little off-putting until you've gotten familiar with what all of that's talking about. But you don't want that to be the stopping point there because one of the things that you really get from using whatever a vendor provided stuff is really support. You don't want to discount the amount that having a vendor support staff on your site can help. And I know how painful it can be sometimes to feel like you can be talking to your vendor and you get some first-level support staff and they don't know what it is that you're asking or whatnot, but it is still better than having nobody at all most of the time. And while there's a lot of documentation available for all of these projects, there will be questions that the documentation won't cover. And someone who is very familiar with your tools can help you learn them. And worse yet, sometimes bugs are deliberately left undocumented because the vendor doesn't want you to know about them. However, their support staff probably are very well aware of this. So some things to kind of watch for when you're selecting something that is in between what a vendor is doing is that, for instance, does a vendor provide thorough hardware and software release notes? Looking through these sorts of release notes can give you a really good idea of a vendor's communication level. Like, how do they communicate with you? What level of transparency do they provide? Do they provide access, for instance, to patch notes that can be extremely valuable to audit if you have the resources or the need to do it? You want to know the delta from whatever upstream software projects you're working from because anything that's different that the vendor has provided probably has undergone less review. This is particularly important if you're working on any sort of safety-critical systems. Is there any sort of direct support channel available? You want to go through a reseller. And your needs are probably going to differ because as a smaller organization, while it might sound better to have a direct support channel, it may be very well that in the case that you're working with an extremely large vendor that isn't going to listen to you or respond to you if you talk to them. I can think of a few off the top of my head that I won't name right now. And having a reseller that you go through means that they might have a lot more cloud to whoever the vendor is and be able to get you to the right support people faster even though it might sound more inconvenient. Sometimes it's the other way around. And next, how long is software support normally provided for a chip set? And not just the chip set. You also want to consider all the different components on the board and make sure early on that all of everything that you're looking for is actually support. And don't just consider the big stuff here. Really, it doesn't really matter if your SoC is supported for, you know, 10 years if some smaller chip on your board is only supported for the next two and suddenly there's no replacement that will fit in without having to do a board re-work. So now let's kind of go back into the common features of the sort of vendor provided SDKs and VSPs. Because you're probably thinking at this point what if I have custom hardware? This is all well and good for getting some reference board but we're making significant modifications to the hardware for our application. So how does this translate into building our own stuff instead of using an off-the-shelf turnkey solution? And, well, these reference designs are your friends. If you're a small team, I really hope whatever your design is that is based off of a reference design because if not, you'd better have some serious in-house expertise or you're going to fail horribly. But we're talking, I mean, this presentation is geared at small teams, so if you have that, you're fine. But in which case, what are you doing at this talk? You want to make sure to have a reference board first. I can't stress that enough. It's your baseline and especially since you won't likely have fully working hardware immediately, it's what you need to get started. At Virgin Collective, for instance, we started by getting a reference board from the vendor and getting an application working there with some of the required features set. And that means that we can only worry about developing for any sort of newly-ed stuff later rather than trying to do everything at once. And this has allowed us to save enormously on development time. So once you actually have the hardware that you're planning to use, even if it's only in the early development stage, you can at least start doing comparisons between the reference boards. And you'll want to keep track of these differences and note the impact on your project. The more things that diverge from the reference, the more work that you've got cut out for yourself. And the more important that this checklist becomes. So you want to write everything down. Or, better yet, use an issue tracker. This seems like it should be obvious, but you'd be surprised how often this doesn't exist. Especially in hardware design land. So you want to at least have something, whether you're using something like JIRA, Red Mine Track. It doesn't really matter what it is. Get it in there. For instance, what we do is that all of these sorts of differences get turned into feature tickets with subtasks as we need to have them. So that way we don't lose track of them whenever garbage is changed. And you can put together an established process to do this. And it also helps out to start planning out the schedule to work on each of these changes. And gives us an idea of how long it's going to take to work on each of these items. These sorts of time estimates can give a sense of what impact would be if any of these change, which is particularly likely at a small organization. And this lets, in other words, letting you really reduce, cut down on scope creep, that most dreaded of two words that you'll ever hear while you're trying to meet deadlines. And because the last thing is you want software development time to be the surprise in the chain. Ideally, you push it all off on hardware. And so... Sorry. So once you have your reference design for comparison, you're going to want to take note of some of the specific ways in which your actual board diverges from the vendor reference. For instance, if you have a similar board schematic, then there isn't too much change, except for any sort of added peripherals. You're likely to at least have the same process or memory as the reference board. You might go up and down that, but it's going to be pretty similar. But it's very common to have, for instance, a separate, a different setup for storage and most other peripherals. And something that more closely matches what your program needs. And you're probably not going to see as many changes, even though you'll probably be reduced from what the first iteration is. But presumably you're building something interesting. There's going to also be at least a few sensors or controlled interfaces connected, whether directly to this board or through some more indirect means. And to get a little more specific about what you'll have to do, you're going to want to figure out where each piece of hardware is connected. So whatever pin setup that you have, you want to be able to alter, chances are you're not going to use the same setup as whatever the reference design is using, either because your hardware designer has decided that it's much more convenient to do it this other way, or it's actually impossible for them to route it similarly to the way that was done because of things that need to be added for your application or just because it may be some cost-related reason. But what it comes down to is most SOCs support multiple configurations, and the reference boards often just don't have something that fits what you do, because they really are trying to show off different features of the board rather than, you know, make a final product. So once you've taken down all of this, what you're going to want to be doing is updating the device tree on most device tree platforms, board files, and others to describe what the hardware is that you do have. And while the details of updating that are pretty out of scope here, the reference design is going to have one, and you're pretty much going to be basing it off of that to just describe where your design differs. And then once you've kind of gotten that in, you can update, add in whatever drivers that you need for your added peripherals. And this is where getting involved in the hardware development process early on can really help, because you can work with your hardware designer to make sure that to find out exactly what sort of peripherals they're adding, whether there's any sort of upstream drivers already available in the other kernel version, and really find out then, does the kernel version that you're planning on basing on already have drivers for that? Are you going to need to backboard them from a newer kernel release? How difficult is that going to be? While in an ideal world, you would always be working with upstream, as if you're following what I was saying earlier about using the software provided by a vendor in order to save yourself time, chances are you're not going to get exactly the latest. So you'll want to make sure that you have that. It's a little bit... You're going to have more work cut out for you if you're doing more cutting-edge stuff. And on that note, you're doing a lot of kernel hacking. You're going to want to use a separate kernel repository if you diverge significantly from the reference, so basically having your own downstream tree. And the reason for this is that while all of these build systems provide a method for adding in patches on top of different packages, if you're adding in a bunch of drivers, this is rapidly going to become unmaintainable. Because as good of a tool that Quilt is, for anyone who's used it before, it's nowhere near as capable as actually using a tool that's designed for this purpose. And you'll end up spending a lot more time doing that than just using a tool that's good for that. So once you've done all of this and finally put together a system that supports the hardware you need and whatever else, you're going to actually... And whoever's working on the application side, if it's not you, has already been taking some time to work on the parts that they can in the meantime. You're going to want to start integrating that into your main build system. And this is obviously hugely dependent on what it is that you're trying to build. But most interesting applications are... There's a lot of common features here because most interesting applications have some number of dependencies between them. So you're going to want to note whatever compile and run time dependencies you have. And in the likely case that you're building something that shouldn't require user interaction to start a program, you'll probably want to be creating a service that starts up when it's intended or possibly even read starts itself when it goes down. Also we're thinking about early on is how it is that you want to deploy. For instance, for us at Virginia Galactic, one of the key things that we follow is test as you fly, which means our one of our goals is to minimize differences between use and development versus production, which is for us the flight environment or working on a test stand. And that means that in this case we actually don't want to have differences between what you have in all parts of your deployment. But if you have a different case where, say, you know that your target system is going to have a smaller amount of memory or storage space than what you're using for early bring up or for debug, then you might want to have multiple different builds, one for development and one for production, or just provide a setup that's easier to use for debugging. Essentially taking advantage of whatever options the build system provides that can help improve your workflow. And particularly note examples that these build systems can provide for making a package repository that your target machines can grab from. I know that at least both OpenWRT and Yachto make this pretty straightforward to create the package repository. I think OpenWRT is mostly O-Package and Yachto seems to be able to use whatever you want, whether it's RPM or O-Package or something else. But at any rate, you can use that in order to get your other developers moving faster. Speaking of packages, though, none of these is really a substitute for instituting some simple best practices. So now that we've kind of got all the things put together for how you want to build up your system, some things that you want to do is to use version control. Another seemingly obvious thing that seems to fall by the wayside when it comes to build systems, because it seems at this point that a lot of people have accepted that version control is a reasonable thing to do for a lot of application software, or working on the kernel or anything like that. But then when build systems, it turns out that people are like, oh, I received the tar ball or a zip file. Clearly, that's the fine way to do things. Let's just make copies of directories. I can't emphasize enough that not using a version control system is asking for trouble. You don't want to fall into that trap. Your teammates are going to make mistakes, and you're going to want to make sure that you have something to come back to and be able to figure out what went wrong and when. But even better than just using version control, because anyone can just throw things in a repository and do have something that's nearly useless, is use version control like the upstream project does, whether you're working with open embedded or open derivative or whatever else. And that usually means making small modular commits, reading the commit log to find out how they've done this in the past, because all of these projects have had people learning from the past mistakes, essentially, and put together an established process for doing this. And doing acting like upstream helps in a lot of ways, because you also get to benefit from possibly pulling in new changes from the upstream project and making that easier, or pushing back in the other direction, which can be a hard sell sometimes, especially if you're working in an industry that traditionally hasn't been doing this. Which can lead to this question, which I mean, people give entire talks on this, I'm sure there's at least one here. But it's like, why is upstreaming important? And at the high level, it boils down to you get reduced maintenance costs over time. It pays dividends if you do it sooner, because importing local changes required to support whatever hardware you're creating becomes a lot easier. You start seeing higher code quality, because when you've got more eyes on the code, upstream reviewers are going to know the code base better than you probably, if you're working on a small team with only so many areas of expertise that you can have, and can tell you how to improve more specifically what you'll need to change before they'll accept it. And honestly, the one to really sell to your higher management is that it's actually just really good, low-cost, positive PR, because after all, your organization is benefiting from the work of others, and it's just kind of the right thing to do to also give back to the community. And really, ultimately you're using software that is built in collaboration of a whole bunch of people, like a lot of them who are here. And if your organization isn't involved in this process and your competitors are, you lose out on an enormous opportunity to really get involved in the direction of the project in a way that is beneficial to you and your company and whatever it is that you're doing, while anyone else who's using the same thing gets to take full advantage of being involved in this process. And over time, the cost to maintain these local changes, if you're not involved in this, is going to add up substantially. I know one project that I was involved in a few years ago had a comparison between a few different groups, one of which was heavily involved in an upstreaming process and one of which was not. And the time delta between the upstreaming process was on the order of months. So you want to be involved in that. And speaking of local changes, you're going to want to take steps to make sure that upstreaming is easy to manage and one great way to do that is to set up a build system for your build system or a continuous integration system to handle these sorts of things. So it doesn't really matter what tool you use to do this, whether it's Jenkins, Buildbot, or whatever, you want to have something. Because especially with fewer eyes on it, it's going to be in a small team, it's going to be easier to make these sorts of mistakes. So you want to be using an option to use local mirrors for whatever build system you're using. Because you don't really want to be dependent on any sort of third-party sites and say one website goes down and your build breaks. It's not really what you want. So doing things that way can really help. If you take advantage of whatever sorts of shared caches your build system provides, this can also really cut down on build time and reduce the time that developers are waiting to find out, hey, when I made this change and I submitted it, did it work? Is this ready to be reviewed? And there's just an enormous value that you get from having working automated builds. You can make sure by doing this that there's almost never actually any broken builds. Because if you don't do that and you end up with broken builds, it's going to make it a lot harder for anyone who's tried to track something down and tried to use bisect to solve a problem and found out that a whole bunch of builds during the bisection process failed. I know you're pain. And because bugs will be introduced if you're trying to move quickly and you'll waste a lot of time tracking down the problem if a bunch of commits don't build. You want the history of what you do to work for you, not against you. And for tracking down bugs, that's really the only way to do it. And lastly, deploying to a TFTP server using this sort of setup can make sure that everyone on the network on whatever network you're using can easily boot the latest version and save time that way. And finally, the last common sense item that I'm going to add here is to do... Make sure you're doing code reviews for this. For some reason, this is also a common blind spot when it comes to build systems. Usually it's considered obvious that code reviews are necessary for applications and kernel development. But a lot of people seem to feel that distribution build system recipes, scripts and so forth, and everything associated with that are somehow merit-special exclusion from this. But if your operating system build system doesn't work, it doesn't matter if your application is perfect. Your product still doesn't work. And another selling point is that the overhead for doing this is pretty low when you're working with a small team, less than five people. Code review can go pretty fast this way. But if you start getting large teams with diffusion of responsibility, in some ways it's good because you have more people to do reviews, but in the other sense, it starts meaning that the volume just starts increasing, and it can be more difficult to figure out who should be doing what. And if you're going to be doing reviews, you want to make sure that there's at least common standards that everyone's aware of, whether this is going to vary by project, of course, but if you're making patch modifications to, say, Busybox is going to be different than you're making a patch modification to the kernel or for GLC or for, you know, one of the different math libraries. So you want to use whatever code style is appropriate for that. So you want to streamline this process using various checklists and scripts. Anything you can automate is good, whether there's things like checkpatch available for the kernel and just really make sure that the scope is recognized by everyone involved who's doing reviews. So in summary, if you're putting together a NE system, you want to make sure that you're involved in the hardware design early on as a software developer, so that way it reduces the number of surprises for everybody. You want to make sure to use whatever reference boards are available and use that to determine how much delta and figure out how much work there is to do and how long it's going to take and use the vendor SDK because they've already tested it out with most of the setup that you have, even if it's not identical to what you're building. You're going to want to use version control because that's mentioned, history should work for you and not against you, and you want to work with upstream projects as much as possible because that's just going to save you time in the long run. And if you're trying to build a successful project over several years, you want to think in the long term. So that's all I got with that. I hope you enjoyed it and it should be off to a good start. Anyone have any questions? Yes, please. I've seen both. So the question was, if someone is upstream, do they typically go through their vendor or elsewhere? I'd say it depends on whether that vendor has a habit of moving things upstream or not and whether they're interested in being involved in the process, but it's definitely a case-by-case basis, at least in my experience. Anybody else? Going once? Going twice? Okay, great. Well, thank you all for coming. I really appreciate it.