 I'm going to get started. I don't know. People seem to still be filtering into the room. I'm Sam Pavlis. I work at Boll Airspace. I don't know. A lot of you probably recognize our logo up there from the glass jars. That's not us. And I'll get into that a little bit later. So I had a professor tell me once the best way to say to people is to tell them what you're going to tell them and then tell them. So I'm going to start with that. I'm going to start with a little bit of an overview of why I'm here, what I wanted to get across on this presentation, what I was going for, when I signed up to speak besides the free entrance. An overview of the company I work for. Boll Airspace is a subsidiary of Ball Corporation. We make cans for most of our business. And they're pretty interesting, but probably not from a Linux perspective. Aluminum cans and Linux, not really the same thing. I'm going to give you an overview of my Linux use and my career. Because I want to make a point about how I use Linux and how people like me use Linux. I'm going to give you an overview of the face array antennas in general, and then the face array we were building. Give you an overview of our hardware. Tell you why I wanted to use Linux. Then give you some Linux lessons and some general embedded development lessons. I know that a lot of you probably already know these things, but I think it's important that we talk about them. And then I'm going to go over some of the advantages I found when I was actually using Linux. So why am I here? I really wanted to, I was really struggling to pick an audience for this presentation. So I hope I hit the people out there, and I get across what I wanted to get across. But I hope you get something out of my presentation. I think I'm largely going to be targeting other embedded developers that use Linux, but I do have some things in here for developers, the people actually producing Linux. There's a lot that you guys do that I really appreciate, and I hope that I can help you understand what I need out of your product. From the embedded developer side, I want to talk about, at ball, Linux is considered a huge risk. We are historically, our real time operating system company, we run either bare metal or something like VX Works, very expensive operating systems. And to propose Linux on a project at ball for me was very risky. I am going to tell you that in my case the risk pays off, and I think it in general will, but it depends a lot on your project and your skill and how much time you have. I'm going to tell you that you'll do a lot more reading of code and reading of documentation and a lot less writing of code, which I think is a really good thing. There's a lot of smart people out there writing code. And if you're using their code and not trying to write it yourself, that's a huge advantage. And I'm going to promise you that things will come together, but there's no way you can come back to me if it doesn't, so that's what I'm going to say. So ball, we started out making the glass jars used for canning. You've probably seen them. You've probably seen them around quite a lot. They're quite common. We don't actually make those anymore. We make the aluminum cans to get your beer, your soda out of. And then for some reason, they bought a technology division, and it spun up. It didn't really help with the glass jars, but it helped with satellites. So we're building satellites now. It's a small segment of ball's business, but it's what we do, and we're pretty good at it. So to that end, I'm going to speak about a couple of our projects. This is a slide off the corporate deck, and I'm not allowed to modify it. Just so you know, time starts in the lower right corner, goes up, and then goes left. Don't ask. I'm not allowed to modify it, so I'm going with it. We started out with the orbiting solar observatory. That's a program we built as a series of satellites we built that are designed to look at the sun. That was in 1962. We moved on to the infrared astronomical satellite. It was the first-ever space telescope to perform a survey of the entire night sky in infrared. Then we moved on to the solar backscatter ultraviolet radiometer 2. That's an instrument on a satellite that helped discover the ozone hole. So you can see that picture is actually the ozone hole. I have it up here on my laptop, too. I'm working on a project right now that's sort of a follow-on to that instrument. We built Kobi, which is the cosmic background explorer. If you've ever seen that image of, it's a blue and pink image of the cosmic background, that instrument helped prove the Big Bang theory. So that's a pretty big thing. We worked on the tactical high-energy laser. This was a laser system designed to shoot down warheads and missiles in flight. I don't think it actually ended up being built, but it was a developmental system. We built HiRISE. That was the first orbiting platform to return high resolution images of the surface of Mars. It was actually able to pick out where Viking landed like 30 or 40 years before it existed. So that was a pretty cool thing. Orbital Express is one of my favorite satellites. It's actually a pair of satellites. We built one and another contractor built another. And our satellite actually went up into orbit, docked autonomously with another spacecraft, and was able to refuel the other spacecraft on orbit. So that's a pretty cool project that we were able to build. Deep Impact is a pair of satellites. We built the sort of mothership satellite. And then it sent a little impactor that had an instrumentation on it into a comet. And that helped discover what the comet was made of. And it actually kept going and was able to rendezvous with another asteroid and image it and brought back some really cool science. Kepler is something that's been in the news lately. And that's a really neat spacecraft. I get to work on really neat spacecraft. Basically, it points at one point in the galaxy. It points at the same spot continuously and just measures how much light is coming back from each of those stars. And then as planets come in front of those stars, they reduce the amount of light coming out of those stars. And you're able to detect that. How you can detect a planet the size of Earth, including a star from light years away, is I don't know how that works, but it's pretty awesome that it does. And then the most recent thing I think we were in the news for is the Ralph camera. That was the thing producing high-resolution imagery of Pluto. I think they launched it before while Pluto was a planet. When it got there, Pluto was no longer a planet. We can discuss whether that's a planet or not later offline. So I'm going to talk about my career. And this is to give the developers in the room an impression of how I use your product and why I am not an expert in your product. I'm not an expert in Linux. I don't have the time. I wish I did, but I don't. So I have, in six years, I've worked on nine projects. I'm a jack of all trades. I've done a lot of things. I started out on a spacecraft instrument running on a spark. I don't know how many of you are used to running on spark platforms, but it's a pretty old platform. And it was running VxWorks, so that very expensive real-time operating system I was referring to earlier. I then worked on a spacecraft simulator, an entirely in software simulator. So we're simulating how the processor emulating the interface of the processor for the software. Sorry, that's a very complicated way of saying that. I did some algorithms translation that's from MATLAB to C++. I did a Star Tracker running on PowerPC, running VxWorks. Again, all of these projects have used Linux only as development platforms. They're not embedded Linux in any sense of the word. Then we started to get into the time I was able to actually do embed Linux, or at least what I call embedded Linux. I was building a camera system for the Navy. It used a full-up x86 computer. It's a fairly powerful computer, but it is technically embedded Linux in my worldview. So that was a fun experience. That was my first entrance into the embedded Linux world. Then I built an aircraft, and then it was intended to go to the International Space Station. It was a LiDAR that was measuring wind speeds. I don't think it got the final cut to go to the space station. It's very hard to go to the space station. And I don't think our instrument made the cut, but they're still flying it on an aircraft. I helped build a camera system for the aircraft. This is probably the strangest embedded system you'll ever hear of, because it was eight or nine server-sized racks that were embedded into the aircraft. I don't know if that's sort of stretching the definition of embedded, depending on who you are. But I called it embedded. I then worked on the phased array. So that's what I'm going to talk about largely in this presentation. That was a pedal Linux build on an arm. And we'll get into that later. And then now I'm working on a camera simulator. So I'm not even working with Linux anymore. That's sort of the nature of my career. So what is a phased array, and why do we use them? So antennas started out pretty much as wires. I don't really know the history of them, but they started out as wires. We decided we needed to directionalize them. So we built things like dishes and yogis. And they're big. They're heavy. They have to move to target a new thing for detecting or communication. And so they have to move between the two targets or the various targets they have. And that's slow, because mechanical things are slow. So then we went to phased arrays. And what that allows you to do is steer the beam much more quickly, purely in electronics, which we all know can be much faster. And so you can actually talk to multiple targets effectively at the same time. Not actually at the same time, but for all intents and purposes at the same time. And that's a fantastic ability for communications and radar. So this is a slide a little bit on how a phased array works. It's a bunch of small antennas working together to create a directional signal. In our system, and this is not true in all systems, but in our system, the RF, the radio frequency signal, state analog the entire time through the entire system, we were just sending digital control signals to analog electronics to control where they were pointing. That involved amplifiers, attenuators, phase shifters, and time delays to direct the beam. And the control computers obviously didn't need to be nearly as fast as if they were controlling RF. So the commanding rate is in the tens of hertz for steering. It's not that fast that I didn't even require real time constraints, because for Linux, tens of hertz is trivial. So this is the system I was working with. I was the programmer, really the only programmer on these blueboards. I had a little bit of assistance from my boss, but he was also my boss, so he was doing a lot of other stuff. These are, the blueboards are micro-zeds. This is an Avnet platform that runs a Xilinx processor that I'll get into in a little bit. The RF comes in here, gets translated into fiber, optical fiber, gets translated back into RF, sends a sort of a bulk transform system that does some very gross things to the RF, and then it gets sent onto the sub arrays, and there's anywhere between 50 and 100 of these in each one of them has in the tens of antennas on them. It's all controlled by the central, again, embedded x86 processor. Your definition of embedded may vary, but it's a full up x86 processor that a friend of mine, another developer, was working on, and he wrote all the stuff that commands the whole system from a central location. So I don't know what the experience level in this room is, so I hope I'm not talking down to everyone. I know there are some Xilinx engineers in the audience, so they'll know this already. But what is an FPGA and why do we use them? So FPGAs are fundamentally reconfigurable silicon. They can be controllers, they can be memory, they can be, they're probably the most expensive way you can possibly get memory, but they can do it. And they can be custom processors, so DSPs, visual processors, pretty much anything you want. They're reconfigurable all the way to runtime. You can actually pipe a file into a special location into Linux and change how your hardware is configured. I've never done that, but supposedly it's pretty easy. They allow external hardware to be swapped and added extremely easily either in development or in runtime, so they have fantastic capabilities there. They provide expandability for future versions. You can start out with sort of a demo version that has the cut down feature set, show it to your customer, and then get them to pay you to do a lot more. So FPGAs allow you to build that into your first version of hardware, and then it's all software and FPGA design from there. Customers love that. And as I referenced earlier, you can connect different daughter boards and hardware to the same board during runtime or during boot time. And FPGA allows you to do that much more easily. So this is the zinc. The sort of silverish stuff here is hard, so you can't change that. It's a dual core arm. It's built in. It's got a bunch of IO built in. It's got SPI, I squared C, that kind of stuff built in. You can't modify it. You can change what pins it comes out, but that's it. And then it's surrounded by this FPGA fabric. So this FPGA fabric is where the power of this system comes to play. You can have a lot of soft controllers. I'll get into it in a little bit, but you can have a lot more IO than this. That's what we used it for. And like I said, you can put different processors in here that do very custom things in here if you wanted to. We didn't really do a lot of that, but you can if you want to. So what hardware was I working with? And I couldn't fit the other picture, the previous picture on the slide, so you're going to have to remember it a little bit. But there's an RF to fiber converter. There's two of those. Those just have one I squared C controller, one GPIO, and one analog, and you go, hey Sam, why didn't you just use a Raspberry Pi? Well, I could have. I have an array driver that's a little bit more complicated of a situation. I've got four SPI controllers on there. Two of them are controlling SPI flashes, and two of them are controlling what are effectively very large shift registers. That starts to get to be a scary number of SPI buses. So you can't really use those. You've got I squared C, two I squared C buses for sensing temperatures and voltages, those kinds of things. You've got analog, same kind of stuff, temperatures and voltages, and GPIO for LEDs, bulk digital transforms, those kinds of things. The sub arrays get really, really big. They have eight SPI buses on them, so that's something you just can't get into your hardware. You have to have the FPGA ability to control eight SPI chains. Again, four of those are chains, are shift registers, and four are flashes. And I'll get into the reason those flashes exist and why there are four of them in a little bit. You've got five I squared C buses. Again, there's a reason there's five of them. You have to have them because of the way the hardware has to be set up, analog and GPIO. And then these are all three of these are controlled centrally by this guy over ethernet. So ethernet is another thing that's really nice with Linux if you have it, and I'll talk about that. So what did I need to do? I needed to receive commands and send telemetry to the central controller for processing and updating of the customer. I needed to control the RF chain at tens of hertz, so not that fast. I needed to store calibration on the daughter board. So the daughter boards had to each have their own calibration. There were hundreds of components that each had their own calibration point on each daughter board. They had to be stored locally so you could hot swap them. So that's a big part of why we have flashes everywhere is because the board itself that has the components stores its own calibration. You need to automate parts of calibration when you have hundreds of calibration points per board and you have hundreds of boards, you get to a really crazy number of calibration points really fast. You do not want to centrally control that even with Gigabit ethernet, so you want to have some level of automation to make your life faster and easier. And you need to protect the hardware from operator, or in my case, engineer stupidity. You don't want to blow up your own hardware while you're working on it, and then you don't want to let your customer blow up the hardware once they get their hands on it. So why did I choose Linux? I chose Linux as I wanted to, let's be honest. I'd been working with it on my laptop for years and years, and I hadn't really gotten to use it in an embedded context. The previous programs I had worked were all other people using it, or other people directing me on how to use it, and this was the first time I got to start from the ground up and use it. I spent a lot of my time. Colorado has blizzards, that's where I work. And we have a campus that's fairly open, and I didn't really like walking between buildings and a blizzard. SSH means I don't have to do that. So I mean, very selfish reasons to choose Linux, but I think they're good reasons. Another advantage is your test team, your electrical team, your RF team all have experiences with Linux. It's out there. They have it on their laptops. They're using it on their phones and on their Macs, and some of them are even using it on their laptops. And so you can easily tell them, hey, this is just a Linux computer, just SSH into it. The big one, and this probably makes me a really bad embedded engineer, but I didn't want to write an SPI flash driver with wear leveling. That did not entertain me. That did not sound fun, and it sounded like it would take basically the same amount of time it took me to bring up this whole project in Linux. So maybe I'm a bad engineer, but I didn't want to do that, so that's one of the reasons I chose Linux. And at ball, somebody had to. We haven't done this level of embedded Linux before. So somebody had to start it. So what are my company's concerns with Linux, and my concerns as well? Speed. Linux is probably about as fast as VX works, but it's harder to prove it. Everything in my industry is about proving it. Can you prove that Linux is going to hit its deadlines, is going to do everything it needs to? We don't have that experience. I know those tools are coming out, and they have probably been out quite a while, but we don't have the experience with those tools, so Linux is risky from that perspective. And support. You don't have a dedicated support channel that you're paying for with Linux. At least I haven't found one. I know that a number of companies are now offering that, but we have not experienced that. We don't have that in our company yet. So why are we reluctant to accept it? This is a sort of continuation of the previous slide. I heard my boss once describe VxWorks to a VxWorks sales rep as sort of the Cadillac of operating systems. You know VxWorks is going to stand behind their product. You know it's going to work, and they will make sure it works for you. You're going to pay them for it, but it'll work. Linux is more of a Lotus, and you're bringing a Lotus to a Cadillac show when you're bringing Linux to ball. It's a different product. It has a lot of advantages over the Cadillac. It has a lot of advantages over the VxWorks. But there's also risks involved. The support is not as good. You're going to end up doing a lot more of the work yourself. We also have a lot of experienced developers at ball. I'm probably one of the youngest developers at ball. I'm not anymore, but up until two years ago, I was the youngest developer, maybe the fifth youngest developer at the company. And so we have a lot of history built up, especially with Linux. A lot of our developers have used it a lot. And back when it wasn't nearly as polished as it is now. And so they're afraid of it. They've been bitten by Linux in the past, and they fairly don't want to use it. They're also used to being constrained by hardware, which is something that we don't really have a problem with anymore. Linux is bigger. It's got a lot more requirements from a hardware perspective on it. So you don't want to use it if you don't have to if you're running when your launch costs are in the thousands of dollars a kilogram. And they have a better knowledge of history. So a lot of the developers I work actually work in assembly still. It's not that common anymore, but they're writing assembly. And if you're writing assembly and someone throws a Linux at you, you're going to go, how do I do all this? There's this huge system built up behind it. So there's advantages there and there's disadvantages there. But that's why we consider it a little risky to accept Linux, and it depends highly on the project. So now I'm going to start to get into some of the things I learned about Linux and the device tree and any number of things. So some of these are going to be general embedded techniques. I'm sure most of you know them, but I'm going to go over them anyway. The device tree plus an FEGA is fantastic. I discussed this earlier in a slide, and this is sort of what I'm talking about with the daughter boards. If you have a calibration routine in hardware here and a calibration flash here and here, and then your Z board is sitting on a sort of mother board here and these are connected over wires, the device tree makes this really easy to do and really easy to swap hardware. So these Z boards in my system are actually all running the same software image and actually the same FPGA image. And they just, depending on the pinout, change what they talk to, and they're able to detect that based on various parameters. Device tree overlays, this is my one thing I really want out of this presentation. Device tree overlays are not well documented to my knowledge. I cannot find them. I talk to some friends at various companies, and they're like, we don't know how to use device tree overlays. I know that the Beagle board organization has sort of done that, and they've used it. But for FPGAs, it's really hard to use. It's hard to overlay a device tree at runtime. So if anyone out there is either willing to write documentation or willing to make it easier to use, I encourage you to do that. I don't have the time, but I'd love to see something happen there. Another thing I learned when working on this project is device tree parameter standardization is hard. So we had an ADC that was listed as compatible with the device tree, and technically it was. You could plug in this ADC, put it on the I2C bus, and it would show up and give you readings. The problem was if you changed anything from their defaults in this ADC, like the reference voltage, the device tree did not support that. So what supported by the device tree changes depending on who you're talking to and what they're talking about and how badly they want to commit their code. So that's an important lesson from a device tree perspective. This is where I got bit really hard for about a month, and I can see some guys laughing, so I suspect that they know what this is about. You can see it right there. Caution, this file is automatically generated by pedal and test decay. Do not modify this file. That screams to me, this should be working. This should be tested. We should know everything about this file. It's not. This file is not correct for the board I was using, and it cost me a lot of time. When you're sitting on an air, you don't know where to look. And when your flash is broken, you really don't know where to look. That was the change. So there was a minor part number difference that made it look like it was going to work, but it didn't. So don't trust autogenerative files. They can have errors too. It's just an unfortunate consequence of the industry. This is a funny way of saying it, but Linux doesn't know your SPI chain. So I'm using a giant shift register. Always check your return values. We're only sending data out our SPI chain, and we didn't have any practical reason to read it back. We didn't need to read the register's values back, because we're just writing to it. But you have to do that. You have to read your return values, because especially in RF heavy environments, wires are going to get signals on them, and bits are going to flip. And you have to be able to prove to your electrical engineer that it's his fault. Because if there's a bit flip in there, and you can't prove it, he's going to go, it's software. It's Linux. Why are you using Linux? That's dumb. And you go, I read it back, and it was different, too, man. And then he plugs in the oscilloscope, and he goes, oh, yeah. Look at that. There's a bit flipping right there. So it's kind of a dumb technique to mention, but absolutely check your return values. And that was a hard-learned lesson for me. Developers will argue about everything. Linux developers more so. And it will be very public and very obvious. I didn't learn what the line between the device tree and the kernel was until very late in this process, and I read some documentation that finally clued me into what that looked like. This also speaks to the get used to reading the mailing list archives. So there's going to be a lot of arguments where you're going to, eminently, I think I got that word right, agree with people on one side of a mailing list four or five years ago, and disagree with the people on the other side four or five years ago. And you want to scream at your screen, and you want to scream at them over the internet, but you can't, because it was five years ago, and they don't care. So that's just going to happen. It's not always about Linux. This is a little bit unused, guys. This one bit us a little bit. There's a glitch filter missing in the hard I2C controller on the FPGA I was using. So that I couldn't change. I couldn't fix the glitch filter. And our I2C chains were originally going through this. So again, back to the hardware engineers. You have to prove to them that it's a hardware issue and not a software issue. And this one actually got a little bit harder because of Linux. Getting to that level of the kernel and getting to that level of the hardware in Linux is harder than it would have been had I been using bare metal or a thinner R-toss. This is a dumb thing, but also really important. Be nice to your techs. I work at a company where if we produce two of something, it's amazing. It's a huge deal if we produce two of something. And we were building hundreds of these boards. This is something we're not particularly used to. I have no experience telling techs how to flash. I don't know how to burn 100 boards. All I know how to do is my manual stupid way. And so I wrote a 30-step procedure for the techs. And they sort of looked at me like, really? 30 steps for 100 boards? You want me to do this that many times? I was able to cut it down to 17 steps, but it was still really difficult and very painful. So it's kind of a stupid thing, but be nice to your techs and try and automate as much of their lives as possible because they have a lot to do. Find the fastest way to burn flash. There's a pretty good story here I'll tell you in a second. You're going to end up doing it a lot, especially when you're working in the kernel and the file system and the bootloader. You're going to end up burning flash a lot. And you're going to get distracted. And that's where the story comes from. There's a really big building at ball that has a lot of different projects going on in it. And it's got a concrete floor. And it's really fantastic for having wheelie chair races. But if the VP comes on a tour with customers, it's not so good. So it would have helped to not be distracted at that particular moment. Maybe not the greatest example, but it'll happen to you and hopefully you'll laugh at it too afterwards. SD cards are a fantastic way to take care of this. But if you're working in a vibration sensitive system, you're going to have to glue your SD cards down at some point. And that makes your life hard again all of a sudden. So be careful with how you're using things and when you're going to have to start building the hardware, you're going to have to start gluing your hardware down. This is a main thesis of, I think, all embedded engineering. I suspect you can all agree with this. Insert yourself into the hardware design process. I had a fantastic electrical engineer come to me early in the project and say, hey, Sam, what do you want? What do you want me to do? What devices do you want me to use? How do you want me to pin it out? And I was like, this is great. I can pick all drivers that are already in the kernel. I can pin it out so the overlaps work in an appropriate way. I don't have to use device tree overlays. This is really great. He came to me. He did it. And he had the documentation or the board designed. But then the electrical team had a design review. And I wasn't invited. So they completely changed the design. And my fault, I didn't tell him why I wanted these things. I just told him what I wanted and not why. And so they completely changed the design, sent it off to the printer. And then two weeks later, we get these boards back. And I'm sitting there going, all right, you changed everything on me. And everything I wanted is gone. Great. So it's really important that, yeah, electrical engineers are busy. And they don't necessarily want to listen to the software or the dumb software engineers. But you have got to get everywhere in their process. Even if they ask you what they want or what you want, you have to make sure they actually do it and the boards actually get printed with your changes. This is another place device trees would have helped overlays would have helped me. It would have fixed this problem. But I didn't know how to use them. So my electrical engineering staff, they're great people. But they kind of hurt me on this one. And then my buddy had to write an interface to a rack. We're talking $100,000 single rack. It's a 380-volt DC power supply. All you could control it with was ActiveX. Yeah, I can see the faces there. It's extremely painful to get ActiveX support in Linux. And so we would have changed. This $100,000 part, we would have changed parts easily because it cost us a ton of money to try to figure out how to get ActiveX into Linux. And we had to use Linux. So it was just very painful. This is a funny one. Hardware will disappear on you, especially expensive RF hardware. The first thing you have to do when you're starting a project is beg, plead, or threaten for a software test bench. If you don't get that, get a welding torch. Because your hardware engineers are going to pull your hardware on you. Your RF engineers are going to pull your hardware on you. Your test engineers are going to pull your hardware on you. And if it's welded to the bench, it's a lot harder for them to do that. No, I didn't actually weld my hardware to the bench. I really wanted to, though. Linux made testability fantastic. This is a huge factor, especially in my industry. There's familiarity for the test team. If they know Linux, they can just SSH into your board, and they can do a lot themselves. You don't have to be hand-holding them like you would with a thinner RTOS. You don't have to tell them how to turn on GPIOs. You can just say, hey, it's Linux. Just use GPIOs. You can run a lot of your software tests on commodity hardware when you're running Linux. It's a lot harder with VXWorks, especially from a licensing standpoint. And access to some sort of scripting language on your hardware is really, really fantastic. If you've got to turn on and off an LED to determine if your GPIO is working in another room, the ability to write a batch script that just flashes it on and off every second is really great, because then you don't have to have another engineer sitting at the desk sending the commands to turn on and off the LED. You just write a script to do it, and then you walk over to the other room and see if it's actually happening. So testability is a huge thing that makes Linux a lot better. This is another silly thing that I learned. Keep a log of everything you do, especially when you're working with embedded Linux. You have a ton of different things you're working with. You're working with the kernel. You're working with UBoot. You're working with the device tree. You're working with your application. And you're going to switch between them, because one thing's going to cause an issue in another. And so you need to keep a log of everything you're doing, because if you don't, you're going to forget. You'll be working on the device tree for a week or a couple days, and then you'll get switched to the application, because something came up and you have to fix it. And you'll come back to the device tree. You'll forget where you were and what you were doing and what command was working for you. So keep a log of everything you do. I know, dumb to say that, but I think it's important to say. You're going to repeat trials if you don't do this. You're going to have to write a user's guide. Everybody will eventually have to write a user's guide for their product. So if you have that log of everything you did when you were developing it, it makes your user's guide a lot easier. And other developers are going to come onto the project. I was actually pretty lucky. I was working mostly alone, but I did have a boss who was helping me, and I had to keep some way of letting him know how to take over when I had to leave and move on to this next project I was on. So keep a log. I don't care if it's a paper log or a blog for yourself or something, but keep something so you know everything you did and every command you typed. When you're working on a better platform, you don't always have persistent storage. Your history doesn't stay there. So copy those commands over Serial or SSH to your log so you know what you did. So these are the advantages to using Linux. I've mentioned a lot of them already, but I want to go over them a little bit. I2C hardware is really easy to move in Linux from the hard to the soft when you find a bug in the hardware. SSH was invaluable. Walking through blizzards is not fun. SSH makes you not have to do that nearly as much. FPGAs allow for hardware commonality. That's a huge advantage of the FPGAs and FPGAs plus the device tree is a lot of magic, right? The way that the hardware appears to Linux with that device tree and that FPGA makes your life so much easier. It just shows up and works for you. GPIO from Userspace is magic. I'm sure we're all used to writing this sort of embedded hello world. Writing that from Userspace is trivial nowadays. You just write a bash script to do it and you just flash an LED on and off. Fantastic. I've said this a lot and I'll say it again. I didn't have to write a ware leveling SPI flash driver. That alone made the whole project worth it. You get a lot of boot options. It's not a Linux thing. It's a Uboot thing. But you get many ways you can boot your OS with Uboot. You can boot over TFTP. You can boot from onboard flash. You can send an image. You can update images. That alone has a lot of advantages when you're working in a system that's going to be updated. You've got expandability in Linux. You can write new user applications easily, upload them easily, and test them easily. Same thing with FPGA. You can write new processing cores easily. You have the built-in tools for upgrades, either your application image or your kernel image. Both are incredibly easy to update. They use standard tools with Linux. You have the ability to patch. Same concept, right? A security vulnerability is noticed. You can patch it. The other systems, you know, VxWorks makes that a little bit harder. There's modularity built-in. So a lot of the engineers where I work, and I'm guilty of this too, when you're writing tightly-coupled hardware with an RTOS or on bare metal, you don't do the modularity like you should. You couple your hardware and your software to a degree you really shouldn't. The way Linux is built, you don't do that unless you try really hard. If you run from user space, you don't have to... You will force yourself to write modular code, which is a great thing in most cases. And as lack of dedicated support channels as there are, the support for Linux is broad. There's a huge community around Linux, around Uboot, around the device tree, and as long as you're willing to do the research, Linux has those support channels. So thank you. I appreciate all the work you guys do to contribute to the code. I appreciate you guys coming to my talk. I hope I provided some value. I know it was a lot of things you probably already discovered for yourself, but I think it's important to say it. I'm happy to take questions or comments now, but if you're uncomfortable talking in front of a group, I'll be around all till Wednesday afternoon. You can email me at that address, and I'll try to answer your questions. I would really appreciate any feedback you have for me. This is my first presentation in front of a group this large, so if there's anything I can do better, it's a skill I'm working on, so I'd like to get that. Anything from my hair looked funny or my fly was down, which I think we're good on. To how wrong I am about the device tree, please let me know either through this email or right now. So I'll open up to questions. How did I configure the FPGA? Yeah, so I was doing it with UBoot, was doing it automatically for me, so I had a pre-built image UBoot would boot up from on the arm and then configure the FPGA very early in the boot cycle. Does that answer your question? Okay, yeah. To my detriment, yes, I did. I was on Petalonyx 15.4 and about three of the bugs I found were fixed in the newer version of the kernel by the time I was done with the project. So there's advantages to sticking, but there's also advantages to operating. 3.10 I think, 3.10 or 3.8. I tried running 4.12 on it because I actually have code in 4.12 and I wanted to see if it would work. It actually, I fixed some things I wanted to fix and it didn't work because I don't think I patched it quite appropriately, but yeah, that was very close to the end of the project and I just wanted to see if my code would have worked, but it was fine. Would I do it again? Oh, absolutely. If I could have a career doing this kind of stuff, I would do it every time. I learned so much. I got to learn so much about other people's code and it worked. It came together. All I can say is it came together. I don't know how. It was magical, but things came together and in the end, we have a functioning system and it was great. Yeah, so the comment there, sorry, I'm not repeating questions. The comment there was, he's seconding my device-free overlays, which I'm going to make really loud. Device-free overlays should be documented better and more usable. Anybody listening who can do that, please do. I think that's a good point. I'll repeat that. You have to set your expectations appropriately when you're using Linux. The support levels are different than when you're using a paid operating system and I think that's a very good point. So thank you. I did kind of, that was the version before they adopted Yachto. It was Petal Linux and then they adopted Yachto as far as I know in 16.1. And so I did, as I learned what I was doing, I slowly moved away from their environment and slowly started doing it on my own. At the end, I was building my own device-free blobs manually. I was building the kernel manually. Obviously I was using their patches, but I was building that manually. My application was built basically straight from the cross-compiler not in their environment. It just depends on what you know and as you learn it. No, no, no, no. I never brought up Eclipse, thankfully. I have a lot of friends who love Eclipse. It has its advantages, but for kernel development kind of stuff, it's not my favorite. Sorry, I'm going to repeat your question after we've already discussed it, which is, did I use Eclipse to build my code and the answer is no. That's a much longer story. It turns out I'm an FPGA engineer as well, largely because I wanted to get integration. So Xilinx is kind enough to provide a scripting interface for their Vivado tool. So you do have to launch Vivado, but then you just run a script in it. And if you run the script, you can get control of the script. But yeah, if that was the thing I was talking to the Xilinx people about, yeah, better get control on the Vivado stuff would be greatly appreciated. How do I answer that? Itar is certainly a pain when it comes to Linux. It's very hard to ask questions when you're asking Itar things. I didn't worry too much about open source things because the way I've had it explained to me means it doesn't matter. I'm sure I'm wrong. But I didn't think about it very hard and just sort of tried not to think. So that's a very fair point. A vague law with criminal penalties. Yeah, that's a good way to describe it. What do you mean by flight software? No, so our satellite business is still very much the Cadillac of satellites. We're still very much doing the VxWorks things. And I would love to fly Linux and I think we're getting close, but we have a lot of heritage and a lot of customers that have requirements that do not allow us to fly it on orbit. So I guess I slightly mis-described. The question is how do I track hardware bugs? So we have 50 or 100 of these boards but they're all the same. And there's only four different boards and if one of the types of board there were 50 to 100 of and if we misprinted it once, all 50 of them got reprinted or all 100 of them whatever got reprinted. So we didn't have that problem at least in our version. I don't know how I would answer that question in future versions because I'm sure that will start to occur and I believe the calibration is on board on a lot of those boards so that's a nice idea of it. You can put a version number in your calibration flash. I don't see any hands going up so I think we're done. So again thank you all for coming.