 Okay, thank you for your patience. We are going to get started and with that I'm going to introduce Tim Byrd. It's going to take it away Hi everyone, welcome to this session of Embedded Linux conference and embedded open source to have some representatives here from the space and aerospace industries and I'd like to just do some quick introductions sitting next to me is Dr. Lenka Triscova, I hope I pronounced that correctly and She is a lecturer and researcher at the University of Libric here in Prague and also head of the Linux for space project which is a new initiative around Creating a distribution of Linux for space next to her that is David Vom Lane who is lead flight software engineer at Astra and If you saw the keynotes the other day, he gave a great keynote talking about some of the challenges for for space missions and then joining us remotely is Steven van der Leest and he is the Chief technologist for Boeing Linux if I am not mistaken so that correct Steve Okay, so Great Glad you could join us remotely. I know it's early. He's on the west coast of the United States So it's a pretty early in the morning for him But we really appreciate these people getting together To talk about Linux in space and in aerospace missions. Oh, and we have one more panelist joining us this is Robert Bochino From NASA jet propulsion labs. Thank you Robert for joining us I know it's early where you're at and We had just a little bit of technical problems getting him cooked up, but I'm glad he could make it so welcome Robert Thank you. Thank you for having me. Okay, so I guess my first question is Hopefully this is a softball question what projects with Linux in space or aerospace. Have you been involved with and Were they Production quality, I don't know if that makes sense in this context Or or were they research or and were they successful? So, I don't know David you want to tell us sure it was off. Let's see Rocket satellite satellite I don't know if it ever became a satellite because I worked in a lab doing the Linux part and a bunch of guys in uniforms We're in another locked room. I don't know if that ever got launched a Lutter lander and rocket so that's the quick summary they all used Linux the First series of two satellites was successful the rockets were successful Well, I won't know about the other ones till a little while the second pair of satellites were not successful And like I said, there's one I have no clue about and unfortunately the company ran out of money on the Lutter lander So we never got to launch so but we're not going to blame me those failures on Linux, right? Absolutely not actually and I sweated that out for both of them, but turned out to be hardware guys right So Steve, how about you? What's your experience with Linux in in aerospace? Sure, my my experience is more on the air than the space side But if we want to talk a little bit later about what those differences might be, you know, we can't get to that I'll talk first about a while back maybe 10 years ago I was at a small company called Dorner works and we were doing some work with implementing Linux for safety critical applications Especially in in air-end space But mostly research it was looking at what was the feasibility of using Linux and what would be the cost to actually flight Certify it. So there were some interesting results there that resulted in some some good initial work today. I'm at Boeing Boeing has used Linux in Space in air in a variety of applications Today on the civilian side Certifying under the FAA to my knowledge. It has only been to level D Which is one of the lower levels of assurance the project that I'm involved with to To develop a a Yachto Linux specifically for a civilian flight We're aiming for higher levels of assurance. So to get to CB and ultimately a So that project is ongoing We don't you know, we'll maybe be back in a year to tell you how whether it's been successful or not But certainly we have extant examples at the lower levels. We're looking at what it would take to go to higher levels with Linux Great. How about you Rob? Hi, so Yeah, so I work at Jet Propulsion Laboratory been there about 10 years and I've worked on a CubeSat a 6u CubeSat space telescope in lowerth orbit called hysteria that that ran Linux So that was a very successful mission. It deployed in 2017 fall 2017 off the ISS and it successfully demonstrated high precision pointing for For exoplanet detection in a very small satellite form factor So it was basically the smallest satellite ever to be able to have that kind of precision pointing and then it was also used for Several extended missions. So after the main mission was Successfully completed. We still had the the cubes out up there and we were able to upload new Software capabilities and test them having to do with Advanced autonomy So that was very successful and then another a big success involving Linux and space at JPL is the the Mars ingenuity helicopter So I didn't directly work on that mission But I did develop some of the software components that are on The helicopter because both the hysteria cubesat and the helicopter use the same flight software framework It's called f prime. So there were some hysteria components that were adapted and repurposed for the helicopter Yeah, that's awesome. I'm I'm a bit of a space junkie. I've been watching ingenuity with great interest It's like the little helicopter that could and and I know F prime is interesting. It's also open source, right? So available for people to look at on github and and contribute to so if you contribute to f prime You might end up on another helicopter on Mars sometimes. So just saying and then Dr. Triscova I'll call you Lenka Lenka What's your experience with this? in 2020 I think I was consulting some cubesats for mission here Which is from by NASA and yeah, we were thinking how to build up because there are cubesats a few of them based on the computers running Linux, so we were consulting how to build it up and Yeah, I started to grab the information then and yeah, we find out that there is no let's say come on platform for Hobbies interested in Linux in space because plenty of people can build up a cubesat maybe But there was not some let's say come on platform So later on we met some people who already have satellites you can see later on In the demonstrations and we started the last year Yeah, great. So David talked about in his keynote the other day the space is hard. That's kind of the standard mantra So what are some special requirements? That might be unique to the space or the aerospace industries that other embedded Linux might have So I I was talking to you earlier today and I said well power budget You know all kinds of IOT devices have power budget issues But what are what are maybe some other things that are special challenges for for space missions or aerospace? Well, I think I did cover them to some degree, but I mean Power budget is is an issue. Generally you have a mass budget and a power budget and you've got to meet both of them Reliability is very key That how that actually manifests itself in terms of how reliable you have to be and what what time frames you have to Provide services depends on whether it's a satellite or whether it's something that's either going up like a launch Server or whether it's coming down like a lunar lander Certainly those are key Radiation is big huge and I would love it if there were a standard solution for that both hardware and software Certainly that's something that I hope to be well know that I am pursuing at present I hope to Be able to make something that maybe we can open source some of at least we'll have to see Those are very key Another one is bandwidth and this depends upon your particular particular detail but I will say that the lunar lander that I was working on had a data rate at one point in its mission of 500 bits per second That's not a lot of bits Gonna be a little hard to manage that so it varies that is Even in low Earth orbit that's going to be the case for quite a while Until we're allowed to do things like plug into Starlink or something equivalent I do hope that that happens soon because it will Revolutionize a lot of what goes on with satellites being able to essentially live stream that the data down Okay, and yet anyone else want to make a comment on any special requirements in your sector I'll jump in I Not only do we have these special requirements for environment radiation vibration temperature and so forth, but also fairly rigorous requirements to Prove that the system is reliable that is providing evidence of assurance And as you start talking about having people involved passengers say civilian flight operations That that level of rigor is fairly high You need to provide, you know sufficient evidence that we're confident the system is reliable That is it does what we want it to and it's safe It does not do what we don't want it to do and proving that is is is very challenging I'm it generally forces us to want very small footprints Having too many features means extra cost of proving those features are safe So we tend to focus on exactly what features do you absolutely have to have and I'll prove that those are are correct Yeah, so one of my questions I wrote down for later is Why does why does our toss have such initiate inertia in this space? And I think that's one of the answers to that is that the provability of assurance on it I attended a flight software workshop earlier this year and People were talking about a hundred percent coverage on the lines of code in terms of testing and That's somewhat shocking for a project that has 25 million lines of code. So Okay I know that the Linux kernel has you know upwards of 25 million lines of code But any particular implementation won't use all of that a very large portion of Linux is is Drivers the next largest section is the different architectures. So on a particular system You're only using one architecture. You're only using very small subs subset of the drivers So it's not 25 million, but it still might be a million and that's a lot of lines to prove Actually, I mentioned one one particular thing that comes up is there's a simple code coverage where you just make sure that all the lines Are hit but that's not actually equivalent to feature coverage Which is making sure that not only do you hit the lines? But you hit them in the sequences in which they execute for real when you're doing real commands and that's a Different thing so it's a hard problem Right So, okay, so it's so it's a hard problem. So why are we here? What why why should we be using Linux in in space and in aerospace as opposed to alternatives? Well, okay, I think that on your cupers CubeSats We have decided to use it as a payload. I mean as a not mission critical system, but it brings you a lot of software you may use for example for a Processing the images and processing the data you have and you have a lot of availability of software stack Which is already existing? I mean implementation of nearly all the drives or protocols used in space or any other protocol And also a lot of people is let's say capable of programming for Linux. So again It's easier to hire people for the project maybe And yeah, it's a lot of fun Yeah And anyone else want to chime in what are some of the advantages you've seen or what so Rob, so why did you choose Linux for the hysteria mission? So I think one reason is I mean similar to what was just said it's it's just kind of an easier environment to work in I mean, you know, you have a familiar shell you have a familiar tool set and It turns out also that If you're willing to tolerate the risk you can actually use the shell on flight so we did that in hysteria I think they also do it in the morning. I've done it in the Mars helicopter So I think those are some of the benefits and yeah including also just the the reuse of Software tools as was mentioned One of the things I noticed and I don't I don't know if this is true for our tosses or not but People at the flight software workshop. We're talking a lot about AI Stacks in space and I was wondering well, it's pretty easy to get an AI stack on Linux I don't know how how easy it is to get it on some of these other our tosses What types of services you need in the operating system for for that type of processing? I think the other point is that Linux just provides a a Collaborative environment for innovations whether it's AI or anything else you quickly see that technology fielded by someone and then proven by a crowdsourcing so it gives you Cutting edge technology more quickly. We have to prove that it's correct But when we need that feature, it's there in in Linux and when vulnerabilities are identified They tend to be fixed much quicker the the the security side is becoming more important Especially as as we're becoming more interconnected and so the ability to quickly patch a vulnerability is important The Linux community, you know the crowdsourcing effect gives us that quick fix We have to prove that that patch is safe as well So there's always that you know background activity of proven correctness, but starting with Modern technology starting with with you know The the community best practices is a great foundation for the for the work Also, I mean I think that Very very much one of the most important things is the fact that there is just so much Linux out there so when I talk about that functional coverage about executing these particular lines in a particular order that you're gonna need them It's kind of done that most everywhere When we come to developing software space for space and this is true in many other places, but processes paramount what it comes to getting to reliability and the Clearly the Linux kernel processes the the testing the fact that people do tons and tons of testing on tons and tons of platforms that exercise different things Really drives the reliability up so there are you know very rigorous mathematically rigorous ways to develop software We didn't do that with the kernel. I think you know But on the other hand we've come up with another process that produces excellent results, and you know demonstrably excellent results So So I haven't heard anyone mentioned real-time, so I got to bring up real-time I know that at least there's a perception that You know Linux has the preempt RT patches But there may still be some concerns about whether or not Linux can hit real-time deadlines So I have kind of personally I have two responses that was one is like for a Cube sat in lower Thurbet I was shocked. I just got into kind of space stuff this year It's like provisioning the satellite takes months and and so or commissioning the satellite And so it's like it doesn't seem like there's a real-time Aspect to that part But are there is that a concern? Is that a barrier for for Linux the the real-time Issues and and what types of vehicles does that apply to or what types of missions does that apply to? So in my experience on hysteria, so now this was using an older version of the kernel With the preempt RT patch, but I still think it's likely true today That Linux is just never going to be a hard real-time OS So if you have hard real-time Requirements in your system that you can't miss then don't use Linux for that But on the other hand, I think there are many cases where soft real-time is good enough and You could also partition the system. So for example, this is what they did on the Mars helicopter You can have part of your system, which is hard real-time and it's not running Linux And then another part of your system. That's soft real-time and is running Linux and that works fine Ability to split your system into real-time and and non real-time is Very nice when you're trying to hold down the costs because all of a sudden you just have one processor that can handle that load I'd quibble a little bit about how hard hard is in Linux real-time. I'm really it is if you see graphs of latency from a Well-tuned system It actually has pretty well-bounded latencies. They are higher than a real-time system Generally, so if you have a microcontroller, for example, it's gonna have smaller latencies And that's nice if you're trying to prevent something like an explosion Other things though a lot of the guidance algorithms. They have target Basically target number of cycles per second that they want to run but they degrade fairly in a fairly well-known Reliable way, so it's for mathematically smooth if you miss a cycle or two by not too much It's actually not gonna cause your vehicle to go out of control or anything So but but there are risks. I can't remember whose talk it was probably Steven and Anyway, it doesn't matter the key. The important thing is what he said was was that? Tuner system make sure that you're doing the right thing It is at least theoretical theoretically possible. You could choose the wrong file system You could choose the wrong drivers and they would blow your response time out of the water So you got to check it. There are tools for measuring latencies and stuff and just do it check it out But if you do that you have a nice very capable system that should react pretty well So I'm working on rockets now. I worked on rockets before Linux real-time. Yeah, that's controlling So I'd actually extinguish between high performance that is it's fast It has low latency quick boot up time and so forth and for for me real-time actually means I can prove Deterministically that even the worst case response is within my requirement. So real-time doesn't necessarily mean fast It just means I can prove it meets my my worst case execution time For a lot of our applications That's measured in milliseconds and Linux can perform at that level Whether I can prove it deterministically depends on not just having the preempt RT patch But having that evidence that for my particular Use cases even the worst case time Even if it hardly ever occurs if it can occur I need to prove that still is within my requirement You have a really hard requirement. That's true I actually kind of steer away from that term prove when it comes to the Linux kernel Demonstrate that's why I shoot for Yeah, I say prove I mean provide evidence to a sufficient level of confidence and depending on the criticality that that Evidence is stronger or or or maybe not as strong but sufficient So my experience with Linux is that the worst case times can often be pretty bad And so maybe I mean maybe that could be fixed with tuning But so I guess what I I agree with everything you just said so my experience is that if you have an application Where you can tolerate a misdeadline every now and then then it works fine But if you can't tolerate any misdeadlines in my experience, it might be hard to use Linux I would agree as well that if there is anything which is truly mission critical Let's say avoiding explosion stuff like this then it's better to use the RTOS But there are plenty of applications which are not so critical and then you can enjoy the power of the Linux and all those software stacks and stuff around Okay So we already well we talked a little bit about Well, I'm gonna I'm gonna switch gears. So I Think we talked a little bit about the availability of a talent pool for for Linux How about development cost does Does it cost less to develop a system on on Linux? I mean when when I started in embedded Linux, we talked a lot about time to market That's for consumer products, right where you're trying to hit a window trying to get a product out by Christmas or something Is there a are there time to market are their development cost issues that are considered in the aerospace industry? And does Linux hurt or hinder that? One my my theory is that if Linux requires less work on the operating system That maybe there's more time than to do work on the science payload or something So is that do you agree with that disagree with that? I definitely I definitely do So Yeah, the sheer Massive number of features that Linux comes with means that there's and the ecosystem overall So you have a lot of other things Including one satellite running Python, which surprises a lot of people I talked to but So it was a science satellite. They wanted to analyze real-time and just send down Just a little bit of data that bandwidth issue But what I find is that it is pretty easy to find some really sharp Linux people And if you find somebody really sharp Actually the aerospace they may or may not have a aerospace background most of the people I hire Probably at least three-quarters do not have a previous space or aerospace background, but they come in they know Linux they You know, they're gonna work with people who are experienced with space and It works out pretty well There you know, I've hired a lot of really sharp people so Any other any other comments on on the development time or cost of development? Is it less or more with Linux or is it hard to say it depend is a project dependent? Well, I think we already mentioned this but I think if you have developers that know Linux Then I would imagine it's it's less right than having to train them and some other environment So we talked a little bit earlier about regulations and I you know if you have to prove well and in particular one of the regulations I saw Or I thought I saw was about the process used develop software and And you know Linux has not followed that process from day one you know a rigorous design first process and so Does is it does that mean it's impossible to to certify? Linux for certain space applications. Is it just a non-starter or is it something that we can work on? I think Steve you said that you're kind of working your way up certification levels Yep, that's right. Well, it's not impossible. We have examples of Linux being certified at at fairly low assurance levels And it's a very hard problem, but we don't think it's impossible. It's it requires some innovation of it's possible the the fact that Linux has been Developed by a community does present its challenges because for example do 170c guides a civilian Flight certification in the US And it Expects that there's that process in place that you start from requirements which drive a design which drive code And and of course Linux hasn't been that process-driven But there are Guidelines in place from the FAA and others about how you would reverse engineer Those artifacts because there is a design to Linux. It's just emergent There there are implied requirements for Linux and you can tie those to your particular operational requirements So there is a way to do it we think and we're we're driving towards that That's great We have one, you know, it's it's a potential competitor So I shouldn't name names, but there is a large rocket company that you're very familiar with which is launching people and They are based on Linux they have a two-tier system which a lot of the systems are in space So you have a Linux sort of at an executive level and then you have microcontrollers Handling the things that you really have to have a lot of responsiveness, but so this Unnamed company has broken a lot of ground and they are flying people You know, we we can name of its basics I didn't say it So I believe Rocket Lab uses Linux I know that Astra uses Linux and we've been using Linux So now we are not human rated human rated is that's basically as high a bar as you can get and they spent Years and years with NASA to get them confident on what their architecture was and whether it would be safe but obviously they've demonstrated that and I think we're all moving forward in theory I too. I'd loved that it was developed with the a 178 Strict strictly adhered to but it isn't on the other hand. It's really reliable And maybe somebody will know somebody will never reverse engineer it and it's in C. So it's essentially unprovable leap Unprove you can't prove it correct So that's an unprovable thing. Well, there is a new runtime verification system in the kernel Yeah, and it's got to accept it. So but but the the cell for folks know how difficult it is to actually do Mathematical type verification hard problem. It's a very hard problem. Well, I don't I want to hug all of the questions. So We should have another mic somewhere in the room. So if you have a question Raise your hand or let us know or maybe maybe the best thing is we've Got a stand over here if you could queue up if you've got a question or raise your hand Otherwise, I've got more questions. I could be here for a long time because I find this super interesting Okay, so any questions right off the bat if not I've got one So what about cots in space? Okay, so one thing that I was really struck with at the satellite presentation yesterday was They're using cots Commodity off the shelf or consumer off shelf or whatever that c stands for But it was carefully curated, right? They went through a bunch of iterations to select parts and then test those parts to make sure that they were space worthy One does that count as cots? I think it still does because you're leveraging You know an ecosystem that produces parts that are cheap and does does that really matter for space missions? I mean space missions are so expensive Already that like if you shave a few cents off of off a processor. Does that matter? I mean it does for radiation hardened and I I think it mattered for ingenuity but What are your thoughts on that I? Feel like I keep going first Cots is good because you can lay down them and get a little nap in space That is not what you were talking about. Okay Cots matters a lot. I mean there's like any market. There's a brand in lower end and Astra's lower end, so We aim to be very very inexpensive as well as highly responsive But that expensive is a big piece and When we have a choice between something which is cots something which is radiation hard We're looking at something which is a few dollars for an FPGA versus $30,000 for an FPGA so even though space is expensive you get a few of those it really adds up and Where you know part of our low-cost stuff is high volume, so that hardware you can't amortize that cost over Some some rocket you pay at each and every time you launch We certainly do Qualification the biggest thing is radiation We love to use automotive grade parts because those have generally been burned in which is You know a lot of that the early Called new born failures Those happen they weed them out of the supply of chips And so you have things which are not likely to fail at least under normal conditions And then we check them out similar It is interesting to note that sometimes those you know orders of magnitude more expensive chips are in fact The same chips off of the same die If you know how they make integrated circuits They've just been tested a lot more And then they mark it up and we pay it I think it's worth mentioning Maybe 20 25 years ago there was a shift from having customized Processors where it wasn't just simply selecting for reliability, but it was actually designed with Redundancy and so forth built in those specialized processors were incredibly expensive and hard to keep Updated and so there was a shift to Commercial processors at that time and then the redundancy that we needed was built at a system level And so instead of building redundancy into the chip you use multiple chips And so if there's a failure in one you you had redundancy that others could could take over and and that reduced the overall system cost And it provided innovation that we could tap into the volume production of processors being used for automotive or other industries So that was the case for For hardware for computer processors I think the use of Linux is trying to follow a similar Path rather than having a specialized operating system that is hard to keep up to date That's that doesn't enjoy the the the volumes that that you that you might see otherwise We get the benefit of crowdsourcing for Linux and we have to figure out How do we build safety and reliability on top of it in some way so that we can get the benefit and Get the safety and reliability we need Yeah, I'll I'll note that The Starlink design which uses a number of Linux processors is using I think configurations of six processors You know in a fault tolerant configuration And they seem to have done okay with that in terms of reliability. So yeah, it's a really important point I mean what he's talking about there were for example to go back into history Those processors there are certain mathematical attributes that will let you tell you whether a multiplication of two numbers is correct but you know without having to double the number of Transistors that you need to do that so that that's what people would do and they do that for addition and subtraction every every operation It's nice. It's really cool And it's also very difficult and the math, you know, not everybody has that level of math It's pretty picky and it can you have to add on top of that all the glue of everything together Which is very difficult to come up with Proofs that that's correct, and I mean proof in a mathematical sense. So what we've done and this is really cool Is it up level that so that processors check processors? So they run things in lockstep for example That can be done in software or in hardware and it turns out if you're designing Hardware to be fault tolerant. There are I did a quick check a few months ago There are at least five manufacturers that provide lockstep processors So it's two processors that check each other. These are microcontrollers, so they're not full-blown processors But I'm hoping that will emerge soon. I'm hoping that might come out of the HP spacecraft computer initiative but That what that does is that lets you just take some off-the-shelf stuff and have it be highly reliable You can bind combine two lockstep and therefore self-check processors So take two of those things and one of them, you know that the probability that both of them fail is really low So that's really really cool The other comment that I've learned when we were Consulting the her mission is that I did grew up in automotive and if you are building a hardware in automotive you are counting every single send and The main physicists of the missions told us all those cubes as they cost nothing. It's just I don't know million of euro because the main Satellite was costing. I don't know how many so if you are building a hardware for such an expensive mission Then you spare money for expensive hardware But there they are they were looking to keep such that it cost nothing so maybe we can have two for us and You'll buy cheap hard from their point of view you buy cheap hardware. I mean at least alternative proven and You simply have two boards Ten boards because it's from them. Yeah, it costs nothing Or if you want to go ultimate you could have two satellites Some people do that. Okay. We're getting close to the end here any any questions. Okay, John you want to Thanks, this is actually maybe a little bit more of a comment, but As someone who is a part of the the real-time Linux development team I just wanted to clarify. I've actually heard it a couple times at the Zephyr group here That the real-time Linux team is committed to hard real real-time. So we don't By any means consider it soft real-time. So a single missed deadline is a completely failed system, right? So and we actually believe with our preempt RT patch that we have a hard real-time system So if you know someone is unable to you know have this reliability Then it's probably misconfigured as we've mentioned here Or it's been incorrectly implemented in user space in the applications and really the preempt RT community wants Feedback we want people to help people because we really are interested in hard real-time We don't want to hear this word soft real-time And so really I would also encourage the space People if they if they don't believe it or they're having problems or their proof of concepts aren't working They should reach out to the preempt RT community because we have a real-time kernel that we're working on really That's all I wanted to say sorry not really a question. Okay Any any other questions Okay, I've got I've got one last one That I don't know how interesting that is but you know Linux has an open-source license that requires distribution of source code to all of the People that you give the binaries to So I'm not sure how much that applies in space missions There's there's no one on Mars who received the Linux kernel binaries But how much how much does the license does the license have any impact on How you use Linux or how you? archive your source code or what How big of a factor is the the license or in particular the gpl to to space missions or aerospace? I Gpl talks about conveying so giving the software to some other organization which Conceivably is why you actually don't have rocket companies. You have launch services companies So nobody sells the rockets At Astra we do not really have anything special with the kernel and I would be delighted to give back there are a few areas that Hopefully I'll get to at some time and would be delighted to get to give that back And actually we our team is encouraged to participate in the various open-source communities because Although we have restrictions on things that we can talk about legal governmental restrictions something called itar And others So yeah, we want to we really want to play good, you know, we want to be good good open-source citizens Okay, we're just about out of time. I want to thank everyone I know especially Rob and Steve on the on the west coast of the United States It's a pretty early in the morning over there But I really appreciate taking the time all of our panelists to come and talk to us about a Linux in space and aerospace and some of the challenges and Actually, I think the latest statistic that I saw was that about 50% of cubesats are estimated to use Linux today It's kind of a hard number to get I wish people were more open with what they were running on their flight stacks, but But I think we have a bright future and hopefully if there's as people get involved with community Hopefully we can address any any concerns or any issues that To make it even even more amenable to this industry So thank you very much