 This birds-of-a-feather session is hopefully going to be slightly more interactive And but what we'll do is we'll just sort of set the stage for what's there and then Hopefully we can have some discussion going. My name is Kate Stewart. I'm working at the links foundation and I've worked on embedded You know trying to figure out how we can make the embedded projects be dependable And then my co-presenter Kelly. Hi, I'm Kelly. I'm the community manager at the Sun project so we're hoping to sort of you know talk a little bit about what what the various projects are up to and then how And hopefully you've got questions about them or you know of things. We don't know Ideally if we've forgotten something or you see a problem, please let us know and that's what we're hoping to get out of this session So with that starting So these are the open-source projects I'm aware of that are working actively towards being part of a system with functional safety working on it And there's work going on to integrate them within Yachto as a framework to make it possible to do reproducibility as well as Have S-bombs and things like that generated automatically with these projects So I'm just going to go through Zephyr Kelly's gonna go through Zen I'll go through Alisa and then we'll just open it up for questions So if you want you've got thoughts as you're hearing things, you know jot them down And we can certainly have a open discussion after that So when you look at those three projects, it's important to understand that Zen is fairly small Zephyr is fairly small and then Linux is not Linux is not a small Project, I think Greg gave you guys some stats this morning. I think 13 million lines of code It's what's in my mind my head right now. What could be wrong? But the code complexity is obviously a big factor on the other hand Linux has been involved over 30 years is well deployed across the entire ecosystem there has been a lot of testing as it's been going along and Each of these projects is taking a slightly different approach To how to work with the functional safety in the set in the safety standards and specs Zephyr is Has a safety working group as well as a committee. They're looking at adopting an open source requirements tool called strict doc Documenting the requirements there and then connecting up the code to the Basically the crime is to the code to the test through that tool and making that Possible to start to do analysis and then some of the artifacts were making available to only to the members But everything that's sort of in the code and around the code including the requirements is going to be open Xen is Basically looking at mr. Stuff and Kelly will go into a lot more details later and then On the elisa project everything is completely open and but we're not really going after certification But we're trying to just figure out how to do it and make it visible to people as methodologies for working with the kernel And pulling up the analysis and game buy-in That this makes sense So that's where we're starting from so For Zephyr for hands up who's heard of Zephyr? Fair number pants good anyhow just for those who haven't it's just it's an RTOS that is modular and configurable and is Designed to work where Linux is just too big links doesn't get smaller than three Meg This is 10 K and up type of deal or 15 K depending on how you talk to However, we are actually trying to go after supporting a V model compliant development process the V model analysis level and So this is basically coming up with the requirements and then doing the traceability at these different levels as well as you know Being able to take it back up to validation throughout So we're trying to figure out how we can actually do this and work well We've got a plan and we've discussed it with to suit this point in time We've got basic buy-in that we're going in a direction They'll probably has a chance of leading working and so we're starting to work on this evidence in the open at the point This is actually all the safety collateral including the source code We're being very very trying to be very transparent as to work in progress of what's visible to who what's not and who was the owner And you'll notice that all the source code is owned by the TSC So we're working with the upstream community on this project to make sure whatever we do can be applied And then so you can you know look through this slide you can sort of see the safety committee Which is of a members only but then the safety working group is anyone in the community can show up And if you have interest you're welcome to show up to these meetings And then as you say a lot of this stuff is going to be public Otherwise some of this stuff may go to like you know just the members Because quite frankly the members are the ones that are funding this and So we have to show them some return for being a member. This is not cheap To have a functional safety manager hired to work on us. Nor is it cheap to go through the certification And so that's part of why we did the split the way we did in this project our certification focus initially and we'll be showing this in all public with the requirement traceability is just the initial kernel portion of it and We will be going after 61 508 sill 3 With the root 3s and we'll be also we have the option to do a 2626 to certification with the same data that we've been collecting So we'll probably go in that probably do both of those at the same time It's it's some additional tables. We have to generate and everything else to be there for that based on the hard discussions with two and Right now we're basically putting a lot of public requirements We've derived from the code up into the public through these living working with the strict doc tool that's the current working plan from the safety group and So you'll be seeing anyone who's sort of interested can watch and see what's going on or quite frankly join in and help document requirements and if you're interested in things outside our initial scope once you see the path pattern We're using for the scope feel free to put in the coverage for our New requirements in other parts of the code and their code and the testing for them because this now will potentially give us a way We can crowdsource beyond just our kernel scope So you want to start with this initial scope show it's possible and then oh, I really care about this USB driver Okay, you can go and work through the same methodology to put the USB driver into something that can be analyzed So we're trying to figure out how to instantiate The pattern and then build from that with this project and what's happening now there So the safety queen is the one that does the Scope of the certification that we take this tube because it involves the money The committee does things that involve the money and you know are specific to the members the working group is the Safety calls and certifications They're enabling it working with a code base working on the required documentation evidence saying the requirement management tooling and it's open to everyone and There's a lot list at the slides at the bottom. There's the anyone's welcome to join that mailing list to start participating if you're interested in it with that This information in the slides on how you can get engaged I'll turn over to Kelly So hi, everyone. I'm Kelly. I'm the community manager at the seven project. I'm also a part of send server So send server they specialize in server virtualization and they're also one of our advisory board members on the project What does my role involve so as a community manager? I wear a lot of different hats so day to day that can involve anything from conflict resolution to event planning To making sure that everything in the community runs smoothly so some of those include Advocating for the developers and making sure that I give developers a voice and how to elevate and promote the growth of the community I also work closely with the advisory board members to make sure Everything that they want in terms of strategy gets presented in the project as well So here you can see that There's an image that I've put up So open source it can mean different things to different people But one thing we can agree on is that it's important But it doesn't mean it's easy. So a lot of the times with my role It's to reduce those barriers of entry within open source projects We all want more open source maintainers, but it's not always easy And I'm sure I'm not the only one that struggles with this The main aim of my role is to elevate that community So I want to get more people talking about it and more people working within that project as well So before I get started I wanted to read through the some project mission statement So the mission of the Zen project is to advance virtualization technology across a wide range of commercial and open source domains By providing a powerful and versatile hypervisor the project aims to enable innovation, scalability, safety and security and virtualization solutions So you might be wondering what is Zen and what is Zen project? Well, Zen is a type one hypervisor. So it's also known as a bare metal hypervisor and sits directly on top of the hardware Zen plays a central role in providing isolation between those different software components A little bit about the history. So Zen has actually been around for 20 years It started in 2003 at Cambridge University In 2013 a decade later it decided to join the Linux Foundation and Zen itself is widely used for safety and security first environments The flexible architecture itself Allows for all those different applications and services to coexist on the same hardware and because it's an open source project There's a lot of different Sections within it and that's why we have different sub projects. So the main one being the hypervisor We also do Windows PV, SAPI and Automative as well Things like x86 and ARM are already supported and in our community There's a diverse range of committers and maintainers some of who are from Amazon, SUSE and Zen server I'm just going to run through briefly on the Zen architecture So how Zen works is that there's Dom0, which is also known as the privileged domain That is the first to start within this Zen hypervisor Dom0 then asks Zen To create the other VMs as it talks directly to the hardware and then it allocates resources such as memory the other VMs don't have access to To like the hardware like Dom0 does What does that mean for security? Well, if someone wanted to exploit then it makes it a lot more difficult because the way the architecture is set up It gives it such a strong isolation that if one to you know where to fail You can't jump VMs and it can't get exploited as easily So moving on from this So to show some of the great work that we've done within the community And how far we've come along since the previous slide. This is to show Dom0 a system configuration So what's different about this one is you'll see that Dom0 is optional and actually all the VMs boot in parallel This means a faster boot time Which technically it means you're not waiting around for things to start and actually it's a lot better for things like your embedded systems In terms of cost savings you actually save quite a bit in terms of you don't have to get send safety certified And Dom0 is optional There's lots of work being done on this Dom0 less configuration at the moment and the project hopes to bring that more towards the future So some of the current status the community has recently done our 418 release and We've had multiple developers from loads of different companies work on this So some of the key things that we were able to share in this release were that we've done We've got a new risk 5 port and a new power PC port The send GitLab CI was also vastly improved and for safety the project has adopted a number of miseracy rules So again a heavy emphasis on improving security within the send hypervisor, and it's already more secure than it already is So send and safety. So send and safety actually work hand-in-hand Sends chosen for safety Because it goes through a rigorous review process So a lot of the maintainers that are actually in those projects they're experts in their field and they have a long history within working there So usually when people choose then or talk about then it's known for its maturity and security features You can also configure it to allow for real-time scheduling of VMs And it allows critical tasks to run in within time constraints, which is really important if you're talking about safety critical systems The project also aims to implement some features that improve real-time and reduce interference as well So for things such as automotive or industrial applications, you need real-time capabilities To meet those strict timing requirements So what are next steps? Well for send and safety they because they work hand-in-hand together The community is working hard on improving the miseracy rules and it's a long process Which you can probably all appreciate, but it's a step in the right direction for safety Some of our project members are actually working on getting sent Certified so in terms of safety sort of certified. There's probably a lot of potential there So you can only imagine what it can do. It's already adopted in a number of projects So I've just put some examples there and we have a number of different users as well More importantly though, we work closely with the Southern and Elisa project So as you can imagine sending this hypervisor if it does get safety certified that has a lot of potential there for different things like embedded systems And finally, how do I get involved? Well First of all, I'd recommend that everyone visit the website. It has a majority of the information there So mailing lists are the heart of the Zenn project community a lot of discussion still happens to this day on mailing lists So I'd recommend if you're not already on it. You can either follow it. We do have a few different channels If you want to reach developers or people in the community directly, I'd highly recommend you join matrix It's free to do so and we have a few different channels So Zenn project is mainly for your users So if you want to play around with Zenn or you getting started Go here Send it out and that's everything developer related So if you're really keen on trying to you know make a new patch or you've got an idea that you know That you want to suggest go here and finally we are a community So we have a bit of a social channel for everyone It's mainly to share news if you've got a local meet-up or you just want to introduce yourself to the community Feel free to do so here and LinkedIn that's our place for key updates So with our 418 release and other things like that you'll see it on our Zenn project blog But at the same time you'll also find them on LinkedIn I have also popped the QR codes above if anyone wanted to directly join At the same time I am around today So if anyone wants to connect with me on LinkedIn or have a chat feel free to do so as well so the next one we're talking about is Alisa and so Alisa sort of has started off with the goal of you know Supporting safety certifications with Linux is in there and Linux plays a variety of roles When you look at something so one of the projects we started looking at was like The open APS system whereas links is running on a Raspberry Pi Its job is to communicate with the hardware and that's about it Yeah, doesn't everything do with the algorithms and so forth. This is just a substrate So it's actual safety functions is making sure there's a reliable platform for the algorithm to run on And that you know something hasn't been misconfigured in boot-ups and things like that To where it would be something an integral part of and then it you know varies in scale to a whole bunch of places Where it could have a lot more impact at the architecture level and the interference in the memory subsystems and so forth so we've got a wide range of things that Linux could potentially be used for and so There is no one-size-fits-all There is no one safe Linux because there's so many configurations in each of these configurations makes effectively the different Linux So we're trying to figure out. Okay. What are the processes that what's the analysis that people should go through? To actually understand have they done the right level of thinking to figure out if the links is going to be safe and so we've been meeting and figuring out best practices and trying to understand parts of kernel and work with the community and the kernel community to improve the documentation of things that are going to be relevant and Like say figure out techniques that work for understanding how to trace the subsystems and things that the safety assessors are going to be looking for So our strategy is to get do analysis be able to do open analysis so people can see the methodologies and potentially repeat it and Then making sure that the like I say we work with a couple of safety assessor communities They're participating in the meetings and they're giving us feedback as we're going along And what we're trying to get is things ready for system integrators who are using Linux So what this looks like in practice is like we've got an automotive use case Which is working with the AGL Ecosystem is a medical use case, which is the open APS and we're working on aerospace right now So we're starting to bring up that and the idea is these are interacting with these other groups inside Alisa one of which here is the systems group and Then there's also a tooling group and so The engineering process group is looking at how to work with the links processes safety architecture as a subsets That was in the Linux kernel the features are all the configs. How do you actually config things? How do you build it? And then the tooling is we have a CI flow That's continually building our reference systems and the systems is actually helping to design this reference systems with the components of Zen and Zephyr together with the Linux kernel together with some of the AGL applications So we can have things that people can plug and play into and Then we can test out some of the requirements stuff So the system working group is the one I want to talk about today because this is what the piece that we finally figured out was missing Which is we wanted to basically put Linux in context and start to do the system level analysis That was me. That was missing people didn't you know people were fixated on Linux the box and Understanding when you build it you're building it all those things but certain requirements for when you're building a system Mean that certain things are important things other aren't important And so what at least what we're trying to do with this group is be able to have something with that We can publicly talk about And so we can explain the definitions and how we can actually Come up with processes and recommendations on how to do things in a way that the assessors will find useful So that we can start to look at the analysis of if you're using Linux for this system Is it safe? And if it's not and what could go wrong and then how can you remediate it effectively? Like you need to put something in on the side to watch dog to make sure it hasn't crashed on you Potentially do a reboot. You know things go wrong and can you recover? And it's an important for you to recover. You just reboot reset. Sometimes that's fine It just depends on what functionality you're trying to do at what for the calorie level so what we're doing is Those other working groups are participating in this and we've got this rough model where we basically using the octo as our tooling infrastructure around everything and Then on a set of hardware We started off with the AGL boards from renaissance and Working with Stefano from the then project We have we had a basic reference. We're trying to move it over to some other hardware options right now And then also look at putting different applications with safety functions We can start to move it down into that, but we've got basically the Linux And then we are potentially looking towards containerization with this as well But this has been putting into our CI flow And so we are building it every, you know, we're building it on a periodic basis And whenever changes happen and we're trying to make it available so that this is the framework We're going to use to do that to figure out paths for doing system analysis When Linux is part of it and so that we can basically make it visible in public to people So people can quite frankly throw rocks at it because that's how we're going to learn It's it's at this point in time Everything is mostly done behind NDAs and in silos and inside companies And so we don't really have a way of understanding what group consensus what what really works And so we're hoping by having something up there that people can throw a rock set We can figure out processes so that people can talk about what works What doesn't in a neutral way and not be worrying about NDAs And so that we can actually collaborate on figuring out how to make this useful for everyone because it is such a big beast as you see It is We need to come up with strategies for approaching it We're also very much planning on working with like you say the other projects which you're seeing here And we're also looking at working with pretty closely with the automotive GreenLynx people so Philip Oman from Bosch has engaged in the Sophie and the SDD eclipse things as well So we're working out with these other communities because they've got the same problems This this problem has been solved by the industry yet And we want a place where we can have two things in the open so we can try to find a solution That is the goal of this and you know Philip likes to say you know if you've got an apple and I have an apple and we exchange we each have an Apple but if you have an idea and you have an idea and we exchange the ideas we both have two ideas So we want to try to basically reach out and make sure that we can figure out what these ideas are that might work and Then basically have them documented and have people evolve the technology here and then the last sort of piece on this of this insistent that we're looking towards is Yachto and Yachto is another district but it actually create distros and one of the nice things is it's a cross-compiler Which makes it it builds a cross-compiler, so it makes a very good for working the things that have a safety critical because to a large extent having the compiler is One of the elements that could go wrong if you're building a safety critical system and most the standards are looking for it So Yachto already is doing this and Yachto is also able to quite frankly generate s-bombs and do fully reproducible builds so that those think those criteria are sort of met today by this and It's being used by a wide range of embedded platforms already today and It's maintained by a highly skilled set of people. So it's there. It's being used. It's probably the number one Way that people are working on creating products with Linux today in the ecosystem So having this as part of the equation is I think a key factor too So that those are the pieces we're looking at today They've got the reproducible binaries Yachto supports the sbx s-bom generation the system view though It's done by a master index. So they have all these s-bombs and they just put a master index on it They don't actually link it together in a product line bomb And so they're working and so the Yachto project has been working very closely with the spdx project of the build profile and We'll be seeing when the sp3 comes out. They'll be ready for generating out these product line bombs Or they put all the pieces together and they're doing a lot of work as well with They'd like to get more help actually linking up the p-test results for some of the components They've got a lot of test data that's being generated for coverage and for quality and it's not really being linked up So the question becomes is okay. How does one start looking at, you know Understanding if something changes is it still good or not? Are the requirements satisfied when the requirements are not written down? so This is a boss What else is out there? Why are we not thinking about to have people of ideas here on how we can get things better? You know, are there other open-source projects we should try to bring into part of an argumentation aspect So you don't have questions that also works too Hi actually It's not a simple questions, but can we say that this gen is the better or best solution for the virtualization. I mean for partitioning as a safety application so We actually I'm researching it for a while, but It's it's very difficult because there are lots of the decision criteria for deciding what the Solution is the best For example, we can think about that like a as a computer KVM or Another solution for the virtualization like containers. So have you ever research It before or any yeah, there is lots of the research results in the paper, but I'm not sure about that. So yeah, great question. So I think with Zen it's been around for over 20 years And then is known for its safety and security features again I can't comment specifically without knowing the technical specifications of the projects or the research that you're working on But with Zen as a hypervisor. We have because of the isolated VMs If anything, you know where to happen It's in isolation and the safety and security aspects of it is what Zen is known for so You know for an example one of our board one of our project members They are working on getting send safety certified So that's more automotive, but it doesn't mean it can't apply to things like aviation with things such as the misery Rules that's already a step in the right direction in terms of things like security Again to your specific example I think you know, there needs to be probably a wider conversation in terms of the context that you're referring to I would definitely say send is one of the key hypervisors out there. That's known for its robust features Hi, so first a comment. I mean Zen is very close to my heart because back Like long time ago when I was in the academia Zen the Zen paper it was given to me as part of my qualifying exam PhD qualifying exam and I had Studying that because yeah, our systems T like professors are super hard So I had to study a lot. So anyway, so that that's my question is about like from a tool wheeler's perspective You mentioned about the mistress see Vacation have you identified like how much effort is needed or like Basically, I'm just curious about for a project like Zen which is pretty mature Which doesn't have mr. C compliance. How much effort is needed to make it mr. C compliant? Oh I think that question is kind of it like how long is a piece of string sometimes with code You know a bug is a bug sometimes and because the the history of the Zen project has been around for so long It will take time to adhere to those mr. C rules Yeah, and you had the smooth stats and so they've they've taken a very intelligent approach to it I sit on the safety committee for the Zen, which is why I know these details, but One of the things that they did is they basically had training on mr. C for all the maintainers and that training Basically made the maintainers understand what the mr. Roles are trying to do is reduce variability Which is improving quality and predictability and they got it then and so rule by rule They are moving the code base forward to be compliant in a structured fashion So more rules are being added each time and it's being done, you know released by release and so they're doing grouping of rules the Zephyr community tried to Move to mr. We've tried with your sort of the LTS version But realistically we have to do things upstream the way Zens doing it. I think and so we're going to be in Zephyr trying to Start looking at adopting things rule by rule and make you should the TSC is happy with that Then before they get applied and then try to get the code base coherent that way too for us And in terms of Linux kernel, I don't think that's going to happen in the near future. Okay But anyhow, so yeah, so that's kind of that's sort of what's been happening there. Sure. Thank you And for the record I passed the qualifier and I've got my PSD. Thank you Yes, thank you. I'm using Santa as well very very well especially if it's once on an MV8 architecture, it's very great but my question or adding I would take is I Noticed that especially the bootloader is very important for security critical Booting of sin and my question is do you implement that in your concept as well? We have not put a chosen a bootloader in the code set right now We want someone from one of the bootloader communities to come work with us If someone comes to work with it, we'll laugh we'll have them Okay, because there are many many things you have considered to make that secure very much Especially when you're talking about booting up into secure enclaves and things like that So yeah, no, we definitely it's a part but what we're trying to do is get something that we can plug and play into and if there's you know people with Interest in boot loading in the boot loading paths in the secure bootling of paths if they want to come in and Pre-instantiate some stuff for us work it into the framework Great as the same applies for me to Kubernetes or docker container running on the SVM as well Yeah, that's why we're talking to like the Sophie folk Because they're very much focused on that part of the problem and so, you know It's not completely unheard of at some point until we may try to get certain things working all together as well there and I don't know if Walt has thoughts about the Application layer and making because I know a GL is aiming in some of that direction too. I believe right now, too But at the top at the application abstracting the hardware layer that dad was talking about yesterday Yes, thank you. Okay, so it's mostly with bird IO, which is mostly what Sophie is doing too. I believe So we're using bird IO for the abstractions any other questions. Oops. There's a few more in the back. Okay Yes, hello together. My name is over a buyer from AVS often functions and And I understood that this is somehow a generic approach, right that we that you set the base Which can be used for any kind of projects. So my question would be so which AC level do you target and And how and how can you ensure that that the users get the mature base because open source? It's not so easy right to Everything is shifting sound so what we're trying to figure out is rather than lock it down to one mature base or one Perfect base what processes should you go through to make sure you've got confidence in your base? Which is not exactly the idea that people want to hear But I don't think anything else really scales. I think there's commercial organizations that will give you a solid base But I don't think the upstream projects can and certainly putting things in reference So the best we can come up with at this point in time is to do the processes for this is what you should have these are Your checklists to make sure that things are Um configured properly structured properly so that we know the behaviors and we have the things qualified Not the ideal answer I agree But What's out there at least it hopefully is a step forward and maybe some will come up with a better idea After we move it a little bit forward, but at least it will give people a way of talking without NDAs and Right now We've had many we've had about 10 to 15 years of people trying to deal with safety Within these constrained and rease and they talk with only in their own company and everything is portrayed secret No one can talk to any other so while this is just a skeleton At least it's something that people can build off and swap pieces in and out So we're trying to use modularity and we're trying to basically surface up enough requirements So you can say oh I'm building this system These it's how this is participating this way and so that you can put different applications on the top that are using these parts You know, it's a different argumentation for level depending on what the criticality is for in terms of life You know, there's a scale here. I think in the aerospace We're sort of working to get in the quality side From some of their scales and in the automotive probably we're around at the same place right now quality going forward a little bit Depending on which pieces of code have which safety functions like what functions have been allocated to which pieces of it is also pretty key And that means a system you need to know what you're hooking it starting it with because certain requirements take effect when certain things happen like Your requirements for an EV charging system. Are we going quite different from your requirements for a drivetrain on a car? right and Potentially they could all both be using Linux. So we have to figure out processes here. I think that's what's going to come down to Do you have any other thoughts here or do you disagree completely with me? It's in general is a super difficult approach right seven. Yeah, I know the world also from From classic projects not not open source and if merely super difficult to reach ACLT right everything process compliant as Some of the things are some of the higher a so So like things like the RTOS or things like that then Maybe able to be part of that when putting the safety side of it there and the Linux stuff not have that safety Criticality is doing your infotainment system, but it doesn't have like you know some of the other things that are there so the partitioning is going to be key and There'll be other open source projects participating in this type of event Infrastructure some of which may be super critical like there are some other open source kernels They've gone through formal verification that may be important for play for certain from certain pop you know in certain functionality I'm thinking the cell for kernel has aspects that may be necessary for some of these higher levels because they have the full formalization already there But on the other hand if you're talking with Bluetooth, you may want to use something like Zephyr Because it can you know it's done enough to basically handle the requirements So and the trouble is we don't have a discipline and open source of Documenting the requirements as things get added in There's always a good reason which is effectively a requirement But that unless you're mining your history of commits and so forth and trying to reverse engineer it We don't have those access to that and so figuring out methodologies to say okay if I'm using this function This is a requirement for it to be considered to be working You know that's part of why you're seeing focus on requirements coming from both Zen and Zephyr right now In terms of okay, how can we surface some of this stuff up so we can know we're know we're done And we can do system engineering properly when these things are part of it Okay, okay super. Thank you. Thank you Another question, okay another thought Oops. Oh, we're being told we have to stop but go ahead. He says we can do one more question. Thank you I guess building off of your comment about like Tracing back code to the requirement The ultimate dream that like you can automate like the CI and have like some kind of like Conformance that is actually going to be doing the verification for you And we'll be trying to penetrate to make sure that you were actually doing what you say you're going to do That is the dream. That is why we want to have the tests That's why we're actually working with the spdx community to basically use the relationships and create knowledge graphs So that we can reason about things and build up from there If we can automate this we can't keep up with the scale of vulnerability fixes and that's going to be key So for if we're basically yeah We want to go there. So help is welcome. Definitely Okay. Well, thank you. Oops. I think we've been told we have to stop and come catch us afterwards Yeah, thank you. Thank you very much everyone for participating and