 So my name is Anas Nashif. I work for Intel. Currently I'm working on ZIFAR. I've been working on ZIFAR for the last four years. And today I will start this presentation with some thoughts and things that are happening right now in the ZIFAR project but trying to cover those in general terms regarding functional safety and certification, something that we always want to do within the ZIFAR project. But first I will start with a disclaimer. I'm not a safety expert here. So I actually hope that at least the title of the presentation has attracted some safety experts here so we can talk with you and we can probably tell us if we are on the right path or not. So I'm not a safety expert. And I will be talking as a software engineer here and somebody who has been involved with the ZIFAR project for a while. But I know enough that you should not be doing something like that. In a car you can just go get a clone or pip install and expect things to work. That's not how it works. In this industry and in other industries. So at least I know that and most of you know that. So today in cars you can probably install an application on your console or something. There are all different things. But when it comes to functionality of the car that's not something you want to do or your users to do. So when it comes to open source software this is something new. I mean it's happening right now. There's a lot of discussions about open source and certifying open source stacks and open source code bases. Must prominently is the Linux kernel. So there are a few projects out there a few attempts to make the Linux kernel used in safety critical applications. But there are also a lot of other projects used in this context. So the short answer is yes. And probably we don't know about it because the amount of open source software out there is amazing and there are so many things that you can do with open source software without actually going and making announcements about it. So I mean my expectation and the way I see things that open source is being used all the time by commercial entities for commercial applications and sometimes the owners or the authors of these projects don't even know about it. So I mean it depends on the license of course. But something that happens in the Zephyr project we see a lot of people just pull the code start developing products and they don't even tell us what they are doing. So the idea that open source can be used in a functional safety context I mean my guess is yes a lot of people don't want to rewrite things and they would just go and take something and adopt it to their environment. So they do a transformation behind closed doors mostly and obviously this is not what we want to see but based on what I will be talking about a little bit later that is becoming a necessity because there are a lot of things that are involved when you try to certify or get a certain piece of code to be safe, to be operated in vehicles and aviation and medical instruments and so on. So mostly I mean what these open source projects to get to this level they need to deal with a lot of standards and most of these standards were written without having open source in mind. So just a snapshot, safety standards a few of them there are probably more depending on the industry and every standard has its own variations and so on and there are really too many standards and most of these standards are quite old have been there for a while probably before open source even was as it is today and so there are quite a lot of them and as Stanium Brown said the nice thing about standards is there are so many of them to choose from and so I mean looking at something like Zephyr which is an Altos I will be talking about it a little bit more in details later an example is this even possible to get something like an open source Altos with a small trusted code base and I'm showing the right slide with that small code base and safety oriented architecture security, POSIX depending on the application you want to deploy on top of the Altos deterministic threat scheduling etc etc is that even possible and the question is or the answer is it's really not open source is not really the limitation the main problem is really is that software, open source software usually is developed in a way which contradicts what most of these safety standards expect so most of you probably are familiar with the Cathedral and the Bazaar model so it is, I mean most of open source projects are developed based on the or following the Bazaar model and when it comes to industry standards they expect you to do something completely different to follow a completely different model the V model here in this case require you to have specification of feature comprehensive documentation, traceability you need to trust your contributors and know who your contributors are and the other problem is that as I said earlier mostly the certification authority will not be familiar or accustomed to this type of development so this is the V model here showing how things work and how things are expected to be developed when developing for safety critical applications and probably some of you are familiar with the model or concept or the different software development models but I haven't seen many open source projects being developed using this model it is quite tricky it requires a lot of process a lot of documentation and a lot of tools and it's not really straightforward but this is to get to the level where you can actually have your software even come close to what the standards expect you have actually to start transforming your development model and your code base and tools towards this model or some similar model that allows you to get the kind of traceability and the connections and the gates that are implemented in such model but even before we go there into the software development model and what the standards expect quality is actually the most important thing I mean quality exists as a bricquisite before you even go into functional safety and when I say quality here it's actually more than just my code has no bugs it's about coverage, it's about documentation about testing, test reports and lots of other things that you see implemented and done in certain projects mostly that are backed by commercial entities but when you write code as a hobby and put it on GitHub nobody will hold you accountable if you don't have that if the code works I mean you have contributors they file bugs and so on and you follow a very flexible and really with that I mean you are really nobody will hold you accountable or anything like that code is usually provided as is and if you find a bug as a user people will tell you file a bug or fix it yourself, contribute and so on and this model kind of doesn't work if you actually want to go up the pyramid here so the expectation from most of safety standards is actually to have a quality managed project or code base that will help you get to the next steps so as I said before functional safety considers quality as an existing bricquisite so you can't even go to the next steps without that so some examples of what is required there I mean at least for the quality managed status or even to go further with certification or traceability you need to be able to go and trace first you need to have requirements and a lot of open source projects are developed on a very need basis I need this, I need that so it happens sometimes somebody tells you on IRC hey how about this and you go, oh nice idea you go implement it and within a few hours you probably have this feature without any proper documentation and probably even without testing and that usually works but when it comes to going to the next level you really need to document the whole process you really need to have the design and the requirements and the implementation and all the testing, functional unit integration testing, all of that and you need to have the dots connecting so you really need to so every single line of code has to be connected and this is like a commercial tool LBRA that actually shows you an example of how is this done so you can go all the way from a requirement to the implementation to the test reports and you really need to show evidence that every line of code that you have in your code base in your project is well tested is well documented it's based on a requirement that goes back to a customer or a user and that's something we don't do in open source projects usually I have seen that demonstrated to me a few years back and it's amazing what they can do there and I don't think anybody developing stuff on GitHub would be able to afford this this is very expensive stuff happening here mostly it is used for commercial application by people who can afford that but I have bled like with the concept of traceability in the context of the Zephyr project and I find it actually really useful because you can see immediately what is missing and what tests you need to add or which requirements are not being covered but we have done that in the Zephyr project using existing tools I don't have an illustration here of that but I looked at our project we are using Doxygen and we have test cases we have requirements so I use Doxygen because Doxygen scans all the code I use Doxygen and I did like custom commands in Doxygen and I was able basically to go all the way from requirements to test cases to test results so there are ways to do that in open source and there are a few attempts and scripts that you can find across the internet that try to do something like that but obviously it will not compete with a tool that does everything for you but it is possible the problem is that it is not being done widely in open source so another thing that some of you probably are familiar with is Misrasee for example so Misrasee is basically a recommendation it's not a standard but at least the standards are asking for following some style guide and Misrasee although it comes from the automotive industry the A in Misrasee here is automotive it's actually the M motor I think it's widely adopted in non-automotive application and depending on your views and how you look at Misrasee you might actually just hate it because it restricts you with what you are doing and any attempt actually just haven't been looking at Misrasee and looking at what people say about Misrasee in open source project it's not or at least some rules of Misrasee are not well received a lot of rules actually make a lot of sense and I really understand why it is very important to follow some standard at least to when more people, when companies when more users are using your code you don't expect them to read into your mind and understand what you have thought when you were writing some certain line of code there are different ways to do things and there are there is code that is easy to read and immediately understand that you can write exactly the same code in different ways and following a certain one guide that is, I would say it's a de facto standard helps with making the code unambiguous and easy to read and so on obviously it's also find out the way some rules are helped with functionality as well it improves not only the readability but it can also help with making the code actually more secure so the challenges with Misrasee there are a few rules as I said before that are really very controversial so we really need to figure out a way how to deal with them you really, I mean Misrasee gives you the flexibility to go and deviate ignore some rules if you document them so we really need to decide how we deal with those and which rules we want to deviate from and we need also to incorporate that into the guidelines not everybody knows which rules are what so how do you do that in open source when Misrasee, the specification or the document is proprietary, it's not available you can just go and browse for that you have to be 15 pounds or something like that to get the document so that's a short stopper actually for an open source project because in the review process if you go and tell someone contributing to your code base that this code does not comply with Misrasee some people will tell you what the hell is Misrasee and where can I read more about that what is the justification what is the rationale behind it so that's a challenge for an open source project I have the document, I mean a lot of people have the document everybody who is contributing to your project to go and pay 15 pounds to actually get the documentation so this is a major challenge and we have to find a solution for this also the other thing related to that is we really need to find an open source tool that integrates with our project which is all publicly available so how do we integrate that with CI so we started the project four years ago or something like that with like we integrated Checkpatch like from the Linux kernel and that's like well integrated in our CI if somebody submits something that doesn't comply with Checkpatch you get a minus one automatically from CI and that works really nice we have a few other things like forget comments and so on and we want to add more and we want to be able to give people input or feedback about their code immediately before it's even checked in and with MSLC this is going to be a problem right now for scanning Zephyr we are using commercial tools and mostly we are using that behind fire walls and in our companies so that's not something that we can share and trying to do a scan on a full project like Zephyr that can be a lot because you will get a lot of violations there so you really also need to decide which parts of the code are important to you to run them through the scanner and also as I will be talking later I mean not only for MSRA in the MSRA context but also for certification so the scope is also very limited the scope is very important sorry okay so just an example of a rule that is controversial it's like a function should have a single point of exit at the end I mean this is a lot of you are probably familiar with that I mean based on MSRA it's like the master readable structure it's like you are honestly omitting function exit code but this is something that you could probably decide to deviate from and create a deviation the main problem is that this is required by many safety standards so this is something that depending on what you are implementing you can just go and say hey screw that I'm not going to do this you really have to go and look at your code and sometimes it's easy to resolve this type of issues sometimes it requires major changes and that's a problem so this is exactly one of the things that is also mentioned in the MSRA C and also in general most of these standards or guidelines make sense if you start from scratch basically when you start developing your project you apply these rules immediately having to do that after the fact is always a problem because it means that somebody will have to go and change the code and probably introduce additional issues and the code is working obviously right now so there is no bug but if you go and start changing the structure there is a risk there ok next one the other thing you can be basically a project with lots of features and almost complete and you have a lot of processes in place and so on the main problem is that you can have the best project out there and the must secure that you have built on your own as part of open source community the problem is that nobody will look at that because there is this whole adoption barrier at least for open source project for a company and OEM and somebody in the industry to go and pick this leader they expect somebody to be accountable they will not look at hobby projects and and use them as is people are looking for somebody to point the finger at what happens, who is liable if something goes wrong so this is tough for a lot of open source project and even if you have a certified offering open or even proprietary you really need to have an accountable entity behind it and and it's very difficult to have early adopters even if you have that because nobody wants to be the first one to use this so the first question you usually get is who else is using that I am going to be the first one, I am going to be the guinea pig and etc so that's a problem that we will face and we are facing on a daily basis also in the context of other things, not only functional safety so how to approach the certification in open source in general so something that I mean you can see and it seems like it seems to be the consensus here is that if you want to take an open source project and drive it through certification probably the best way to do that is actually by snapshotting your sorcery creating a branch and having basically run it under structural control they are validating and control and and try as much as possible to separate it from the continuous open source activities and contributions so basically it's building a cathedral on top or even beside the bazaar so you really have to run two models and the other thing is getting the supported feature set right is very important that limits the amount of code, the amount of documentation process you have to implement to get to your final goal so trying to do for example you take download the Linux kernel and say hey I'm going to certify that that's going to be a lot of work so you really have to go and start removing the things that really don't matter I mean I heard a lot of numbers that somebody was telling me that when it comes to functional safety every line of code is $150 to $200 in terms of engineering process, development certification, validation etc so every line of code counts in this matter and this is general basically it's also from a security point of view it's reducing, getting the same functionality with less code that's always better then you have less of an exposure and exposure surface or attack surface automate as much as possible this is as related to the traceability you need to be able to trace everything and you can use commercial tools but if you can actually manage that within the project using some scripts or some automation tools and get all the dots connected that would be great so you have to think about it ahead of time what tools do you want to use what interfaces you want to expose to be able to generate the final collateral for your project and also the final one here is try to get a concept approval at least or some guidance from a certification authority as early as possible because you don't want to go all the way and be told at the end that whatever you are doing doesn't make sense or your plans are not going to work so you really have to get some consultation and some approval for your concept as early as possible so the ideal project here is has a split development model flexible open source instance auditable and controlled instance the auditable instance is a branch with well defined scope developed with stricter rules and with an entity behind it the flexible open source is business as usual developed in open with the community participation the auditable instance aligns with the open source instance at a cadence that need to be defined based on the entity and certification cost the entity running the auditable code base need to have some kind of experience when it comes to certification on the other side the open source community will help enrich the open source instance and basically drive innovation and keep your community happy in this case here so basically when looking at the whole idea is that the conclusion is that you really can't have use the same model as we are accustomed to in open source to do something like that you really have to have another track for your functional safety activities and that's really important and I think there are by the way I didn't even go into the standards and what type of software they accept there is like proof by use reusing existing code there is a whole lot of rules and documentation about how to deal with something like that we really need to comply with this is the most important part you really have to split the development into two so an example regulating the bizarre you want also your open source instance if you want you want it to run by certain rules you can just have the wild west there so you really want to prepare whatever code goes into the tree in the open source instance has to be regulated to a certain degree so that if you decide to sync or rebase at least have another version that is certified you don't want to do the work from scratch so it's very important to run the project using rules based on our experiences with the Zephyr project the code is available publicly and it's very important that it is scrutinized by everyone we are using code reviews to improve the quality the question is do you have the right set of reviewers who gets to have the final say how do we guarantee that the reviewer is aware of some safety implications and how long for example should changes be reviewed and just to give you an example of where things need to be improved at least on our project this is like a snapshot from data we built from Github the project is running in Github and shows you how many days it takes to merge a build request as you can, this is a logarithmic scale so a lot of our build requests are merged like three, four thousand out of six thousand I think are merged like within the first day so this is like mostly trivial stuff and things that have been probably discussed before the thing has been submitted this is good probably for many projects but now we are talking on the project we really need to establish some rules there I mean I mean several projects are developed across the globe sometimes I submit something North America you know and that might touches some area that is under the responsibility of somebody in Europe or China so it's very important to give people more time so we really need to establish a set of rules of how we want to get things in so we need to change the curve here and try to move it at least have a few days to get proper reviews and then also wait for the right people who are responsible for this line of code to actually do the review and not have somebody else who is not responsible basically have the code on us review the code so we really need to change that another thing is that you face an open source project is sometimes you struggle with the fact that not a lot of people like to do reviews so you see a lot of people submitting a lot of code all the time but they don't review anything review burden is like left to a few people involved with the project and this is another problem and we really need to find a way to solve this I mean I still need we're thinking about a karma thing if you want your stuff to be reviewed you have to review some other people code but this is a huge problem so for whoever participating here in this room in the Zephyr project we are asking for reviews we need more reviewers development can happen at any time but we really need the viewers so going back to what I said before about the two branches this is something that has been done in the past safe artos which is based on free artos was started this way so there was free artos and a few folks I heard decided that they want to take free artos and make it usable for certified so it was done this way and now however self-artos is a commercial application this is not something we want to do we want actually to keep whatever we do in the Zephyr context in the Zephyr project we want to keep it open source so it has been done in the past so talking about that have been mentioned all of that where are we with all of this with the Zephyr project so looking at this we are like at the very bottom we have to admit not at the very bottom but there are 999 steps here based on some information I got so we are there but I think we have the right tools to go up fast we do have the tools and the ingredients to get there fast so it is not going to be one step at a time it is going to be more than that and that is exactly where the Zephyr project comes a lot of people are facing this and having to deal with that so the project is to run something I mean industry wide not backed by one single company or by one individual it is actually to run it as a governance as a Linux foundation project with participants from across the industry and in this case we have the support of Intel Linar or Nordic and XP and a few other silver members here who are equally basically interested in that and participating in these efforts so the Zephyr project and this is like the intro at the end of the slide they are the Zephyr project I usually do that at the very beginning it is fully featured it is not only about safety so there is security and there are a few talks about that there will also be a discussion on Thursday there is a hackathon running there will be a group discussing safety and security in general Zephyr is configurable and modular and that would help us defining the scope and limiting the scope for certification so we actually will go to the bare minimum and try to certify that cross-platform right now six architectures plus being able to run on Linux as a Linux executable obviously open source under the Linux foundation with Apache 2 license and we have a very rich IB stack and Bluetooth stack so it is fully featured this is the architecture here very abstract but it shows it is more than just a kernel and trying at the same time to solve different use cases why Zephyr the main reason why this actually would work for this problem is that it is fully open source it has a governance model and it is not it is not the product of this company or this individual it is a community effort led by industry industry names famous names like NXP Intel Nordic and Linaro and we are trying basically to address multiple problems at the same time that we have faced when we started this project trying to solve the fragmentation and people say ok you are trying to solve the fragmentation by introducing yet another but in this case if you look at it it is trying to do things really differently and when combined with other projects we are basically we have our goals are a little bit different especially when it comes to functional safety so there are features for the next year or what we have accomplished obviously safety and security are on the top priority but there are a lot of other things going on at the same time so it is not only about that we really need to address that end to end when it comes to compilers, to IDEs also to connectivity features so I talked about limiting the scope and that is basically what we are trying to do we are not trying to certify the whole thing we are basically going to take a limited set of features that is comparable to commercial offerings at least so we can compete in the market and certify that so we are looking at the kernel scheduler and a few kernel services on top of that and this would also help us because it is the first time we are doing that and it would give us some confidence to move to the next level and to middleware and application for example adding an IB stack or adding bluetooth, this is going to be significant but we want to do that as a second step in terms of candidate start-ups we are looking at IEC 6158 for security we are looking at common criteria and there are a few things there for other industries that we are looking at as well this is the IEC 61508 is general and it's like that the mother of all other standards if you want so getting that would help us also go and look at ISO 26262 and others ok so Zephyr long-term support LTS this is actually our upcoming release and we are actually going to take LTS and fork it not fork it, branch it and maintain it long-term and from this branch we will actually create the mentioned auditable code base which will be taken through certification the idea is to keep things in sync and contribute to that I mean this is experience with other project this is usually very tough to keep things in sync especially if you are frozen in time in terms of the code but we are talking about a limited set so that should be easier than maintaining the whole branch for certification at least so I mean just to visualize that here in the architecture we are looking at the kernels there are interfaces some ABI's on top of the kernel like devices, there were peripherals and kernel ABI's and some portability lines like SpaceX and a few other things there so the summary here is that functional safety and security requirements need to who exists with the open source nature of the project so it's not going to be one or the other it has to be both of them at the same time everything is open source quality needs to be driven on the project level so we need to showcase our quality process and test it publicly so we really need to start doing more of that publicly to show that we are serious about that and we are serious about that and also drive the adoption through quality management release process I mean by showing that we care about quality we will get more participation from the industry to drive this effort also we want to manage developer and contributor expectations I mean when you are developing in this environment obviously the expectations are a little bit higher in terms of code quality in terms of processes that you need to follow and we will want to continue innovating we don't want to stall and say ok we have done enough to work on code style and fix bugs we really want to encourage adding new technologies and new ways of doing things and that's something that we have been doing really well so we are not really focused in the project on end-to-end solution we are focused that has its own advantages and disadvantages so we are more a horizontal thing where a lot of the features of the value is actually at the low levels if you want an end-to-end solution this is also possible but you have to do a lot of things on your own but we are going to address that as well and then we need to also somehow officially establish accountability and trusted entity to deal for example something that is in the Zephard Charter by the way is to have a certification and somebody who actually drives the whole thing somebody with the right experience to actually drive the whole process and manage it with that thank you very much if you have any questions go ahead if you go back to three slides you have this architecture diagram yes this one there seems to be a clear partitioning between the parts or the modules for which you need certificate and the parts for which you don't need that so I wonder so you talked about the process you want to basically merge back stuff into the certifiable version have you explored an alternative where in the same tree you split by components and you have the core parts that are certifiable and basically the open-source contributions with a less strict process only to the parts that are not aimed to be certified yeah I think this should be possible the main problem right now with Zephard it's like all in one get three and we are trying basically to isolate that to split things and the idea is not a lot of changes need to be going into the kernel we really need to bring the kernel and the core services to a state where it is basically what we want we are just going to fix bugs and drive the certificate so we should be able to do that and the idea is that if we do that you should always go and take like Bluetooth from upstream like from master and make it work on top of it because the kernel on its own will be useless if we just maintain that so it's very important and we will be able to take components from the blue side of things and make them run but what you are saying there is that this is certainly possible but we really need to get there first to make this assessment and see if that's actually a workable but that's obviously a more attractive solution it could be even split into two different trees one for the orange part and the other one for the blue part and that's what we are trying to do right now any questions? go ahead you talked about first adopters do you have anybody and all the vendors the supporters of their own do you have any example where people who have actually deployed this into real safety critical systems? yes but I can't talk about it it would be interesting I need to check but Zephyr was deployed in this type of environment already and the expectation is that once we have something some of the members of the project will start pushing that into their customers so Intel and a few others will be pushing that into their customers because it will be hopefully cheaper and it's something that you can innovate on so instead of depending on third party commercial solutions that's the whole expectation this is going to be supported and pushed by the various members but there are already uses I understand but there is a reason I mean used in a safety related product but we still don't have the certification all the way and this is coming because you can use it partition yourself to model like QM and cell 3 and depending on your safety architecture you certified the lowest level of the S&P you certified the lowest level like power management to interrupt and you mentioned the unit test so how hard is it to unit test those things and could it have been applied to Linux for example I don't know about Linux but I mean this is actually the way we are developing I mean the unit test and the testing is actually bare component already so that's already addressed I'm not sure what would be I mean the testing is done on a component level so you are not looking at Zephyr as a complete solution you look at components most of the time because it's a configurable and modular so not every application will have the same features so most of the testing is actually happening on the component level so you can already do that they mean some things are more challenging to test than others yeah there is obviously there are things that easily tested by just building an application and running it you get the data there but certain use cases would require complex setups like when you do networking or bluetooth or multicore and stuff like that but it's all tested yeah it is tested thank you any other questions so you are talking about replacing certain components with something that is certified or running that on top of something else like if you have a hypervisor which is certified run it on top yeah I mean that's definitely possible and that's actually a use case we want to address as well Zephyr already runs on top of ACON for example and we are working we are talking with the ZEM project to do that as well so it is yeah it is definitely possible it depends on how your safety architecture and how you where you the highest integrity level is going to run okay I guess we are running out of time here thank you very much