 Okay, so hello, my name is Kate Stewart, I'm VP at the links foundation and one of the projects I spend a lot of my time working with is the Zephyr project. And I've been with the Zephyr project since this beginning, in 2015 it was one of the first projects I started working on at the LF. And what I tell you a bit about is I've also been working in open source for a long time and been very involved with the Linux kernel and I liked a lot of the practices of a Linux kernel. And so some of these ones we tried to explicitly use in Zephyr and that's what I want to show you and talk to you about some of these practices. So it's changed a bit since I started but it's still pretty much the same. In the sense that there's a lot of open source R tosses out there. And the question becomes is which ones to use and where should you spend your time. And so one of the things I've started tracking about in 2018 is what is actually happening in them. So every month for three or four years, now I just do it every quarter now, I would go in and I would go into GitHub and all these projects are on GitHub. And so one of the things is if you, and this is where those numbers came from, okay? So you go to the project and you see how many commits there are and the total contributors and that's on the first page. And then if you go into the monthly summary you'll see how many people push things in the last month and how many commits they've got. And if you take the number of commits, monthly commits and you divide by the number of days in the month and the number of 24 hours in the day, you get a rate which is how many commits per hour. And so these commits per hour is a useful sort of indicator to understand, okay, what sort of velocity the project is having. But when you go and look at this and you look across these, you don't even need to get to that stage to really understand. I guess, well, is there a pointer? Maybe? Yes, there's a pointer. Okay, so the first commit in these projects sort of tells when they started. Then we see who controls the commits. Is it a community or is it one company? And generally you can say that if it is a community they tend to do a lot more innovation and a lot faster. In companies sometimes there's a lot of innovation inside the company, but then it's just one person who can make it outside. And so that sort of skews things a bit. But then you see who's actually contributed. And if you look now, there's over 1,400 contributors into Zephyr as of December 1st. Which is, well, probably more than double most of everything else, possibly usually three times or four times. This is the next highest one for double contributors. So yeah, it's at least double the number of contributors in anything else. And in the last month, 183 people were contributed compared to the next closest was Nadex, which is a good community project. I'm not saying anything against these projects, but there was 49 people. And you see now over time there has been over 70,000 commits into the project. So there's a lot of momentum. And there's a lot of features going in and a lot of bug fixes going in. Both are valid commits. And in the last month, for instance, you can see that we've had over 1,200, which gives us that rate of about 1.7 commits an hour. So how does this compare to Linux? Which is what I was taking as my sort of starting point. Well, from the 5.9 kernel, it's about nine, there it is, nine commits an hour. So it's about a fifth or 20% of the rate of the kernel. But given that the kernel is one of the fastest moving in the biggest project for an embedded OS, it's pretty significant how this all works together. So how Zephyr achieves this momentum is the question. And so some of these are achieved by best practices. And let's sort of think back to what Linux was like when it started back in 1991. At the same time, there's a whole bunch of Unix sources available, Minix, VSD, SVR4. And there was a bunch of commercial, different curational distributions. So we had a very wide set of Unixes out there. And to a large extent, Linux was a good second option. It wasn't the best option, certain features or anything else. But it was something that people could contribute to and build on. And now today with Linux, as you see, it's pretty much dominating most of the markets for an open source R toss, except when it's sometimes it's too big. And so what we're trying to do is have some equivalent for those smaller footprint places. And that's where Zephyr's coming in. And that's where we're trying to get Zephyr working. So use Linux. It's got the best features out there, if you can. But in those small sensors, the actuators, things that have to have battery life that last in years, tens of years even sometimes, Zephyr's is worth considering because it doesn't consume as many resources. So in 2016, 2017, I was at the LF here, there was a, I guess, Greg Crow Hartman and John Corbett put out their Linux kernel development report. And in this version of the kernel development report, they started highlighting some of the features that they think were been significant to Linux's success. So one of which was short release cycles are important. Having the hierarchical developer model, having maintainers and a hierarchy, so you know how the changes will flow into the kernel. Tools matter. We all had get by this point in time. And a lot of the velocity of the kernel didn't happen until after it came in place. Consensus model is important. Making sure that people all agree. If sometimes that takes a long time, but it tends to be a good place for people to feel like they have not been shut out. A late factor is no regressions rule. You can't make things break. Early days in the kernel, things were breaking. But once they adopted this no regression rule, this has been part of the velocity of the kernel. The real effect is that corporate participation is critical. Companies, it's not just obvious. It's companies and people doing jobs because they can make money off of it. And there's a virtuous cycle. So having it, corporates participating as well as community people and people scratching their own inches is what makes it effective. And there should be no internal boundaries in the project. Just because if you work on one driver, you should be able to go work on the print K stuff. You should not be blocked into anything. There should be no internal boundaries. So these, Zephyr started in 2016, early 2016. So a lot of these pieces of recommendations as to why it's been successful, we were looking at with the project. And there's a few other sort of lessons that have people learned. Keeping things vendor neutral is probably very important. The mix of companies and individuals by scratching their own inches. Quite frankly, it's not in that set but making it easy for people to do upstreaming work. By using the DCO, and I'm a very big fan of the DCO, you do not have to go through a whole bunch of lawyers generally to do contribution license agreements. And you also end up with something that is not going to shift out from under you. And this is one of the things I kind of am strong. Having the public codes, having the way of signaling whose reviewed things is a useful practice and very valuable. And then we've got the consensus or a decision making the hierarchical development model, no internal boundaries, tools matter, short predictable release cycles with fixed merge windows and then having the stable in LTS. All of these are factors I think that have contributed to the success of Linux over the years. So, and one of the key things to understand is when developers get frustrated, they get creative and they do good things. And Linux is also showing some of that. We would not have get if our infrastructure for version control was smooth. So these are the lessons learned from Linux. And so when we start Zephyr off, we wanted it to be the best in class RTOS for connected devices. And so what we wanted to do is say, okay, this is our vision. How are we going to do this? Well, one of the things we did like most Linux foundation projects do is we focused at TSE for making the decisions about technologies. And the governing board makes decisions about money. So we separate the money decisions from the technical decisions. And I can't stress how important that is. But, you know, the developers, we basically decided to use K config and K build from the kernel because we want to be able to configure and we want to be able to share device tree information and so forth and be able to put in what we need it because Zephyr is targeting from tens of Ks up to whatever you need to put in. But, sorry, too much talking. But this is, no, we have it. When we launched Zephyr, there was a unified kernel. Sorry, there was a nano and a micro kernel associated with it. After the first year, the developers decided, no, we can just do a unified kernel. It doesn't make sense to carry this overhead. So they changed that. We started off with Garrett and Jira. And they went, no, we want to be on GitHub and Issues. So in 2017, we moved over to GitHub and Issues and that's helped accelerate things for Zephyr. Again, keeping it easy for people to contribute is key for people to do it. Our build system went from K build to CMake because that's what the developers wanted to do. So the APIs and the hardware extraction levels have been reworked several times and continue to evolve. The modularization device tree support is key to Zephyr and then we had started it with a view that we would go after LTSs because we saw how successful that was for the Linux kernel. So these are some of the technical decisions that happened. And so taking these lessons learned I was going through from Linux, vendor-neutral decision-making. Well, yes, the project has support from multiple companies. All you have to do is look at the history right now and you'll see no one company will dominate Zephyr. We see a lot of, certain companies have more contributions than others, but we are not, I was very conscious of not having an elephant factor and we explicitly tried to nurture a lot of companies participating. Companies and individuals participate. Yes, our TSC has both in them. We streamlined the upstream process. Again, you can see the contributing document and we use the DCO. So it's very easy to just start contributing and sending patches and pull requests into Zephyr. As long as you agree to the license as per the DCO and the license for pretty much all the code in Zephyr is a patchy, too. There's public code reviews. All the pull requests have to be signed off by at least one, I think sometimes two people, depending on where it is in the kernel we're heading to. The consensus-oriented decision-making TSC, sometimes people will disagree on the issue in the pull request. No surprise. A lot of the discussion happens there, actually. And sometimes they get explained to the TSC for the discussion. But there's a forum and it's not one person making a decision. It gets in front of all the TSC voting members and they get to vote. So the consensus is happening that way. We have the maintainers file. I think we had co-noters initially. We've moved from having maintainers formally. And so you can see exactly who maintains which part of the subsystem. One of the things we're trying right now is the notion of co-maintenors. And so we can onboard new people into maintaining. Because a lot of the times the problem is the pipeline of maintainers and the bandwidth of maintainers. Because they're doing this sometimes in addition to their job. No internal boundaries. Same thing. See the contributing for distributed version control. We have a ten... The Linux kernel has a two-week merge window. And then several weeks of stabilization. Zephyr's chosen to have a ten-week merge window and then two to four weeks of stabilization. And that's working for us so far. As Zephyr potentially gets bigger. They may revisit that discussion. But this is working right now. And then long-term support. We actually had first long-term support. Long-term support one had four release updates. And we're on long-term support two. And every two years we get an LTS for this project. So as you can sort of see we've been trying to apply these lessons we've learned from Linux into Zephyr. And as a result we now have... We start with three architectures. And we now have all of these hardware architectures. We started off with the M-core from ARM, Arxenopsis and Intel X86. 64 bit has been added. A-cores and R-cores have been added. And then 32 and 64 bit risk five. And most recently people start putting some of the MIPS architectures in there. Because they wanted to place to pick up some of the new communication technologies. And so by putting a new architecture in here you get suddenly the value of all these other drivers that are already there and all these sensors that are there. The architecture at this point in time is pretty complete. Or at least has a lot of good communication technologies. There's things we more still want to add. Like for instance I'd love to get... I'm talking to some of my members, some of the members and some of the community people about seeing if we can get matter into Zephyr. I'm getting matter stack into Zephyr. And then there's... But we've got a very good Bluetooth low energy stack. As well as Bluetooth Mesh. And the audio. So the Bluetooth stacks are very well tested, reliable and they're all integrated. You don't have to get something else extra and bring it into Zephyr. Because all of this is sitting in the code repo. It's all under K-config. It's all config like say. You basically configure it in through device tree and it's there in those images you're building. Which makes it very fast to go to market and to get new things working. This... We follow a structure of our code repositories where we are getting the development happening at that rate of 1.7 an hour. And then every two years we cut our long term support. This will get if there's a vulnerability or a key bug fix. This gets updated. And then a subset of this is what we're doing the full traceability on. That's part of doing something taking it to auditable. And this is what we'll be taking to get 6150 weight certifications on. So if there's anything that has to change we're trying to make sure it's all kept up to speed. And this is how we're dealing with the fact that you don't want to take things to safety unless they're moving too fast. So we're trying to work up stream and then have that LTS ready to go. So LTS, like the Linux kernels, product focused. They're keeping KERMA security. But we're not putting, you know, but we're not putting new features into it other than potentially people may want to use it for new hardware or put their own versions of new hardware in. And it is, it has a much longer testing cycle and we try to keep it supported for two years. The Linux kernel has different varied points for their LTS's and then the similar structure platform is even trying to make it longer. If we find community people that want to be maintaining this for longer periods of time, the discussion I think will start. But it's not feature-based and it's not cutting edge. However, when you're making products you don't want that necessarily. If it has the features you need, you don't want it to be changing out from under you, but you do want the security fixes if it's significant. And we basically have demonstrated by doing, and so we've had four LTS updates that have come out in the last few years. And auditable is being worked on right now. The modularity is happening by device tree and Kconfig. And this lets us potentially share some of the device information with the kernel drivers as well as Kconfig lets us figure out exactly what goes in. And so you only put in what you need. It's statically compiled in at build. Okay? So you're not bringing a lot of crop that you don't need. And this lets us keep these images small. What about security? Big topic, right? Zephyr was one of the first adopters of the OpenSSF badge. And we became passing within the first year after it came out. Interestingly enough, you saw the decisions that the developers made about changing infrastructure and things like that. We stopped passing because we didn't update our documentation. And the websites weren't working. So this is kept, you know, has some automatic checking behind. So once we failed us, it's like, oh, we've got to get passing again. And yeah, we may as well try to go for gold. So Zephyr was one of the earlier projects that actually has achieved the gold level. You can see all our processes documented. And there's currently about almost 5,000 projects right now and only 12 have gold badges. So like I say, we're taking these types of best practices very seriously. You know, the nice thing about doing badging and I would encourage any project that you're working with to try for it is it gives you a framework for figuring out and adding best practices. It sort of guides you through improving your security posture. And it also helps you create onboarding material for people in the security team. And it is pretty transparent. It lets you be transparent about attesting to what you're doing in the project for other people who are coming in. So encourage you to go look at Zephyr's badge. Go look at what we're saying. If you think something stale, open an issue. We'll fix it. But this is sort of like our public documentation of the things that are important for security. The other thing that the Zephyr project has done is we actually registered as a CVE numbering authority. So CVE numbering authorities are companies. But we felt it was important that we manage because there isn't a lot of companies and it's multiple companies, it's not just one company. We wanted to manage our own vulnerabilities. And so we have a security team that brings and gets the information for vulnerabilities as and has a project security as a response team or p-serve. It's volunteers from our various members in the community. And they are the ones who are reviewing this security architect and then deciding if things get fixed or responding back and getting more information. We want Zephyr to be using products though, right? So when there's a vulnerability, people need time to fix it in the field. And what we wanted to do was make sure we have an embargo period so that things don't get exploited or shared before their time. So the Zephyr team is committed to fixing within 30 days and then give vendors 60 days under embargo so they can update the products that they're building on those issues. So this type of information I think helps people balance between what's upstream and what they've got for products. And that's what we're trying for. So to sort of summarize how the involutions has been working, security team started right at the start of the project. It meets bi-weekly. The security practices were documented. Sorry. We've registered the CDN numbering authority and been working with the upstream vulnerability management systems. The best practices badge was met in 2019. We're doing a lot of automation. A lot of testing that goes on with Zephyr on every commit. And we're doing periodic coerity scans and misery scans to ensure the code base is kept clean. We did some improvement on our vulnerability management in 2021 as a result of a bulk vulnerability report. One of which was changing our process for the embargo. The other of which was making it so that people who have products so if you've got a product that's using Zephyr and you want to be notified you don't need to be a common member. It would be nice if you did to help support the infrastructure but you can register for free. You just have to let us know and point us to your public product. And then you can effectively get the notifications under embargo, the embargo period. And then in 2021 Source S bombs have been created by Zephyr since 25. However, automatically generating a system bill of materials for the built image and built infrastructure for Zephyr. So Zephyr right now with one command line change you can basically generate two Source S bombs. One for the Zephyr sources one for your application sources and then a build S bomb which points back to those Source S bombs. So you have traceability exactly which source files made it into your libraries and which libraries combined to make it into your final ELF image. This level of traceability is key and embedded because what we need to do is not try to fix things that aren't broken in places. An example of this was Fnet. So Amnesia 33 happened a few years ago and it had a version of Fnet in our LTS1. TIP didn't have it anymore but Fnet was in LTS. If you just looked at the version numbers you would think okay I've got to go and replace everything. Well actually when you go down to the source file level and you see it none of those Fnet sources that had the vulnerability were actually in the Zephyr tree. If you just looked at the version number you would have wasted a lot of time and effort which is why we need this traceability because updating things and embedded in the field is expensive. It's not just pushing a button that suddenly goes there. There are systems that are working in that direction but being authoritative and knowing you're fixing only what you need to fix you need to have this level of information. So what are the results of applying these best practices? Well there's 4,400 forks of Zephyr out there right now and so part of our challenge is figuring out what are people doing with some of these forks. Some of them you get by word of mouth and some of them you get by looking at things and digging around. There's a board in Brazil that's working off of Zephyr that we learned about over the summer from talking to people and we also see smart devices so we're talking to people creating devices and so people are adding AI to Zephyr like TensorFlow Lite and so on and so forth. So this one here is electrical grid monitoring so they're looking for an on-lease electrical grid and this box here has to be working for 10 years because you don't want to be climbing up the pole to be changing it but it's sending information. This one here is sitting on the top of garbage cans big dumpsters and it's actually in there and are they full enough that you have to get a truck to come by and dump it or not? So you're not running around all the time dumping things that are empty but you can be a bit more selective and then any... Show of hands, who's actually been on the ICE trains in Germany the Deutsche Bahn? Just one hand. Anyhow, these are like the bullet trains here in Japan. Oh no, no, no. No? Anyhow one of the latest ones we've figured out recently is we learned about... there was a talk in German that I couldn't understand but people translated it for me and this sensor pump has a zipper in it and it basically pumps out the wastewater tank and makes sure that it's properly empty so the trains stay in service and they don't have to stop. So combining AI sensors and these systems gives us a way of doing something that is more than just heuristics we can actually train things and then there's a variety of other products that are out there that we found so if anyone here in this room has a product running Zephyr please tell me about it so I can make it visible because embedded is hard to find products but ODECON's hearing aids these ones here are running Zephyr okay Winterbinds are running Zephyr so we've got this whole spectrum so our watch is tagging devices are running Zephyr and one of the ones that you'll be seeing showing up next year is Next Generation and it's public because they gave a talk about it so I can say it but the Next Generation of Google Chromebooks will have Zephyr in it to monitor things in the low power state so they'll be running Linux on part of it and then Zephyr on other parts and we've seen these types of uses where you use Zephyr for the low power side of it and minimal sucking of power and resources for all these applications there's over 400 boards in the repo right now so if you've got a board or you're working with another board it's a good chance it's probably already to have a port to Zephyr there and if it isn't there there's probably something that's close that you can work from because these are all devices and they're standard interfaces and there's only so much I be there so I would encourage you to look in the repo directly given the link is here boards.html pretty straightforward if you're familiar with working around the Linux kernel repositories you'll find a pretty familiar there's the architecture, the boards and so forth and there's over 100 sensors integrated already so picking up these sensors, picking up these boards if I'll be putting you configs in is how we're getting all those commits the bulk of the commit work is happening by new boards new sensors, new technologies integrated in different ways and there's a native IP stack already integrated so you have including you know you've got full IPv4, IPv6 you've got time-sensitive networking you've got BSD sockets, API, uploading so forth and we do a lot of testing we're using the Maxwell Pro for testing to make sure we don't regress in there so we're trying to use some of the commercial services and so this is what quite frankly membership you know membership helps us support because those aren't cheap and so that's kind of what's going on there we have a wide range of protocols common protocols for communicating out and we are supporting a wide range of technologies here on the networking side already and Bluetooth host and mesh technologies are there the low energy and Bluetooth classic are there so as you can see they Bluetooth SIG actually uses Zephyr for prototyping new specs so there's a group that's in Zephyr and also in the Bluetooth SIG and when they're working developing the new specifications they'll work in a private repo to make sure that the specification is solid and then when the specification becomes public they're for Zephyr so we are kept very close to the edge of the Bluetooth and I'd like to have that type of arrangement in some other places too and then low energy obviously because Zephyr really cares about the low energy and don't worry I'll put these slides up after okay but you're welcome to take pictures it's fine and then we have a fairly complete USB stack and you know how I said we have TensorFlow Lite and those smart devices are two samples of TensorFlow being integrated with Zephyr already sitting in our documentation repo to show you how to start doing it one is a very simple sine wave one and this other one here is for like a magic wand so you have the training and you can get the information and so forth so these are here to help people unwrap and help people start to work with the code base effectively and hopefully not have too big a learning curve we're wanting to continue to improve our learning curve just to be clear we know we're not perfect yet you have to download some things you know things like that but this is an area that the TSE is very much focused on and then we cut to S-bombs and one of the nice things about this is on once you say West S-PDX after that your builds with West will generate evidence for your Zephyr sources which is the app sources and then which C's which objects, dottos and so forth make it into libraries and then which libraries together make it into the L so you're given an L-image and you have your build S-bomb and you can trace it all the way back to exactly which files are there and they're hashed so you have that trust level and as we take things that have to last for a long time knowing that you can be authoritative about exactly what is going to be key and if you want to see you don't have to believe me here there's a dashboard I've got the link to it here and at micro is one of the members of the project and they put this renode simulator up there and they simulate all these configurations and so they actually have a dashboard that they keep everything running and they do things built and passed in the testing and anything that says built or passed has those three S-bombs in it because it is that easy and so if you're interested in seeing what a good S-bomb looks like I think there's one of the members of the SPDX community Steve Winslow did the work in CMake to actually make it happen so the quality of these are pretty high for S-bombs and they're just generated automatically so encourage you to look at them so I guess this sort of summarizes what the Zephyr project is and you know it's there to fit when Linux is too big okay it's not a replacement for Linux but it is there to augment Linux and we have a fairly vibrant community we have underneutral governance permissively licensed so you don't have to worry about patents because people are contributing to it saying that Pat Grant sorry too much talking it's modular and it's development and we started off with a security mind and we're working towards proving it out the architecture is very complete so it tries to let developers just focus on their applications that's what we're really trying to do here okay so if you want to learn more go look at our community pages the codes up on github there's a variety of mail lists and a lot of the community works in Discord there's about 4,000 people in Discord right now and this is where the discussions are happening if you click on that link you get an automatic invite and there's various channels for companies and various channels for technologies, marketing and so forth so like you know jobs new products, things like that so I encourage you to go on to the Discord and if you've got a new product tell us a little bit about it that's using Zephyr the developers are interested in learning that their code is being used in places it's exciting for us with that does anyone have any questions go for it so first of all thank you very much this is a really really interesting talk and I'm really glad to learn about Zephyr first off I just want to second the things you're saying openSSF I have two both the Tuff and in total project went through that process also and those standards in particular are actually very useful I've seen other recommendations lists that are very dubious in some of the things they recommend and I'm not saying that everything on the openSSF badging list is 100% perfect but it's all very good and very close it's a good starting question so my question is I took a look at Goliath I'm not sure that I'm saying that okay it is pronounced Goliath just follow I took a look at it and it doesn't seem like they have the concept that a key might be compromised or a repository might be compromised as part of the threat model for how they do updates and there are other open source available things that have that threat model are you interested in having them contributed to Zephyr or okay great Goliath is one of our members and they have certain technologies they keep at house and they've got certain things that they work on upstream like all companies do and so getting best practices contributed upstream to Zephyr great thank you and will we work directly with you or will we work with them you would work with the community okay thank you just put in the portfolio the pull requests into the Zephyr repo possibly put an RFC up and let people know what you're doing and let them have a discussion before you so you figure out how to structure it and not waste time and then start upstreaming and if those issues reach out to me I would very much like to get some of these technologies up there thanks okay any other questions especially in committing to Zephyr the project is very well managed I deeply agree but say in this presentation a bit far from this slide I have two questions about Zephyr first question I know that Zephyr rapidly grows forward especially in the EU region compared to that it may not get much attention in Japan that's why we have printed out so that's very visible to the Japanese to make it easy so if you don't have one please pick one out it's what we've got here in my viewpoint that is because the Japanese industry has unique real-time operating system that grows with governmental support but I want to more publicize Zephyr in Japan welcome to start having meeting groups and things like that for Zephyr users that would be wonderful so please let me know who you are and I will reach out to you afterwards and you may become a Zephyr ambassador if you are supporting Zephyr thank you second question I understand that Zephyr intends to use it in critical systems but we have to use it there we have to go through the safety certification first but on the other hand beyond the project that holding goods to Arduino API to Zephyr I'm keeping an eye on it because it is a large region that can apply to Zephyr there are any area where you expect Zephyr to expand its applications I expect Zephyr to continue to collect cores there's a lot of DSPs out there that really don't have a good support plan and there's been discussions potentially of getting Zephyr used for them I might see that coming in as there's new architectures that are emerging in risk 5 some of these are definitely being followed and looking at getting them in there's a stop signal at the back but thank you for the very good questions thank you everyone and please feel free to pick up stickers at the front and those brochures if you would like