 So welcome everybody My name is Stefan Schmidt. I'm working at the Huawei open source technology center have been working on the on your project for like two and a half years now and I want to talk a little bit about how we are using the fire. So this talk is mostly focused on how we use the fire how we Make it part of the on your distribution how we build things on top of it and how we integrate it and how we Yeah, also use it on a developer perspective how we do like maintenance and so on So yes, we do contribute a little bit to the fire, but tiny I would say But this is more like the user perspective here So the agenda for today for the talk would be I guess not many of you would be familiar on how But what a mirror is and in the context of that what of my harmony is So I will introduce that a little bit to to set the context at the scope Make sure that you know what we are doing here why we are touching on the fire and The meat of the talk will be about five different use cases. I picked from our experience over the last two years and Like the problems we had the solutions we came up with the lessons we learned in the process So that'd be like five very different ones some more technical some more like process oriented and so on and then There will be a little touch on on future roadmap discussions So let's set the scope here. So As mentioned before the question is like how we use sapphire in eclipse on euro and and what we do with it As I said the perspective is very much from a user perspective and we have the needs We have are mostly for integrator integration For having a stable sapphire base where we can base our own features or blueprints or customer Solutions on and how we would do maintenance on top of that But as obvious as always we also have need for new features coming in we need bug fixes coming in So we need to make sure that we are picking new versions of of sapphire from time to time and how we are going to do that So this is like the scope. We are looking at the moment. So let's start with in overview on on euro So on euro itself, it's in a project at the eclipse foundation there's an working group at the eclipse foundation who's driving that and In a nutshell, it's an it's an operating system. We are building or Bringing up this is a lot of like distributed aspects in there So we want to have like a good wireless connectivity in there to make sure that we can have different devices work together form more into intelligent networks and Areas so the idea here is to bring something over which was started on on the Atoms side in China with something they called opum harmony and that is an operating system that was brought up there and Oneros kind of the sister project to that. So that's something we are bringing up in the European area so on euro has This promise that we want to build for devices big and small so for us that means that we want to scale up from MCU's to normal application processors, but also going into like servers and so on so that is obviously a very difficult thing to do and This is something you can't do with just one kernel, right? I mean the choice of Linux was obvious But then on the on the MCU space We need to look for for something else as well So on when we started on the euro we looked around and that fire was the most promising one for us to go along with So our complete build tooling is built around the octo And for Linux, this is a given. There's no problem to do that for that fire There there was and there still is matter that fire, which was a really good starting ground for us To get started and then for for other auto systems So we don't only support that fire We also support other auto systems in this case for example light us But also we experimented a little bit with free articles and so on so we always need to have some sort of In york to speak layer where we would do like integration and make sure that you're using the correct tool chain Make sure that you're building for the right targets and everything So this is what we're having here End of last year we did the only your 2.0 release That was very much focused these two years in the making was focused on building us in horizontal platform Where we have like a good hardware support where we integrate all the different open source projects We want to rely on making sure that we have that all set up and then have a broad Staging ground where we then can go and build either product or the customers or the partners of the only working group can build products and but this year is The focus is very much on having a vertical solution So once now that we have the horizontal platform You want to go a bit deeper and that is where our open harmony would come in So if you haven't heard about our harmony, I wouldn't be surprised So it's a project hosted. It's the open aton foundation in China It's the open source foundation of how many rest which is in proprietary operating system from Huawei. It is Apache 2.0 licensed So there's an easy way to transition code in and out They have a notation of different Systems, so they have many small and standard I will come to that on the next slide and go a bit more deep and the support on open harmony is Linux as well as Light OS so Light OS is their RTOS system. They're having on the other side So why you might have not heard about it. It's not something new or something unused or anything So there's last time I checked there was over 270 different certified products already in the market and These are coming from over 100 different manufacturers As I said the reason you might not have heard about it. It's because it's very focused on the Chinese market But the idea is to expand that and on your is like one of the steps going forward going into the European market So these three different types System types we are talking about an open harmony would be like Maybe not the best choice of names, but that's how they name it as many that would be the level of the Cortex-M It doesn't have to be on their support for other architectures as well But it's like the level of a Cortex-M. It need at least 128 Clibbert Clibbert of flash a RAM Now flash and then that's our system. Then you have small systems that could be our Cortex-A level They could either operate on RTOS or Linux and then you have like the standard system which would only be supported on the long side One of the some of the key features I mean you look before if you look at the architecture diagram There's like tons of different stuff going on but some of the interesting parts for this talk I picked here is the soft bus which is a distributed bus which enables all kind of devices in the same network to work together and There's an a distributed schedule on data management on top of that and the idea here is to form something What the marketing department called super devices which basically means like all the capabilities of the different devices are brought together into a virtual unified device and then that can be used from a user Think about in your home for example you have a TV which has a sound system It has a camera for example, and then you have like your phone with a camera and sound and so on or you have like your audio Multi-room system or something and once you enter the apartment Your phone knows about all these different Things around you and has like audio capabilities of like the multi-room system or the TV and it's been switched Automatically from like having your video call on your phone walking into the room and can switch over to the camera of the TV And so on so you have like all this capabilities and resources available and they are Not specific apps or specific configurations you need to do but they can transition in between with these kind of super devices Open harmony also has a current abstraction layer I mean yes They need this sort of for like the other side to have like things working on Linux and others And they also have something called the hardware driver foundation, which is their approach to find a way to Have drivers working on art assistance as well on Linux That's some of things so now we come to the part where I described a little bit of the use cases we had this fire and Describe what kind of problems we we saw there and what we can do about it So maybe you have seen some of that yourself So our first target was how we are going to build for their fire as I said before we are doctor based for all Other builds so we wanted to have a unified way for the developers to actually go with that So there was matter the fire available for us, which was a good start But the first problem we run into was that the This layer was really mixing together a lot of different things. So it was mid mixing the Basic machine support for different devices different boards But it was also bringing and distro policies and and all kind of other things in one layer And once you enable that it was difficult to like Overwrite things or make sure everything is working smoothly in a in a bigger context that might work easily for you Like your one product or something no problem, but if you're going to want to go to distribution scale up, then it's more complicated So what we came up with Was we are splitting the matter the fire layer into two layers basically one for the fire base P and one for the fire core That solved quite a few of the problems that was a lengthy process that was people involved from our side But also arm and so on everybody had like the ideas how they want to transform that but in the end I think we we landed with something that is looking well for us The other problem we had was that in matter the fire there was only support for a handful of Machines or boards. I think like five six or something like that And if you compare that to what therefore actually is supporting that was like a drop in the ocean so One of the problems we had was like how we bring these wide range of hardware support and Enable that for everybody that is building this on nero and wants to do that for their own products So the first approach would have been we are generating all the machine files and putting that into the metal Fire beast P layer but the maintenance of matter that fire haven't been happy about that because that means They don't really know is someone supporting that is it actually building and so on and if even if they would manually add that to See I on their side that would need to scale up heavily without maybe they're being used around so the middle ground We did or we settled on was that be right a bit of tooling to generate machines on a need by Basis so what we did basically was we enhancing the CMake system On on the fire to this is a patch that is sitting the layer So that's nothing in the fire upstream or something what but it basically does it It looks at all the different boards that we having here and maps between the kconfig options and so on and gives us the The tuned files you would need for the building on the doctor side and that would then again be exposed when we are running West boards then we can look it all up and then from this all the information they're getting at that point We can generate machine files in your to speak That means we have been we had an easy way for any being new boards That was like a matter of minutes basically to do that and Whenever we needed something and we want to bring it upstream then would go through the normal process adding it to see I do a manual Verify everything is working and so on and that is something everybody can do so the there's a command now that you Can actually execute it's a if you run with this bit bake It's just a normal Receipt inside the layer and then you can generate the different machines and see if that is working for you So that was something that that worked well for us Maybe not in the full scale as we wanted it to be but it was a good compromise And for us that actually means that the extra work We did to unify the unifies the way of the developer workflow People could work with the sapphire in the context of a nero that was really worth the work So still that means you can still go ahead and do your application development This vest directly and so on do your testing and so on but if you want to go into production and maybe go to like shipping and deployment and so on then you could still put it in in the normal workflow and have all the normal maintenance process and Security checks update and maybe also like IP clearance and so on As I said before the main work we have been doing a matter that fire has been around splitting the layers We're having adding in more sample applications We have been doing a few tweaks and fixes here and there and then I think we are up to like almost 70 patches That if you contributed there and on our side We have been like regularly updating the layer and making sure that we started with an LTS release on that fire Then we've switched over to a normal release because we needed some extra features and at some point we will a switch back to LTS at that point So that it was like one is the use cases the initial one We we need to solve on our case and make sure that that is working smoothly for us But once you have the the basic things in place like building and then deployment and make sure that that is all working Then you're also looking into what kind of features we are looking for so obviously we need hardware support This is something that is kind of a given We had we had started with a night to day and 96 board and that nowadays The board that we are mostly using on the in the Zefi context is an Arduino Nano, which is a Nordic based 52 base chipset But besides hardware we also have like a few needs for specific subsystems or components and features For us it was mostly around connectivity and graphics So we have a need for open thread We wanted to have co-op on top of that and we wanted to have LVGL for the graphical part as well Once we started to use that we saw quite a few smaller problems I would say that most of them are not really related on to Zefi itself It's more like if you start and use them in a bigger context that might be problematic So for example on the open thread side, we are building There's a radio dongle we are putting in to use open thread also from the Linux side And this radio needs a firmware and we are building the firmware with metazire fire Which is all good that is all working fine for us but at some points I was in version bump in the spinner protocol which is like the protocol going back and forth between the dongle and the Linux host in that case and Given the different versions we have been using when building with Linux for the Raspberry Pi for example or something And on Zephyr said as well, that was like trouble some and not So we need to make sure that we when we bump something on one side We also need to make sure that all the corresponding user space tools in the different layers of on the Linux side have been on the same level That is nothing magic or anything there, but it's something a bit more complexity You need to make sure that you bring that in Kind of similar it was for the LVGL for the graphic part We we wanted to use LVGL and see if we can Build something that where we can build a graphical interface that can be used on the Zephyr side as well as on the Linux side Yes, there would be obviously would be in the application logic There would be differences and you might need to have some sort of abstraction or so But at least the basic interface was something we have been interested in in building out I have we have another use case that actually touches on that a bit later But that was one of the things we started with So why the Solution for the observed problems we're having that was something easy to fix that was yeah some bug fix here and there and something we can we can do in a matter of days and For every gel that was was a bit more complicated because when we started to use it on the Linux side and the Zephyr side was really Far behind in terms of like API level and versions and so on so and then we wanted to to ramp that up and Revamp that on the Zephyr side and there was nobody really like maintaining it and making sure that everything is working smoothly So basically we ended up doing quite a bit of work there to revamp it having the external LVGL model updated to newer versions of LVGL Fixing bugs on the way obviously and then helping also to like whatever regressions came up get them fixed and so on so That was a bigger a big chunk of work. We have been doing there Besides that the main contributions from us was really like smaller things here and say we need like a fix in the device tree for the Partition table to make sure we have persistent storage for open thread and so on But all of that was was was fine basically So the core functionality and the bring up of the hardware was was easy but like if you have like specific features or so that was more complicated for us and As I mentioned before like we only had like 20 patches or so to Zephyr core Contributed so that's not a big amount or something But nonetheless that part work quite well for us to to get us in a state where we can build we can Have it all like nicely working between Linux and Zephyr and then now we can actually see what kind of applications and Things we can build on top of that Which basically brings us to the second as a third use case so one problem one project we have been working on that was called Eddie and it's an Enabling distributed intelligence in one of the areas it doesn't really matter much like what the project itself was doing It was more like how we wanted to use it and and what actually is touching up on on Linux and Zephyr there So we wanted to have a system that can run on the small MCU based the devices to get like some As a sensor data or some like specific locations in there And then you also have like the Linux side and we wanted to have a shared code base as much as possible That's what we we set out this so it was a C++ project But not too heavy so it was fine to also run it on the MCU side So we didn't use any very heavy C++ feature or anything and we wanted to use co-op We have had a need for Jason as a format and we wanted to run the connectivity of open thread And These kind of code sharing between the two is as I said before it's clearly not a problem for the Zephyr community Right at least not for the developer community. That's nothing what you set out to solve But I still think it's good at least to bring it up here because there might be some other people doing the same here So the first problem we saw was that for the for the co-op side and then we Investigated what kind of co-op support is around for Zephyr. You have the native co-op support in Zephyr Which was working well for us, but it's obviously a very different API to the co-op which we have been using on Linux Then the next step is how you would configure your your threat device your open threat device to make sure that's actually in the same network How you would do the commissioning to bring up and so on this is very very specific On Zephyr compared to like how you would do it on Linux on the command line or with a System we service or something like that. So we ended up in a situation where we had a lot of Lot more Duplicated codes and we hope for we did some abstraction were possible But we realized soon that most of that Happened on a way higher level So we always had like specific parts for like I saw Linux or Zephyr. That was not code We could really share in that regard which was a learning exercise on our side as well so maybe we started out with a Too optimistic idea on how we can actually use that so One of the things I was wondering and maybe some people here know about that or if not you can bring that up to me later Is that a it's a common request? Did the Zephyr team or project actually heard about that before that? They it would be good if there's a way to actually have a code Shared between running on Zephyr and at Linux. So this is something because maybe that's just an very niche Use case we are building up here, and we shouldn't really go into that direction Yeah, and as I said before so if I'm not talking about a full abstraction layer or something I know that's very well. That's out of scope for for Zephyr project So coming to our use case number four We have something in a neural called blueprints Which is a little bit more than what you normally would see is an proof of concept So we would do code. We would do a proof of concept. We would do documentation around it having all the the tooling and so and Making sure that we build something that that might could be used by some of the partners in the working group and Go ahead and build a product out of that And maybe we already have reached like 50% or 60% of the way for for the basic stuff for them So they can add their own value on top of that So the the problems we faced here was Obviously looking around what kind of hardware we are selecting bring up the demo hardware and then some very Specific problems we run into on the size limitations on the on the feature set We wanted to build for these demos So as you see in the title the the demos about door lock and cats door lock is pretty self Explaining so I don't think I need to talk about that cat stands for context aware touchscreen So basically the idea here is trafficked a touchscreen that would depending on the devices that are in the network the devices It actually discovered depending on that it would show you different Views on how you would interact with these devices on the on the door lock that obviously means it would show you a pin pad But if you would it would find some some sensors for like temperature or something that it could just show you a dashboard There's some information about that so it's really like depending on the setting it's in in the context It's in it would act differently So this was like one idea we having and we wanted to build a blueprint around that So for the hardware that was pretty easy I would say you obviously need to be a little bit careful what kind of hardware you're selecting But given the very wide range of hardware support and driver support on that fire That was quite easy for us to select the right components here More problematic was the idea to having the graphical part running on that fire as well So as the idea was to have the door lock as well as cats both of them running from the fire So the door lock part is easy, but the cats part was way more more complicated And we ended up in in switching over from that fire to Linux in the in the middle of that So let me explain a bit more about that So the door lock is easy as I said, it's just a motor driver It has a keypad. We have like the the by the ship is inside the So see directly so we have opens red on top of that and co-op and some some basic Application logic for like setting pins unlocking and so on so that's all easy and fits well into the use case for the MCU system here We started out this having LBJ running on the Nordic ships that as well We connected the display and we have we reached a stage where we actually can show some some graphical Versions of like what we wanted to do But we very soon came to the situation where we run out of Resources basically I think the most critical one was really flashy and that regard the we have like one megabyte of flesh was there and This full that fire as an autos this this a feature set We wanted to have this lvgl as an external module and then all the graphical assets for like the the icons And then maybe some logo and then the fonts and all the other things we run into a situation that that wasn't possible We have been pondering for a while if we Switch the hardware to a more powerful one with more flash and so on or actually hook up an additional flash to the board or something That that obviously all of that could have worked But at that point the time was short and we wanted to get the demo over the blueprint out So we decided to switch it over to Linux at that point That was there was a good thing that the AVGA part was easily switched over so that helped us And Then we have to demo the the picture you can see here that was something we showed it and embedded world beginning of the year There's a communication going on so you have like the contacts aware touch screen on the Raspberry Pi site and you could As you can see the motor is opening closing and you can control that from the keypad I said connected directly to the door lock or over open thread and co-op communication back from the contacts aware touch screen so that has all been working but Again, that was more like a problem on the hardware limitation side again again. Nothing coming from the From a Zephyr perspective, so that's nothing I would feedback and say that's something that need to be fixed or anything I just don't know if there are better alternatives on the graphical side So we selected LBGL because we wanted something a bit more flashy than a very very basic Interface, but it obviously comes at a cost in terms of size Yes, the other point we wanted it to like make sure that we can have something switching between Linux and Zephyr So going from the technical use cases we had also had one that is more process or Yeah, more IP driven in that regard. So we wanted to have IP compliance It's a core component of on Nero as a project as a working group We wanted to not only have s bombs generated for the builds we are doing for all the artifacts We are building but we also wanted to make sure that we have an an audit team looking over all the components that go into our builds and verifying that It all works well together. So the license are compatible the components There's no binary blocks and so on so that was something we also had been running We have been developing our own IP compliance to Shane Building on top of Yachto getting all the information from there going over the build artifacts then from there feeding it all back to Phosology and there the a team of manual auditors was looking over all the different parts that have been going into the images and making sure we have no problems on license compatibility or Actually files going in with uncle license or actually binary blocks or something like that When we enabled that for our Zephyr builds we actually run into a few issues. So there have been binaries in the in the repositories that have been pulled in in our builds and This unclear license situation. So they have been halls from NXP and expressive There they have been binaries that have been part of the halls of the vendors that have been Pulled into our builds even if we for example didn't build for an XP or expressive or something So they still ended up in there and that was something we for as passive that was solved without our Any interaction from our side but for an XP we brought it up and that got solved later on Then there was also a problem on the LVGL site There was a font that was getting pulled in that was not there was no clear license Not was not compatible to force to use there That was an upstream product a problem that got solved in every upstream and then In back into the module in the in seems a fork that we're having and on the Zephyr side to get it in There's still a few things with a bit of an unclear license statement. I mean for engineers That's maybe not so problematic, but lawyers really go into the deeps here So if you have for example a file that is licensed on the one hand as a proprietary license But also as an open source license, you might need to have like a specific Expression in there that gives the rights to the Zephyr project to actually allow that We saw some files in there for MCU boot and the NIOS 12 But what we did basically was once we had all the results and for solitude getting out there We we prepared a list of the potential issues we looked over them internally first and then reported them back to the Zephyr project and From the four ones we reported two of them have been already been solved. So that was good That was a very good smooth process for our side To are still open or stay closed. Maybe at this point Where we didn't really get much of feedback or Engagement from the Zephyr project I I kind of understand why that is because I mean if you report a problem for a specific vendor License in the code that's sitting in there and that might need to give a specific grant to the project or something It's very difficult for the open source Project in the team behind that to actually deal with that It would always need to go back to the vendor at some point or another to actually go through their Lawyer team and so on to approve that But nonetheless, I think that these are important things to raise to make sure that whoever builds something on Zephyr Has a good way to actually building a product So these are the the five use cases we encountered basically I mean we did more work and so on but these are the five ones I thought at least worth sharing here and maybe some other people have same experiences here so for us the the journey we are on right now is more Not so much focus on Zephyr at the moment as I said the focus for this year is very much on having a vertical solution Making sure that we can build on top of that So we are looking more into like having application frameworks and ecosystem enhanced having better IDE tooling Profiling the systems and optimizing them and then maybe in open harmony. For example also Placing some of the critical areas with some some other program and languages like Rust and so on But this is all not very related to to Zephyr So the question really is here for us if if there are any interests on having like open harmony running on Zephyr So I've been thinking about how is it that could be done But it really depends a lot on if there is enough enough interest from the partners in the working group or for other people To actually express an interest to to go that way because right now it's not on their agenda or anything It's more like a thought process we're having so if you wanted to do that I mean we had the talk before from Carlos who actually talked about like how they are pulling it all together in the connect SDK from Nordic and a Similar approach could be taken here in terms of like just having external Zephyr module to glow glue these both systems together So we have on the one hand we have CMake for for Zephyr for open harmony If you use it directly the components you would have a GN-based build system Also something for example meta users and so that is something that could be done as well. It could be pulled together Open harmony itself has already a support for Arta So it has support for for light or so that means the the platform specific support that could be something that could be enhanced and Adapted to maybe work with Zephyr as well And there was definitely to be a quite a big apporting effort to do the platform support But also maybe touching things like the current abstraction layer and the hardware device Hardware driver foundation soon, but the point really is all all of these things would only make sense if someone actually Expresses interest and being part of the new working group to do that so in case you have an interest in that in case you really Want to talk about that get get back to me. I saw here or just when I'm around here I'm out of the confidence all week and you can just get in touch with me on that and With that I would already be at the end and I'm more than happy to take questions now Let's start there and then we go. Okay. Okay, so let me repeat the question So the question was about the IP to shame between building and if that is something that only specific for for near Or is that something others could use as well? So when we started out? I mean these S-bombs thing is like what normally projects are doing right? They they go because that's something engineers can solve easily you go and you say, okay Look, we need to make sure that we specify clearly How what what license we are having and all the things are going together and then maybe we can actually Generate an S-bomb out of that saying which fights is having which license you can make sure that they inbound and out point Compatible and so on but this is only like one part of what you would need if you go ahead and want to release products and go out of that so There if you go into the more commercial space and corporate space there then there are kind of like commercial tools available But there's also something called for solitude and that's something how we build things around it So what we needed or what we was a way to get all the information out of our yachter builds Get it through and get it into for solitude and find a way for the audit team to actually look into that And and work with it and give us basically vouch for it to say look this is saying we can release that it's all good So that's basically what have been built They have been like a few tools around that I don't want to go into all the details But there's like tools that get all the information extracted from the yachter side Feed it back into for solitude Then we have a fancy dashboard that tells us this kind of licenses that are in there These are the inbound licenses These are the outboard licenses and these are the specific packages in yachter for example that have that are compatible or not and then People can actually look and look at the different files that are getting direct into that This is like how we can see this specific C file as having this license statement And that might not be compatible with something else and at that level the audit team would go ahead and look at it And then clear all the things for solitude has functionality That would allow you for the next build that comes in that to reapply all the same Things we have been done before and that's also functionality we have been using and the so the way we did the compliance to Shane is that it's not tied to a new itself so they it can be used by other projects as well It's actually its own project at eclipse foundation They can actually do that and other working groups in a cliff foundation actually are going to start using it There's a for example the I think that was expressed interest from the software defined week a group for example They had an interest in that and other project could go ahead and use it as well. Thank you so question So I can I didn't get the beginning Okay, yeah, okay, yeah, it is I mean that's I think that's in for like Almost a year or something. I mean as I said this year We didn't even touch much on that one that was all last year on that part That was like one of the first things we when we needed to solve that now that was that was worked out already So I mean the thing is matter that fire was started and it was maintained by I think one person who intel It was more like a side project So that by itself is very much focused on this and so on and I fully understand that if you do like the application development for For that five it's only that's perfect what you want and but matter that was already there, but it was like Not so much maintained and like in a good shape for production I would say and but all of these things have been solved so we have been sitting together with people from arm and the maintenance of Meta that fire and surround making sure that we find solutions that are actually good working for all of us and the beef P stuff is Factored out and the core layers factored out The only thing that is said is only sitting there as an receipt to generate the machines That is not something where we didn't get the big bump of like more hardware support or something like that But if you have an interest in that and want to do that on your own project or something It should be a matter of minutes to get that added and then it's mostly about like Engaging with the maintainers to make sure that these things are keep working and then you put it in CI and so on No, that's all in there There's no we have I think we have a fork of metal that five is only a few patches But that's more like integration and maybe integration with our CI. So there's nothing big Missing or anything any more questions. That's over there from the internet. I guess from David. Oh Wow, okay Okay, cool Okay, so let me repeat it So David just said that he reopens the one issue we had on MCU boot and he is trying to get like a bit more engagement there Might be difficult if they can actually get them to contribute to that one as I said These are more the two to open issues. I fully understand. I mean that engineers. They don't even what what should I do is that right? You always need to get like your lawyer team involved in that and that is really really not the best thing to spend your time on but To get these kind of things off to the only one Okay, I Don't see any more questions. So I would say thank you for your attention If you've any more questions, I'm here and I'm here for the whole week. So