 Well, welcome everyone. Thank you for joining the talk. This is a Zephyr user story It's about my experience with Zephyr in a couple of the last years and Yeah, shall we begin? My journey begins at ProGlav at the end at the beginning of 2015 And I joined there as an early employee What does ProGlav do? We they develop scanner gloves for the industry. So In the industry for example in manufacturing you're scanning a lot of barcodes and for example assembling a car you Scan parts very often and what ProGlav does with the hand scanner is it saves a lot of time in industry And that's what I developed with them so we prototyped a lot and The requirements that we had were mostly Came mostly from a product. So it's sub gigahertz because in industry you cannot use 2.4 gigahertz especially Bluetooth It's a small form factor as you see because we want to fit the scanner on the back of the hand And it has to be of course a low power and with a scanner engine that takes a lot of power That's a very hard to achieve. So a lot of You need a big battery in there So what we did in the beginning of course were probably the very same mistakes that everybody does It was my first job. So we started off bare metal That's that's something that came out because we are prototyping a lot in the beginning Of course not with the same hardware. We later on develop so the product initially when we made the switch from Prototypes to the actual hardware that we were going to use Yeah, we just started with bare metal because we didn't know any better and we prototyped as we went and Yeah, requirements came in Every softened with every new customer visit, you knew you had a new requirement a new feature you needed to develop and it grew very very organically and Yep, the as I already mentioned the hardware components were very heavily dictated by the product itself I mean it had had to have a scanner engine that needs a lot of energy and a Big battery and a radio module because it needed to be wireless. So there was actually a second device To forgot to mention the scanner of course Sends the data to a receiver and the receiver is connected to a terminal Where the data gets funneled and either through keyboard input or a serial connection So yeah, due to time pressure, of course, we made the prototypes later on the product Which led to a lot of a lot of work And and what we what we've built ourselves even even built ourselves the build system for example or a serializer protocol or especially the the UX so as I mentioned Every feature that a new customer requests that we just built in there because yeah It just was step-by-step and there was no real no real border where we said, okay This is not a product at this was the prototype. That's the product and we do everything new again No, because you reuse everything you have so at some point When we already had a couple of devices in the field we decided, okay, let's let's overhaul the hardware a little bit let's overall the software and there we Continued over we decided to port everything to Zephyr and Zephyr First and foremost provided us with a streamlined and unified tool chain Which was a pain back then when we did everything bare metal, right? So what Zephyr had there you just download the SDK and you're done That's that's it. You you have the compiler you have the tool chain with everything That that you need to get the project running with with two downloads to repository and the tool chain. That's it Um It also is that what what does it help you it of course makes onboarding also easier which in startups happen happens from time To time because they don't have everyone involved in the project from the beginning from the beginning So yeah, especially documentation helps there because in a startup Sometimes you have don't have the time to document everything, right? So it helps to have with essential component using something that's already documented Of course using Zephyr it made the Hardware and the software the product by itself Adaptable for future hardware improvements So we tried of course then to have abstraction layers and move away from from bare metal to a more scalable architecture if you want to say that and of course it it made board support maintenance a lot easier because Board support is very well structured in Zephyr and and it's not like with the bare metal version sometimes you have driver files for some arbitrary sensor lying around in a source directory which is Should actually be buried in some some layer and you don't want to actually see that as an application developer, right? so What was the timeline? We shipped the first product so the first the first iteration the bare metal version within 12 months with mostly myself And I on boarded one developer in the time during the time into the product into development and Later on the Zephyr the Zephyr version we had We made it in nine months of time so that already accelerated even though we on boarded Another two developers and two interns during the time Where the strength of course was the heavy Documentation site that's ever had and still has that helps us just to tell someone okay Look into the documentation everything's there. This is the API's we use And you can go and develop something for the application Why do you actually need more developers if you ship the first product within 12 months? Why do you why do you need more developers later? Well as I said the product grew organically and of course you have devices in the field So you need to maintain that So there's bug fixing testing On-site customer visits because at the time at Progloff it was very important that also engineers went to the to the customer so You don't have 100% time to really develop something but there's lots of different tests and it's not only the features you want to develop but there's a lot around that and Yeah, essentially it's maintaining a technical debt at some point with the bare metal application That's why we needed more people and of course also porting that to Zephyr This took then overall roughly two and a half years From when I joined developing the product the prototypes and the product Up until we actually see certified the second product, which we then also should my next Journey was at Prog at Blick so in Q4 2017 I joined Blick as lead now lead developer and team lead for their hardware and what does Blick do? Blick develops a logistics tracking software so customers come to us and Ask us whether we can track their load carriers and whether we can tell them where their load carriers are For that we have a front-end That tells the customer where these load carriers are so special load carriers for everyone who is not familiar with that is so for example in manufacturing car manufacturing Like car doors or truck axels these are transported on special load carriers so these are actually specific to this one part and These can go in circles So they go from the manufacturer to their customer and come back So every time you produce something you need these load carriers. Otherwise, you cannot ship to your customer These things sometimes get lost sometimes they pile up at one of the parties so yeah, that's what we're trying to solve and actually currently already solving for some customers and We need hardware for that and that's my job so we have a receiver unit that's in point-of-interest areas at the customer and a Sensor node sensor unit that's attached to the load carrier So that's what that's the hardware we develop and the Zephyr for us is running on the sensor node So the requirements there are pretty similar to what I personally experienced that proglove So with the sensor unit we need low power heavily so these devices need to be in the field for years essentially we want to be able to support the life cycle of one load carrier which is it can be up to 10 years for example and From a radio perspective we are using 2.4 gigahertz button 802 15 for So we don't have any compliance issues with the Bluetooth there for example in automotive industry So this time Since this was a new project I started from scratch we decided on and I heavily Voted for that that we use a Zephyr from the beginning which was a very good decision from my point of view and now and It enabled us to go from Prototype to CE within six months to be fair with a with a simple application, but it still shows for me that it Very you can go very fast from a very early prototype to actually CE compliance, which we're doing actually at the moment So lessons learned from from my experience at these companies and so now within these four years I was we shipped industry great products with small teams and startups The at first At first they didn't really scale right so a bare metal doesn't scale you're reinventing the wheel all the time You don't want that So just that just doesn't work Reported to Zephyr so at the time we ported just too many legacy dependencies still like you don't want to depend on Your old bare metal organic code really try to to get rid of that and convert your customers to that because that's actually the only thing that's necessary is get your customers to To adapt in your product. Otherwise, you're gonna end development always be dependent on these old interfaces since libraries you developed for you internally and Also, we actually did not upstream enough code like we had a lot of customizations in the kernel which from From some point of view might make sense because you will build a custom project Some things might not fit your device But it's it's very important to just to just upstream that because sooner or later You want to upgrade the kernel and you just have to rebase that always and that Zephyr changes of course, but it has a very very good abstraction layer which prevents you from Rewriting the application every time you straight change a sock or change to change some hardware component, right and bare metal That's just always the case and Yeah, that generally makes it makes me makes me very sad because that's Usually not fun maintaining like these all these legacy components So later I decided, okay, let's do this with suffer And we are still not upstreaming enough in Blick of although we do So I can't I can't stress this enough like by upstreaming you get so much exposure to freelancers Which can help you in development like we had a couple of freelancers engagement now in our projects That that just helps you get there get a project going because in startups you generally resource constrained, right? You want to have someone helping you with this and upstreaming is it's just a way to go at least from from my opinion now For example We have some some flash drivers that we currently still maintain ourselves internally and which is at the moment Actually, that the biggest the biggest pool of conflict if you want to upgrade to a new colonel on the other hand we made some changes to network to the network stack because it currently we're not really able to use the suffer network stack due to some constraints and That is also not yet upstream. I'm I'm in in the progress of Engaging discussions there, but what zephyr then actually enable us to do very fast is actually adapt the network stack very easily So we just Introduced our own radio API and we were able to use that in the application that was very fast It took me like one or two weeks and then we were ready to go and without any hassle and some sort of conformity in the project that That's not very Yeah, that's not a heavy technical that Of course using off-the-shelf components where possible for example like the a serializer For example that we wrote ourselves. We don't do that anymore Just use an off-the-shelf serializer and it's very easy with suffer to integrate these external libraries into the build system Like for example protobuf if you want to use that it's very easy Zephyr uses CMake and if you find a library that also is built already of CMake It's very easy to get these Integrated as a soft mode sub module in your project and just integrated in your build system and you're done Also shell access for example for testing and debugging If you know like you don't just build the features but also testing components for example and offline testing or Compliance testing for CE for example requires you to test certain functionality on the device and Just to get some IO on the serial port for example the device just use the shell of Zephyr. It helps you you can hook your own functions in there and Implement the testing and the interface is already there. Just use that So yeah sticking together components with Zephyr is really simple due to its very coherent build system in my opinion And of course the build process if you're talking about CI and embedded There's actually not too many Options that are very out of the box for example testing without actually mocking a sensor or an interface for example But at least the build process is very easy to implement in a CI and you at least want to have that for Before you merge something from a merge request Yep, makes me happy. So generally using Zephyr for these types of projects, especially in fast paced environments That makes me smile so Yeah, as you saw most of my experience came from startups and As I also already mentioned startups care about features like it's about sales like you have these big goals Everything that's that's touchy and shiny you can sell right? So actually everything that you can tell to a business guy. Okay, I'm working on that and that and that and he can actually try it out That's very very nice and everyone sees. Oh, you're working on stuff But yeah, that's that's that's not always the case, right? So What you do is you work in a very fast environment so prototype iterations and fast to market But you have to as I mentioned implement the build system the board support Tests for production environment and and that's not very flashy. You cannot show that to a sales guy And tell them hey tell that to the customer. We can do that like no one cares Yeah, and that's this makes progress feel very slow for these people and it makes very hard for you to justify the resources you need for development and This is why you actually want to have something to work with that takes off a lot of work from from your development process especially in low resource environments, so since your backlog is always growing and You want to implement features and your resource constraints you want something that takes off This this general load around it and you can actually at least to some bigger degree as usual focus on these features and Yeah, for that what you want is a solid build system You want to have a solid architecture that or at least something that gives you already the the framework of a solid architect So you can orient yourselves along that These like interfaces serial interfaces radio interfaces networking anything you can think of And the ecosystem of course because you want to have people to talk to if you have problems, right? So if you use some some If you build your own bare metal application, you can't ask anyone, right? You're probably even worse. You're dependent on this one developer that developed the kernel, but he left last week That sucks Yeah, so communities very important and standards are very important like most likely for example in the case of suffer There's already a library supported like MQTT co-op if you want to use that you don't have to worry about implementing that in your build system or general development you can just tell an in turn hate prototype that for me and He will be able to do that because the documentation is actually very good And there's a lot of samples already or blue blue tooth like you won't have any problems getting a blue tooth application running with a That's already supported So this brings us generally to the to the artist choice so it Influences actually road map because you have to think about all these things that you want to develop set up and maintain around your project and Yeah, you want to use something that gives you all the components that you actually want to use in that regard And there's a couple of artists of course We of course looked into them like embed OS contiki free our toss, right? Maybe if you're if you're if you're capable the license especially Yeah, and in our case we we decided for suffer because it actually applied very well to all these requirements in my opinion for Where you have to work on hardware support the tool chain all the testing that you want to do These API's interfaces that you that you should need for for any IOT application in that regard Drivers you need for your sensors or radios application libraries like I mentioned MQTT and so on And especially the license because you want to be able to make modifications to the car and actually Maybe I don't want to want to publish them even though I think upstreaming is a better option But sometimes you don't always want to want to publish them because you want to prototype first and ship a prototype to a customer And you want to be compliant with the license, right and with the effort. That's really easy So, yeah, in my opinion suffer falls very well in all these requirements. So It's it's a very nice code style. So it's very easy to actually read a driver or read Read implement anything that's implemented in Zephyr and Look around and find your way around the code. It's very easy to get used to that and very easy to to adapt it and the documentation is very nice because as far as I work with suffer this Really nothing that's not really documented. So that you find something for everything and generally the API documentation is very well because if I'm not mistaken, it's exported from from the dock searching comments and Yep, on the other hand, it helped me ship products fast, right? So six months from from prototype to see when I see that somewhere else If you have a pile of requirements and want to build something from scratch Yeah, not easy, especially don't people forget about prototyping a production and testing, right? So it's not just about these this Yeah, how difficult can it be to send a packet to and to to another device a second device just a receiver? I just want to do that. Yeah, but eventually if you especially if you work in industry you want to be able to Certify that product and you need to go through testing and end-of-line testing and certification and Yeah features is not everything you're gonna be working on I was able to maintain all these projects with relatively few people So on mudding always was very fast Even though it was painful, of course, if you're in a release window on boarding someone is not really fun Well, it was easy because you could just pinpoint them to the documentation And also with freelancers In the beginning like when I when I worked at proglove it was Zephyr 1.6 I think when we started working with Zephyr was not easy to find freelancers but Anyway, these the freelancers we found were very easily able to to work their way through it and and then actually contribute something We could really use in the product And Zephyr's community is very welcoming like from my point of view if you're chatting in an IRC just asking a question There will be definitely someone who who answers your question Which is necessary because you don't want to be a roadblock by something like that and Upstreaming is also very direct like we upstream the real-time clock driver For one of our socks and it was very easy. We found a freelancer for who was already involved in Zephyr. So it was even better and Yeah, usually people are there even from from the chip vendors like nxp or Nordic where the the board board support For example, it's already very good. So if you want to prototype something Just use use these these boards and they work out of the box usually for for most of the common applications like Bluetooth or 15 for and And if you have questions about that, you're very easily able to contact these persons because they generally maintainers of an open source project Yeah, so in my opinion and my experience what I what I had right now have right now Even even today is that they're very forthcoming and and friendly towards you if you just ask them Hey, why does this driver work? How does this work? Is there do you know someone who can help me with that? So yeah, very nice and Of course also something that comes with upstreaming is it gives you It gives you more reviews, right? So during our upstreaming from the real-time clock driver, for example We had more eyes than just our own to to review that and what what what that what we benefited from that is actually Okay, our driver was better afterwards and it integrated better with Zephyr Right. So and this gives us more experience with working with Zephyr and later on being able to better upstream and In general raising the quality of of the real-time operating system and eventually your own product, right? and your own flexibility in upgrading later or changing something and Yeah, at last I want to thank everyone who worked with me up until now and today and Everyone who gave feedback to the talk and I think it's time for questions Yes Yes, so I think this heavily depends on your so I maybe I repeat the question. So what was most painful with Zephyr? because I Mention a lot of advantages. So one disadvantage, for example, is the network stack. So currently it supports IP and Both IP you can I think you can do a lot of things But if you for example in our case want to interface directly with the radio or directly with layer 2, that's not possible It's it's so there and then on the other hand, it was very easy to to throw in our own interface, right? So if you don't work with with the net stack, that's Probably depending on what you really want to do with with it with your product On the other hand for example regarding low power management low power management is something that's very use case specific I talked to one or two developers some time ago earlier this year and There's no not really low power management implemented you have the hooks for it So if you use the kernel the kernel goes to low power if it enters the idle thread and still you need to take care of yourself to actually shut down all the devices and all the sensors and Then again, there's there's tools for that that make it makes it very easy. For example the Drivers generally support a low-power hook that you can can also call in the in the success spend call and I think in 1.13 there is now actually a template implementation of parts of that correct and That that takes that takes work and then of course depending on what type of low power management You actually need for your applications and for example in our case the devices need to be essentially in low power all the time So then there's the question. How do you use? Zephyr at all. I mean do you need threading and how do you make sure the radio wakes up in a correct state and things like that? So if you're looking to these things that then Probably need some adaptions in the drivers and then it also makes sense to to upstream that in some cases Other than that I think a lot of things have improved really so back in 1.6. I think that was around the time when Zephyr actually got went public two years ago There were a lot of things that Were from a driver's point of view not really implemented to a hundred percent Still I think the peripheral support is very good But sometimes you just need to implement implement another port something like that at the moment I think the support for that is really good like in our case. We didn't really need to implement anything else In that regard but back then and especially if you use some sock, that's not yet supported or fully supported then The advantage in that case would be you need to implement that. Yes, very easy There was a time so so two years ago It was still all Kconfig and they moved to DTS now device tree Which needs some Getting getting used to well if you use to DTS, it's very easy and there's that there's there might be some Some things you need to port to DTS yourself for your board But in general there's like a lot of support from the community if you have questions about that That's really straightforward like adding a custom board for example If you use a sock that's already supported in a dev board for example, you just copy that deaf board and make your adaptions It's very easy Any more questions? Yes, please very Very beginning of Sefer What Weren't you afraid of seeing just so many changes happening at the same time that you were developing your product and we wouldn't that Didn't that scare you No Okay So maybe to elaborate a bit on that because yeah, and just know it's funny But the thing is if you don't plan to upgrade then yeah, if you start developing the product based on a kernel like we do right now It's a question of are you happy with Stability of the system if yes, there's at first no need to upgrade And later on actually upgrading so for at the time when I at proglove We updated from 1.6 to 1.9. It was actually very easy. The only thing we had to adapt was I think the spy API that was that was rewritten but other than that And it was also a new a new products like or so we had the time I mean what means afraid maybe you can can specify a little bit. What what do you mean by afraid? At the time I'm not sure if the micro kernel nano kernel transition had already taken place the cake on fake I already taking place and and so many things you're doing your development Yeah, you're going through a through that heavy amount of modifications It's is that going to be a problem for your for your application or not? Whereas it's interesting to see that you still decided to go with that even at that early stage of the project I understand. Yeah, so generally we always use the stable versions like the versions get get tagged and Because because Zephyrus developed in a way that you have I think there's three release candidates always and they get also patched During that time and if that's tagged, that's more or less stable because there's also a Stabilization phase for the release candidate before they finish the release so in general I would say I Personally wouldn't be afraid of using a stable version if you use a version. That's just master Yeah, I wouldn't be so sure about that then I would be afraid because There might be some components that still depend on not different API and something like that So so these might break things later on in application or your own kernel adaptions, but generally Going with the stable version. I wouldn't be wouldn't be too afraid Any more questions? Very well, and thank you so much