 All right. It's two, and the room is already virtually full, so I think we can slowly get started. My name is Benjamin, I'm a developer advocate for the Zephyr project at the Linux foundation, and today I want to share, so yeah, not super happy about the title, but the short story is that I want to share some tips and pointers to things that might help you be more productive when writing embedded software in general, no matter what platform you're using. If you happen to use Zephyr and the Zephyr real-time operating system, the tips might be even more useful, but it turns out that it's actually just talking about some of the things that can help you when it comes to emulating, when it comes to trying to get and to improve your security story, things like that. So yeah, like I said, I'm Benjamin, I've been with the Zephyr project for a few months now, but I've been doing open source and IoT for many years, did a bit at the Eclipse Foundation, I was also part of the Azure IoT product group at Microsoft, I've been hacking around with lots of open source, open hardware stuff, this silly picture, some of you might have seen that before, like during COVID, I ended up tinkering with Arduino's and stuff and building some kind of artificial nose using like TensorFlow running on MCUs, that kind of stuff. But yeah, these days I'm spending a lot of time on Zephyr and like I said, it's a disclaimer and or good news because you don't have to use Zephyr to hopefully take away some interesting tips from this session. Some of the things that are painful whenever you end up writing embedded software and whether it is quite frankly as a hobbyist or more like in a more professional capacity if you're interested in like building actual end user products, there's many pain points but some of them are listed here, hardware can be a pain because like it's difficult to debug, it's difficult to have like efficient round trips when it comes to testing your software plus hardware might not even be available these days and you need to be able to maybe jump from one architecture to the other or from one silicon vendor to the other because the sensor or the piece of equipment that you were planning on getting has all of a sudden years of lead time because COVID and like silicon shortage have messed up a lot of things there. Of course resources are limited so you need at some point some tools, some ways to make the best of the kind of resources you have so I'm going to try and talk about some of the tools that can help their sometimes embedded is lagging behind in terms of like getting to a point where you do at least some kind of testing of the of your code ideally getting to the point where you do even like continuous integration maybe continuous delivery but like historically this hasn't necessarily been a practice necessarily implemented in the embedded world so I'm going to show also a few things there managing the provenance of your software especially if you end up putting some open source bits and some third party libraries into your embedded system how can you manage that efficiently and security of course like if you care about IOT of course the the S in IOT stands for security right we all know that and again some tools there. Starting with the hardware one thing that I think was really interesting in this survey that Avnet so the big electronic components supplier did think they did that about a year and a half ago a survey of their customers and their community but it's probably would probably be the same findings today the chip shortage chip shortage really changed in a big way the way people build their embedded products and embedded designs and looking at some of the insights there but like people if effectively they say that nowadays due to longer lead times which are not expected to improve by the way people like the way they pick up their their parts for their design is is pretty much driven by the availability rather than the preference and basically what that also means is that when you get started building a new solution a new product you might not even have the hardware at hand at all in the first place and or whatever hardware you pick up for your initial V1 your initial POC might be completely different from what you get eventually and what you manage to secure for for for going production so this is also a reasonably old screenshot but I could have updated it and I'm pretty sure I would have found occurrences of like at some point in time last year I was interested in getting myself this particular developer kit only to realize that the lead time was over a year which is that's and that's not not the only one right so that's but it could very well be that this is like the perfect solution for my particular idea my particular design and I really want to get started how can it still do that well over the past few years there's been lots of tools many of them actually being being open source and or reasonably open source like most of most of them providing being being made available in open source and I want to talk about some of them and show you some of the things that you may not know and I see a lot of people taking pictures so that so it's probably a good sign that there's some new new names maybe for for some of you but yeah like simulating your and emulating your microcontroller is one thing and that's I mean there's many ways to look at it I could have put maybe QMU on that list but sometimes you need to do more when you build some kind of IoT device there will be sensors involved how do you emulate your sensors how do you emulate the connection to whatever whatever it is you use for communicating with the outside outside world whether it is Wi-Fi whether it is Bluetooth what can be done there so in no particular order showing a bit of the tools so this one might throw some people off because this will feel very maker may very DIY oriented except that it's actually really powerful it's called it's called a walkway and should you be targeting arm silicon and or expressive like ESP32 kind of kind of silicon you will usually get pretty far actually with with walkway in terms of testing not only your embedded software but potentially also the interaction with sensors and or actuators displays sort of stuff so really quickly short demo of what it means from a sort of a general standpoint and as well as what would it mean if I were to try and use it for my Zephyr embedded development so let's say I am targeting some kind of ESP32 thingy again right from my web browser and or this could be something also integrated in an IDE such as VS code and maybe a bunch of others there's like there's APIs to do it more locally if you don't want or don't you don't feel comfortable running it from your browser but essentially left-hand side this would be in this particular case an Arduino sort of Arduino based kind of embedded application I can actually run it against the fully simulated hardware all the way to the point which I mean it might be small for some of you folks I'm going to try and make it slightly bigger what we see here on the simulated or emulated display is the result of our embedded app that's running and we kind of guess that it's doing not only like display and GUI things but it's also simulating and emulating connection to a virtual Wi-Fi so like the effectively if we were to look at the code on the left-hand side like there's effectively connection to SSID blah blah blah which is fully emulated in the browser and of course like I could be wiring up more sensors like here we have a push button and whatnot I could do I could be doing more than that and or one thing that I could do as well is opening up the whoops the the palette let me just try and find the shortcut actually I'm confused there we go the yeah the palette the command palette similar to what I would have in VS code and I will just look at and this is small so I'm going to read read it out loud but upload firmware and start simulation essentially what I want to do here is take a firmware that I have already built on my machine which happens to be using zephyr and I think this is going to be just like blinking on led so but just for the sake of the demo let's see like this is exactly the same elf that I would be deploying on my real esp32 except that here well it's running on emulated emulated hardware and we see the the blue led blinking or pulsating and we see in the shell we have the the the zephyr prompt I mean this is one way to to to emulate things and I I kind of like the the way how easy it is to do things like simulate the wi-fi connection it might not be your thing if you you're doing something more more complex than just a couple sensors and then things like that but it's it can take you pretty far something else is so yeah that was for the demo another really interesting tool open source it's called the re-node again if you're targeting all sorts of arm cortex or arm whatever really and a bunch of others it can also take you really really far so this one will work will run fully locally on your machine although you can also use it in CIECD as we will see later if you want to do some kind of like testing and things like that you can also run it in like a docker container sort of thing but yeah re-node is also pretty cool I have my re-node console right here which I will make as big as I can maybe yeah so you have it you have like a common line interface where you can instantiate machines like the description of what is the kind of board that you are targeting like is so in my case I will I already have the description of the scm32 defkit that I am targeting and then I want to deploy an app on top of on top of it and this is going to fully emulate the hardware like this is really so what we see here is one of the code samples that you would get from zephyr zephyr does many things but one thing that it does is that it provides you with the pre-integration with the LVGL embedded graphical user interface framework and so this is the sample of like using LVGL to do UI stuff to the point where I actually have like integration with my with an emulated touchpad I have of course the screen as well and I could I mean I could do many things with as if I were to really interact with the with the physical hardware and one thing that I also want to call out because I think it's pretty powerful to sort of modernize embedded development practices is the frame buffer that you see here that's effectively rendering some kind of GUI you can fully interact with it through an API including using the the testing framework that we know interfaces with because that's the robot framework and you can do things like I'm going to like run my my built embedded application out of my CICD pipeline like whenever someone does a commit I deploy the app on the emulated firmware I wait 10 seconds I take a screen capture and I compare it pixel by pixel with a reference image right and that way I test whether there's been regressions or not in my UI this is the kind of thing that that I mean that we know it makes possible and as of actually the latest release of Zephyr the integration with the robot framework and like making it easy for people to write the kind of tests that I just described is effectively supported upstream where am I wrong chrome where is my actual here it is something else that's this one is actually really really new but it's and it's also kind of cloud-based it's something that ARM provides and that that that runs in in the cloud and I actually only tested it and tried it myself a couple days ago but I thought it was worth mentioning because it happens to just work and it's kind of like similar in terms of experience to what some of the things I've shown with with zephyr sorry with walk we all with re node but essentially and I think I might have the demo yeah like right here right in my browser again like I would be simulating some kind of stm32 defkit and I would have access to the the shell I mean like whatever my zephyr application is doing I can interact with it through the console if it exposes some kind of console interface I can interact with the simulated fake environmental sensors so like I want to pretend that the temperature is 35 Celsius whatever this would be all running on on simulated silicon so that's yeah all those tools I mean some of them will one way or the other they will be frustrating and or they will not if like effectively behave 100% like the actual silicon but they can save you a lot of times and or helps you test a few things that you wouldn't have been able to test as easily potentially a few words on the inside that when you deal with embedded it's sometimes hard to like really optimize your application sometimes well you really want to make sure that it actually fits in what you have and or sometimes it might be more for cost efficiency like you want to effectively make your app even smaller and only to realize that hey you know what now that I've made it smaller I can target maybe a slightly smaller SOC I don't need as much memory and I can save a few cents here and there on my bomb so that could be yeah that could be the kind of things that you want to look at one tool that's really well integrated with and that works just out of the box when you're using zephyr but pretty sure you would be able to use it just the same with any other RTOS really is a pen cover and it goes pretty far in terms of providing you stats with regards to how your binaries look like in terms of like what is the the size code wise what is the memory that your code is going to use what is the what is going to be the worst case scenario in terms of how much memory you're you're going to use on your stack and sort of stuff so just yeah main point really of my talk is to leave you with a bunch of pointers many of the things that I'm mentioning are way more way better explained probably in the documentation but now at least you know they exist and so for using something like pen cover it's really straightforward you just build your app and making sure you turn on the compile flag config stack usage and then you just run the thing and just really briefly so we're done with a re-note now yeah here so building my app so whatever is zephyr hello world and I just want to show you some of the and this is really maybe familiar to some of you but sort of the the thought process when it comes to using those kind of tools I just built the thing and then now I ran the pen cover so it's already integrated within in the zephyr build system like and effectively my browser just already opened up with pen cover enabled one thing that I could use the tool for is I mean I just ran a hello world it turns out this hello world is doing some basic a printing hello world on the console eventually my real application going production like in and building a real end user product I might want to turn off any sorts of communication on the serial and on the uart all together and here with the tool I can really easily navigate to all the bits and all the all the all the modules that make up my app and I can really easily realize that a driver wise the serial thing it's actually take eating up one port for 1.4 case in terms of like code at least I know that whenever I want to go a smaller that's certainly one big area of improvement right and I can also like dig deeper into every pretty much every every module and see like what's the stack usage and things like that and I think it can it can really help and also for security reasons actually we're going to see that turning off and removing some some unused features and or areas of your of your embedded code might might be useful as well but yeah that's for pen cover if you don't care about the sort of the having something as detailed like you can also just rely on the more simplified reports that you can also get out of the box with the with the Zephyr build system things like getting overall RAM usage or RAM usage is also something that you can get through those options. Another thing that can help with regards to optimization and or making the best of whatever resources you have available is there is a tool that you may or may not know exist in Zephyr but I didn't know initially and I find it really convenient which effectively you turn on and it will automatically every 10 seconds or so it will look at how full the the stack for each thread ever ever was right and just telling you that hey you feel you've been allocating this amount of memory to this background thread that's going to acquire sensor data but you probably don't need all that much because looking at the actual execution you never used all all the memory so don't and make make it smaller right and so the this one actually I can also show really quickly in in in ReNode like I'm doing demos with everything simulated here I guess but yeah I have another sample which is yeah thread analyzer so launching an emulated application which is again a hello world with a bunch of threads and like I said every few seconds it would dump it would give you information with regards to A this particular thread used like 29 percent like thread A used 29 percent out of the one K that was allocated to it so maybe like if stays the same over the course of the entire life cycle of the application maybe that's that's too much and you can cut that in half just give it give leave a bit for for safety but that that's I mean that's super easy to turn on in the in the Zephyr build and just enable the feature similarly something that super easy to enable I mean oftentimes it would actually be there by default if you ran some of the existing code samples etc is the Zephyr shell just want to make sure that I call out that it's it's full of hidden gems sometimes in terms of really helping you so of course it's gonna make during your sort of prototyping phase it's gonna make your app slightly bigger but it's gonna also make your life much much easier in terms of yeah I have this this driver this sensor this i2c sensor for which I'm building a driver and my driver kind of doesn't work well you're gonna probably troubleshoot what's going on by just like interacting with your i2c registers like manually through the through the Zephyr shell or if it's SPI you do it similarly or you interact with the ADCs or if you have enabled any kind of like file system you can interact with the file system you have Wi-Fi in your design in your stack then well you can actually discover the SSIDs figure out your signal strength etc etc and it's extensible as well and if you are to build your own app it might be in addition to logging and then to debugging you might actually want to come up with your own commands so that you can make make your life easier there so that's yeah I guess that's another one something else CICD how to maybe again modernize embedded development there one thing that I want to call out because I think it's a really good example of how to get started with it is the example application that the Zephyr project is maintaining and so one of the things this app helps you do is sort of learn a lot about the best practices in terms of how should you organize your your source code and your source tree that yeah what's the best approach to have if you have your own driver like where should you put it and something else that it does is that it shows you how using GitHub actions but really it doesn't matter what what matters is that it shows you the few lines of script of almost if you will to fire up a docker container set up the Zephyr build environment Zephyr SDK Zephyr to change some one dots and then build the app potentially run the test sheets that have that have been that are part of the of the repo for this app and then get a result right build successful build failed test successful test failed so it's I mean I would really recommend that if you want to get started with Zephyr you start essentially with this particular application and then you start tweaking and changing it towards your your actual code but yeah what essentially that means is that if I were to go in the GitHub actions section I can I mean I can look back in time of all my like previous builds whether they were successful or not I can get access to the test results download them and I can even get to the point where I can download the pre-built firmware like that was entirely ran in in a CICD on a CICD machine independently from my own local desktop and this particular firmware I can try and test on the real thing the real hardware or maybe on ReNode and things like that so that's yeah I mean it's there there's yeah just just look at the other code samples and things like that to start maybe improving the way you're doing things something else so it might be I mean the more we go into the presentation the less I'm happy with the title that I give to the presentation in the first place it doesn't have to do with necessary productivity it's more about like you might not know these kind of tools exist and they might save you a lot of time and potentially money when you're doing something in a in a really in a more professional capacity I don't know if you're aware or familiar with the notion of S-bomb of course we talk a lot about bombs in the hardware world but like when you ship a product yes it has hardware but it's also pulling in lots of software components like I said earlier lots of them probably are open source in in in one way or the other tracking the provenance and like keeping track of sort of the manifest of what it is that you put in a particular revision of your of your product might be might be a problem there are tools and there are standards to do that like the spdx standard helps really sort of annotate and decorate open source code with all the metadata that helps track the provenance like this particular source file that ended up being compiled and built and linked into your application while it comes from probably from some kind of github repo so it has a sha associated to it so this is something that can be captured right and so that's something that actually is really easy to capture when you build an app built on zephyr you just again have to rely and leverage the built in built build system to just like do as instructed here which is whenever you prepare your you prepare a build you just need to initiate the workspace so that so as it has the spdx things in it and then you build and then you get a report and I want to show you really really quickly what what's what's the output there so killing the the pen cover thing just just showing you that it's really just taking you seconds most of the time to do to do this stuff so we yeah so I'm going to initiate the spdx thing build to do you build the same sample as earlier there we go so I built some kind of some kind of hello world but now in addition to my binaries and to my my actual application I also have some manifests that could be useful further down the road because they actually captured a lot of information which I mean I there's a tool that I can use to visualize it yeah let's see it's not there I think I screwed up the first step didn't I I mean it's all right I don't have to do this this particular demo let's see train again I'm doing something obviously wrong there yeah whatever the the idea is that in addition to your binary you would have captured in a manifest and in a very standard actually format you have the list of all the individual source files that ended up being making it into your into your application and whenever like six months down the road there is a cv and a vulnerability report for a particular open source component that you know that you're using you can know whether you're impacted or not because depending on you might not be shipping the exact same version but through all the metadata that's being captured there this is not something that you can you can be aware of and yeah last but not least things to help you build more secure so I'm like full disclaimer I'm no security expert but I like to rely on whatever tool I can find out there to help me at least be more aware of things that I could do better to at least improve my my my security posture when it comes to securing some kind of IoT or embedded device well it's essentially to over maybe over simplified things but usually one thing that you want to do before you effectively go production is make sure that you've disabled all the things that can make your device and your solution less secure like if during prototyping you've been using like lots of logging and you've been outputting lots of information on the on the console and things like that this is probably all those and you have like compilation like debugging symbols enabled and things like that you want to turn that off and there might be and again like I will focus on zephyr but I'm pretty sure it's the same in other places there might be features available to you in the framework and in the os that may make things more and more secure like enforcing like memory protection features hardware protections whatever so there is a tool in zephyr that maybe people don't know exists but that I find really really interesting to to help you sort of realize what it is that you might be doing wrong or that you might want to have a closer look at this one is also super simple to trigger just run your build like just build the hardened config target and it will give you lots of information with regards to things that you may not want to you may want to stop using for your end end product and something as simple as a config boot banner like if even if you're not familiar with zephyr I can quickly tell you what that is it's the kind of prompt that you would get first time you you boot the system if there is some kind of serial output it would give you like the the actual exact version of zephyr you're running this might expose some this is useful info for for a potential add occur because they they know that you might be actually running some kind of vulnerable version of zephyr or what not and I mean similarly like are you running are you using the console are you logging are you do you have the build output stripped in my case I don't but what the tool is recommending is that I do right I should probably be stripping the beginning information and it would also warn you if you are using experimental features in the kernel like you you might have very good reasons to do that but at least it's yeah it's it's probably good to for you to double think about that and make sure that it's really intentional whenever you're running that in production and with that I think I left actually plenty of time for questions some pointers there's many ways for you to learn more and to get involved but first link will give you that's here for what is worth but all the the binaries that I run against work we against re node I put them there just for you to just realize that yeah whatever existing binary can be easily floated around in in different emulation tools and it should essentially just work out of the box the community is really really active on discord so if following this talk or any talk really this week there's more that you want to discuss with the community please join the discord channel and yeah everything else did the usual contact info and with that I will yeah take questions if there's any and if there's anyone from the project that feels that there were some useful tips that I forgot we can also talk about that all right yeah just shout the question and I will repeat it oh there's a mic even better uh yes I think it's working oh it's working so uh so can we get this presentation somehow yes I should have probably uploaded it in the system I haven't but I will yeah just like in the next hour or so I will make sure that the PDF is there yes yeah that's the wall point like there's tons of pointers you you rather have them yes thank you there was one here thread you know you mentioned the thread analyzer and I've worked with it a little bit but what I'm missing in it and maybe there's some configuration to add that it only prints the the data once so if I have some application that's running and later on maybe I connect to do some sampling or something else and I want to see these threads I don't get that data anymore so I think it's by default it's like every 10 seconds or so and or you can trigger it manually but like just look for the k-config options around around it like it's uh yeah it should it should be more than once right otherwise it's not that useful because you yeah yes all right whether questions online I guess not I didn't see any let me double check before all right well with that thank you for your attention feel free to reach me online and I will post a slide yes