 So, hi everyone and welcome back day one of Embedded Open Source Summit, second talk of the day, so hopefully you'll get a nice opening to this week full of very cool events. Let's start with our title. This is when we originally planned this presentation, this is the title we came up with, and I think it's pretty catchy, porting an AI-powered, wearable health monitor to Zephyr on open hardware. Our goal for the project was to embed ourselves more into the Zephyr project and learn more about some open hardware we wanted to try out and software. However, reality of embedded software development and computer science in general is that, well, as you most know here, plans don't always exactly go as we want them to. So if we were actually to retitle the presentation so it more accurately reflects what we spent most of our time working on this project, it definitely wouldn't be this. It would be sort of something like this. But again, this is not as catchy and it's definitely a bit too long, but it definitely kind of encapsulates what we worked on. So, Guba, why don't you tell us what will we cover today? Yeah, sure. So our presentation follows a standard war story flow where we touch upon each subject and then cover how things were supposed to look like and what they actually were in reality. So who are we? So this is me on the left. I'm currently leading a team at Tietoevry for Undisclosed Client. And we're doing 5G networks, L1, L2 stock. I also graduated some time ago from MSC and embedded systems. And in my free time I do some sports and, well, climbing, surfing, windsurfing, etc. But I'm currently after my new ACL reconstruction, so I might be slow to walk around. And this is Shimon. Yeah, hi all. And this is Jakub, by the way. So I'm an open technology engineer at Avanad, which basically means I get to play around with some open source during working hours, which is always cool. I'm currently co-chairing the Carbon OSDK project, which is a green software foundation project. Feel free to check it out afterwards. There are links, basically all the blue hyperlinks will be links to the actual project. And I'm also a final year Masters of Engineering, Computer Science student at the University College London. In my spare time I dabbled in the areas of game development, embedded systems, IoT, and I'm pretty crazy about windsurfing. So what the motivation was for our project? It was a goal to create a health-wearable monitor that can measure the blood pressure in a non-invasive way using photoplatismography. I'll tell you in a moment what's that. So why choose this problem? Initially it was a part of the Design for a Cause competition that we took part in and it was during the pandemic and we all know how everything was overwhelmed back then. I guess everyone has already fed up with this topic, so I won't dig into it more. And what it allowed for? It allowed for monitoring elderly people and even ourselves with our smartwatches, for example. It was a non-invasive way to monitor our vital signals. And of course we chose it to learn new things, so that's why we ported it to Zephyr. So the original plan, original project was basically looking like this. It was based on closed source Arduino Nano 33 IoT that was connected to Max 30102 PPG sensor. It also had a small LCD display and some buttons for actuators. It was a low-power device, so it went into sleep and deep sleep, etc. And since it wasn't capable of having a full AI model deployed on-premises, we used AWS Lambda for the inference part and it communicated using MQTT. Also since not everywhere Wi-Fi was present, a companion app was necessary to be able to communicate with the Wi-Fi. As I mentioned, it was originally a part of a competition. Basically on the right you can see a nice 3D model we did back when we were developing the project. And then on the left you can see how it actually looked printed out and fully assembled. So in my opinion it looks really nice. It's pretty bulky for a health monitor right, but bear in mind this was just a prototype to test the theory of ours. And all of the blogs and info about that project you can find through the URLs here on Jakobsblog. So obviously the next step for this project was to bring this to open source. Open source RTOS and some open hardware. So say hello to Zephyr and Qwiklogic EOS is free. Kuba, why don't you continue introducing this a little bit? Yeah, sure. So our new hardware was Qwiklogic Qt+, which was running EOS S3 SoC which allowed, well, it combined and embedded FPGA with ARM Cortex M4 MCU which was capable of deploying machine learning models directly on the board. That's why we basically chose it. The reality was a little bit different. We'll tell you a little bit why in a moment. So what was the key selling point to us? What were the key selling points? So it had sensor fusion engine, so flexible fusion engine and sensor manager that basically offloaded computations on sensors and data collection. It had the embedded FPGA. It was low power SoC, apparently allows for I2S always on PDM microphone data collection, so it sounds pretty cool. So we all should know that all fans change when met with reality. As a quote of Helmut von Moltke, I found from 19th century says, well, no plan survives first contact with the enemy. In our case, the enemy was, I guess, trusting the branding side of the SoC and not actually looking too deeply in the Zephyr source code before committing to the particular board. We may have mentioned or we haven't, basically they stated they had full Zephyr support for the board, which there was Zephyr support, I agree that, but full was a tiny bit of a stretch. Don't get me wrong, we really, really like the SoC. It's very fun to work with now. It has a lot of very cool features. It's really low power and it's really powerful for how little it is, but it was definitely a little bit difficult to get started. However, in hindsight, if we didn't have to deal with all of these issues and this project didn't turn into kind of a war story, we'd never learned so much about the actual hardware, we'd never be able to add some Zephyr support for it and we wouldn't learn so much about Zephyr. So the first issue that came about when we got the board was, well, with board splashing, pretty generic. When we received the boards, we were very excited to get started and start coding right away. Unfortunately, after trying to flash some, cast some cortex and four applications and doing some digging on GitHub and seeing that, well, nothing was really flashing to the board, we realized that the bootloader of the board was completely worked. It just didn't work, the one that came with it. So this was the first issue we encountered and luckily we found some correct sources on QuickLogic forum and GitHub for reflashing it with a jailing debugger and what to follow. They had some specific program to load up on the board and this worked perfectly fine for Jakub as he had a jailing with him. Unfortunately for me at the time, Jakub was in Warsaw, I was in London, I didn't have a jailing. All I had was an ST link and open OCD. So I decided to take an open source alternative route and try it out. And yeah, so I went for open OCD and tried. Unfortunately for me, I quickly realized that the flash driver was not upstreamed to open OCD and, well, I'm guessing everyone is aware of the open OCD debugger project so I will not be introducing it. Recently I jumped to open OCD's IRC and checked in with the community where I could find more info about it and was pointed to a patch by Paul Furster that actually added an EOSS-free flashing driver. So I rebased, it was a little bit outdated because it didn't get merged in at the time when he wrote it. So I rebased it and tried it out, basically, yeah. Unfortunately, as you might have guessed by now, it did not work. It failed on flashing when sending a read ID, RGID, SPI flash command and never actually oh, never. It didn't receive the proper ID of the onboard flash when doing that, of the quick thing, plus board. So instead what I actually received, if you look at the red line at the very top was a bunch of zeros. And what I wanted to get was GD256Q16CID, which is written out down there. I spent, well, basically a week trying to understand how open OCD works a little bit more because I never actually had to dig into it and what was wrong with the driver and I tried even forcing the driver to select the correct flash device no matter what, so just to pick that flash device and try writing on the board, see what happens. As you might have guessed, since that command didn't go through, this didn't go too well, I completely destroyed the boot load that was already on board. It just, when I plugged it in, nothing happened. Yeah, that was not cool. So at the time, I was a bit, well, pressured by time because I wanted to get this done before this event as well and also I didn't want to continue sitting on this without much progress. So I put it aside. I grabbed myself a J-Link which I'm not most proud of, but I'm planning to get back into this and actually upstream the fix for the patch. So future folks working with this won't have the same issue. So yeah, so I moved on from that and went on to using a J-Link. I flashed it, worked fine. So now I have a bootloader, Kuba has a bootloader, ready to go, ready to go into Zephyr. Yeah, so about that, well, it turned out that some peripherals were not really supported, we had Clocks and UART and then we decided, yeah, let's connect a sensor. NGPOs. NGPOs, yeah, well, mostly. And yeah, we went on to adding the support to sensor manager. Well, it turns out that this is a very low-power DSP that has its own programming language and it turned out that adding support for that in Zephyr might be a little bit too much. So, yeah, what we did, we thought that, yeah, I square C devices are mapped on the wishbone bus, so let's just use the I square C directly. And yes and no, it turned out that it also had some steps in order to work. And we decided to ditch the sensor manager for now. Maybe we'll add support for it sometime in the future, but Robert from Quick Logic told us that only a handful of their clients were using it successfully and it had an obscure technical reference manual, so we decided to do that. I square C. And yeah, I started to port the hull of this board to Zephyr and coded the initial code for I square C support and then Shimon to cover it. So I'll cover that. So basically, what did we end up doing to improve the boards and the SOX support on Zephyr? We started by reading a lot of QuickLogic's Quark SDK code and previous Antmicro's sponsor of this conference. So shout out to them. Work on the entries effort to support, which we're very, very grateful for because it gave us a very, very solid starting point. The Quark SDK is basically a VRTOS fork developed by QuickLogic. And we added I square C and wishbone bus support to the hardware abstraction layer module for Zephyr. And then we added the actual I square C driver for USS3 in Zephyr. Yeah, so I'm just looking around it to see the screen as well, if I'm not missing anything. So why did we actually have to do that? As you can see, we're just jumping here back for a second. This is the sensor posting subsystem on the SOC and there is what's called a flexible fusion engine. And that is connected to the wishbone buses. And that's basically how we are communicating with the I square C bus for the wishbone bus for the flexible fusion engine. So the first blocker we had when porting the HAL that was already there for the VRTOS version to Zephyr was that the flexible fusion engine was not enabled. It was being enabled somewhere in the VRTOS version, but was never done for the Zephyr port. So what we did here is we read through that. We looked at the technical reference manual. And we modified the code in the SOC's initialization, which is located, I put the reference to it. And SOC, the program that runs on SOC initialization. And we added, first of all, enabling flexible fusion engine power domain, line 63 to 65, then waking it up on like 67, setting up all the clock sources, and setting up the clock divider ratios, which, yeah, on line 78 through 81. And then finally, routing those clock signals to the actual I2C peripherals. After enabling the flexible fusion engine, I actually got to test the I2C HAL on Zephyr. And I first introduced a small change to it to fix a bug, which I discovered at the time, where when I was testing the HAL on Quiklogic VRTOS. Basically, after each wishbone bus transmission that actually went out to the I2C devices data register, there was a tiny delay for the message to actually be transmitted or received. And the actual bug happens when there is another wishbone bus write or read command right after that. And then the FFE is still busy handling the previous command. And according to how the code was wrote at the time, it just crashed. So the I2C command failed. How I actually discovered the bug was pretty funny, because I just added, well, the classic way of adding print statements after every single wishbone bus send. And this introduced enough delay for the message to start going, because I was like, OK, it fails at this print. So let's see what happens before the print when I remove it. And then I knew step by step in which section it failed. So what I did is I added a busy loop that waited for the register to clear up after each such command. And at the moment, that's how I added it. It worked fine for me. So for now, I'm keeping it. Maybe there is a better option, but I thought this might be the best one for now. And after that, I was very excited to get to the actual driver adding stage, having worked a little bit with Zephyr before. I beforehand tried adding support for a board called view terminal, which I managed to add some enable support, but then someone else picked it up and actually now have support for the board. So thanks be to them. But this gave me some experience to know that after having some drivers, it goes pretty much downhill from there for up development. So I implemented the driver based on the how that you've seen previously. And we found some really great resources from last year's Zephyr project developer summit on driver development by, sorry if I put your name, Gerard Maru-Paretas. And it really helped us a lot to get a good understanding of how drivers works in the Zephyr system. It was a bit tricky for me being not really a kernel deaf and the Zephyr newbie to implement the driver, but manageable. So we followed the logical, like the normal flow of modifying the driver's CMake list to build the files we want for the driver, add the kconfig file for our driver, modify the main kconfig file to import that other kconfig file, then add the DTS bindings in the proper location for it, and finally add the actual source code for the driver. And well, at the very end, update the SOC's DTS to include the new peripheral, the new I2C peripheral. A quick note of what I had an issue with at the time is I mistyped the binding, so it did not match the DT driver compact macro defined in the C source code. And well, I've learned, however, that this is very much crucial for the actual driver to ever be linked properly and to be included in the DTS. So finally, after adding the actual source code for the driver, it was time to get to the actual adding of I2C peripherals, I2C devices. So, as predicted by us previously, and what we are very happy with, adding the sensor, the SP02 sensor Max3010 support, was basically as simple as adding a DTS definition, right? We just dropped it in the device DTS with where the I2C is defined. We had to change a few configs just to configure our device. Yeah, small thing, you may notice that on the right, it's 30101 and we use 30102, which are pretty similar, so that's why we decided to use the existing driver. Yeah, there is a, thanks, Kuba, there is a driver in Zephyr 3 for Max30101 device, which is yeah, similar enough to Max30102, there's just a difference in the first one has another LED, which we just don't have to use, so it wasn't too big. And we just used that, we then set up the driver, enabled those two LEDs we wanted to use, so IR and LED, we changed the current of the pulses of the LEDs to not be maximum, because we realized that when it was max, our values read, always hit the ceiling, and we didn't get the actual resolution of the sensor, and then we changed some sampling conflicts. Similarly for the display, again, I was really amazed by this, that after I added the driver, just like things worked, which for me was like a very great achievement, I was very happy with myself. Adding the display support, the SSD-1306, was also as simple as adding a DTS definition, there already was support for it in entry, there were some great samples for using it, so really kudos to whoever added that, I haven't looked at the Git blame, but we'll do afterwards and add them to the list of people that helped me, and it just needed a definition in the DTS, a few configs, and we were ready to go. Oh yeah, we had to add dynamic memory allocation for a specific sample we used. Yeah, it's one small pain point that we encountered was that there is apparently a USB to UART on board, but it requires a FPGA IP component that's being loaded on boot by the bootloader, and the driver for that is not in Zephyr, it's written in free RTOS, so when we started working on this project, it turned out that, well, we cannot use the USB for printing logs, so, but thankfully, hardware pins for UART were exposed and it worked out of the box, so quick tip, I bet you all know that if you don't have USB to UART converter, you can simply add another board as a listener, and yeah, and listen on the pins there. So, I guess you're pretty tired already, so I'll try to be brief here. I will talk a little bit about deploying a machine learning model on EOS S3, so our initial goal was to deploy the model on the embedded FPGA, it turned out that it wasn't powerful enough, it only had two 32 by 32 multipliers, so it wouldn't fit any sensible machine learning model, it's rather used as a companion, as I said, for USB to UART or some other offloading operations. Thankfully, we could deploy the model on the Cortex-M4, and there what we did was, we had to choose, use CEN-CML or use just raw TensorFlow Lite for microcontrollers, and as for CEN-CML, it's an entire suite for data capture, labeling aggregation and whatnot, and it turned out to not be as straightforward to perform just the inference on it, so we decided to not dig deeper into that, it looks really solid from the point of data collection, labeling, aggregation, et cetera, but for us it was too much, we already had some data from an open data set that we wanted to simply deploy, so thankfully TF Lite worked out of the box, there is a support in entry of Zephyr for that, it uses ARM, SMSYS, and then IP operations for acceleration of these various machine learning operations that might be present there, and what we had to do was to add quantization, and if you're not familiar with that, it's simply remapping values from one range into the other, so in our case it was floats to int8s as that was our model was using underneath, it simply uses a flood buffer of N, of Chars basically, yeah. Yeah, so during our talks with Robert from QuickLogic, we were introduced to a different project from the open hardware group called Core 5 MCU, and in general the Core 5 family of boards, and it's a RISC 5 board with a much large embedded FPGA, and from what we understand, we would be able to drop ML models on that FPGA, so if there's someone here from open hardware group, we really much would like to get our hands on the board and to try out getting a machine learning model deployed on FPGA running alongside an RTOS, it would be very cool, so I know there somewhere around here, if they're not here we'll find an afterwards, but yeah, just a quick interjection. Finally do what we were supposed to do with this board. So yeah, let's talk a little bit more about blood pressure prediction, I guess you all know what blood pressure is, I guess at this time it might be getting lower as we ate our breakfast some time ago, and yeah, photo platysmography called also PPG, do not be very verbatim, is a technique of applying, well, infrared or red radiation from LED onto our skin, and then measuring the reflected wave. So this way we may see our current heart rate, for example also our blood oxidization, and as in our case, we can also infer the blood pressure from it. So yeah, biological signals are pretty complex, as you can see on this graph, I did it myself, I'm pretty proud of it, we have systolic peak for example, dichrotic notch, diastolic peak, and what we basically did was, instead of putting all sampling period samples to the model, which in our case would be for example, 200 samples, we decided to extract data from one such period. In this case it was for example, the peak ratio, cycle duration, and some deltas that are defined in the nomenclature, in the literature of basically biological signal processing, so it's a known problem, and I will mention about the accuracy later, but I recently read that CNNs, which stand for convolutional neural networks, are really good at processing these features. So yeah, instead of 200, now we have six inputs to the model, which makes it around one kilobyte, so it's really small, but still we need to do data filtering and pre-processing on the MCU, which is, let's say, we have a classic space and execution time trade-off, so we need to sacrifice some processing speed to have the samples processed on the MCU. So yeah, one thing that's not on the slide here is that I struggled a little bit with the accuracy of the pre-processing on the model. It turned out that, well, C++ implemented some things differently from Python in this case, and the data set for training will be mentioned by the end of this presentation. The ground truth for this model training was from arterial blood pressure, which is an invasive method of measuring. So yeah, depending on your needs, you may choose a smaller model and more data pre-processing, or you can go for a CNN, which definitely won't be one kilobyte, I would say more like megabytes, and not all edge devices can afford that. Accuracy was pretty standard at the time I crafted the initial model, so mean average error is basically how different is the model regression value from the, or the predicted value from the ground truth and just averaging it. So for systolic blood pressure, 16.5 for diastolic, it's 8.3. Yeah, some results of us. Boris, yeah. So this was actually what we originally wanted to present on this whole small section that is encapsulated in basically a single two slides is all including the AI on USS-free part. Currently all of this is just a prototype sitting on a breadboard. As you can see on the far right, we have the quick think plus with the tiny USS-free sitting on it, the max 30102 sensor connected to the screen, and then this is my setup, so USB 32 as the USB to UART converter or interface. And we're planning to polish off the software with a little bit, so it's actually even more Zephyr-like and using more of the Zephyr features we wanted to experiment with and redesigned the casing, so it's a little bit nicer and looks more like our previous project, so we can actually stick it in our hand and try it out, test it maybe for a slightly longer periods of time. Yeah, so just for that, this is a big case and it's not a PCB that's crafted by us, it's just connecting some components, but if we were to do it professionally, I think we should design a PCB and put all the small things on. Okay, that's another thing today. Yeah, I know, like the smart watch presentation might cover that. Yep. So as always, each project has much room for improvement and extensions, so let's mention some of the things we'd like to cover, so apart from, this is me committing to it, submitting PRs to Zephyr to upstream the I2C drivers for the board and some of the support for the HAL module of it and also the OpenOCD flashing driver. We have some things to figure out and debug. First of all is whenever we actually configured the user button on board as GPRO input with Zephyr, for some reason it stopped working. When we make pinmuxed it and we didn't configure it, we were able to read the input from it the moment we configured it, it's gone. Input is just always one or always zero, so that's something to figure out. Bootloader on the board might be broken, at least the one we got was broken. We had to do a little fixing of it or maybe just skipping some safety features of it to get it fully working as we wanted, but that's another thing to explore. What we were not fully happy with and we want to maybe look into was the TinyFPGA programmer since the board is a bit more complex. It's not just an MCU, it's an MCU of an embedded FPGA. The programmer is written specifically for its TinyFPGA. It's in our experience sometimes just hanged or crashed or on macOS or Asakili nukes on the M1, it just never really competed and never brought it properly. And the last thing we would want to do is actually using the FPGA on board the USS3 and doing some maybe data processing on it. One more thing, the flasher had some weird quirk that for example, when we were flashing a constant size binary, everything was fine, but then I tried putting much larger binary and it simply wouldn't go through and it suddenly bricked our board. And this is kind of weird behavior, right? So we might want to debug the upstream and see what's going on there. Yeah, our fix for it was we flashed the whole thing and try flashing the program with the bootloader. So like once the bootloader is flashing at the same time, flash the program, do it a few times. As I call it, warm up the flash and just eventually start working fine on every single flash. It's just, I don't know why. So this is something definitely worth exploring. Oh yeah, also there is a re-note system emulator that we didn't have time to explore, but apparently it's quite interesting. Yeah, it's a very cool system developed by Antmicro and they have I think a like a Zephyr dashboard for boards with Zephyr support and that you can also emulate fully in re-note. So as much as I understand it, you couldn't emulate the entire SOC with it, which sounds very, very cool. We didn't get to play with what I'm guessing Antmicro will have some presentations today or during the week talking or at least mentioning it or you can see it at their stand. Talk to them at least. So some acknowledgments, bear in mind this is not everyone, I look forward to put everyone. It's probably would be quite a few people from the Zephyr project and others adding support for the things we already used. But big shout out to Chris Fritz and Henry Riggs Anderson and a few other members of the Zephyr community for amazing help during the project development. They were a great help when I just jumped in on Discord and were quick to respond and reply with help for it and hoping for further help when PR guidance when pushing it. Paul Feritzer from Open OCD for guidance in debugging Open OCD and folks from Antmicro for paving the way to USS-free support in Zephyr. Yeah, you can find some of the links for further readings later on here. So feel free to close the slides and look at it and quickly mentioning some of the takeaways because I know we are close to time. Yeah, so when choosing a board to develop, check for peripheral support in Zephyr several times at least document your progress. You'll thank yourself later. Yeah, we told you how we approach adding I2C driver. You can use TF Lite and yeah, when drivers are in place, Zephyr just works. Exactly, that's a very nice takeaway I think which I think we should all be proud of and happy with. Yeah, and that's it. Thanks a lot and if you have any questions, yeah, you can have a minute or two. Yeah, any questions? Does it work? Yeah, it works. Oh my, maybe actually, it's been working. Might have to slip. Hello, does it work? I think it works. It does. No, it's me, sorry. I'll repeat. Okay, so the question was that we used the FPGA for the model and no, we did not. We were considering using it and trying to flash something either way but we were discouraged from doing that. So again, yeah, in this project we didn't use it in the end. Okay, I have a second question. Yeah, it's okay. So you used this optical sensor but then you had a quite a limited feature set. So six features. So in my opinion, machine learning or neural networks always thrive if you have like more features and you're not that strict in selecting and then I would be really interesting if you just broaden that and make the feature vector a bit bigger and maybe like more raw values and if you try to then interfere or interfere also like, since you had blood pressure but you could also go for oxygen levels. You could also go for like some hard anomalies. I don't know. So that would be really interesting to see, I think. Yeah, thanks a lot. Yeah, we... I can, yeah, I can. So answering that, initially the model had 30 features. We did some, okay. It's, I think we're... Yeah, let's just grab the question afterwards. So thanks everyone again and PR socials if you want to grab them. Let me just wrap up for you guys here that there's a chat session as well that you can engage in and ask additional questions and I'd like to thank you again for your presentation. Thank you. And yeah. All right, cool. Thanks everyone.