 Hi, and welcome to the Zephyr Developer Summit. My name is Yuval Perez, and I'm going to talk to you a little bit about time to market for embedded applications using Zephyr and Pigweed, and I've added a little bit of an extra twist to it, Firebase. If you need to contact me about this talk or any other talks I've done, please feel free to reach out to me via email, discord, or GitHub. So I've worked at a couple of startups in my life, and really I've found out that nowadays, more than ever, every company timeline is everything. And this is true for even personal projects. If I have a pet project that I'm working on, I don't want to be spending a lot of extra time on boilerplate logic. And once you've launched a product, even iterating on it or tweaking it is hard enough as it is. This talk kind of came about because I've used tools such as Firebase. It's not the only one, but it does kind of bring everything together in one place. And it's a really good way to not have to predict exactly what your users are going to do once you've already launched the product. And this is something that is available to most Android iOS, web applications, servers, things that are very far removed from what we do in our day-to-day. But we're going to see how it's not. So some highlights that we're going to focus on is remote configs, being able to use our kind of Firebase account to change how our EC behaves. And that might mean turning on and off a feature or maybe tweaking a threshold for some calculation. We're going to check user engagement in real time as well and kind of track what features are being actively used and how users are using our application. And then we'll take a quick peek into crash analytics through Firebase. This, I will give a disclaimer, does not work yet, but it doesn't mean it can't. So the bridge between those two, between our Zephyr embedded application and the Firebase console is Pigweed. And we want to move quickly. And the question is, can we have both of these, both the embedded application and this quick iteration? And the answer is yes. And we're going to build a product right here in the next probably 15 minutes. So what we're going to need to do is decide what the product is. I've got one in mind. And then we're going to set it up, configure logging, set up the application logic, such as tracking events, setting up remote configs for how we're going to modify our algorithms at runtime, and then give a peek into how crash analytics could work. So the product is something I've been wanting to do for a long time is building an automatic gear shifter for bicycles. One day I'll actually build it. And the inputs are a bunch of sensors on the bicycles. We can measure cadence, power, speed, gravity vector, and whether or not the user overrode our decision. And the output is the gear selection. We have a rear and a front cassette. So we're going to start off with a much simpler algorithm. And what we're going to do is we're going to take the user's cadence here, which you can see fluctuates just under 100 in this case. And then the amount of power that they're outputting. And then we can effectively divide it, right? We can get energy per rotation of the pedal by taking the power divided by the cadence. And we can see that. And we're going to set a simple threshold that I think was 160 or around 160. And if we cross that threshold, we need to shift down, basically. I mean, the user is not spinning faster and they need a little bit of help. They're pushing a little too hard and they need the lower gear. So the workflow that comes through here is we're going to collect a bunch of data, compute the energy per rotation. And then here I also included this optional step for computing the slope. And then compare against the threshold, which is the green line. So this is to detect if we're going uphill, maybe. And then if we pass the threshold, we're going to shift. If we fail the threshold, we're going to wait. So setting it all up, we're going to need Zephyr and we're going to need Pigweed. So we're going to add these to our manifest. And then we're going to enable C++. I believe now this actually became deprecated. So don't do that one unless you're using an older version of Zephyr. Now we're going to initially start off with a really simple main CC. And we're going to just log. Hello for Zephyr logging because we're going to need to debug things and we might want some logging along the way. When we go to build it, we're going to need the Pigweed environment. So every time you run West Update and you pull the latest version of Pigweed from main, you're going to need to update the bootstrap.sh. If you're not updating the SHA that you're checking out, you just need to run this the first time. And then every other time this is a much faster one, it just activates the virtual environment. If you're setting up a virtual environment for Zephyr, I would suggest just leverage the one that's already set up in Pigweed. So once you've run Bootstrap, that's when you do the pip install Zephyr scripts requirements.txt. And they will basically just merge the two virtual environments into one. When we go to build it, it's yeah, that's it. We just build our application, just like any other. So the new kconfig we're going to add here is we're going to add this tokenized RPCs. And what this does, it does quite a lot. It effectively starts up a server on our EC that is going to be the host to one or more services, RPC services. The first one is going to be this tokenized RPC service, which will automatically get added for you. And any log commands that you have will be tokenized, meaning the string is going to be reduced to a four byte hash. And it will automatically get passed through to the RPC service that logs it. So this is not 100% required, but it is the most optimal way to get started, really. You can feel free to look through the kconfig for Pigweed logging. We have plain Zephyr logs, which means everything just gets passed through to the Zephyr backends. And we have plain tokenized logging, which means that the logs are going to come through as a base 64 encoded tokens. So Pigweed is going to actually intercept all the Zephyr logs and it's going to create those hash based tokens. And then it's going to put everything into a log entry message and pass it into a service stream, meaning effectively the EC is going to just push this out through the UART. And if nobody's listening, it's just going to go into DevNol. If you want to grab it, what you need to do is basically just run this Python 3. There's a console involved with Pigweed already. You have to be inside the Pigweed environment to do this. You pass in the device. In this case, I was connected to USB 1 and then you need to give it the token database so it can decode the tokens that were come through. The other alternative is a web console. If you want to use the web console, you're going to jump right in here and you're going to have this really nice web UI. You can connect directly to your device. You can upload different token databases. There's a couple of settings in here to allow you to change how you want to connect to it, whether it's plain logging, base 64 log encoding or RPC tokens. There are extensions coming soon where you can even send RPC commands through this web console so you can actually pretend, kind of mock the host of your embedded controller. If you want to decode anywhere else other than these consoles and the Python and the web consoles, you can very easily do this through all of the Pigweed RPC-supported targets which are in Python C++, TypeScript and Java if you're maybe using your phone to control this gear shifter. The actual core application logic is we're going to dive into a little bit of math. It won't be too bad. We're going to put things into a buffer of these Q31 values. Now, if you're not familiar with the Q31 values, I highly suggest checking out the Zephyr DSP Digital Signal Processing Library. It will make your life much easier when doing some of these fixed point arithmetic. The key thing we need to know about it is really it just has a range of negative one to one and a bit shift value for scaling. So if the shift is one, the range is just multiplied by two, right? If the shift is two, the range is now negative four to four. If the shift is negative one, then the range is minus half to half, so on and so forth. And now we're going to get the current threshold and we're going to use Remote Config RPC through that, which is the new thing that we've added for this talk. This is not an official pigweed thing, but it is built on top of pigweed. And then we're going to iterate through the buffer and compare it against the threshold. And then whenever we get a gear shift, we're going to register an event with Firebase. So how do we track these events? That's the most important. We want to know when the system is doing things in general, while abstracting away the personal identifying information, PII. So the way this works, we've created a simple service, right? This is going to be a Firebase event service. We're going to create a service stream here, return to the client. And this works very similar to how the logging does, except for in logging, we don't mind losing events in the event service. We probably don't want to, we want to avoid missing them. So what we've done is added one little bit of caching here. If there is no host, right? If the user isn't using their phone actively to connect to this device while they're cycling, we're going to save things to an event cache. And the event cache can have different rules, right? This just depends on how the application, how complicated you want your application to be. Maybe it can be first in, first out, right? You can evict the oldest, evict the newest events. You can even have different signals to kind of turn on maybe the Bluetooth if the event cache passes a certain watermark so that we can connect and dump it and then shut down the Bluetooth connection to save power. But the general idea here is we're going to pass one or more of these events over to the host. And then the host is the one that's actually responsible for passing these over to Firebase. So the idea is the EC does not need a live internet connection. But the application can pretend like the connection exists. And in the actual Firebase console, you can see here, I actually got this fully piped and we have connections, first visits basically every time, new devices that came online, right? It attracts these page views, but those are kind of different areas of the application that I called pages, different sessions. And then you can see two unique users for this because they're two devices. We don't get any PII obviously, but Firebase also comes with a debug live view, which is really, really nice for debugging these events. So you can see here when I basically first came online, I went into the original, the kind of entry point for the application. The application configured itself to transport mode, provided RPC as the mode here. So this is a parameter to the transport mode event and then synchronize this database, the token database, and then got the full connection. And so these events were all kind of cached and then it was all dumped right here. So once we have events, we probably want to see how different users are going through things. And having remote config is the way to do that. And this is just the kind of inverse service. It pipes data from the server to the host. And then when the EC gets a connection to the host, we're going to pull the latest and then cache those configurations. And Firebase supports technically four. I've only implemented three because I couldn't think of a good reason to have a blob type of thing here. We have these ints, booleans, and strings, and then we get client stream. So we might get one or more of these values when the host connects to the EC. And when we look at the remote config console, we can see we can add them here. We can add new parameters. We can tweak them. We can then select also what percentage of the users. And this is where it gets really interesting. So in our example of setting a threshold, I'll go back to our graph here. So we set a threshold and maybe 160 joules per rotation in this example. And we maybe are seeing that we are getting a lot of events where the user manually overrode. So we shifted maybe up or down a gear. And then the user shifted back up immediately afterwards, right? Those are events that we can track when the user shifted in the opposite direction within maybe five seconds of when we shifted. So our goal is to reduce the numbers of those events. And what we can do is set this threshold as a remote config, right? It's a simple int remote config. And we can then use a B testing to set maybe 50% of our users to have 160 as the threshold and then 50% of the users to have 165 as the threshold. And then we can compare the events from those two groups and see which one worked better. Once we decide, we can push the same value to all of our users as the new baseline. The other thing that we can do with the Boolean one is maybe add a flag that guards it, right? So one of the earlier flow diagram that I provided showed that we want to maybe take into account the gradient. So we use an accelerometer to measure the down vector relative to the bicycles forward rolling. And we can say, OK, well, maybe we want to tweak the threshold by percentage depending on the gradient. And we can then maybe just enable or disable that entire code path using remote config. So crash reporting, like I said, is the experimental point. And this works with kind of the cache being the assumed path, right? The previous ones are if there is no connection, then go to cache and then flush it on connection. Now what we have is when we crash, we're going to hit our trap. We're going to log all the crash report information. And then when the EC boots back up and it gets a connection, the Crashlytics server is going to stream basically down. If we're connected, we're going to get pull the cache data and we're going to flush it into the host. Like I said, this doesn't quite work yet because Firebase specifically did not have a free form crash reporting. It's possible I've started looking into maybe using their Android NDK, the native development kit to kind of piggyback and have a almost a backdoor into Firebase. But that I couldn't get that to work quite yet. There are other crash analytic platforms that will allow you to do this. I just wanted to keep everything under the same umbrella for this talk, which is why this wasn't quite working yet. So finally, getting this all together, what this looks like, we've launched a product that makes a lot of assumptions on the threshold, has maybe a couple of different configurations on the code path for making the decision. And the beautiful thing about it is we can make informed decisions not based on a small study inside of our own company, which is not actually a company in my house, a pet project, and spend a lot of time where we can ship this to users and then use AB testing to configure these things and use analytics based on events and to drive our product into a better and more fine tuned location, which means that we've actually hit the market before having to spend a lot of money on research, a lot of time on research, and so on and so forth. The addition of crash analytics will allow us to make sure that users are getting the reliable service that they're hoping for. So in our example, right, we've added an event for manual gear overrides. We talked about that when we shifted and then the user shifted to correct us, or maybe the user shifted without any of our shifts. Meaning we missed an opportunity to do this correctly. We used Remote Config to set the threshold and we can improve our algorithm in order by AB testing and making sure we're getting the user behavior that we wanted. So if you have any questions, please feel free to reach out to me. My contact info is on the second slide of this and I'll be happy to answer any questions that you might have. Thanks for everything. Bye.