 Hey folks, I'm Aaron Massey. I'm a software engineer with Google on the Chrome OS embedded controller firmware team. You may also know me from being a maintainer of the Zephyr FieldGage API, as well as of emulation in the testing framework as well. So we have a goal. We want to scalably verify peripheral drivers. Now let's dissect this a little bit. Specifically, we have, let's say some chip for IC. Let's say it's a FieldGage as part of a smart battery pack. Well, we have some driver that knows everything about that FieldGage. It knows about its registers and it knows how to talk to it. What kind of, like is it SMBus, is it not, what have you. We want to verify that driver, what the piece of code that knows explicitly and is specific to that chip. We are not interested in verifying the bus controller driver here. That's sort of out of scope for what we're going to be talking about here. So just the driver that is specific to a peripheral chip. Now we also want to do this scalably. So what does that actually mean? Well it means we want it to be cheap and terministic so that we can run in CI. And what we're going to see later in this talk is that we can get really good quality and scalable testing by combining emulated testing with hardware testing. So let's get into it. We're going to talk about peripheral emulators and we're going to talk about when or why we should emulate. Like, is it worth the engineering cost? How do we create an emulator? And once we've created one, how do we actually use it in a test? Then we'll have some final comments and I'll have some time for Q and A. All right, so what is an emulator actually? And by the way, I've gone from off-chip peripheral emulator to just emulator because saying off-chip peripheral emulator is a little long-winded and I don't really want to say that over and over in this talk. I also want to talk about this and talk about emulators and these peripheral emulators in an abstract sense right now because well, there's simulators, there's emulators, there's fakes, there's mocks and it's a little confusing which one we're talking about and it's a little overloaded. So let's start with this just so that we are all kind of on the same page about what we're talking about here. Okay, so let's start with a test. This is a test that has a hardware flow and an emulated flow and it goes from left to right. Cool, we have this config emulator here that basically controls if it's emulated or not. So let's step through this. So we have some tests. Let's say we're testing a field gauge driver. Well, that test is gonna call some field gauge driver API or it doesn't have to be a field gauge, it could be sensor, it could be a lot of other things. So it's gonna call some driver API as a test would, as any client would really. Well, that control and data flow is gonna go from that test into, well, the driver that implements that driver API. So that driver API or rather that driver is probably going to want to do some kind of bus reading or writing to its associated peripheral. So in this case, we're talking about I2C. It's gonna want to go ahead and do some I2C reads, just checking, hey, what's the temperature of the battery or what's our charge or capacity at? So, okay, cool. Our driver's gonna go ahead and talk to our field gauge. By the way, I'm starting with a hardware flow first. So emulation is disabled. This is a hardware test flow. I wanna start with this just so we can all be reminded about what a hardware test flow looks like. It's something I think we're all a little more familiar with. Anyway, so that driver is going to, well, try to do an I2C read or some kind of bus read. Well, it's gonna ask the bus controller driver to send these signals. The driver doesn't actually know how to break apart the I2C messages and send them on as little ones and zeros along the wires. That's kind of outside of the peripheral driver's purview. That's more of the bus controller driver, the part that we're not really interested in testing here. So, okay, the driver goes ahead and says, hey, please send this to my peripheral. And it does just that, right? In this case, for an STM32, it just goes ahead and it takes those I2C messages and sends them on the bus and goes over the wires directly to the peripheral. Cool, this is a hardware test. A lot of us are pretty familiar with it right now. So far, so good. Let's go through the emulation flow. Well, we have our driver test and we call some driver API. So far, no difference from the hardware test, actually. Well, that driver test, obviously, that driver API is gonna be implemented by some driver. And again, from the driver's perspective, there's actually no difference. It's gonna have to make those bus reads regardless. So, okay, the test, the driver. Now, here's where things get a little interesting, a little different from the hardware flow. We have emulation enabled. So what that does is that allows us to use a fake bus controller, right? The part that we said that we weren't really interested in testing here, that was kind of out of scope. Well, we're gonna include this fake controller that's gonna take these very abstract I2C messages from the driver and we're gonna forward them directly to the peripheral emulator. And then this peripheral emulator, it takes those I2C messages and based off, you know, is it read, is it write, goes ahead and modifies its internal data or just responds accordingly based off whatever its internal state is. And we're gonna get into pretty deep detail on these last two blue boxes on the right a little bit later. I just wanted to kind of go over an overview to make sure we all had like the right mental model of what we're talking about here. The main takeaway here, though, is that from the test perspective and the driver's perspective, like logically, there's no difference between a hardware test and an emulated test. So there's no difference in that source code. And the other thing here is that these emulators are emulating at a really robust transaction model level of abstraction. So they receive an I2C message of like a stop or a start or a write or a read. And that's the level of abstraction that we're operating at. So we have a loose abstract idea of like what an emulator it is in the context of peripheral emulators and Zephyr. So before we go any further, like is it worth the engineering cost here? Should we even do this? Here I have like pretty rough comparison of emulation and hardware and testing. So let's just go through some of the things that we get from emulation. So first, you know, it's hardwareless. Look mom, no hardware. I love writing that joke. But what's cool here is we can take the emulator and start prototyping the drivers against the emulator before we even have the hardware available. There's some earlier comments on that in earlier talks actually as well. That this allows us to at least get started and not sit there twiddling our thumbs while we're waiting on parts. Another nice thing here is with emulation, we get really scalable testing. We can run this NCI and it's pretty consistent. It's pretty deterministic, right? At least they should be. And it just works. It just runs. It's not slow the way hardware tests can be slow and hardware tests are also often non-deterministic because hardware is frequently full of non-determinism. Another nice thing here is code coverage kind of just works, right? You can just use L-Covered, G-Covered. It's a lot harder to get code coverage on a actual physical device. You might need to have a special test device that has extra memory to include and record that extra information. It's just extra setup. And again, we're looking to be scalable. So I don't know how scalable that is. Another nice thing here is when an emulated test fails, it's a lot easier to debug. You just take out GDB and you just run on your developer machine or maybe some remote machine possibly. It's a little more involved to get debugging set up with an actual hardware board. You have to either simply have a J-Link or connect some probes or maybe all you have is like you are. So it's a little more involved, a little more work. And you don't get the same kind of control that you would in a emulated environment with GDB on your dev machine. Another really important item here is when you're doing emulation, it's a lot easier to go through and kind of fake a certain kind of test scenario or environment. It's a lot harder to do that sometimes in hardware. So as an example, right? Let's say you have a battery and you wanna test whether or not it's a drained battery or it's at some certain percentage. So you kind of have two options in hardware typically is you can use some specialized hardware that kind of fakes the battery readings or you can sort of just wait for the battery to drain. One requires specialized hardware. The other one lowers your life of your battery. Neither is really like I mentioned, scalable. Both emulation, we can just say, hey, pretend to have a low battery or if you have a temperature sensor, hey, be on fire. Be boiling. That's a lot more expensive to do your hardware. But we have one problem, right? We're not shipping the user an emulator. There's all these nice things about emulating. Good to know. Yeah, feel free to reach out to me a bit later and we can talk about that in scaling. So that is an approach. You just have to have people who can put that together and set it up, cool. So like I said though, there is a problem, right? We're not shipping the user an emulator, right? We need real hardware tests. We need real accuracy, right? We are shipping the user a hardware. But what we're gonna see is that we can actually use the hardware tests to inform our emulation and our emulators. And we can get the best of both worlds. We can get the control and velocity of emulation with some of the accuracy and fidelity of real hardware testing. So let's see what that might look like in a project. So here's a project lifecycle. We're gonna step through this both on a pre-hardware side and on a post-hardware side. When we don't have it available and when we do have it available. So first, we're gonna select some part. Maybe it's because it's what our user wants and something cool, but more often than not, it's because of some advantage in supply chain, right? We have something set up to actually get that part easily to us. Well, while we're waiting on the parts that would be put together and while we're waiting to actually receive them, we can get started, right? We have the data sheets. We know what other parts we're gonna use. We can start writing emulators and start prototyping our drivers against those emulators. Or maybe we have some stub driver from a manufacturer, but we can take that and we can actually just start prototyping and running some emulated tests with it to make sure it's doing what we want it to do. Like at least roughly, right? This is still before hardware, this is still emulation. I also want to mention right here, this is the best time to write an emulator. It's when you're writing your driver and you're most familiar with the chip and the data sheet. All right, so we get our hardware and we start actually using our driver. Awesome. Well, explosions. It doesn't work, or maybe it doesn't work as well as we'd want it to, right? We prototyped this driver with emulation and when we run against real hardware, maybe we're seeing some kind of problem. Maybe it's not being registered correctly. Maybe it's, who knows what? It could be a lot of things. Does that mean that, okay, this was a waste of time? Not quite. All right, well first we were able to get started and start talk about the design and development and prototyping of the driver before we got to that part, this part I mean. So at least we've got that covered. But the other thing we can do here is we can take our failing hardware test and we can use that to inform our emulator. We can use that to patch our emulator and our tests to actually improve our emulator and make it more realistic. So our hardware test actually becomes a test for our emulator as well. And so what we do here is we can patch our emulator to make sure it fails the emulated test the same way a hardware test fails. Well, once we have that emulated test failing for the same reason, at the same time we start patching our driver to make both tests pass. What we've done here is we've taken our hardware test failure and we've created a regression test. We've created a more accurate emulator and we'll catch the same problem before it actually makes it out to a hardware test lab, which is gonna be a lot cheaper. Now, something else that's really cool here is as we go through this process over and over because we're going to find more and more things we're actually becoming more aware of the types of mistakes we make in our emulators and we're using every cycle of this graph we're actually improving all of our emulators because we're learning how to emulate our hardware. What's also pretty cool is that this is basically test-driven development. We're taking or we're using the tests to drive our firmware development and our emulator development. So we have our red tests and we're turning them and creating and making them green. Okay, so now we have kind of a rough idea of some of the benefits we can gain from emulation and where it kind of belongs in an entire project really like the whole project. So we've been talking very much in the abstract, how do we actually create one? Let's get into it. Quick reminder, this is the best time to create an emulator. It's when you're creating the driver. It's when you're most familiar with the peripheral and with the data sheet. All right, so we want to create an emulator. Well, how hard is it? It's actually not too complex. It's about 400 lines of C code typically, documentation and whitespace included. We're not emulating an SOC. This is not QMU, all right? We're not cycle accurate to the peripheral. We're really just implementing a bus functional model to handle bus messages. Not going too complex here. Well, okay, so we've written the source code. We actually instantiate emulators pretty similarly to how you would instantiate a device driver in Zephyr. So we have the device DT define with its various parameters. Well, for emulators we have emulator DT define. And they share a lot of the same parameters. You've got the node ID and initialization function, some private runtime data and some static configuration data. But there are two parameters that are specific to emulator DT define that we also have to have. Well, actually you only need one. The other one's just very useful. That's the bus API and the backend API. So the bus API is the implementation of how that emulator responds to the bus messages. It's how do I respond to an I2C read? How do I respond to a write? Or basically, how do I respond? And the backend API, we use this to allow the test to reach into the emulator state and set up test scenarios like low battery. Let's start with getting a little deeper into the bus API. So when we're implementing the bus API, we're actually implementing this last blue box on the right here. That's what we're doing. It's this function here. Oh, that was really loud. And this is what that implementation looks like. It's only about 50 lines of code, really. Again, I'm keeping it pretty abstract, but we just implement a function. This is for I2C that takes the emulator, the I2C messages and some base furniture and we just implement, hey, is it handled? Is it read? Is it a write? Is it something else? Should I explode? What should I do? So that's the bus API. Now, what about the backend API? Well, let's get into that. So we've talked about how you can have a test that calls a driver API, which then gets implemented by a driver, which then asks a bus controller to forward messages to the peripheral, in this case, an emulator. That's great, but a lot of the time, in fact, I'd say most of the time, especially with sensors, our peripherals are telling us about the world. We're not telling them about the world. If we have a temperature sensor, I don't tell the temperature sensor, it's boiling. It tells me it's boiling, though that'd be a lot. So how do we handle this? Well, we provide an API to allow the test to reach into the backend state of the emulator. That's why we call it a backend API. What we can do here is the test just says, oh, emulator, I want you to pretend to have a low battery or pretend you're on fire or you're upside down or something else like that. And then the emulator implements that API and implements, this is what my internal state and registers are gonna look like. And then when the driver next kind of pulls and talks to the emulator, that's going to get percolated to the driver. So this is an example of a test actually using the emulator API. So this is the BC12 test. Here up on top, it's including the BC12 header, emulator for that emulation API. And it's just saying, hey, emulator, pretend you have a BC12 type SDP charger connected. Cool, so it does that. And in the middle here, in which I've kind of hidden out, is that the driver is regularly talking to this chip and it's checking, hey, do you have a charger connected? And if it does, the driver goes ahead and updates some state here with a callback. And that's what this test is verifying. Did that driver actually work? Did it do what we were expecting it to do? What's neat here is nowhere here do you see anything that's specific to a particular kind of BC12 chip. This is at the API level. So we're not writing specialized tests for a particular driver in a particular peripheral. So this is just an example of what that emulator BC12 header would look like. Just an API struct and the header that you end up, or the function prototype that you end up calling from your test. And this is what an implementation would look like in an emulator. Not a lot of lines of code. It just takes the emulator and the partner type, set some internal states, which will then get reflected back to the driver when the driver talks to it. Asking the right questions, that is. All right, so we've talked about the bus API and we've talked about the backend API. Let's see how both of these get combined together. So this graph here shows kind of both at work. It says test there, but think multiple test executions, but the same test source code. What's happening here is a test talks to a driver API and based off the device tree, it's either talking to a MediaTek driver and thus a MediaTek driver or MediaTek emulator for the same chip, because the driver and the emulator are both specific to the peripheral and thus they're both coupled. Well, if my device tree says, oh no, I have a diodes driver, I have a diodes device, well then it just runs the same exact test but includes that driver and then that associated emulator. And then if it needs to test, oh, have this charger connected, both can just share the same API. So you can have multiple test executions with different device tree overlays, but not change your test source code at all. So that's how we create an emulator. Talked about the backend API and the bus API. So all right, we've created our emulator, we've done the work. How do we actually use one in a test? Well, first thing we gotta do is enable emulation, right? But we also need to include two device tree nodes. We include the bus emulator node, which is the part that intercepts driver messages to the peripheral and then goes and finds the right peripheral emulator to forward them to. And we need a device driver node for, well to include the device driver code, right? That's actually no different than a regular hardware test. So when you're adding this node here, the bus emulator node, you're adding this blue box here. That's what you're including. And all the work's been done for you, you just need to include this device tree node. So you don't need to write any C source or anything like that, we're done. You get this for free if you're using a native POSExport, that's why you'll see those tests very frequently used with emulators. But you can just as easily include this in a QMU. And I have some examples later that'll show links to different examples that actually use a QMU with a peripheral emulator like this. There's some stubbed values here. They're not actually used right now, but they're just there so this effort thinks this is an I2C controller. By the way, we have also SPI and ESPI as well. I just chose I2C because I think we're all pretty familiar with it. And then we need the device driver node. Just the same as a hardware test, so we just add that in. And when we add that in, we get the device driver, but we also get the emulator. Because the emulator is coupled to the device driver, when we include one, as long as the emulation is enabled, we get the emulator as well. So here's some examples I've got that I was alluding to earlier. I don't know if the PDF slides actually have working links now that I think about it. They were links, but I don't know if that survived the whole PDF process. But those are some search terms to get you started. Feel free to ping me too on Discord if you have any questions. I'm Aaron Massey. I think I'm Aaron Massey, underscore Google actually. Well, before I get into Q and A, I wanna talk also a little bit about some improvements that I'd like to have in the emulation subsystem with Zephyr. So this whole function that we need to implement to sort of parse the bus messages and respond to them, there's a bit of boilerplate there. It's not a lot, but it can still cause duplication of mistakes. So I created a GitHub issue on this. I would love some comments and some help. I think the initial idea is sort of emulators that are implemented device that uses SMBus. Maybe they can all use some shared abstract logic for parsing that. But just an idea, please comment. I could use the help, we could all use the help. Thank you. Another thing here is we have a lot of drivers, but we don't have a lot of emulators. Now I picked three drivers here. I'm not picking on any of you or anyone's driver, by the way. I just picked these pretty arbitrarily. I just wanted them to have different APIs and be sort of updated in the past year or so. It'd be great if we had emulators for them. And especially if someone who was like more of an expert on their associated peripherals would be really great if you could contribute an emulator for them so we can get CI and regression testing for these drivers. I think eventually, and Keith actually alluded this to this earlier in the keynote, it'd be great if we started asking contributors of new drivers to just contribute an emulator and emulate tests with that new driver. They're already pretty familiar with the peripheral and with the driver, so it's sort of the best time for them to create that emulator. It's when they know most about the peripheral. And then we get CI and code coverage just from the get go. All right, so one final thing, I'm willing to talk about how using emulators has been a success at Google. So as Keith mentioned earlier in his keynote, we started at 30% code coverage and then we went to 90% code coverage. We started with a pretty large pre-existing code base and we took drivers that already existed and we implemented emulators for based off our data sheets and the existing driver code. And then what we did, what we got from this, when we moved from 30% to 90% code coverage is we caught various bugs in drivers. We also detected dead code that was not link time optimized out, meaning through emulated testing, we were able to make our binary smaller. We also blocked regressions from changes. We blocked them before they made out the hardware testing. And we even went through the process of prototyping new drivers when we didn't have the hardware available. So here's that big, pretty graphic that I spent some time on. Does anyone have any questions? Yes. Ah, Marty. Oh, hey Marty. Nice to meet you. Nice to meet you. First of all, oh, yeah. First of all, thank you for this wonderful talk. I would have really appreciated knowing all this information when I was reviewing some of this stuff for the first time. So thank you. I really appreciate your work. I guess I have a question, and this may be something that's just sort of out of scope of this level of testing, but it seems like at the level of the fake bus driver, you're sort of intercepting at the level of like the struct ITC message. Yes. And so that seems to make it impossible to include test cases for handling situations where there's corruption on the bus that are not representable as a valid struct ITC message. Do you have any notion of whether that is something that could be tested via the backend API, or do you have any thoughts on how to address situations like that? And you know, I didn't get to writing this RFC in time, but we actually have started talking about this about, hey, let's send some corrupted messages, let's intercept the individual messages, and then send them over to the emulator. We've actually thought about maybe we can have some kind of stack mechanism where you can have multiple sort of interceptions of this bus messaging, and you can kind of change it on each side during the test. That's a fantastic question, and that's something that we could use to improve it. So all right, I'll get cracking on that RFC, or maybe I could use some help. Maybe you can help me out with that. Thanks. Awesome. Yeah, once again, thanks for the brilliant talk. I've also got one question. Any ideas, plans already on when peripherals would also have interrupts like an IMU that would trigger on the fall detection stuff like that? Well, you can include a lot of that in the emulator code itself. So in our emulators in Google, we actually have that implemented and then quite a few of my emulators were like, oh, this value changed. Go ahead and fire this interrupt. Go ahead and set this interrupt as high. So you can totally do that. It's the emulators. They get compiled as part of the Zephyr binary, and so you can rely on device tree or what have you. Did I answer your question? Okay, okay, absolutely. Hi, my name is Hugo from BlueNetics. My question is, I saw some statements, Zephyr statements, so is this part of a Google fork or is this already? This is all in Zephyr. Oh, sorry, I should have let you finish. Is this publicly available in the Zephyr tree done? It's just we need to activate or use it. Absolutely, it's in the Zephyr tree now. Actually, all of Google's emulators are also open source. They're just in our Chromium OS tree as well, but it's obviously easier to use if it's in Zephyr. So, but absolutely, yeah. To make this easier for, as you mentioned, requests as we have new drivers and so forth, have you thought about sort of a way of from the tests, having a trace that, so we, I could see a version of this where you just have it like be a data, like a description file of register read address. There's some way of describing this for that peripheral as opposed to having to write all this code necessarily to kind of have a state machine and sort of emulate the whole device or do whatever that may be versus just saying, hey, okay, I've got my test. I ran my hardware. Okay, I got this version where I enable this K config in Zephyr that dumps out a trace log. I can take that trace log, put it in a file, and that's my emulator. Okay, so you're talking about, can we just take the logs of register reads and writes and then just say, okay, as long as we have these reads and writes, can we just say this is our emulated test? Yeah, exactly, right? I mean, you may want to put some additional things like you have, like for being able to signal right from the test if it's kind of saying, hey, I want to emulate or act as if, that I'm the low battery condition or some of that stuff, but some way where we could take a trace with some other information, and then that would just generate my emulated driver for me for a lot of cases, because then that makes it really easy for someone to do this as a low hanging fruit. There's still the option if someone wants to write a full driver, emulated driver, they can do that, but the quandary I have with us having to do this and expect this from people is, if I'm a silicon vendor, I'm doing this, I've probably got a bunch of other things I'm already doing for simulating my system. Now you're asking me to do yet another different one and trying to explain the cost of that to somebody that's, hey, do this Zephyr specific thing and then, hey, you're doing a system C model and I'm doing this model, I'm doing that. It's like, how many of these do I have to do because everyone's got some different way they want it to look? It's just, I mean, I've got to convince management to say, hey, do this 16 different ways, right? So I think we have to figure out how to make this cheaper and easier if we want vendors to actually provide that as part of driver support. I understood. So when it comes to the engineering cost of actually creating the emulator, we don't have to make a fully fleshed out emulator at first. We can kind of more iteratively do it. Like I was adjusting the trace, right? Right, so what I was going to add into, so you mentioned the trace. That's an interesting idea, basically recording what the trace is and then using that to generate an emulator, right? Is that what you were saying? Yeah. Okay, okay. He said yes in case anyone, man. That's an interesting idea, especially for prototyping or just getting started quicker. I think it's the very least worth exploring into an RFC, what have you. At Google, we didn't end up doing that. We found it easy enough to just create an emulator and sort of patch it as needed, but I don't see any reason why you can't just take a trace and then try to have a tool that then generates some kind of emulator. Obviously, when you have generated code, make sure it's at least somewhat readable because you may wanna have special implementation in your emulator that is, I guess, maintainable. And so generate code and maintainable doesn't always work together, but that's interesting. Let's talk more offline, like that's very interesting. Hi, thanks for the interesting talk. How do you run the tests in the CI? You run them, like compile them for POSIX or run them in QEMO? You could do either. In fact, in those examples, you can test in native POSIX or in QEMO, just use Twister or just works. Thanks. Great talk, thank you. One thing I suppose is worth pointing out here is when you have, you're just showing one driver, but taking an upper level, say you're trying to, you have a system which has a battery, it has a charger, it has temperature sensors, it has other things as well. You can just wire all that up and run logic on top of that that uses these things as emulators. So you can do higher level testing of your code basically without doing any additional work. Right, so Simon makes a great point. I didn't really talk about it too much in this presentation, but you can make the emulators for the peripherals and you can use the drivers of those emulators, but you can combine and use multiple drivers and emulators and create just a layer of testing above it. So if you wanna test, let's say your USB power delivery policy, right? We actually do that in Chrome OS in our open source code and there's like several layers. So, you know, good question, thanks. Hey, I have actually two questions. The first one is currently this relies on using the transport API in Zephyr for the different drivers. Is there anything that you've done to explore memory mapped IO? So like using some extra abstraction layer through Zephyr to do register reads and writes through memory to be able to interject that? I haven't looked into that myself. I don't know, that sounds interesting. I'm not sure. I haven't really looked into doing that. I don't think it's been necessary so far, but can we talk about this more offline? That's very interesting. What was your second question? Sorry. No worries, we all answered the first one, so that was perfect. Oh, I'm sorry. And the second one is can you actually use the subsystem also to do a high level application development before you have actually your hardware? So not just for testing, but also just running your application code already in your target devices. Is that something you can use for that? Absolutely. You absolutely can. You can just start up a native politics board and run it. In fact, we've also done that in Chromium OS. And yeah, so you can totally do that. You can just start up a board and you can have emulators as part of the binaries. And we'll just talk to the emulators instead of real hardware. Amazing. Thanks for the introduction to the emulation. Actually, a very simple question. Which bus types are currently supported? We have I2C and we have spy and e-spy. We can add more. So here's an example of the e-spy node that you would add and the spy node that you would add. These are all native politics for free and you can also just include them in a QMU. And we've got a question from the virtual audience. Would this also work on fully embedded devices such as the Nordic RP2040 or Raspberry Pi? I don't see any reason why it wouldn't. This is just some code that gets compiled into the binary and you can just run it. Maybe I misunderstood the question. Yeah, I'm guessing maybe they're asking could you do emulation of like an entire board? Oh, well you can. So this is specific about emulating the various peripherals attached via some kind of bus. But because you can run it under QMU, you can kind of do both and combine it with other tools. So, so you guys have two minutes left. Yeah, so let me get here. Hi, thanks. You mentioned about three different specific kind of peripherals that you were looking for implementations of. But it might be worth talking about other generic versions of like so you had an ADC, you had a GPIO, and you had a sensor. But are there not generic ADC emulators and generic GPIO and maybe even generic sensor emulators that you could just say give this sensor value with this type or give this ADC value? So there is an ADC kind of, I guess, vague. It doesn't actually use the emulation subsystem where it's doing this whole bus forwarding thing. There is something out there is the ADC, I think underscore emul dot C. And that might be, you might want to take a look at that. So that's a little different because it's not actually doing this whole, you know, parsing of bus messages, right? So that's a little, the parsing of bus messages is a little bit more involved than that. As for in terms of more generic emulators, I mean, that's part of kind of what I'm asking for in up here is maybe we can get some kind of at least generic emulator code to kind of improve the emulator development and we can get more emulators quicker. You know, I've been working on the fuel gauge API and just having a smart battery emulators, pretty nice. But I've been considering changing that into sort of a library so that folks can use the emulation for very specific chips that use, that implement the smart battery spec, but maybe they have some quirks or something like that. But we want to reduce boilerplate in that emulation. You know, that was just an aside. Good question, thank you. All right, yeah, we're out of time here, but thank you very much, Aaron. Thank you, folks.