 Hello, everybody. Yeah, my name is Tom Burdick. I work at Intel. This is my talk on kernel timers, jitter drift and expectations, some hard lessons were learned on my part and others to sort of build up this knowledge and understanding or sort of intuition about kernel timers. I really wanted to kind of share that with with many of you and hope in hopes that, you know, I might save some time or help help build some some intuition about curl timers for those watching. So, I guess, let's let's talk about what the idealistic kernel timer might look like in Zephyr, right. Well, we create our timer with either a macro or a timer in it. And then we go in and we want to set up a timer to to occur with some sort of delay or period. Maybe we give it a, you know, a delay of one millisecond and a period of one millisecond and then the expectation then is, you know, we got our function callback at exactly one millisecond after we call starts. There after every millisecond passes, we get a callback in our code runs. And really the, in the ideal scenario, the only thing that would ever cause any sort of problems with, with, you know, the exact one millisecond period or delay would be in the harbor right the clock source is actually, you know, oftentimes a crystal or an RSC oscillator those things have, you know, small, small variants and timing. That would be the ideal scenario right exactly one millisecond every time no problems. No delay is no drift no jitter. That's not reality. We live in an imperfect world. So what is a realistic kernel timer look like well, the delay is really a minimum. So you can say you know hey I want to call my function in a millisecond. And getting some, some feedback that I don't have the mics really hot. But yeah, so. But yeah, the, you know, the delay is really only given as a minimum. So, you know, you say I'm going to call my function in a millisecond. Well, it's really, you know, your function will be called guaranteed after one millisecond, one millisecond or further into the future. And that's true period as well. And really the the accuracy of, you know, how accurate can you can you set your delay or period is really the accuracy of one kernel tick. It's, you know, so we, you know, in the kernel timer API right you can arbitrarily pass in a timeout value could say oh I want to timer in one microsecond. But the reality is, it's, you know, it's going to be rounded up to the nearest tick. And, you know, the other thing about a realistic kernel timer is there's very likely some jitter involved in your timer so you know trying to rely on a kernel timer for, for, you know, exact periodicity, periodic behavior. You know, there's some things to look out for it's not that you can't make it work well in that in that scenario but there are a lot of sort of gotchas and you know there's some things to kind of keep in mind when doing that. And, and last but not least there there's there's there's there's actually a strong possibility that there's some drift, depending on your hardware. So, you know, if you have, you know, some some agnostic code that you're trying to run across lots of different hardware, and you have for example, a timer that's supposed to run every millisecond and you're you're relying on the fact that well okay like on you know board acts that's perfectly fine like it's it's about, you know, almost exactly one millisecond every time. You're on a different board and it could be a little bit surprising that oh no that's it's not. It's not a millisecond it's it's a little more than a millisecond, and that can cause some drift and some issues so again something else to keep in mind and look out for. So I guess yeah like what is what is clock jitter I mean I think probably many of you already know but some of you don't send. And what are the sources of jitter so jitter is sort of a loss of precision so if I, you know, set my timer for one millisecond, and it's sort of like, you know, here's one millisecond, and the reality is it's like it's kind of firing you know in this this window of time after and the wider that window is the less precise my timer is. So that's clock jitter if you kind of imagine like oh the timer was scheduled here but happened here here here here over and over and over again. That's a loss of precision. And that can cause that for kernel timers. Well there's many potential sources of your and in Zephyr with kernel timers interrupt masking mantras jitter which includes if you go to use like an IQ lock or a spin lock. If your priority interrupts than the timer interrupts in your hardware. That can also prevent the timer interrupt from firing and then there's some jitter there's some delay and getting back to you to the timer system and getting your call back and all that. From flash can also, you know, another source of jitter right hey like my timer interrupt fires I got to go fetch code from flash to then load into SRAM. Okay, well that might take a minute will not a minute reality but you know, a little bit more time than the maybe expected. So a little bit more time from when the the timer actually fired to when your code runs. And then there's just, you know, so see it's right where maybe you go to try to touch some hardware some memory. Right there's some some bus arbitration that needs to happen there's some delays variants and memory access time you're trying to you know touch, you know, tightly coupled memory versus SD RAM off often off on the board somewhere that might might take some time. And of course you know the hardware clock itself can be a source of jitter, but probably not a realistic one. In most cases, most people are probably going to be using crystals. What's using like a, you know, 555 or RC oscillator or something then all bets are off right. And so what you know what do I mean by clock drift clock drift is really a loss of accuracy. So, you know, if we kind of go, you know, I think a lot of people think about that. I'm sorry my daughter came up to say hi. Yeah, so if you think about what what you know, precision and accuracy might look like. You know, we kind of think about the dartboard maybe an accuracy is, you know, like the center of the bullseye would be perfectly accurate and off to the right we're hitting all our darts in the right spots, or we're hitting all our darts and a nice little tight circle but it's not in the right way and that's drift. And, and that can be caused much, you know, the only thing that I know that that would cause drift in in Zephyr kernel timers is really the grounding so the minimum and also yeah like you the minimum unit of time that you can really schedule things is one tick. So if you if you, you know, if you try to schedule a timer for one microseconds. That's, that's, you know, maybe that works on some hardware with with, you know, like a mega her clock or more, but then you go to run it on some different hardware, maybe the clock is a lot of no three 2.7 state like is on a Nordic board for the most part or maybe on an XP board with the low power clock. You know, you're not going to get a one microsecond scheduled timer, it's going to be all sorts of, you know, off so. So then, right, the conversion from from six to clock cycle can cause. And, yeah, so what are the expectations of current timers then, you know, given all of this right. They should be schedule every tick. And this is really important you should be able to schedule a timer. Every tick of the, you know, every every single tick and kernel ticks. And why right well, I mean, the idea, you know, a curl tick is really a unit of processing time so you can think of you know I want to do something every tick that's that's that should be the expectation. You should expect, you know, some drift if you're using time units rather than ticks, and especially you're trying to look for you know code to work across multiple boards with with K timers with with time units of the behavior may vary. And this is something so it's not it's not always hardware agnostic to simply say, I have a K timer with a one millisecond time out. It may vary behavior over over a different heart. Some small manager, regardless of what you're doing. It's, it's, it's almost it should be expected, unless you're very careful. And, and there's a nice test that sort of like shows all this behavior and, and sort of give some nice statistics that I created that I'm going to demo here at the end. And you can kind of see what your hardware does with with this. What one moment I apologize. Yes, working from home has it's a benefit. Anyway, um, Now, let's do the demo here. Yeah, so what is this kind of look like here. I need to do sure. Okay, so I have, I have Nordic board right next to me that I'm that I'm running live with the test, so you can see the actual output of this time or just time or dirt test. Yeah, so it's going to, you know, it kind of like runs through it collects samples and it does some statistical analysis and we can see on this Nordic board for example. And, and, you know, the so we've set a timer for one millisecond. This is actually, you know, a not an integer number of cycles that that ends up being because it's a clock of 32.768. And, you know, we ended up with like a minimum, a time a timer of about 1.1 milliseconds, and, you know, a maximum of 1.2 so there's some there was some bitter. And, and there's definitely some drift right if we if we set like a timer for one millisecond over time will be drift and in fact like it's expected, you know, in this test case that we're going to see that trip, and let's kind of know that we can also see this, you know, on the hardware as well. You know, I changed my test a little bit so that we can get a API open toggling every time the timer fires. And here we can see, you know, this is the last thing it's a demo but. Yeah, I mean we're having statistics the same kind of statistics that that are on from the test here. We need we need to do a little bit of math because this scope I couldn't I couldn't set it up to get the, you know, the period and frequency based on both edges but you know, again if we divide this by to the time period here, it's about 1.01 milliseconds. Again, we're going to see, you know, from some grips. And if we look at our standard deviation here. Sure we can divide by two but I believe we can hear. Not quite, but there is going to be some standard deviation right there's going to be some variants from here in our clock as well. So yeah, I hope I hope talk was was really helpful and sort of maybe you know building up some intuition about kernel timers how what to kind of expect out of them. They might work on different hardware, you know, they're going to work differently on different hardware. And yeah, that's it. Any questions. Great. So just looking at the text output from your tests it the non rounded value is 32.768 cycles per interrupt or whatever. But the minimum was 36, like that's a substantial difference. It is, it's, it's, yeah, so in this case for sure that again. This is a, you know, this is a Nordic board and if we look at so this is kind of tied into the kernel create. So if we look at the kernel create here. The kernel tick rate is going to be, you know, it's actually higher than the clock rate clock rate is 32.768 kilohertz but the kernel tick rate is actually 8192 Hertz. And that's, I believe that's because on Nordic boards, the timer can only be set like once every four for timer cycles I could be wrong I'm not a Nordic engineer, but I recall seeing that in a PR. So this is sort of tied in right exactly how, you know, this is all set up and again kind of going back to the point right where you can only set a timer as as accurate as it is. You know, right I think it's going to be one over 8,182, but it's going to round up the next. Right. No, yeah, I mean, but, but yeah, exactly. It's kind of. Yeah. So these things can be, again, I think kind of, I mean, it's like it's all sort of there in the back here are minds I think right but the, but it can still catch us off guard. It catches me off guard sometimes. And especially when when going from different hardware, where, you know, again, like, I want to set a timer for one millisecond that works great on my hardware and then I go to run it on some other hardware. And, oh, it's not one millisecond that's not, you know, that could that could cause some chaos. Things can go wrong. Another question as well. Sorry. Sorry. I think, oh, there we go. I just can't tell directions. Hold on. Okay. My question is like a that actually like two ways that you can have like a, for example, you can have your K timers. Other one is you can have some of your work you write them and then you can and you can do some scheduling. Right. So, as you think, so which method would have the lowest clock jitter. So, the other part, I mean, I heard about the K time, was it schedule will work is that is it was that the, for example, like a, in addition to using the K timer, you can have a working item and then you can have this thing. Like a, you can do some scheduling after some delay, and then inside the work you write them, I mean, after this, the work you write them get executed, you can actually do actually reschedule again. Right. So then in this way. Yeah. So, so this is kind of like another interesting part about curl timers. So I mean, I kind of, I kind of directly talked about, you know, K terminal, a timer start and, and having callbacks and that happens directly in the, the interrupts the timer interrupt. So this is about as precise as you can get in terms of like software, you know, driven timers in Zephyr without without using. There's another subsystem that lets you directly tie into like a different timer compare. And that's cycle, you can get like cycle accurate timers with that. But basically every other timer in Zephyr is built on K time out including K timers. So for example, like your schedule will work you item that's built that's built on. That's basically built on K timers, like semaphores with delays, right where you where you can say I'm going to wait for my semaphore for like a millisecond that's built on a K timer for for the, you know, the wait time. So, so all of those things where all those kernel API is where you can, you can pass in, you know, time, or you can also pass in tick values typically, and the accuracy precision jitter drift, all those rules still apply. And so again, that can, that can kind of, you know, if you need something more accurate, right there, there, there is like a hardware driver API that you can tie into and get cycle accurate timers. Yeah. Okay, we have another one. Yeah. If you want to have a real precision where you want to have especially zero drift, would it be an option to accumulate the elapsed time and then compare to your ideal time value. And then basically, slightly tweak the the next delay. Is that supported? That's absolutely so it's not directly supported by any kernel API, but you can do that. So like the here, you know, on my screen, right, you see cycles and all this stuff and you can actually get there. There's, you know, a cycle get API is where you can get the cycle accounts of the timer and and base your, you know, base your next time out on that so rather than setting like a period so exactly like what you're talking about right but there's no, there's no automatic periodicity like try, you know, like, there's no API to do what you're talking about where it tries to, you know, if you if you go too far it kind of shifts it back. There's no sort of like API to deal with that in the current I think it'd be a cool API as sort of like a build, you know, a library that you can build on top of kernel timers but the way it is now. It's not done that way I think predominantly because that adds more logic right there's more code that needs to happen to make it work that way. But yeah that would be a really cool addition as like a, you know, a little layer on top of these, but no it does not exist, but you can build it yourself in your application. Okay and we do have a virtual comment that someone wanted to add which I think relates to what you were previously talking about but they say just wanted to add a comment that setting your scope to infinite persistence is also a nice way to eyeball jitter on clock. Absolutely. Yeah, in fact, yeah right exactly. So if you do that, you'll start seeing like the, you know, kind of the fuzziness of where the edges are sort of spreading out. It'll kind of look like, I mean, ideally it'll look like a little bit of a Gaussian right like it'll be really bright in the middle and then fuzzy as as sort of the window of jitter is there. And you can see that. Definitely. Yeah, it's a cool way of doing it. We're doing a last check for any other questions. I think we are we are we have gotten them all in Tom if someone wanted to reach out to you are you on discord are you in any kind of community they can get into what are the best ways to reach you. I'm on I'm on discord. My, my name is Tom B. So yeah, feel free to reach out to me. Just ping me. I'm pretty active and mostly discord channels so you should be able to find me. Wonderful. Thank you. And I'm going to do one more final check. I think we are all set then Tom. Thank you so much for your time today and for joining us virtually and from wherever you are and we understand that life happens. So thank you so much and we'll we'll see you soon. Thank you. Thank you.