 Hello, hello. Can I get going? Hi, folks. If you're watching this video after I've streamed it live, make sure and check the notes doc or the description for time codes for different things we talk about. Yeah, got lots to do, lots to talk about, and we'll answer questions. So check the notes doc for that and we'll get started on housekeeping here in a couple minutes. But we'll say hello and answer any questions folks have as well. Let me rearrange my windows. Happy Friday or Saturday if you're Australian or on that side of the world. Hey, Bruce. Hi, Anthony. Hi, unexpected maker. Hey, David. Welcome back. And now I want an epic sleep. Hi, Beata. Hey, Daniel. Hi, Mark. And Emma. Hello, Emma as well. Hey, PT. Hi, G3 Holiday. Hi, Yanni. Hey, Randall. Hey, Dave. Twitch is up awesome. And hello, Davidessa as well. And David Buchanan. She waved. Perfect. Awesome. Hi, Ham's Lab. Hope you're well. Hey, Charles. How is everything on the West Coast? Pretty good. We had a bit of a smoke earlier this week, but it's okay. Now we got rain last night, which is awesome. Hey, Doctor. I'll do my best to distract you. My green ball. Yes, it does. That shadow. Get your mind out of the gutter. Awesome. We'll just hang out a little bit. I do have some work I need to do. I have a PR that I'm almost done with that I need to finish. Hey, Mark Gambler. Or Gambler, I guess. Um, this is classified as work. It's true. But I have, so we're making really good progress on getting to release candidate, actually. We just have been cruising through issues. And so I think I have the last, I'm working on the last one we have to do before we were to do a release candidate. Unless folks find something over the weekend. Don says, this is my favorite part of the show. I wonder if Scott is old enough to remember romper room. I don't remember romper room. I'm 35, so maybe not. Hey, Jeff. No rain in LA, of course. Yeah, we haven't had too much rain here. It was actually kind of weird. Mark says, I've finally ordered my first RP2040 boards assembled by JLC. Looking forward to playing with them. It's a test jig controller thing. I'm a youngling. I'm glad you think I'm young. I'll take it. Definitely got some folks in the chat that know romper room. My Saturday morning shows. Doctor says JP made me spend all my money. He'll do that. That's why we keep him around. Very rainy in Sweden, says Mark. Has the ESP32S2 hot reload issue been resolved? I've not had a chance to look at it, but RC means maybe. Do we have an issue for it? We've, we have fixed a couple S2 issues recently. And I wanted to talk about one of them that was a doozy that Jeff and I figured out earlier this week. Jeff says, yeah, some places too much and many too little rain. Germany was horrible. Oh yeah, I saw those videos. We'll get rain. It is Seattle after all, but we're pretty dry during the, well our summers are pretty dry. Hi Patrick. Oh look, it's 203. I should get going. Oh, I have an addition. I have peanuts in case I get hungry too. Hey Gary Z. Ham's life says you are young. I'm old enough to have a colonoscopy this morning. My hunger after was sated by some awesome Thai food. Awesome. Thai food sounds good. And Jay for seeing says, hello, I'm here for that ESP bug breakdown. Yeah, Jay for seeing was helping us with it. Don't worry, peanut dust cannot go through the interwebs. So doctor, you're fine. I mean, I was doing ESP. Well, we were doing ESP stuff. I think it's thought it's in pretty good shape. Never too old to learn embedded. You're exactly right, Jeff. Never too old. I only learned it, I guess five years ago now. Bruce is making frozen pizza. Okay, here we go. Hello, everyone. This is Deep Dive with Scott for what is it? August 27th, 2021. We do this every Friday, nearly at 2 p.m. Pacific here on the Adafruit Discord server. I work for Adafruit on CircuitPython. CircuitPython is a beginner friendly version of Python designed for microcontrollers, which are these little tiny computers that are very inexpensive and you can just make it do something you want and then set it aside and let it do that forever if you write, if you want. So Adafruit is an open source hardware and software company based out of New York City. I work remotely for them. So I'm here in Seattle, Washington, which is on the left side of the US if you're looking on a map. If you're from outside the US, if you want to support me, you can support me by supporting Adafruit who pays me. They pay me to do this work, including the streams. So you can go to adafruit.com and purchase stuff there and that helps pay for them to pay me. If you want to chat with me and a lot of others, we do have an Adafruit Discord server, which is the middle box here. You can join that by going to the URL adafru.it slash Discord. So there's the Discord chat is like all all year round or all all the time, not just when the stream is live. So that's really nice. If you're enjoying the folks that are in like the YouTube chat, but you want to chat with them later, that's great. Yeah, next week will be on Friday at 2pm. We do shift it to Thursdays occasionally. But yeah, that's about the housekeeping. The last bit is that the cat cam that doesn't currently currently have a cat. He's in here, but he's being a scaredy cat. So Spook is in here and he is, I don't know, maybe I can take the cat cam for a ride here. We'll probably have to move it when he moves, but it's cleaner day today. So he's he's hiding out like a bum. So there we go. He has no choice. And it's kind of chilly. So he's all curled up. I have the window open. But yeah, so this is Spook. He is epileptic, which means occasionally he has seizures. But since we started the stream, we put him on different meds and it's been awesome. He's had way fewer seizures and he's been feeling better, obviously, and all of that. So it's been really good. I just want to warn people briefly that like there's a slight, slight, slight possibility that that will happen. But he's been doing really good. So I don't really expect it. And let me know if he moves out of the cat cam and I'll repoint it. But yeah, kitty don't care. All right, so got lots of chats here. So let me catch up on the chats before we go further. So yeah, unexpected maker for the hot reload issue. Try it out. And if there's not an issue for it, make sure there is one because we're going by the book for that. Mark says, what's after a circuit pipe on seven? Will you look at basing it on Zephyr? The number of new boards is just insane. What about the new build system? That's a good question. For me, the work that I'm planning on trying to do is after this BLE workflow stuff is done, or I spend a little more time on it. I'm actually, I really want to do the Raspberry Pi support. So that's kind of next on it for me. But we don't have any concrete plans for when we're going to go to eight. And we don't have any concrete plans to base it on Zephyr either, or do the new build system as much as I would like to. The library for the ST7565 isn't in the library collection. How do we get it in there? How do we get it in the bundle? To get it in the bundle, you have to add it to the bundle repo. I think that's what you're talking about. Let me switch away from cat cam. Make that window a little shorter. Yeah, so if you need something in the bundle, and it's an Adafruit library, you can do PRs to here and then it gets auto-released. My friend from your state is coming to visit us in October. That's awesome. 5132 and it's been closed. Okay, great. Yeah, I'm excited for Raspberry Pi and CircuitPython. I just ordered a little Wave, not WaveShare, a little HDMI screen to use with it. There's a new Adafruit board, the expressive pad. There's no support for it yet. Is that the thing that I have behind me? Are you talking about this? Is this what you're talking about, Jeff? The expressive S2 thing that they made? There's an open pull request for it. There's an open pull request for it. We're just waiting for a USB PID from Espresso for it, and then we should get it in. Yeah, if the library for the display you're talking about is not Adafruit supported, then it will go in a different bundle. But we have a CircuitPython.org bundle, which is a number of people, first stuff supported by a number of people in the community, and then we have a community bundle as well. Hi, Keith EE. G3Holiday asks, when will Whippersnapper go public? That's not really a question for me. I know Brent's the person that's been working on that a lot. I think soon, Hamzab says it's got a way to go before it's ready for release. Yeah, Brent's a better person to ask, or Lady Aida. I thought they were going to try to be not beta soon. There's two main people, Lauren and Brent, who were working on it. I think Lauren was on vacation or something, so they were going to wait. Maybe things popped up then. If you want to participate, you can always ask Brent and he'll let you in. Just because it goes out of beta doesn't mean they won't keep working on it. Oh, before I get started, I wanted to plug the RealPython podcast, which I think David has linked, so thank you, David. This is a podcast I was on. We recorded it just a few weeks ago, so it should be pretty current. Public beta, no invite needed. I think soon, next couple of weeks is my understanding, but it's not up to me. I just did this podcast. It's nice and relevant and nice and new, so check that out if you want to hear more of my background and higher level philosophy stuff, although I'm always happy to talk about those as well. I got one of the few STEM at QT boards that does not come with a CirclePython library and discovered it too late, which I do. Make one, please, or check with Katnney first to make sure that she, Katnney would know if there's a library in the works as well, so Katnney's a great person to check with first. Pierre says, great podcast, listen to it this morning. Ah, thank you. Yeah, and dropped it in the chat too. Thank you, thank you, thank you. Yeah, thanks to, I think, yeah, thanks to Chris for having me on. It was a really good discussion. I really enjoyed it, so I always like doing podcasts, so thank you to folks who have recommended podcasts for me to be on. Okay, so I think I'm not super set up for it, but we have a bunch of ESP folks in here and we, yeah, there's this bug that was a doozy. So the way that we know that we're close to, the way that we know we're close to doing a release candidate is this is the list of open issues for milestone 7.0 and we only have five open, but I think that this one, these bottom two are actually kind of like, we think they're solved already or we want to retest them. So we really, what we're waiting for is we're waiting for this one to get fixed, which has a PR already for it. And then I was just working on this hard faults on count IOD in it, and then this one should be pretty straightforward too. So, you know, next week hopefully we'll do a release candidate for 7.0, which would be great. And the reason we did a bunch of alphas, and then we just did like, we have one beta out right now, and we probably just go straight to release candidate. And we chose just to be in alpha for a while. Yeah, David, it's your bug. The reason we were in alpha for a while is because that we wanted to stay in alpha as long as we were maybe going to change the API still. Hey, Paul. So that's why it was like a long alpha, and it's going to be a short beta because we're hammering all these things down. And Jeff and Dan are on vacation for the next two weeks. So it's going to be me doing a release candidate, hopefully, and then maybe when they're back we'll do a stable release. So okay, so let's look at the closed issues. And we've been cruising through these. Alpha, beta, what's the difference? Yeah, so for us, alpha means that the API still could change, whereas beta is like, we might add things, but the things that are in there won't change or probably won't change. How long was seven in alpha versus how long revision is typically in alpha? We had alpha six. So we had six releases of alpha, maybe five. We had alpha zero, which is not actually release. And then I think alpha one was not actually released as well. It's probably a better metric to know how long it was, but I don't actually know. But yeah, there was just like a lot of, so that's really what drives a major release is that we want to change some of the APIs. We were there first. We call the alpha, beta first. That's fine. We won't get to delta. So yeah, we'll do the next step after beta is release candidate. And the way that release candidate works is we think this is stable, but we want to have some people test it before we call it stable because we do actually have a pretty big, a lot of people actually just use the stable release, which is good. But it means that we do want to make sure that we've tested it. So the way this, we'll do a release candidate and we let it soak meaning we let people try it and use it for like a week. And if nothing major comes up, then we'd say, okay, this is stable. And we just mark the same version as stable. We just like rebuild that version at seven or as a stable release instead of as an unstable one. So yeah, there's a, yeah, so that's kind of the process. So seven O is pretty close. And a lot of the work for the stable release actually kind of ends at the release candidate phase, especially if no bugs come up after that, which I'm pretty confident in. As Randall says, I've been using seven, it's pretty good. Yeah. So like seven's been pretty solid. And it's going to just, especially like alpha six, beta zero, and then the next release, like, we went from like 20 plus issues, then we'll be down to zero. So that'll be really good. Okay, so the, the issue that we're looking for, you can see all these issues were just like, are pretty recent. But the, I forget what I titled it, what did I title it, like mega issue or something was, so I fixed the teensy four one. And that was another thing I did crash on auto reload. That sounds like what unexpected maker was referring to. I forget which one it was. It was just fixed today. Ah, this one, the mag tag vaccination tracker from David, who's in the chat. This is a weird bug. Mag tag vaccines. Yes. Thank you. Thank you, Jay for seeing. I haven't had time to trust test the crash on auto reload. See if it's fully fixed. All right, well, folks here should try it. If you find anything like a hard crash, let us know. Should I show this issue? Yeah, I think I have it set up still. But this is, this is what David said is that this demo code from this learn guide, it used to work in alpha three, and it stopped working in alpha four. And it has a crash loop. So what it does is it says blah, blah, blah, it gets it. And then it resets the serial and it starts over. So we tried to fix it and didn't. And we found that it stopped and connect. And then Dan's asking for a more minimal example. Jeff has this more minimal example here. And then Jeff says, Oh, this is the watchdog. So it says here, like the pro CPU has been reset by the watchdog. Yeah, I did not try. Yeah, it was. Yeah, so this bug, Jeff took two days to look at it this week. And then I took like a day and a half, maybe two, to look at it and also just like hit a wall. We both hit walls on it. So what we did yesterday is Jeff and I got together yesterday and just got on a video chat and looked at it together. And we used J for scenes code. So yes, see, here's where Jeff said like, I didn't get any traction on this. I'm going to like stop working on it. And then J for scene was working on it and gave yeah, and then Jeff picked it up again, because I had a different fix that thought we thought it might might work. But then we had this minimum code here that that still caused the crash. So I think what I'll do is not to give it away. It's a we figured out why it's a bug and we we know that it's going to be rare given what it is. It has a lot of preconditions, but it's and it's a bug that's it is present in Micro Python as well. And we have a fix for Micro Python too. But then again, it still takes like a lot of details to hit and it's it's more likely to hit with Micro Python than Micro Python. So first, I guess we should talk about what the watchdog is. So what a watchdog is is it's it's a separate piece of hardware that runs separately from the CPU. And what it does is it's just got this timer that says like, I'm just going to count down. And if I ever get to zero, I think it's down. Anyway, it's counting, right? And if it runs out, if time is up, what it does is it resets the CPU. So what it's doing is it's saying like, if you don't feed me, meaning the way that the CPU can say like, oh, I'm still here, don't reset me is it writes to a register saying like, oh, like I'm still here. And that resets the timer. So what a watchdog reset is, is that the CPU failed to tell the watchdog peripheral like the separate peripheral electronics, like, hey, I'm still alive, which basically means that you're like see your CPU is not alive anymore. And therefore you reset. So let me show you. It's got all these debug prints. And I think what I'll do is I'll go over all the debug prints because I think people want to see, want to see it. So I'm an nrf land. So let me just switch. This is the bug it so I should talk about that one first. And then I need to check out fix GC 32. So one thing I did do is I tried to update the IDF thinking maybe it's not our bug. So that's why I'm when I switched I have to have to get it back updated. So now we're on clean, we're on this get GC network. I'm going to build it for the mag tag. And while I do that, I'll show you the mag tag that I've got. So my setup is a little wonky. Let me show you. So here is my mag tag. And this is why I get get the products is that I have hacked it. So these the green and yellow wires here are coming off the the UART that is the debug UART. And then I have this plugged in just to get ground. And then that ground is going into this chip, which is a USB to serial converter. So what that allows me to do is it allows me to see out debug output from the ESP, even though the USB is not connected, which is really, really handy for debugging. So like generally the approach that we took there isn't a debug port at least a special debugging version. So usually with boards, like the chips and the boards, boards are not that different. And so a lot of times where you have a case where you need debug access, you can actually take the code that causes a crash or something and move it to a board that does have easy debugging access. But this was a weird enough bug that we did actually, I think need to have it on a particular thing. So what am I doing? I am plugging in the USB to serial converter. The nice thing about having this separate is the USB to serial converter can like pick it up from the very beginning. So this right now, this actually has the code fixed on it. So if I just plug it in, we'll start to see output. And that's I just hit hit into the bootloader here. But this is like the debug output that you get when you normally power on. So this is like ESP ROM, this is code that is burned into the chip. And then from here, this is bootloader code. And it's giving us some information about like what version of the IDF we're using, how we're setting up the flash chip, how we're setting up our memory tables, blah, blah, blah. And then I hit it again. That's why this happened is that I hit the reset button. And we started up again. But this time we started up and we went into tiny of two, which is the bootloader. So we have that. And we don't need this. That's Jlink from NRF stuff. Oh, and my it was complaining because I had the wrong ESP environment. Yeah. So if folks have questions, feel free to ask them. I really dislike how console history doesn't just isn't shared across all console histories. It doesn't make any sense to me. If I tried to do a make command now, it's not going to know what I'm talking about. It's like, you know what I'm talking about. Talking about debugging, I was trying to control the Neopixel on my Feather 840 with a guide with no success couldn't find Adafruit Bule Services Nordic in docs to help troubleshoot the Bule UART. Adafruit Bule Services Nordic is going to be in the Adafruit Bule Library. If you're using five, or using five, using seven, there are now two UART. If you're making your own UART or you're using code that does, there will be two UARTs, which could confuse things that that's something I need to look into next week, because there's now a UART that circuit Python controls. I decided to have 7.0, but the mag tag I was going to update happens to be running the vaccination tracker. Is the fix for this in the beta zero? It is not, but we merged it earlier, so it might be in S3 by now. I think it was merged. Oh yeah, so I have some code in here that I need to undo. I was trying to fix it, and I'm trying not to give away. So I'm actually going to do a git reset, soft, head minus one. So what this does is it steps one back commit, but it leaves all the changes. When you use this with n new object with final errors, or does the class have to have a final error, or does it handle it automatically? We looked at this, it looks, it does handle it automatically. It looks for the del method, and if it doesn't work, it's fine. So that was not the problem. If I discard that. Spoilers. Fish history merged, that's exactly what I need. Let me open that in a tab, because I'm not going to do it now. It drives me nuts. It drives me nuts that I had to like magically get the right terminal session. It makes no sense to me. All right, so we're building, and it's going to show up in ports ESP 32 S2, mag tag. Seems like it was once part of fish shell and removed probably for performance. All right, so I'm dragging it over. So this is the broken version should be. So we'll take a look at the output here. Is that doable? Can you see that? I know the colors are a little bit bad, but I could make it bigger probably. So we restarted. Oh, even more spoilers. Hold on, let me, don't look, don't look, don't look. Okay, so I'm going to put it back into the bootloader. So it is doing the same thing, but it's doing it for a different reason. Rebuild. I want it to crash. I left this branch here so we could see all the printf debugging that we had done. Oh, trying to use code from the guide. Yeah, I need to take a look at that for sure. Okay, so here we go. So let's take a look at this. Hopefully folks can read it. 100% here because I like hanging out, not because I understand it. Can circuit python do quad and or octo spy? It can definitely do quad spy natively. You can't like set up quad spy from within circuit python code. But if you have a board that does quad spy, you can and you octo, we don't have any boards that do it, but it probably wouldn't be that different. Yeah, Pierre, go ahead and file an issue to remind me. I may not. We can always close it. Okay, so this first one is ignorable because that's just me hitting back into the bootloader. So it doesn't just keep going around and around. But this is the reset that we're interested in here. We have this, it says TG1WTDDWDT. So what that's telling us is that the reason that it reset is for timer group one, the watchdog timer went off. And the IDF uses this timer to make sure that the CPU is being responsive to interrupts. So there's two, there's two watchdogs that the IDF has. One is a an interrupt one. So it's making sure that like interrupts for Wi-Fi get serviced. And then the other one is like a task ones that's like further down into the real time operating system, making sure that the real time operating system is sharing time, sharing CPU time with things. So this is the case where it's actually the interrupt watchdog timer that's having a problem, which is pretty wild. So what I found here is I turned on debugging and I got this got IP event done. And then what I found is that we were starting to do a collect. Collect is the process for collect is the process for finding what memory is no longer used and then freeing it. And so let me I guess let me just tour the code with this. But these are all my nrf files. It's okay. So we're going to not be in nrf we're going to be in. So where we're where we are right now is we're in main here. And there's this, there's a lot in main. Oh, I ended where I want to do. So there is this GC collect call. So this is this is the process of looking in all the places that memory still could be referenced. So we do collect start and then we look in the CPU registers, we look in the the mount table, then we do background objects, we do objects for different circuit Python things. You can see here we have these prints. So these ESP early log W's. These are print, these are print f's basically. And the the W here is like level. So you can see I is informational W's warning is error D is debug. It's a way for like you to leave stuff in and control like how many messages you want. But I tend to pick it just because like what color I want it to be. So if we look here, we have this collect BLEIO collect USB HID collect to Wi Fi collect after Wi Fi. But if we look here, we see that oh, we're only getting to collect USB HID. And then basically all the stuff below we added later. So we kind of started from the top level, trying to figure out like, at what point are we going into the state that it's just hung and then the watch time timer kills it. So we said, oh, well, we're doing USB HID. So Jeff, who was working with me on this taught me this trick on how to print out the file name and the line number. So if we go back here and we pull up that file, it's going to be in shared module USB HID init C. And in fact, we can look at 249 is the line number. So 249, you can see we have the imports for the logging. And then there's this USB HID GC collect. And Jeff had this additional trick, where he does a define of X and that does the print. And because the print has this file and line on it, you can put it in multiple places in the but the print will be different. So what we were doing is we're saying, okay, well, the problem is somewhere in this function. So let me add prints to it. And lots of prints. These are separate ones because we're printing device ID and stuff. See if I can make this a little bigger. Sorry, I'm not used to working on a tiny screen. Yeah. So you can see there's a lot of prints here. Like, we didn't add these all on one go. We had to do multiple iterations. Like remember, this is this is at the end of our debugging. This is not the full debugging process. We spent three solid hours doing all of this stuff. But yeah, so we did these prints here, we figured like, oh, we're getting in the loop. But now once we're in the loop, I wanted to print device ID. We actually found an issue where this index here was wrong, and it's been fixed separately. And it wasn't the cause of this bug. And then we figured out like, oh, this is working, but that somewhere in here is not. So yeah. Yeah, I just keep adding more prints to solve a problem to figure out where that problem is. So what is this telling us? Let's take a look back here. So here we can see like, okay, we're getting into it. And we're on iteration zero and iteration zero. And then this is the memory that we're doing. And then if we look back, so the nice thing about here is like, this same thing occurred earlier. So here you can say these see these same exact prints. So what is this line 264 doing? Let's pull it back up. So line 264 is this like, okay, we're, we're printing out the in report buffers at that index, which is then passed to GC collect pointer. So what GC collect pointer is, is it says, okay, this is something on the heap. So I want you to remember that we're using it still. And then furthermore, I want you to look inside of that memory. And if anything looks like it could be another pointer, recurse and keep all those things around too. So let's pull that up. Pi GC, collect pointer. So just GC collect pointer calls GC mark. And mark is here. So and you can see there's this more, more prints here. Okay, so we have GC mark. And then the very first thing we do is we say, Hey, let's make sure that this pointer that we're marking is actually on the heap. So this, we know the bounds of the heap. So we can say like, is it in those bounds? And is it not null as well? So generally, this should keep us from like doing anything bad. But then we have these prints, these prints are guarded by that particular address that we know is the one that causes the problem. So what we'll do is we'll see file online number here. And then what we're doing is we're doing ATB get kind at head. And then we're doing some more stuff. So ATB is the what's the A stand for allocation. So there's, there's always a cost to having dynamic memory, right? So this is why, for example, you buy a hard drive, it's, you know, 512 gigs. And but then when you format it, you get like 400 and something gigs instead, right? Like, one of the reasons that is, is that you have to have room for all of the metadata about where all the information is. So for the heap, in micro Python, there's, there's two, one or two pieces of metadata that you need in addition to the memory, the, the data itself. First, you have the ATB, which is the allocation table. And what that's tracking is that's tracking for this, for this set of blocks. So instead of tracking each individual byte, we actually say, like, we're actually, so that would take a lot of metadata memory to track every byte. So instead, what we do is we try, we track 16 bytes at a time. So we call 16 bytes at a time a block, and that's the minimum size that you're going to allocate on the micro Python heap. So the ATB has information about the state of each block. And it keeps track of things like whether it's free or used or marked, marked means that you will keep it around, we won't free it. So that's the first thing. And then what happens after is if you have it on, there's a thing called the finalizer. And so a finalizer is a function that's good, that gets called when memory is freed. In Python, this is the dunder del. So underscore underscore del. Generally, we don't recommend people to use it because we it's not clear when that code is run. Because that's the when that code is run. Because that code is run when the memory gets freed. And you don't know when that is, right? Like Python manages when memory is created and when it's destroyed for you. It's not explicit. It's implicit. So there's so in micro Python, there's the ATB table. And then there's also an FTB if you have it on. And the FTB is just a bit array that for every block, if it's a one, that means, oh, it has a finalizer. And then after the FTB, there's just the memory that's used for the user data, split by into 16, 16 byte blocks. Okay, so what this is doing is it's kind of hard to see because there's all these prints. But what it's doing is saying, okay, first, let's check the pointer. And if it's the pointer is fine, we'll do this log on 371. So we see the log on 371, we then see 376 381. And then 385, this is the block ID. So this is the basically the the start, the the address minus the start divided by 16, or something like that. So if we look at 385, we can see that we're printing block here. So we we did block from pointer. So we got the block number. And then we did ATB head to mark. So this this is said like, mark the head, that means that it's used peer either either way off libraries or a circuit Python assigned, not the bundle, don't do the bundle. There's a couple issues on the BLE libraries already that I have to look at. And then there's this GC mark subtree of the block. So that's 385 and then 389. So if we go back here, so we get the block and then the two so so notice the line number is very different. So let's go. This is this must be in mark subtree. Let's go check take a look at what mark subtree does. And this is what I was talking about mark subtree is going to take a look and say, are any of these things are any of these things on the on the heap as well. And therefore, we need to keep them. So we get the block in and this is not usually here starting block. Usually just mutate spot but I wanted to I wanted to keep starting block around so I could do these prints. So first what it does is it figures out, okay, how many blocks am I looking at. So it says, starting with my block, do I need to look at more blocks after me for pointers. And then check this blocks children. So first we convert this block back to a pointer. So this is pointing to where the block starts in memory. And then for so I is going up to the number of. So this is the number of blocks. So in this case, it's two because we printed out the two times bytes per block, which is 16. And then we're dividing it by the size of a void pointer. So this is going to be four. The magic number 1212803 is just a block number that it's the block number that is causing the hang. So something is wrong with that block number or something related to that block is wrong. Not to get that away. Or yeah. So what we're doing here is for that particular block, we're going to market subtree. And then we're going to go look at every pointer. So every four bytes a pointer is four bytes. Yeah, it's very specific to this debug issue. We're going to we're going to count down from the this is a way to just loop the number of pointers. But then this pointers plus plus, it's it's starting from the start and working its way up checking each thing through a pointer. Yeah, that all these ifs are confusing. Sorry about that. It'll make more sense. So this is just saying if the starting block is the one we care about printed out, we don't want to print this out for every block. Otherwise, we get like tons of output. So we're going to print out the the where pointers is pointing. And then what happens here is we dereference the pointer. So we read the value at that pointer. And then we print that out as well. So what we hear see here is we're reading this pointer here. And it's zero. Okay, that's fine. It's not on the heap where we're done. We keep going, we look at this f four, it's zero, we're fine. f eight, it's zero fc, it's zero. And then the last thing we do is we try to read three f eight, 000, 000. And it just stops, right? Like nothing, nothing continues to happen. And eventually the watchdog timer comes along and says, Oh, something really bad happened, we have to just reset. So that was kind of troublesome. Yeah, exactly. Dexter has the kind of the explode emoji. One thing we did look at is if we go, let me not show you my downloads directory, but let me pull up the technical reference manual. So reference manual for the ESP 32 s two. Oh, and we're on the right page too. I'm glad this was so easy to fix. Yeah, right. 258 could be a problem. Folks like that. I haven't given it quite away. Okay, so what we have here is this is the address mapping. So the way this is the idea of memory mapping. So there, the CPU can say, I want to read this memory address. And that memory address may not actually be in RAM. There's like different ranges that go to different places. We talked about the different buses before, like we did the system on a chip kind of layout stuff. So typically with there, you'll get like one memory bus will have like one big memory range. And then it, that memory bus will then split that range up into smaller pieces for different things that are connected to it. So the place that we are at is we're here in this two and a half or 10 and a half megabyte chunk that goes between three f 50 000 and three f f seven f f f f. It's 10 and a half megabytes. And it it's a way to read external memory. But it's actually cached. Now the weird thing is, is that, oh, look, this is the address that we're trying to read three f f eight 000 000. ESP 32 is Harvard architecture. Yes, it is separate buses. That's what it's showing me. Yeah, it looks like it is the the extensive CPU that's in in the s two is. Yeah, so so oh, oh, no, like the memory address we're trying to read is marked reserved. So usually what will happen is when you do this, you'll get an exception immediately. Usually what will happen is on ESP 32 s two, you may have seen this before, you'll get like a load prohibited. And when that load prohibited happens, you'll get the panic handler. And then the panic handler will give you a back trace. And that's really handy and helpful. And it's very clear like where your code was when it did the thing that it shouldn't have done. However, in this case, it looked like we're not actually doing that. What's happening is that the the memory bus itself is like just saying like, I'm working on it perpetually, which is stopping everything from happening. So interrupts aren't happening. No code is being run. And eventually the walk watchdog timer kills you. This isn't the first time I've heard this happening. Damian had warned me and told me about a bug that he hit on an STM chip. That's a Cortex m seven, where he had the similar thing happen where the CPU was trying to load an address. So on the Cortex m sevens, it will try to start memory fetches before you actually need them, which means that like for all the data that you're looking at, if it's within the ranges that thinks it can prefetch it will. And so he had had this problem that was really tricky to debug, where he was just doing some computation like SSL computation or something, and it was hitting some value that the CPU was trying to prefetch, which was then shutting the whole thing down, like shutting the whole memory bus down, preventing everything from happening. And so I kind of knew that like, Oh, one failure mode when you read a bad address could be that like everything just stops. And so that's the case that we're seeing here is like, Oh, we're reading, we're trying to read this three FF eight, 000, 000. And so my next reaction was like, okay, well, are we setting up? Usually there's a one way for that failure to happen is that there's what's called a memory protection unit. And this is in the higher end Cortex series stuff as well, where you can like the IMX, I did all the code for the IMX that says like, Oh, like your RAM or your flash could is two megabytes, but they've allocated say 16 megabytes of address range for that. And so you have to tell the memory protection unit like, Oh, only allow me to read the first, the first two megabytes of that range. And if I accidentally read outside of that, just crash. Yeah, the bot, bot rate is 115 200, I think that's what mine defaults to. So, so yeah, I would like originally said like, Oh, let me let me see if there's a memory protection unit that's configured wrong, but I couldn't find it. And so I actually reached out to Ivan, who works for Espressif and he was very interested in this as well. We chatted, Jeff and I chatted with him yesterday. And he said that it really shouldn't happen. Like, the fact that like it does the system on the chip got hung based on a memory read is kind of bad. And that it shouldn't do that. He said, it should either do a load prohibited or it can cut like there. If you hit the cache in an area that the cache is not servicing, like it will raise an exception and cause a panic as well. And then maybe maybe when you read a bad address, you just get like zeros back or something. So this is a really weird case where we're reading and we're causing it to hang. Which is bad. And so the thing that I commented earlier when I said it was failing early was one thing he asked us to check is that that if we read this address right when we start, it still causes a problem. It's not dependent on like the sequence of reads that we did prior to that. And that was the case. So it should be easier for them to debug. They don't have to like get this all working to get one particular sequence. If you read that address, you'll, you'll hang everything, which is not great, I guess. But they're, they're going to take a look at it and figure it out. So now the question is, is like, okay, why are we reading this? Like something is wrong with circuit Python still that is causing us to read this invalid memory. So let's go back and take a look at our output again. So what we did is like, okay, we're in GC collect. Now we're collecting the USB HID stuff. We're looking at this particular pointer. And we're looking through it to make sure that anything else is here. So we've looked at the block and we've said, oh, we need to look at two blocks worth of data for that. And then we'll go through that. But notice that the fifth thing that we do, so one, two, three, four, five, the fifth thing we do is where we have a problem. And a block is four pointers. So if we go back up here and take a look, when we did it last time, when we collected this memory before, so here's that same address is here's the same block. But now, oh, wait, like the length of this allocation used to be one. It's one, and we only did four reads. We didn't try to do that fifth one that screwed us up. So this one is a problem. Or this one is correct. The two is a problem. So we looked here. And we said, oh, you know what? The thing that occurs after the ATB, or no, what we did is we said, okay, why, how do we come up with this too? So we looked back at the code and we said, oh, there's this do while loop that's n blocks. So we say, we do one increment. So it's always going to be one. And then we're going to do ATB get kind of the next block. So not 1212803, but 128004. But then what we did is we printed out, we found this print. Where is it? GC unit. The cat is on the move. Yeah, so there's this print here. So we said, wait, well, like, is there any room between the ATB and the FTB? Maybe we're actually, when we're reading that very final ATB entry for the block after the last block, we're reading into the FTB. So that was what the problem was. So if we look up here, right, so here's the GC layout. If that evil read has a kind, then maybe create an exception to skip it. Yeah, we weren't doing it correctly. So here's the Alec table. The Alec table is at this address. It's this many bytes long for this many blocks. And then the finalizer table starts at this address. And as this many bytes and for this many blocks. So what happens if we take this number, and we add 32001 to it. So Python, hex, plus 32001. And we get 3FD88129, which happens to be the start of the finalizer table. So we said, oh, I wonder what the difference between those two collects must be that something is changing the state of that first entry of the finalizer table, which we're inadvertently reading as the final block, or the block after the final block. So then we go to the code and we say, okay, well, how many places do we mark the finalizer? And there's just one place that we actually set that memory. And it's in GC Alec, we do the allocation, this is the code that searches for a place to allocate. And then we say, oh, if finalizer is enabled, and the memory that we're allocating has a finalizer, then we do this FTB set for the given block. So here you can see that we have this debugging, we say, if the starting block of the thing that we allocated is less than eight, meaning it's at the very start of the heap, we know that we have this problem with these two blocks. So let's just read the kinds. Sorry, I hope it wasn't too loud. So let's read the kinds of that last block and the block after the last block, if the thing that we're setting is under the first eight blocks. And that's what we have here. Let's scroll up again. So that's what we have here in green. Thank you all. So yeah, so before this allocation that we're printing out, we can see it's zero and or one and zero. And that's the case where it's going to be length one. But then we say finalizer for block one. So that's the block that we're editing. And then after we set it, it's now one and two, which is what will be now interpreted as that last allocation being length two instead of length one. So this is a doozy. So the problem is there's a couple of ways to fix it. So if we pull up MicroPython, MicroPython is where we found it. And I realized like maybe it's not actually MicroPython issue, but they do actually have the same problem. You can see problem in GC mark sub three in specific circumstances, right? So a lot has to happen for this to be a problem, right? It's like the thing that's allocated in the second block has to have a finalizer in order to get the correct value so that it's the tail. And then the thing at the like, then you need a thing that's allocated at the very top of the heap as well, which CircuitPython, that's where CircuitPython will put things that are long lived. MicroPython doesn't do that, which means it's like less likely that something happens at that last slot. Although anything that runs to or that uses that very last slot could be problem. Keith E says so there are conditions where the amount of memory GC needs to free is size so that crash doesn't happen. There are conditions where the amount of memory GC needs size so that crash doesn't happen. Not sure what you're thinking. So the problem is that when the ATB table actually needs a final entry, right, the ATB table needs an extra entry for the thing after the last block. Or if when you're doing the check for it, you could check if you're at the last one. So here you can see what Jeff said. Like this is a problem. I said actually this could only be CircuitPython because of the way we do the long lived. But then Jim said I believe the same issues exist in MicroPython. Mark subtree will check getKind on block plus one. But I don't think it ever could be triggered because there's no way in MicroPython block zero will have a finalizer. I.e. it's very common that the final block will be ahead. But zero chance that the first block will have a finalizer. Having an empty byte at the end of the ATB might actually be the cheapest way to solve this. The alternative is to do to make the do-while loop at the start of Mark subtree. Also check that it hasn't gone past the end of the heap. But that comes at a code size and performance cost. I missed the reasons for the finalizer. Could you review what finalizers are for? So finalizers are a way to have code run when something gets free. So for example, you know, we do a lot of pin in use tracking. And we don't do this very well. But what we could do in theory is we could say, if you create a digital in-out, and you're using a pin, and then you delete that or you overwrite it so you no longer have access to that object, we could actually call dnit for you and give you that and free that pin at some point when you're like, it's like a destructor. Yeah, it's a destructor, but you don't know exactly when it runs. And this is a, it's a Python thing. Context managers are the proper way to do it in Python. But you can't actually do it as a, as the Dell method instead. Yeah, so the actual fix is that, yeah, so Jeff says, my gut is the cost is to pay an extra byte of ATB space for one eighth of all the possible GC heap sizes, right? So this is only a problem if you fill up that last byte of the ATB, because the ATB is only two bits. The ATB is two bits per block. So if you have, yeah, so I guess it's not, it's one in four GC sizes. So if it happens to be divisible by four, then you're going to have this problem. But yeah, so basically you always need to have a little extra space for that little, that like end block plus one state to just always be zero so that you end. And that the finalizer state doesn't impact it. So yeah, here's the fix that we merged in. So here's a memory check. Jeff did actually reproduce it on the UNIX side. But basically what he's doing is he's saying total byte length minus one. So set aside a single byte if we're doing the finalizer thing. And then that causes it to always have a little bit of a gap. So yeah, it's kind of an off by one error as well. But it was a lot of work to run down this pretty simple thing. Key says that was what I was poorly phrasing. I was trying to replicate it but hit a point where I couldn't replicate the bug and couldn't figure out why. It must have been I hit a lucky didn't fill the last byte of the ATB when I was working on it 100%. So you know, I said I spent like a day and a half two days on it. And the place that I got stumped was Jeff kind of jumped back in and said, Hey, if I turn off long living, it doesn't crash. It works okay. And so that was like, Oh, weird, like long living is what allocates to the right hand side of the heap. And I just fixed another bug where if you take something that you allocated at the left side, and then you long live it, what happens is you move it to the other side. And if that's okay, if you know all of the references to the original one, you switch it. But there could be bugs if you move something and something else was pointing it to it in the first problem in the first spot, like that can cause issues. And that's what the the i2C display thing was a problem. Confused left and right. So let me find on my YouTube channel, I've got some really good visualizations for this. And I'll just play it because it's like a minute long. Um, yeah. So let me see Grover even made. See Grover even made some sound. So I'll turn on the desktop sound here. So this is this is how circuit Python still allocates. So this is in 2x. You can see it just starts from it starts from the left hand side and grows up. It's too loud. Okay. So this is this is how MicroPython still allocates. And this is how we allocated in 2x before we added this this long living stuff. So low memory addresses are on the left and the high memory addresses are on the right. And you basically kind of like fill things up as you go along. And that's where things will stay as long as you refer to them. But then once you hit the end and you run out, you have to go through and figure out all the spots that you all of the spots that you're no longer using. And that's where you can place the next set of stuff. So here you can see like it's got kind of like a Swiss cheese to it all. And this is why people can get really frustrated where they say I'm trying to do this allocation. It says it's this long. And then when I do mem free, it says that it says I have more space, right? But this is this is what we call heap fragmentation. Like you have a lot of free space, but none of it is next to each other. It's all kind of split into smaller things. Hi, Alvaro. I forgot to say hi. I saw your message though. So let's let's keep watching this. So yeah, so now this is with apparently in 3x we added long live. So it's been in there a long time. And we still are finding issues with it. It's a little tricky, but I'll show you why I'm still happy we did it. So this is like a like a very basic version of what's called a generational garbage collector. So in that first case, after we had swept everything, what we could have done if we had the ability to is we could have squished everything together. And that would that would be called kind of a generation. So like you when you started, you didn't know how long somebody something was going to be used for. And therefore you treated all the same. But the moment that you can go in and say, here's the stuff that I'm no longer using, but everything else I know that I've used for a while now, then I can, like if you have a layer of indirection, you can squish it all together and get rid of that fragmentation. But we don't we don't have indirection in micro Python and circuit Python. So what we have to do instead is we have to actually choose to be deliberate about where we place it when when at the start. So so one thing we do now with this long live stuff is we assume that if you import something, it's going to be around. And so what you'll see is that and there's a couple other kind of rules for things that we know that are going to stick around like all of this queue strings, for example, we know that those those data structures don't get freed. Like when you import a string and it gets queue stringified, like it sticks around. And this is actually the topic of like one of my like my first deep dive that I did before I started doing it regularly. So there's a lot more time of me talking about this. But let's keep watching this just to show you. So what you'll see is that there, there will be some things that are just allocated on the right hand side, which is the long lived portion of the heap. And that's where that usb hid block was causing trouble is we were we were allocating that long lived from the get go. And so that's why we kind of guaranteed that we were going to have that something in that spot. And which is why the bug disappears if you turn long lived off. Okay, so let's play let me play this. Right, so so this is the case. This is the same code as the as the time before. And what you can see on the right hand side in this long lived area now is that we have like very few gaps, right? Like, it doesn't guarantee that there are no gaps, but there are very few gaps, which means that on the left hand side, the free spaces that we have are larger and more contiguous, which means we can do larger allocations later in in the evolution of the program. So yeah, I'm really happy that we have this long lived stuff, but like it is a source of bugs. But yeah, let's see what else it does. So what a lot of this is, it's a lot of like importing modules. So all the stuff you create as you import on the left hand side until import is done, and then it gets moved all to the right hand side. And we actually do a collect after that import happens to because if we can keep stuff like as far left as possible, then we leave this like if you ever want to do a big allocation in the middle, we'll still have the space. So yeah, I think I'm really happy with this visualization that I did. I think it's pretty neat. No, I'll link to it. I think that covers it. Does that cover it? But yeah, the planets have to line up for the bug to appear. Yes, it's it's definitely like a rare bug. It is a bug, but like, it's been a micro Python and sacrifice them for five years, right? Did you discuss what needs to be done when things are moved from left to right in an earlier podcast? Maybe there's only a few places that we actually do it. We like we could just find those. So it's like, like we were just looking at these so you can just search for like, if you just for look for like long lived. Here's the script that generates that visualization. And then you can see like Pew Pew is making something long lived. Here's the mechanics from the GC. You can say like, oh, like object, object module, like the global's dictionary for the modules put there. Here's another dictionary that we're making long lived. So like, assuming somebody's pointers needs to be updated. So yeah, as a general rule, if you're what you'll see here is like, occasionally you'll see it in like a port. This is actually not everything. There's also allocators that have LL in it. Look at LL. And let's just look in Pi, like see all these variants. These are variants that that do long lived applications as well. So like, if you're if you're creating a you are, for example, and you're doing a buffer and you're going to hand that buffer to like ASF or like whatever the the vendor how is like, if you're making a pointer that you need to pass to the vendor how or you're you're there's a second place that's going to be stored besides a Python dictionary, then you want to make sure that it's long lived. Because because the second copy won't be updated, we only know how to update the pointer copies that are in dictionaries. So it does compaction if needed. So it doesn't really compact so to speak. What it does is it will move like dictionary trees, it will move dictionary like trees of objects from dictionaries like when a module is imported, it will move those to the long live section. And it's hoping that it's doing it before there are any other references to it, which is not always true. And that's why you get some bugs around it. But yeah, kind of like you have to do it early. So all of this long live stuff only impacts it only impacts the allocation process, it does not impact the freeing process. So the freeing process is all the same. It's only the out it's only the decision of where in them in the heap to put it that that is different. But yeah, so it's a bug that would not it would be harder to tickle this bug without long living, but it is not actually a long living bug. The I squared C display bug that I fixed this week was not a long living bug either. There's another case where we move memory. And that's for displays. So if you create an object that we use for display, because we keep displays alive after the VM goes away, we move memory then as well. And we had a bug with that for the I squared C display. So yeah, that's definitely in the weeds and it took us a while to figure out and it was pretty relatively simple when we when we figured it out. But yeah, thanks for the report, David. And it was thorough enough that we could reproduce it. Oh, and J for seeing has the the og deep dive. All right. Any questions? Any topics you want me to cover? Or otherwise we could, I could jump into these nrf things that I was working on. Any other directions you'd like me to wander before we do that? Oh, yeah, this is a unrelated. I pulled this up. This is a right to repair bill for Washington State, which I didn't know about. But for those of you in Washington and want to do that, let me know. Oh, yeah, I positioned the camera again. Dr. wants a cat check. He's in his bed now. I'll curl up because the window's still open. That's got to have been a pretty awesome feeling once you get it all working and figured out why it happened. Yeah, we actually like, I immediately pings Lady Ada and said Lady Ada, we figured it out. Because she was like, let me know when you figure this out. These bugs are really tricky. So yeah, we were and then we talked to Ivan about it too. So yeah, is this a kitty break? Peanut break. Should I mute it so you don't have to listen to me? I could play the video again. My debug output is garbled, although I have one 15 200 set. What are you debug? What is on the thing that you're trying to listen to? Because I think that we have, I think we set it in the circuit python config for one 15 200. Vedran asks, have you tried Scythem for the project? I have not. Keith E says, man, that bug was messing with my sensor super bad. I'm excited for tonight's night label. Oh, you were seeing it with a separate thing? Don't understand your question. So what if you're trying to get debug output on the ESP, what code is running on the ESP? I hope we fix it, Keith. It is consistent though. And that was the saving grace with it is like, if it happened, it would do the watchdog timer and start over and do the exact same thing. So we can see if, see if it's available yet. So if you go to Adafruit circuit python, and then yeah, so this is the pool fix. So if we hit here, we can see it's still queued up. So yeah, it's not an S3 yet. This will tell you when it's going to, it'll be an S3. This is the code that will put it in S3. Oh, you can't see the desktop. Thank you. At some point, I'll make a thing where you can just control the view of mine. Yeah. So the way that I got here is if you just go to the front page of circuit python and click this yellow dot for the latest commit, and then hit details, you can see the build status for the merge, right? You have the debug. Yeah. But you have to, you have to enable the debug output for the ESP32. So when I was doing, when I did my circuit python build, I did debug equals one. But I don't know for sure that you're putting circuit python in it and wanting to see circuit python debug output. It is not, it is not the strings that you print from circuit python. Those do not go to the UART. You can make it do that, but it doesn't by default. You have to edit the code. Keith says, I really need to make an advanced debugging cheat sheet for all of these things. Yeah. I'd be curious to know what you think should be on there. I did long ago make the, the GDB learn guide because I figured I should document it. It's stale, but it still applies. No, plus, plus. Yeah. So here's, here's my, my workflow, which is still my workflow for GDB. It's for just the M zeros, but it, I use it across basically up everything, but the, the ESPs, I use this workflow. ESPs are different. What did you use to make the heap graphic? Um, the actual image is rendered with the graph vis pie graph is just a bunch of, um, oh yeah, Mr. Dalgarde says excited for the inner fixes. That's what I want to get to. We have half hours still. So I think I'll go back to that. But, uh, yeah, it's graph is and I just made like a bunch of PNGs and then put it all together as a video. So, um, I hooked up a way to capture every, every change to the heap. So I had this, uh, and it's documented in the, I assume it still works. I think there's a document in tools, build memory info, analyze heap dump. Right. This script renders a graph of the micro Python heap at the given point in time it was dumped, takes the binary dump around the binary of circuit Python on the linker map file. So this is using pygraph vis. So that's what actually rendered it. So basically what it does is it just dumps ram every step. So basically it runs GDB by setting a break point at, uh, every, there's a break point or a function call you can turn on that any, like any action on the heap calls that function and therefore you can break on it. And so you can get this, a snapshot of memory at every point that the heap is changing. And then you can render it all and that gets you a video out of it, which is cool. I haven't looked into it in ages, but it was really helpful for when, when I did the long live stuff, it helped me understand like, Oh, these things I could move over and how do I do that? Yeah. All right, let's talk to nrf. So Mr. Dalgard filed this issue. All right. Hopefully I'm saying it right. You can try to correct me if I'm not saying it right. I'll try to say it right, but I may not be able to get it. So this is the issue is open yesterday. It says crashes to horror fault handler when code is run and the alarm pin is triggered. If you don't do the trace back module stuff, it won't crash immediately. But I think it's actually you don't need the trace back stuff. You just need this counter DNA is what's actually crashing connected to PC via USB. So light sleep is fake sleep. So light sleep is always fake sleep. It's only deep sleep that we fake. But yeah. And it turns out on nrf we don't actually deep sleep. We kind of fake we even fake it when we're supposed to be doing it for real, which I found out Dan reproduced it and said like this, this stuff looks suspicious. Adding your finding initializing a pin alarm and sleeping on it will actually break count IO, which will no longer increment. Yeah. So I'm kind of in these weeds. And there's a separate issue for the RP 2040 that we're not going to talk about. But there's a number of issues that I've found and I need to finish it. Generally, the problem is, is that when you there's the nrf library for the GPIO TE, which is the GPIO thing that allows you to interrupt when pins change. So that's that's kind of the common thread through here is like, these are all listened to an external pin and interrupt me when a pin change comes in or a pin value changes. So pulse in is one of those things that will use it. Count IO will use it and pin alarm will use it as well. So those three things are kind of all interrelated and that what they're doing is they're listening to a pin to figure out when it changes value. Um, so let me just get out of here. I actually don't need this branch anymore. You know this. I only kept it around to keep you all up to date. It explains why I see the same power usage regardless of sleep mode even time to sleep. Yes, I was actually surprised by this. But it doesn't go into system off. Um, I'll, I'll show you. I'll show you. What did I call it? Fix nrf GPIO TE crash. Okay. And this doesn't have that IDF update. So if I want to make it clean, not only do I have to switch it, but I have to update all of its sub modules as well. Okay. Other nrf 52 express clean. Let's take a look what I've done so far. I won't maybe go into so much detail as I did last time. So this is sublime merge. This is a way for me to take a look at what I've been doing. So let's see what I've done so far. So in pin alarm, um, it had this configure pins for sleep. And this is called in deep sleep or light sleep. And the very first thing it does is it says if the GPIO TE is initialized, then uninit it and re initialize it. And so this is why I believe, um, uh, Mr. Dahlgard said, Hey, if I have, if I do a pin alarm when count is like, can I, I doesn't work once the alarm wakes back up. And that's because the alarm was just like completely resetting everything, which it didn't need to do. And so I deleted it. And then there's this error checking here. And so this is a, this is going to be a theme. This nrfx GPIO TE in a knit, uh, returns you an error message. This is the thing that I still have to do is that, um, here, for example, in counter IO counter, uh, it's not checking that this actually worked. And this is important because I think here there's this rotary IO, raise an error if more than four encoders are used. So, uh, this is actually the test program I have currently loaded. And, um, it tries to initialize, this is from Pierre, I think. Yeah. Um, it tries to initialize five encoders. So this is the test code I have here. I tweaked it just a smidge because I don't have these names in board IO. So I actually imported microcontroller.pin as board and it worked. So basically like it doesn't crash when you try to create, I shouldn't have had those peanuts. Uh, it doesn't crash when this, this fifth one is allocated. But if we look here, micro dev says like, oh, there's actually only eight channels on this. So in each rotary IO is using two of them. Thus you could only actually use four. Um, so basically there's a number of places that we're not checking that it actually worked. Um, and this is one example of that. So if this in a knit doesn't work and then you call in event enable, it actually causes an assertion error. Um, so yeah, so the, the the remaining to do I have for myself is to fill in this, uh, check, um, to, to raise an exception there. So that's what I've got to do still. It's, I'm not allergic to peanuts. It's just they're still in my throat, which is why I said how I shouldn't have done that. Um, so I, I fixed the count IO thing by just deleting the thing here. So that was Mr. Dollgaard's thing. And then, um, I looked at rotary IO to implement count IO. Um, yep, I'm guilty of that too. Uh, so this I think fixed the count IO issue where we weren't like initializing like this. It's a little tricky. There's this, this GPIO TE in it. That's the whole thing. And then in the other things, there's just in a knit, which is an input in it. And so that's for a specific pin, which is definitely different things. Um, and then for processor, wait, so that's a separate issue. So I moved the, that global reset, which was in pulse in, I moved it into port. So it always happens. Um, we had a, a, a related issue where if you, if you were relying on that, but then turned like, if we turned pulse IO off, but had the other two, for example, like it would not reset correctly. So I'm just always resetting it here for now. Um, this was just a compiler warning that I fixed needed to be initialized. And then you can see here in incremental encoder, I'm doing the same thing where I like, I've got to check this error message state. Um, and then raise an exception. So the next thing I have to do is figure out what exception I'm going to raise. Um, and then as I was testing this, like the original issue from Mr. Dalgarde was talking about light sleep, but I also wanted to test deep sleep. And when I was testing deep sleep, I discovered that when waking up, um, we wait the two seconds, we wait the one second for safe mode, and we wait another second for the BLE workflow stuff, which it's not supposed to do. It's not supposed to do that if you're actually waking up from an alarm, a deep sleep alarm, it's supposed to start up a lot faster. Um, and the way it does that is through, not this, this. So this is the data sheet for the nrf. And there's this register, um, make it bigger. So this is in the section for power, power management. This is called reset reason. The reset reason register will tell you why something reset. And this, we were seeing this earlier with the watchdog reset on the ESP as well. Like all most microcontrollers or all of them have this reset manager thing. So from here, you can tell kind of why you started up. You can say, Oh, was it the reset pin? Was it a watchdog? Was it a soft reset or a software reset? Was there a lockup? Kind of like also kind of like a watchdog. Are you waking up from system off? So this is what we thought a deep sleep would be. There's also this LP comparator. And that's also if you're in system off. Diff is a debug interface from system off, and then NFC from system off. And then Vibas. So this is like if the, if the power on, if the power got into a good state, that's what started us up, for example. So those are all the reasons the chip will tell us. But if you look at what I fixed, actually, I should show talking about the sleep state. So an nrf, common, how alarm, this is where we go into sleep. So there's a alarm enter deep sleep. This is the thing that that's actually supposed to do it. What we see here is it says, Oh, prepare for deep sleep. And then set a timer and then call system on idle until alarm, which I thought was originally when I looked at it, it felt like it was an nrf thing, but it's actually not. It's actually further up in this file. But what you'll notice here as well, is that after this function finishes, it actually then calls reset CPU, which means that the micro controller when it's starting up is actually seeing it as a software reset. It's not seeing it as like a deep sleep reset. So if we look here, if we look up at idle until alarm, so this is the deep sleep on the nrf right now. And the reason that I looked at the PR for it, and the reason that we don't actually do system off is that the real time clock doesn't work when system off. Only it's only pins, I think they can wake it up. So instead they chose to because it's a really good system on sleep is still really good. The decision was made in that PR to just do kind of a faker deep sleep. So this is keeping track of the timer stuff and then it's actually just running like all the background stuff still, which it might leave USB on actually, although it won't happen if USB is on. Anyway, it's weird. It is turning off a flash, which I think is actually like one of the bigger pieces of power saving. But it actually does do that for other things as well. Deep sleep is just a matter of what you think deep sleep is. It's not the deepest of sleeps, if you will. It's not system off for nrf, but the number is like where is that? Hey, Johnny. Pull request and the author was John Tussac. So this is where it was. The deepest of sleep sounds good right now. Anthony says what you think it is. Yeah, so Dan is showing here that like system off is just under a microamp, but we're in like it's still in the microamp range, which is really good. That will still burn through current like crazy. I didn't think so. I think I thought it was still pretty good, but well it's not a loop. It is doing it interrupt to interrupt. I think it's pretty good. So like here's the actual loop. We run background tasks and we just do this port idle until interrupt. So that will do that will do wait for interrupt. So this is essentially where you'll end up when you do time.sleep or other sleeps too. So it's pretty good I think. Doctor is asking is there a good guide? There is a good guide that Dan wrote and it actually has numbers in it so we can take a look. I use the wake on DPI detect and current is low. Yeah so this is a good guide deep sleep with circuit python posted in the discord. It talks about the different alarms that can wake you up and pretty happy that we call them alarms. I'm happy with this API. So time alarms, pin alarms, touch alarms, how to tell what woke you up, sleep memory. This is memory that will be will remain over and then there's this power consumption tab. So these has charts. So here's the ESP32 S2 and we can see it's like 50 what is this say 50 milliamps and it's a little under 230 microamps when it's sleeping. And this is that's for the S2 and then pin alarm. Oh maybe there's not nrf numbers. No. Well other chip families. Oh I think wait is this this is a draft. We should look at that. I don't know if we realize that it's still a draft. I think Lucian did these numbers. So you can see that like ST is down to 700. Light sleep is still six milliamps. I thought people said it was much better than that. This has six milliamps for nrf which seems really high to me and it's also on the blue fruit. Yeah that's not a good example. Anyway if anybody cares about power consumption you really should check it yourself. You just said that about time sleep. Does that need an RTC or just based on an internal clock? It's usually based on an internal RTC. Pierre says six milliamps that's the nrf cp running full full on. Well it's on a board that's got a bunch of stuff on it like sorry I didn't I don't think this like this page is actually marked marked draft so we need to need to review it and get it published. I'm logged in as myself so I can see draft pages. Mr. Dahlgard says the tested the nrf to 0.3 milliamps in sleep for the feather nrf. Oh Dr. can see that page. Really? It's it's red here. Maybe if you put the URL in. It's fine. But yeah power consumption is one of those things where you can just like yeah you can see the guide but not that page. You probably can't see this other chip families tab. Whatever remind me remind me to get that looked at. Okay let's wrap this up. So I think to wrap this up I've just got a check. I'm changing the URL because of 404. Okay well good I'm glad it's protected but bug me about that we'll get that published or not. Wrap it in a bow and ship it. Usually usually new pages get reviewed that's why it hasn't been publicized yet. John Franco deep sleep yeah and I'll plug I just like to plug this if if you care about power you should make sure you have a way to measure it and if you don't already own a way to measure it the recommendation that we have is this Nordic PPK2 which is what those numbers are coming from. It's a really good way to really good inexpensive. This is way cheaper than a lot of the other options. When are the super chats coming to this channel? What is a super chat? I don't know what that is. Yeah exactly so doctor says I have a project I want to leave in the woods in low power consumption with your grant for that. This is exactly you need something like this to be able to monitor the power usage. Super chat is a way to throw money at you on YouTube. We used to have that. People have done that to me before but maybe somebody turned it off for our channel. PPK2 yeah if you want to make power consumption and circuit by them better please do but it's a never-ending task. Minnesota Mendes says I would do that Scott. I do have a github sponsors page if you do want to sponsor me personally but honestly like just buy stuff from Adafruit they paint me well they treat me well yeah adafruit.com is how you super chat. Yeah and I do have github sponsors if you really do want it to go to me. Okay we need to figure out an exception for this because that's the last piece of work I need to do so I want to get this out the door today so yeah that's that's how this deep this deep sleep works on the nrf so if somebody wanted to go in and actually make it deep sleep then I would be okay with that um and you get stuff too for your Adafruit Super Chat. I order free stuff from Adafruit all the time so don't worry about that but yeah oh yeah true and if you do adafruit.com as the super chat then you get stuff too yeah it's a win-win okay so what exception do we want to throw uh I think we do value error let's take a look at PWMI what errors will we throw if it's busy uh this is a bad one to look at invalid pin all timers in use that sounds very similar to what we're gonna do so let's take a look at shared bindings WM out invalid pin all channels in use that sounds good thoughts thoughts on what people think this text should be ideally we would reuse an error message because then didn't I add the that's why it's super I guess I didn't do the checking there yet I had deep sleep stuff on my Arduino nrf 52 I had to go down to the register level to turn it on but it really goes to bed I'll have to look it up and see how I can help that'll be awesome I think having good deep sleep in circuit python is a huge game changer because most of the time you have to go to the register level but what circuit python has going for it is that it's circuit python that's keeping track of what is being used right now which means that it can be really aggressive in turning things off because it knows what you're using whereas in like c code you have to in c code you kind of like you write the c code and you have to write what what can be turned off or on but because circuit python is more dynamic it can also be more dynamic in power so I think has a huge potential there are two different things that can come out of this it can I was looking at it here so it says in and it can either give you success busy meaning the pin is already used but I think that's going to be very unlikely given that circuit python tracks pin usage already so I'm not going to differentiate between these two things because the thing we're really checking against is this GPIO TE channel is it is no no channel is available so like that's basically what we're saying so what I'm going to do is just add that so we're going to add that check to pulse in no soft fail messaging what do you mean by that what do you mean by soft fail yep have to keep track of everything well yeah that's why like circuit python does that for you which I think is really candy hugely beneficial okay so this is an interesting well more interesting case so for rotary I rotary encoder we actually have two and if the second one fails but the first one doesn't then we actually need to free the first one before we raise the exception because we don't want to hold on to it and then counter needs to do this as well all right so that should be good and if we look at this is kind of annoying I'm just going to cheat and amend my last commit and now I can kind of see it all in one place so here we can see like okay in pin alarm we're deleting this complete reset that which I guess we need pin alarm to check to does pin alarm check and pin alarm it doesn't so I think that because it does its own stuff which might still be a problem I'm going for better not perfect but it's actually well I guess it is so it's doing this in a knit so I think we actually want to move this and I guess we need config too although that's not gonna work no that's not gonna work so configure pins is also called for deep sleep and deep sleep will require it to be set again so what I think I actually want is there's a way to just check but it's not maybe I don't need to do that I don't know don't know I think I'm gonna cheat and not solve that I'm gonna leave that be so we're not gonna correctly check for pin alarm don't tell anyone we weren't before so not worried about it so for counter we're now checking correctly we'll say all channels in use with a runtime error and then so for processor I I didn't fully explain this so this is the change that allows us to to correctly say that we woke from deep sleep even though even though it's a not the list you looked at the last word error checking returns correct so I was looking at the pulse or PW PWM out is used internally so its API has returned returns an enum or or defines values for different errors and then they get mapped to error messages but here because these these things aren't used internally I just raise it directly I think that answers your question James okay pulse in so we're gonna fix this reset thing but we're not gonna well no pulse in okay we are checking as well and where we moved through reset so for rotary or for incremental encoder we are checking both A and B and we're always resetting it now and this build thing so we should be able to make it load it on here so I've got a yay glad you understood you so here's my feather nr 52 840 which is not the right one there's one on my desk here that I was using and I've got these fancy two wires that I touched together as a way to wake it up but apparently I haven't built here yet so I need to source do do do there was a dog the neighbor dog is really loud maybe that's what you heard implicit declaration all right let's figure out I think it's in pi runtime actually so that's for counter so encounter will do include well it doesn't have a license header it might have been me and need a feather wing with one large button redefinition because I copied and pasted so we'll test this and then I'll go the PR out and people could tell me I didn't do it quite right I need a neokey feather wing I have one somewhere that's a good idea I don't have one on my desk though that is a good idea though why why why have a fancy feather wing when I just need two wires and yeah mr doll guard points out that there is a user button which there is again too hard I like my wires if I use the button I would have had to figure out how it was connected okay ports nrf build plug it in double tap to get the bootloader the blood blood blood blood where's my to window change what I'm connected to it's blinking too red like it crashed which it should have done but to is not seeing any know why that happens this is a bug that has been on here forever that I have not been able to figure out there is some timing thing or something on nrf where this auto reload is on it's like 16 characters gets sent back into the repel sometimes just just very weird and I don't know why it happens it's yeah it annoys me but yay we're getting our exception we're getting all channels in use this is what we want and this this is what the test for the five rotary encoders so I was in here so I figured I would knock that bug out too so let's um assume that it's all good and oh we're in commit amend fixes fixes this one 52 40 and fixes this one as well 52 11 and I think that's it it's those two I was looking through the nrf um issues earlier and I actually was able to close a couple other ones too I like this uh theme uh pier so yeah I was looking here so this is mr. doll guards one that will fix this one's weird and this is the one we'll fix too this is an interesting thing for Thomas who I don't think is on the stream today talking about how you switch between light sleeping and um counting which is interesting I think we may need to blur the lines a bit and maybe like you could sleep on key matrices or something because that's what it's you sure I think um this sleep thing I'm not going to look at rotary encoder is hard this is weird and I wonder if this is not I wonder if this is fixed I don't know there's lots of issues docusaurus with a few css changes the user button is of course trapped underneath the display yeah it's kind of under there okay um let's get undistracted and okay those are good ship it make a pr y'all can test it tell me I didn't fix it and in fact reviews would be really helpful because uh Dan and Jeff are on vacation next week so now's a great time to get more reviewers and let me just label it it's not beauty what am I thinking circuit python api milestone seven great um now if we look at our issues list again four seven we can see that we have five open three of them have poll requests uh one is for tax still and then there's this uh rp2040 does not release count i o pwm slice on vm reboot so okay I'm going to stop streaming but I might actually just hammer that one out now because it should be really quick um any other final questions before we get out of here I know it's late and we do actually have people drop off as it as we cross the two hour mark um James says we learned and enjoyed doing this so one day down the line someone will give a buck doc you saw us I have to look at that yeah walk is a good idea doctor I might actually do that too all right uh have a good weekend everybody uh as uh let me just recap quickly the the spiel um I'm sponsored by Adafruit to work on circuit python if you want to support them and by extension me please go to adafruit.com and purchase some hardware there um I think there are adabox spots as well so take a look at those um if you want to chat with me and a lot of others outside of the span of a stream make sure you're on the discord server by going to the url adafru.it slash discord uh deep dive is happen every week 2 p.m pacific uh on fridays next week will be on friday as well and I think that's it I like that doctor put a discord link in the discord um Keithy says james says thank you minnesota mentat near doc thank you all um I'll put the cat and get out of here thanks again and uh thank you to david dcd for taking notes uh I really appreciate it and I hope your vacation was good time to wake the cat up did you know did you know I was gonna come all right have a great weekend everyone bye