 Hello hello. Can you hear me? Can you see me? Am I too loud? Too quiet? Looks, it should be the same as last time. Cat is up. Maybe she's unhappy that I'm talking. We'll just get going here in a couple minutes. We'll say hi to folks who are in the chat. Sorry, I always get the wrong. Cat Kim is being finicky as always. We'll see if it works a bit quiet. We'll see if it works a little bit later on. I don't understand why it doesn't work. Who knows? Okay, a little quiet. Although I sighed and I got up in the red, so let's wait a little bit longer. Vin is like, why are you talking, dad? Hello Dave, hello Beata. Thanks for hopping in and watching the stream. Excited to take it going. I've been doing a lot of housekeeping because I was out most of this week. Do you want to go out, Vin? Alright, I'll be right back. Let this cat out. If you watched our streams before, Vin, one of the two cats, she likes to knock the garbage over when I start talking because she'd rather me be quiet. Alright, so we're just over time, so we might as well get started. Hello everyone, my name is Scott and I work for Adafruit on CircuitPython. Adafruit is an open source hardware and software company based out of New York City, but I've worked remotely for them for a while, since 2016, so seven years in the wrong channel. Hello Clever, hello Dishapu. Totally lost my train of thought. Adafruit's in New York City, they pay me to work on CircuitPython. If you want to support me, support them because they pay me quite well. And you can go to Adafruit.com to support them. Buy some stuff that you see here, all the little toys and gadgets, and a lot of them run CircuitPython and make them really useful. So devices are a two-part game, they're both software and hardware. So like I said, CircuitPython is a Python version for microcontrollers. It's based on MicroPython. It's meant to be really easy to get going and run code and make some microcontrollers and whatever is connected to do something. So hello Gerard and hello David. We'll get to the ESP32 stuff here in a little bit. I always like to take questions and go off-tangents as well. Hello DCD as well. But a bit of an update. I mentioned last week that I was looking for office space and I've signed a lease. I get the keys next week, which is exciting, so I won't be streaming next week. Fomiguy will be streaming in the spot instead, so keep an eye out for Fomiguy next week, 2pm Pacific. I think it does the same time here on Adafruit's channel. And then I'm hoping that by the next week I'm in the space enough that I can work from there. So it's like a mileage away from here. It's going to be a very different thing because I've actually worked from home since 2015. So it's going to be really weird. But it should be a good commute. I don't have to drive. I can walk it if I want to. So I'm looking forward to that and we're looking forward to getting this space in the house and fixing it up and then using it for other stuff. So yeah, these deep dives tend to run about two hours. I have a bit of a cold because my one-year-old's in daycare. So we may just go the hour. We'll see how I feel and how we're going. Are there any questions from folks watching before we kind of dive into what I had in mind? Is there another baby on the way? No, there is not. There is not another baby on the way. I can see why you would ask that question though. We might choose to. We haven't decided yet. Talking about the new office space, if folks have suggestions for maybe I could do green screen behind me but I could also do some sort of setup or sorts of stuff. If folks have ideas for things that should be behind me during a stream, part of the reason my brain is gone is not only do I have a bit of a cold because the last few nights I've been having trouble getting to sleep because I've been just thinking about all the things that I have to do for the office move and what I want in an office space. So it's been going quickly and I'm looking forward to being moved into the other space. David asks, on the I2S library, does it do both left and right channels? Yeah, so the audio system in CircuitPython should be able to do stereo. If you have a mono sample, it will just copy over to both channels. I2S will always have two channels but if it's a mono sample, it will do the same sample to both channels. Eric asks, how big is your new office? Good question. So I was looking for something about the size of this room, but maybe a little bigger. So this room is like 10 feet this way and like 16 feet this way including the closet. So this is actually like a bedroom. So there's a closet to my right here that has a bunch of electronics and old Game Boys and that sort of stuff in it. I've got like a bunch of like tons of Adafruit hardware, tons of DevKits, that sort of stuff. So this is about 160 and the new space is just about that as well. It's a little more square than this room, so we'll see. And it's also got a bit higher ceilings, so we'll see how that goes. Hopefully I can fit everything because I'm probably going to bring a couch with me. So that's going to be a bit bigger of a thing. Clever said I could do both, a green screen on a pull-down curtain like a projector screen. Yeah, I think a green screen was probably a good option. Phil PT, who does Adafruit, suggested the Elgato one that they use. So I might just get that just to have. We'll have to see how much I fill up the space. David says, do you need to duplicate the computer or will your remote connect or totally not work when at home? I think generally I won't work from home and if I do it'll be on the laptop because I won't have a space. I won't have, at home I won't have a monitor and keyboard and stuff. It'll be on the laptop. What about the cats and you're now off as Dave asks. The cats will stay here at home. So I think in the short term all the cat stuff will stay here and they can hang out in here. But we are looking like there's a roof leak here that we've tried to fix numerous times and we haven't managed to. So something that we're looking to do is just completely take all the drywall off so that we can finally figure out what is wrong and maybe redo the roof above this room as well to finally hopefully fix that. So the cats will stay here, they won't go with me to the new office space and it'll be a big shift for them. I think that's one of the main things I'm worried about because they are like nine years old so I've been here eight years. It's only been that very first year when it wasn't even a full year where they had the house to themselves. So we'll see how they do. It's been a lot of changes having a one year old around the house too. Ari got ran in here this morning and spooked his dad. It was this morning or last night so I spooked like hissed him and Ari was just like trying to pet him. Yeah, figuring that out. Clever says no more cat cam then. Yeah, I don't think I'll do. Yeah, we won't have cat cam. David similarly says why did you recently invest in a new cat cam then? Well, I bought a used cat cam and I didn't know exactly when I would be moving. I didn't spend on a really expensive thing. I will have kind of a view so I was thinking about instead of cat cam doing window cam, out the window, weather cam. But we'll see. I don't know. That's why I'm trying to rethink the setup on the sidebar here this way. Rethink that setup and maybe green screen as well. If I green screen it then we can do the other stuff. Besides that, I have a big desk here and then I have this desk behind me. But I do want to get another desk I can have kind of to my side to hold like debug probes and stuff like that. And the hover cam, like the top down camera that I've got. It'd be nice if I could do that kind of more permanently. So I think that's something I'm thinking about too. Yeah, be careful of the attic spaces and insulation above rooms. Yeah, I think we'll probably find somebody to do it for us. Set up a cat cam live stream on its own channel and then embed it. I think those probably already exist. I don't know if we need these kitties. These kitties here, although Spook is quite cute for sure. And that's what Dana is asking too. Can you do a remote link to cat cam for your office? So do you all just watch the stream for the cats then, I guess? Anyway, yeah, so it's going to be exciting. It's going to be good. Right now, Ari's daycare is a drive away. So we're doing a lot of driving back and forth today care. But when he turns two or is about two, we should be able to switch to a daycare that is walkable. Walkable or bikeable. And it's kind of like this new office spaces between the two. So there's a future where I'm able to drop him off and walk to the office and do all that sort of stuff, which will be quite nice. But right now we're driving. So we'll see. DCD says, I'm here for the technical content. All right. Well, let's take that as a segue. And like I said, find me on Discord if you have ideas for streaming setup and things like that. I was thinking about getting a second light, for example. I don't know. I might make the lighting a little bit less shadowy. I'm continuing to look at my camera or not at my camera. Yeah. So find me on Discord. If you want, if you're not on the Discord already, you can go to adafru.it-discord. And we have the live broadcast channel there. And then we also have CircuitPipe on Dev. Those are kind of like the two main places. So we've got a question. Retia says, I've made some deliberately vulnerable web apps in CircuitPipe for the ESP32-S2. And I want to eventually do the same with the S3 and have an app using Bluetooth to set up the vulnerable app. I found the way CircuitPipe then handles writing to the disk makes it hard to create a setup Wi-Fi AP where people can add their Wi-Fi credentials to allow CircuitPipe and Project to persistently join a Wi-Fi AP. Our expectation is that when people load CircuitPython, that's when they load their credentials. So that's why CircuitPython doesn't do the join my AP sort of thing. That's one reason. The other reason is we don't think it's a very good user experience because you tend to lose internet access when you do it. It's pretty clunky. So our focus has been when you get CircuitPython on it, you also provision the credentials. And if you see the web serial loader that's available on CircuitPython.org slash downloads, it can save the Wi-Fi credentials to your drive for you as well. So that's kind of what our approach has been to it. But if you have changes you'd like to make to CircuitPython to make it easier, we're totally open to that as well. So the thing that I wanted to at least start with, talking about today, but he says, I wish I could make an IOT setup page to configure once and let a Honeypot or Vulnerable app I'm teaching a lab on, lab work, lab on work after the first run, but I understand. Foamy Guy says, one option is you could store credentials in NVM storage. I've used that type of mechanism before for end users configurable Wi-Fi. Michael Bishop says, something I don't see people using enough, you can put a Wi-Fi name and pass it into a QR code, but then your device needs a camera and a ability to scan the QR code. Thank you. I was just wishing a self-hosted page could save something like credentials to the disk. Yeah. Web pages should be able to write to the disk, but like the web workflow won't start until you have those credentials. I've not done much access point stuff. David points out there's a tiny code reader for Musil sensors that reads QR codes and can be talked to in iSquared Z. All right. Let's, we'll keep chatting. That's no problem, but let's move on to this bug that I told Katani I looked at today. So here's the issue. It is issue number 8288 if you want to follow along. It's the Metro ESP32S3 watchdog timer expired safe mode when various demos. And you can see the different versions here. Here's a very basic code.repl, which seems kind of bad. So we'll try that. Here's the original demo. Here's the iSquared Z pin identifying script. And it also has the same issue. So I had a theory when I was thinking about this, but I was only thinking about the second example. This first example seems, doesn't, doesn't go my, go towards my theory. So I, oh, sorry. I'm a new parent. So I have a Metro ESP32S3 here. It's a prototype, but I wanted to see if I could replicate it. So I've, the first step to fixing this issue, and I think Katani's been hitting this a lot. So I want to, this is why I want to take a look at it and figure out what the issue is. So the background is that the, there's a couple watchdogs that run and one of them, I'm trying to remember. It's been a while. I think there's one that's interrupt. So like making sure that the interrupts aren't blocked for too long. And then there's a second one that is yielding to the idle task. If I remember right. So, so in circuit Python, we have to basically like give up our free RTOS tasks so that the ESP IDF can run other stuff. And if we don't do that often enough, it will raise this error. So that could be it. So the theory that I had. So the other example is like searching through all of the different I2S pin combinations. And in that case, you're going to be generating a lot of objects. So my theory that I'm coming into this with is that somehow on this board, which has eight megs of RAM, it, we're hitting, we're actually filling the, we're actually filling up the heap. And we're doing garbage collection. And when we do garbage collection, we're not yielding back to the free RTOS enough to satisfy this like internal watchdog. So that's my theory. But this first example that fails doesn't match that theory. So let's see if we can replicate it. I've got what I did before is I did get a new branch going. This is up to date with the latest circle Python as of today. You can see I just this new board from the shippu I just merged in. So we've got that other things of note is that. So this is not the idea five update. This is with idea four still. And it is this is this has the micro Python 119 in it. Now, I think, you know, catney might actually be working on. So this 756. Oh, see, see 756. It's possible she's on the hello unexpected maker. Yeah, we're, we're trying to hunt down this watchdog timer expired safe mode. Problem. I think she may be on the eight to X branch actually. Let's just take a quick peek here. And see if we see it. See those commits on the eight to X branch. So one was C 75756 is in both of them. So there's one. Yeah, so she's been doing these like pin definition updates. So it's possible it's possible we won't we won't be able to replicate it. On main, although I don't think main has changed that much. So we should, we should just start it. Let's just see if it, if it, if it has a problem first. So I'm going to press the boot button. And then the reset button going to make flash. So this is not a debug build. First, we're just trying to replicate the issue. And then once we replicate it, we'll start debugging. If it's not present on eight to four, then is it a problem? Does it say it's not on eight to four? They can't even say that. It's not clear to me that it's not on eight to four. Eight to four would be somewhere near the newest branch of. Hey DJ Devon three. Okay. So we flashed it. Now let's hit reset. I don't even know what's on this board. And we got the boot loader. Metro S three boot. Hmm. I wonder what that is. This should have copied over all of it. Let's try it again. Test on the stable release first. Yeah. If it, if we can't replicate it on main then we'll try it on, on the other branch. I don't think a lot has changed though. Huh. I don't know why we're just getting the boot loader. Let's do a clean. Maybe we need to wipe it. It should, it should be able to find the like second. It should be able to find circuit Python. This is how I usually do ESP testing when I'm working on it is by using this flash command. So I'm looking in the make file just to see the commands are off set. Yeah. See, so flash should be. Yeah. It should be doing both because there's also this flash circuit Python only. So it should be flashing everything. I was going to say it's been a minute, but I picked up a Metro S three and 16 are eight. So ends flash are as Ram. Like that has a built in SD card and gobs of Ram yum yum yum or Ram. Have you seen this issue at all DJ Devon three? Have you seen this watch dog timer problem? I haven't seen it, but I've also not been doing a lot of testing. Most of what I've been doing is either IMX work or ESP compilation stuff and configuration for the IDF update, which I set myself. My goal is to get a PR out for the IDF five by Wednesday at the end of Wednesday next week. Thursday is when I get access to the office and I think I'm going to try to move all that like over the week over the long holiday weekend. It's a holiday weekend in the U.S. And it's flashing and then it stays in the boot loader and we hit reset. Those two still. Hmm. It's not working. I wonder if I have a different. I don't have a production version. It's possible. That's the problem for me. I think it should be. I wonder what happens if I just copy over the because it should have built a U of two as well at your last three or bin Elf. Hmm. It did not build a U of two. Oh yeah. Unexpected maker you came late. Yeah, I'm moving my office out of the house so that we can fix up this room and use it for other stuff. I'm going to release and I get in there on Thursday next week. I pay for one one day's worth of August. They build me. They build me for August 31st. Okay. Why aren't we getting a U of two? That seems very strange to me. But oh I guess that's because we're doing the flash target. If we do the regular target, it does the U of two. Okay. Let's copy there. This used to work and now it doesn't. Okay. So now it's copying over. I can see the Neopixel flashing. Never used watchdog timer. I did see the reported issue on it but have nothing to add. Yeah. So this doesn't require you to use the. Yeah. See it. It just went right back into the boot. Is it because I'm I do have this connected already. I'm only getting the boot loader. Huh. That seems strange. Let's do debug. Let's do a debug build and see if we can't get some outputs. Reset us back into the ESP. Hmm. Hard faults on the matrix portal S3. Hmm. So maybe the next thing we do is we switch to the eight to X branch. Because maybe my stuff is not working for some reason. Like maybe it's been fixed to there but not on main. So eight to X got merged in the main not too long ago either. We can switch if we if we want to see. It's possible this was this is a separate problem that was just broken. Main is pretty main is where we're starting to work on 9.0 and we have turned that corner because Dan and Jeff did the 119 micro Python merge which is great. And we want to do the 120 merge before we start like calling it 9.0 officially because the MPY versions will change with the next merge as well. So if you are using main right now you'll have to use the Python version of libraries. We don't have MPY files currently for main because it's going to change. Otherwise once we're settled on the MPY version we will do all of the tooling for 9x stuff. Hmm. Hmm. We're flashing debug and we'll see if we get some output. We should. It's labeled TX and the IDF upgrade too. Yeah we'll do we'll do the IDF upgrades in 9.0 as well but it's not blocking us like making all the MPY stuff. Like the bundle and 9x bundle and stuff that's blocked on the micro Python stuff not on the IDF. The IDF upgrade should be more internal than the micro Python one. Hmm. So we're still getting the boot loader and we're not getting we're still not getting output. So let's do ESP tool S3. I want to do I want to obliterate it. Now it doesn't feel like to me doesn't feel like it's sticking. There are a few warnings during the build process does your CI process force warning clean commits. Um these warnings are tend to come from the IDF not from circuit Python. So we end up being stricter about circuit Python code than we do about the library code that we're using. DJ Devin 3 says I'm generally excited for 9.0 but it's going to get messy on the way there. Yes it will. I selected the wrong device. We want the expressive JTAG. Alright we'll erase the flash. Now we'll do the flash. And if this doesn't work then we'll try doing A2. There we set again. Oh there we go. So I got um I'm getting neopixel blinks like it's circuit Python now. And there's the circuit py drive. It took a while to pop up. And we're still not getting any output on here. Did I not connect it correctly? I think it's correct. Although who knows if these these wires that I've got going to it are good enough. It should I don't know actually I don't know if this board during debug will automatically do. So A root metro S3. Neopixels working. Double tap. So one area of optimization is I think a lot of well this is for flash mode. But I'm not convinced that we've actually optimized the flash settings. The flash and RAM settings for a lot of these boards. A lot of the ESP boards. Where do we do the... So this is 16 megs of flash. It's a it's an N16R8. Takes a while to boot and appear with a 16 megabyte of flash. Yep. I think that's part of it. And that could be the source of this problem too. Like something some sort of housekeeping stuff is taking forever. Okay. Why aren't we getting... Because we're not... I'm expecting to get debug logging out. The IDF has untracked content, but it looks pretty clean. What board... What other boards would have this? Ah, here we go. This is what I'm talking about. Never reset these pins. 43, 44, S3. Oh, I should have copied more of it. Yeah, I wonder why it takes so long. Have you investigated that? Just in the nick of time. I would love to fix this issue. Staying in bootloader. And we're still not getting any UR output. Here we can see is... Oh, TX and RX are 40 and 41 on 43. Oh, there's debug RX and debug TX. Where are those? Oh, you know what? I think those are... I'm looking at the board here. And I have an older board. I'm plugged into the wrong place. Let's pull up... What's new? S3 Metro. Yeah, so I have older silkscreen. But I'm connected... Currently I'm connected into this TX here. And I think I should be actually here. There's this little debug label there. So if I just do... I don't have a great way to connect to it. And this is something that I'm going to have to figure out because it's like... I need to solder onto those things, but in my new office space I won't be able to solder. I don't know which one's which. Great, now I'll just hold it. Ooh, actually... I can bring out my big stick. This thing. Yeah, I could take the solder projects outdoors to do minor changes. Oh, you know what? No, it's not, David. I know what you're talking about. There's one board with a switch. For switching it, but I don't think it's this one. Yeah, so I'm just connected to the wrong pins. So let me... See, this is what I'm thinking I need a separate thing for, is that if I had a separate desk, I would have more space to set up this jiggy thing, and then I don't need the solder to it. Sorry, you can't see. That's another thing I'm thinking about. How you'll be able to see. We have error messages. When I brought it up ages ago, when we were seeing the double starter circuit python over mounting, I was just told it was something OS related. So I never looked further. Yeah, let's just take a look. So this is TO modes. So these are timestamp modes. And it's super handy because now you can see how long it's been between messages. You can see the timing of... These are prints based on the neopixel flashes when it configures the pin and when it de-configures the pin. So you can see the amount of time that is happening between the flashes and how much time is spent kind of flashing. Which is pretty neat. So maybe what we do is we'll just hit... I'll just hit reset here so that we can get all of the messages from the start. And then we'll just take a quick look at the timings of it. So this is the bulk reset that circuit python does. This one second is probably the one second safe mode wait that circuit python does for all boards. Do not use for mass production. Timing parameters will be updated in future IDF version. Cool, cool, cool, cool. Yeah, and so this 2.3 seconds is not... Is just the difference when I press the button. So I don't actually see... Like this bi-ram test took almost half a second. I think this one second is probably the safe mode blink. But I don't see any others here that are significantly... That are long. Oh here's, you know, this 5 seconds is just the blink. Is reset the same as power cycle? Maybe not. Should we try a power cycle? There's a... So I unplugged it. It still seemed pretty quick. It was just like the first time after the flash seems pretty slow. And like if you erase the flash and make a file system on it like it does have to do like add all the fat FS housekeeping stuff onto it. So yeah, I don't immediately see anything of note. So let's go back to the original problem and we'll keep an eye out for any slowness. But Katni says that this basic example causes this internal watchdog expired timer. Which basically means it's getting wedged somewhere. So we'll mount. And the mount takes a while. But I feel like that could be like the host computer takes some time to... The host computer takes some time as well. Ah! So I think we already... We just hit the restart that Katni saw. Like here's the PSRAM warning again. Yeah, and you can see that the both CPUs were reset by the watchdog and this TG1 watchdog sys reset was hit. So we are able to reproduce the problem that we saw. And look here, we have a saved program counter. Which must have been the program counter when the watchdog timer occurred. Maybe? So let's just take a quick peek at the... So this is me pulling threads and seeing where they take me. So let's open up the build directory and we'll look at the firmware map. And we're looking for 4037. Which doesn't... Oh no. So basically what... Like we're getting stuck somewhere. And the question is... So that's very weird. So I don't see... Oh! 4020. Yeah, I don't actually see that there's any code at 403. Let's look at the linker script. Which I guess we get from... Which is annoying. IVF. Did we get the linker here? So I don't really trust that. I've got my ways. The circuit by then do any JIT. No. It's not that smart. I don't think. Cool. Well I'm happy that we're able to reproduce it. I'll tell you that much. So... Let's... We're off in the weeds. I don't trust that. So let's figure out where we are in circuit by then. So... I suspect, and we'll just do that by adding ESP logs. The code.py here is this digital in-out creation. Which should seem straightforward. And it should work. It's a very strange... This is a very strange issue. But it could be like some sort of memory balance thing. So common how... Digital in-out. You know this could be some sort of like cache issue. So here's the construct call. And this is what happens. It's been a while since I've done a... Is that what it is? I think so. So I just press the buttons to enter the bootloader again. I'm only changing circuit by then files at this point. So I can just rerun this. And reflash. And there's the output we're getting from the bootloader. The ESP bootloader. Oh, yeah, I have obs jump. I'm just not using it. You would have to know which one to use. Like there's an extensive version, but there is obs jump. I could also try to do like address to line. Okay, reflash. So let's start up again. Looks like we're boot looping. So when we boot loop, we'll just get it back into the bootloader. Reserving zero bytes of ESP. So we have to remember that dioconstruct... I don't think it's getting that far. It could be an issue. So here's the load and here's me pressing the button. We go along. We do the dioconstruct, but those are not... These are dioconstructs for the blinking. Like we're not actually into code.py yet. Code.py yet. And then we get to this. Reserving zero bytes of PSRAM. And then we hit the watchdog. So let's add another... It's called this dio. And then let's copy this. So the other place that it's doing it is busio. And it's spy. We're doing spy construct. I don't think we shouldn't be... So what I'm doing is I'm trying to figure out where in the flow that the user code prompts are we getting? And basically follow that down until we figure out what is. So the watchdog issue is the problem that you're getting into somewhere that you shouldn't. You're getting... You're not going to the code that you need to go to. Or you're hanging basically. You're preventing everything else from running. So this tag thing is like the prefix that you get like the I2S or ESPIDF is the tag is. It allows you to like filter to see messages only in a particular module. My assignments are getting stuffier and stuffier. They were actually pretty good. I went for a run. It's pretty clear getting gummed up again. Okay, so we've loaded and now we're boot looping which is interesting that we're boot looping because we're not... The next time we start up as in safe mode we shouldn't rerun the problem. We shouldn't rerun the problem code. So we got construct and construct done. But we didn't get to the digital TG1 watchdog timer. Like there is the earlier DIO construct. Let's look at the IDF source or the docs. This should tell us which watchdog is used as what. See there's like this int watchdog timer and this task watchdog timer maximizing execution speed. Watchdogs, this is what I want. Make sure the free R test task switching interrupt isn't blocked. Runs in an infinite loop with interrupts disabled or hangs in an interrupt. The default action of the interrupt watchdog is to invoke the panic handler in timer group one. So TG1 and it will be the display. I thought we should have the panic handler on there. Can you disable the watchdog and observe correct behavior? Good question. The other thing I've seen is that you can actually change the timeout on the watchdog. The task watchdog timer I don't think we have on. I think we turn it off. Hi, Dinkleburg. We're not getting the panic handler. Jacques Wilson says, thanks for doing this. Sorry, I joined late to get me up to speed. What program slash problem are you trying to troubleshoot? So we're doing circuit python. We're on the main branch and we're running this code. It's a data fruit circuit python issues 8288. And we're running this seemingly simple code and it's causing a watchdog timer to expire, which is weird. We're trying to figure out why. So let's just see. Let's take DCD's suggestion and component config. Did I mention how I really dislike cake and fin? See, panic handler is supposed to print registers and halt. But somehow we're screwing it up so much that it's crashing. So here's the interrupt watchdog. And it's got a 300 millisecond timeout. So here what we could do is we could just say 600 or 1,000. So let's set that to 1,000. Let's place the panic handler code in IRAM. Maybe if we put the panic handler in RAM, we'll actually be able to run it. So I think that works. Now I'm going to do a clean because if you rebuild, it won't rebuild the IDF code. But the SDK config can impact it. No catcams now. It's just watchdogs. Is catcam working? It is. It's giant. Whoa, I don't know why it's so giant. Resolution. It's an HD. How big is it? Good. I was hoping we would have a catcam for the final stream. The final stream with cats. Catcam is big? Yes it is. Oh, and it's frozen. Brilliant. It's big and it doesn't work because I moved it and it did not move. Okay, well, back to the desktop. We'll check on catcam later. Can't say I'm too sad about it. Honestly. If I had a dog, would you call the cameras watchdog? Okay, so we're flashing. We'll hit reset in just a moment and hopefully we can get the panic handler because that would make things simpler. If we could get a back trace, that would make it way simpler. So it looks like it worked. Oh, nope. I jinxed it. I just pressed the button to stop it from resetting. So here we can see that they were reset by watchdog timer. Do you like boot mode is in dual? So there's potentially a lot of optimization we could do. So by flash has modes of how many bits come in at once. And DIO has two lines. QIO, quadIO will be four and then octal would be eight. And right now it's actually booting in DIO. I don't know the history of watchdog. Okay, so we're still having this problem. And it is interesting that it's three seconds from the time that we do construct done to when it resets. We set the watchdog to be a thousand, which is a one second, but it may actually be like trying really hard and doing multiples of that. So let's just add some more logging and see if we can't, you know, find how it gets further and further and further. The next place I think I want to log is in shared, like we should be getting this DIO construct because I think that last line is needed to reproduce it. So let's go into shared bindings and just see if we can get some more. Should be QIO for runtime flash use, DIO for when you're actually flashing the chip. QIO for flashing can fail quite often for some reason. Yeah, and I think that's why we're on these like safe settings. But there is totally room to optimize, I think, the actual runtime. I just don't know, like nobody's actually done the work. So in digital IO, digital in out, there is a little code that runs before. So construct is where we have a print. And let's just do another, let's just do a ESP log tag. I also want to do, I'm going to do two burns, one stone. I'm going to do bus IO as well. So one thing that's weird about the bus IO initialization is that it's using the standard pinout, which we may have to, it just returns. The question is where is it? So we've got a couple more logs here. We could be in USB? It could be a USB issue. Actually, that's probably worth testing to see what happens if I disable the USB bus and then do it. I've got it active because I'm flashing the device over it. But I think we did see the timeout on, we saw the timeout even when safe mode was active. And that's when it doesn't, safe mode doesn't road code.py. So maybe code.py doesn't actually matter at all. It could be more of a like issue with the flash. So it could be like flash reads takes so long that we hit the, flash reads takes so long that we hit the issue. But maybe it's, so this is the the ESP chips are very similar to the IMX and that like code is also, there should only be the one watch dog active shouldn't be too. So let's reset and see. There it was again. And we did get the construct done construct done, but we didn't get that. And it's three seconds still. Let's do the test to see if we get it without. So I have these cables that have the switches on it. So I just hit the switch. That means that it won't enumerate on USB. And I'm just going to hit reset again. Yeah. So it's still, it's run code.py. It's still hanging. Now it's doing the blink and it's okay. So it is code.py related. But what is it doing? So it could be background task related. So that will be my next guess. So like, you know, we're looking at the code.py. And we're basically saying like, this is complete. This finishes, but we never get to this next portion. We know that because we don't get those prints, although there is a little bit of buffering that happens and maybe that's the issue. But let's just do a quick, let's go to background tasks and see if we are running background tasks while this happens. So background tasks are like housekeeping sort of things. Like filling audio buffers is one example. Rendering display, display is another. So it's possible that we're there. Maybe I need to try to JTAG connect to it. Maybe that would actually help me figure this out. Background callback run all. It's possible that like we're blocked waiting on some sort of lock. This will happen probably a lot. I don't think that PC is valid though. It went off into the weeds. This is why I want trace stuff. Thanks Dave. Have a good night. Okay, let's see if we're in background tasks. So what are we going to do? I got to turn my USB keyboard back on. I'll flash again and I only have to do a partial rebuild because it's circuit pipeline. Do you have an MMU or MPU? I think there is an MMU, but I'm not sure. A lot of that gets taken care of for the forest by the IDF. That is more promising. That looks separate because we're not printing out. But abort was called at a 403 program counter. Yeah, I don't know what's at 403. It might be ROM actually. I could look at that. So this is good to document. So if we do this, we copy this back trace and we do Python tools, decode back trace, data fruit, Metro S3, we paste that in. Hey-oh, it gets corrupted. But it is a lock acquisition that is aborting. I have a memory map. I can look at it. You want to fix this for me? Go ahead. Right. So this is a problem with a lock. And it's getting squashed. And then there's this 40050965 address. Yeah, this Python script uses address to line to decode the back trace. But 4005 I don't think is in our memory map or in our memory map files still, which could again mean that we're in ROM. It's so weird. It's so weird. This is a weird bug. Metro S3, firmware health map, 4038E137, 4037. So the lock stuff is in IRAM. No, it's earlier than that. IRAM 070400. There is DROM, linker script in the memory map, 307A. But this 4005 is still weird. 4005 is lower than that. We're executing code from IRAM. All the panic stuff is in IRAM. The stuff for it is none. This is why I don't like other people's linker scripts. It's all this linker magic for two. Well, address to line couldn't find this one either. And we panicked even earlier. I think I want to just run it again and see what happens. I mean, I wonder if these PS RAM, is it the PS RAM problems? Timing parameters will be updated. Is that possible? Hobbs jump T, easier to read. Yeah, bug in the back trace code. I can't unwind the stack. I wonder how we... You're assuming, Diggleburg says, I wish I could look at those lines of code and understand right away what's going on like you guys do. No, I don't understand what's going on. I wonder how we change the PS RAM. Do not use for mass production. There's a warning there. Maybe we can find more info if we just find what this warning is. Do not use for mass production. Spy flash timing tuning. PS RAM 351. IDF. Maybe this will tell us what we actually need to use. IDF components. By flash. By flash timing tuning. 351. Here's the warning. Do not use for mass production. Needs tuning. PS RAM needs tuning. OPI RAM, PS RAM. Hardware support. Maybe this is what we should just... Maybe we had to turn off auto detection of PS RAM 265. PS RAM enabled. This just calls it. This thing says do not use this. Instead, we could... Let's see if they've gotten rid of us. Let's look at their 4.4 branch components. Interesting. Here, they no longer have that warning. If you were to bring a personal project to live to life, what development environment would you personally prefer to use for the ESP32 platform? This is kind of the best way I know... This is the best way I know how to do ESP32 stuff. I wonder if we just updated the IDF. PS RAM access failed when DFS is enabled. And timing tuning. Remove 80 MAG DDR warning for 4.4. Ah, so this removed the... It moved the warning. And basically, if you clock it at 120 or 240, it'll warn you. So they removed the warning, and then they added flash timing tuning index, and then they also say PS RAM timing tuning index. So it will output... Let's just update the IDF. It gets a little hairy because we have some fixes that we carry, which is annoying. This is a suspect. Actually, let's see. Let's see what this changes when DFS is enabled. Update IDF to 5.1.1. We're getting there. This is not... I'm talking about updating to newer 4.4. Although you know what? Socks, bi-mem, support timing tuning. We're still hunting. Right now, the theory is that our PS RAM settings are causing data corruption. And there's this bug fix that is ESP32 S3 PS RAM access failed when DFS is enabled. I don't know what DFS is, unfortunately. Maybe it's like dynamic frequency tuning. What do we think DFS is? Merge requests. Dynamic frequency scaling. Yeah, that's a good guess. Dynamic frequency scaling. We're sleeping. This sounds very suspect, right? These changes seem quite relevant. Fix PS RAM timing tuning bug. I'm not sure what of these we have. Maybe let's go... We've got 30 more minutes. I think I'm... Can I turn off the timing tuning in menu? I think I can. I think I could. Yeah. Maybe that's what I need to do. Is that it's messing it up? I really get confused by... You don't manage this well. What branch are we on? I don't know. I thought it was this. So we haven't updated it since January. 630. So we can go here and go to ESP IDF. Yeah, we're on 630. So we haven't updated IDF 4.4 since the start of the year. So we have eight months worth of fixes that we don't... So let's... Expressive. Expressive. That will probably take a while. Courtney says, Maybe my question is not familiar to the best. I saw the circuit by the repo and personally I'm only familiar with C in the IDF framework. But I know a college that is working with MicroPython and I understand why it exists, but would you prefer it for personal projects? Oh, would I prefer a circuit Python over C? Yes. I think... Yeah, I mean, I have a bit of a trap personally for using circuit Python for personal projects because I always find bugs and I end up just debugging circuit Python. Which is like my job. So a lot of my personal projects end up being kind of further afield and more Python-y than circuit Python-y, unfortunately. Because anything has anything to do with circuit Python, if it's a project I can basically call. I can call it a... I can call anything related to circuit Python work. Fusing. I think, like, not only is the development workflow for circuit Python much easier, but also, hey, which applied without complaining to me too bad. It is updated in it. What is the problem here? Oh, my init's in the wrong place. All right, there we go. Okay, so we've updated our IDF. So let's... Let's try it. Maybe it was fixed. So we're going to do a clean again. I will get a main boot litter motor already. The other thing that I was going to say why I like circuit Python is that it's portable. So, like, a lot of stuff that you write can be applied across device families, not just one particular one. So, like, your code defetched your... Like, I have this inky frame here, and it's, like, it's Wi-Fi code, and an init code, and it's running on a pico... pico w, but, like, the APIs are the same, so if I wanted to move it, you know, say, an init Wi-Fi board with ESP came out, like, I could just move the code over and just be off in a way. So I generally do, like, you know, I believe in the circuit Python way. And I would argue that that's probably, like, circuit Python itself is a bit stricter on APIs, so it's easier to do that with circuit Python than it is with micro Python, but only barely. And, obviously, I have a bit of a bias as well. All right, so we'll see if this fixes it. That'd be cool. And this should also include the tuning output, so it should tell us what settings it's going with, which would be neat, too. This is kind of a long shot. It is totally possible it's our bug. But I don't think many of our ESP boards use octal... octal RAM, and I think this, or a flash, I think this one could be a bit unique in that regard and explain kind of why we're seeing this issue here, but not on a lot of other boards. But, yeah, I think we could turn off timing tuning in the menu, too. We could just turn it down and see if the problem goes away. No. We, so Jacques says, I might get me banned from this channel, but what you say is the pros and cons of CircuitPython versus MicroPython. Also, micro.server and MicroPython will run on the same... I assume CircuitPython. We won't ban you. Generally, the Adafruit philosophy is that we don't care what software you run on our hardware. Ultimately, at the end of the day, Adafruit makes money from selling hardware, so if you find MicroPython on our hardware to be awesome, then that's great. And CircuitPython wouldn't exist without MicroPython as well. But we've done a couple things specifically with CircuitPython on top of MicroPython to make it better in our opinion. One is that we are stricter about our APIs so that you know that the code will work across ports. There's a technical reason because we factor out that API layer and MicroPython doesn't. And then in CircuitPython, we've really put a lot of effort into what we think of as the first five minutes. So this is... And so our workflows are pretty good, I think, too. So I don't trust this flash because we're seeing the same error. This is interesting. It initialized caches in normal one core mode. I imagine there's a world where why is it in one core mode when we're on an S3 and we have two cores? It seems bad. Yeah, I thought that we were removing the version, too. I'm wondering if I didn't flash everything successfully. Like, I did a clean. Does it show me the version of ElphileShaw? IDF630. So this doesn't look like flashed successfully. Or I updated the IDF in the wrong... Yeah. I'm in CircuitPython 8. I'm in the different directory. That's super annoying. So I think what I do is I go in here, check out. I have to do the same thing because apparently sub modules are different. I don't feel like my shell history has ever worked the way that I wanted to work. And I've tried to figure out how to get it to work. All I want is a search, I think. Get fetch on shallow. Whoa! I just got a notice from the daycare saying, I thought he just woke up. He slept two and a half hours. Okay. I'm in the same thing again. In a slightly different... But this merge worked out pretty well. So we didn't put that much energy into it. Yeah, one core mode sounds suspect, doesn't it? Control. Like anytime you're passing buffers between the two that could cause problems. Which would happen with Wi-Fi stuff, but I'm not using Bash, I use Fish. Okay. DA18. Clean. And it doesn't work. Great. I have a love-hate relationship with. Generally, we've skirt around vendor. Oh my goodness. How much did they change? ESPBT. I'm getting deja vu here. I hope this fixes the issue. I just... We know what I could do is I could just don't have to fix it properly. Deactivate didn't do anything. It's supposed to deactivate your Python virtual environment. Which I know that the export.Fish sets up, I thought. That's why I tried to get myself out of the virtual environment before getting into another one. But apparently they don't set it up that way. They have to do it differently. Waiting for download. I hope this fixes it. A lot of people going back to the micro.server thing, like, I know that's a Python thing, but a number, a couple people put a lot of effort into HTTP server stuff as well. Okay, well this explains what we're looking for. It is an ESPBT, but we're not on test IDF size. I guess the expedient thing is just to comment it out. It is just me or the screen freeze. I'm just taping. Okay. Yeah, I thought about turning Bailey all the way off, but I got lazy. Path of least resistance. Oh, I changed too much. Then I have to recompile the IDF too. I generally only, like I have a 30 or 16 core Ryzen here, but I generally when I'm building the IDF, I only do a single core. Because I don't trust the make files to actually concurrently build correctly. Because we have this weird hybrid of like make files plus the ninja ESPB IDF libraries. Okay, so we still aborted and we corrupted a very similar place. So it didn't fix it. Let's verify that we are IDF 445, DA181. So we are on the newer IDF. Cache is in normal one core mode still. Which again, sounds bad. Here's the timing tuning index 5. Have you found using multiple cores compiles incorrectly? Oh, for most other builds I'll use all my cores. It's just the IDF one that I've had problems with. It won't like destroy anything. It just won't work. Or like, yeah, the build won't succeed if it's in the wrong order. Cache is in normal one core mode. That is very suspicious to me. Let's do some menu config digging and see if we can see cache config. I don't know why. I think this could be a rabble. I don't know what this tracks tracing feature is, but that's the sort of thing that could 240 cache config. Here's spy ram config. Let's look here. Auto attacks. Cache fetch instructions from spy ram. Allow external memories and argument to task create static. Ram clock speed 40. Integrate RAM in memory test. Alright, let's just turn it down and see what that gets us. I didn't see any config for the let's just turn it down and see what happens. Is there a text file that menu config uses that is easier to search? Menu config is derived from a bunch of cache config files scattered across the entirety of the project. It's something that I've been working with for the IDF5 update. And then the outputs get written to the SDK config, but that only gives you the config line. It doesn't give you the descriptor stuff. Jacques says, I ran into problems with HTTP server with MicroPython. Is that something CircuitPython addresses with your references to Adafruit's work on the APIs? Thanks for not banning me, by the way. We ban people, but we ban people for bad behavior, not for asking reasonable questions. I've only really watched the HTTP server stuff, but it's like Adafruit Python HTTP server. And folks have put a lot of work into it. Maybe Fomega will cover it next week as well. But it's not really when I'm talking about standardized APIs, I'm not really talking about that because I'm, for standardized APIs, I'm talking more like hardware interfacing like iSquaredC and SPI than I am for then like like socket level stuff. All right, any final questions? We are running up against our 2-hour window and I really have no idea how long this is going to take me. It's great that I can reproduce it but it's very weird. It's a very weird problem, so yeah, maybe it'll just work here because we slowed the flash down. Nope. We still aborted from a correct corrupt stack trace and it's even happening earlier in our code that it used to so it does do a memory test that took almost a second. That's interesting. So I don't expect the makers watching. That could be part of what slows startup down is the SPI RAM test. This 7 seconds is between waiting for download and repressing the button. So that's not interesting. I suspect this factory will give us the same thing, but we could just double check. Yeah, it's like in lock acquisition. I wonder if it's related. I wonder if it is this. So we have this callback critical. I wonder if we can't log during critical stuff. I wonder if we cause ourselves the problem because before the logging was pretty generous. I bet you that's what the actual backtrace is. Is there a lock logic errors? Probably. And that's what I'm thinking with this logging is like maybe the logging is causing lock contention like under the hood. There's also log early which may not use locks which could actually give us more information. Let's just see what happens here. It's still going to crash. So we still had the problem, but we had the original problem. We're back to the original problem. We didn't log the background. Yeah, so like spy bindings construct done. We're not in background tasks. You can see here that this is where the background is. The GitHub issue is number 80288. UnexpectedMaker says it generally had the men test off. Maybe not in CircuitPython. Yeah, it looks like it's still on. That might be part of it. Corne says, and here I am still needing to wrap my head around that it is possible to use high level programming languages on embedded devices. Really awesome what is possible makes it so much more accessible. Yeah, that's the goal and, you know, MicroPython deserves a ton of credit for making that a reality. CircuitPython really builds on the VM that MicroPython created. All right, well, I don't know what to do. I might keep poking this. It's also Friday, so I may not. But I think I should wrap up the stream. So let's do wrap up. If I thought I was close, then maybe I'd keep going, but I have no idea. So Catcam is still still frozen. Still frozen and in my way. All right. So hello, or hello, goodbye. This is why I'm not going to make any progress. Goodbye, everyone. Have a great weekend. This has been Deep Dive with Scott. I'm Scott. I work for Adafruit on CircuitPython. If you want to support me, support Adafruit, they pay me to do all this stuff. You can do that by going to the URL or to adafruit.com to purchase stuff. You can join us on the Discord chat, which is the middle chat here. Anytime. That doesn't disappear after the stream goes away. You can go to adafru.it slash discord to join that. Next week, I will not be streaming, but Tim Fomegai will be streaming in this slot. Tends to do deeper dives in the library stuff, which is also super cool. And hopefully I'll be back two weeks from now in my new space, which will be cool and it's going to be a lot of work. I have a lot of stuff here to go through and to bring over to the other space. But yeah. Thank you all for watching. It's been great. And we're working and maybe I could let's do one more unplug. I kind of want to sign off by giving Spook some pets here. So we'll see. Nope, it's still frozen. I got so excited when the image showed up. I don't know what the deal is. Sorry, no cat cam. Good riddance. Good riddance cat cam. Anyway, I suspect this logging panic is actually due to this print. But yeah. Welcome to the weeds. Welcome to the deep dive in CirclePython where we don't always find our issues. But it was nice to be able to reproduce it. Issues that you can't reproduce are just extra impossible. So let me just take one the last look. Mike Jones says high level programming languages is why I don't understand anything when I'm doing. I'm enjoying actually turning on the timers and setting things up manually and now looking at how CPUs actually work. That is totally, I totally agree with you. That's exactly why I like working on CirclePython is because there's no OS below us. That's pretty much it. And we have looked at or we've started to allow you to do memory map access so you could actually hit it straight from Python too. So that's interesting to you. Take a look. We have it in a couple ports but we can add it to more. It's pretty straightforward to implement. Anyway see you in two weeks hopefully assuming all the internet and moving goes well. And I'll see you on the discord before that. Thanks everybody. Have a great weekend and we'll see you next time.