 Alright, good afternoon. How's it going, DJ Devin? Thanks for dropping the link there. Can use you for folks to get in if they're in the chat here. And let's see. Browser still. Get comfy here. Browser. So welcome everyone. My name is Tim. For anybody that might be new, I go by FomeGuy on GitHub and Discord. This is the DeepDive, which is a weekly live stream program where we are diving into CircuitPython. It is most typically Scott, the lead developer of CircuitPython that does this stream these days, but he is out for a couple of weeks and so I am filling in here. So if you are new though and you don't know really what I'm talking about, let me take a minute to give you kind of the high-level summary of what's going on. So we're working on CircuitPython stuff here. CircuitPython is a version of Python that runs on tiny computers called microcontrollers. There's a bunch of pictures of those here. Basically, these are little devices. Most typically, they're going to plug into your computer with a USB drive. They'll show up like a plug-in with a USB cable, and then they'll show up like a USB drive on the computer. You open that drive, you modify a code.py file on there, and then when you save it, the microcontroller, the computer that's on these little boards, will actually interpret and execute that Python code for you. So that's kind of the high-level... Let me check something here real quick. The high-level version of what's going on. Let me do this also. There we go. It's an open-source project. CircuitPython is, so anybody is allowed to make projects with it. Anybody is allowed even to make their own hardware and add support to CircuitPython for it. The company that pays for it, though, is Adafruit. This is their website, adafruit.com. So they're paying the team who works on the project. Scott, the lead developer, like I mentioned, as well as a couple other folks are paid by Adafruit full-time to work on the project. And there are folks like me, a handful of us who are paid to work on the project part-time. You know, both doing codes and reviews and live streams like this one, plus projects and maintenance and all kinds of different stuff goes into it. So a huge thank you, of course, to Adafruit. And if you want to help support CircuitPython, one of the ways you can do that is by just heading over to adafruit.com, purchase hardware from them. They manufacture hardware here in the U.S., based in New York is where they're at, and they have many of the microcontroller devices like we talked about that run CircuitPython. They also design and manufacture lots of different sort of plug-in modules that you can plug into your microcontroller to give it access to different stuff, different sensors, inputs, outputs, all kinds of nifty different things. So head over there, get yourself some toys. Thanks again to Adafruit for making all of this possible, and thanks to everybody who supports them by purchasing hardware. Now let's see, how's it going, Beata, over in the YouTube? My friend, nice to see you. So today's, ooh, I did not mean to do that. Okay, one second, let me... One second, hold on, we need to get that back. My browser. This is also not the right one. Oh, I got it somehow. Oh, thank goodness. Okay, sorry, this is a side from the topic at hand, but I accidentally closed the wrong browser, which had a bunch of tabs that I kind of would like to keep open, so I'm trying to get it back open. I was able to make it, and this, boy, this cat here is like running back and forth as fast as he can tearing ass around. So hopefully he's going to chill a little bit before we get too much further into this. Boards, boards and add-ons, galore, and CircuitPython. Yeah, for sure. How's it going Dexter? Nice to see you as well, my friend, and the Shippu too. Nice to see you, everybody. So the specific thing, the specific topic at hand tonight is actually the same thing we worked on last week if you happen to catch it, which is we were working on Circuit, which is a command line utility. This is a utility that you can run on your PC, and it allows you to interact with a CircuitPython device. So if you have a microcontroller plugged in to your PC, Circuit will allow you to install libraries on that device, manage libraries. I always explain it kind of like pip, my goodness, kind of like pip, but for CircuitPython libraries instead of Python, like CPython libraries. You know, I never noticed before these little underlines right here. It's kind of interesting. It's like they got turned into a link even though they aren't text. I wonder if there's like a space or something there. And so specifically within Circuit, what we worked on last week was inside the web workflow, adding support or enabling, you know, implementing the code to make it so that it could use the dash dash auto command. This is what we worked on last week. What we're going to do now is based on feedback from Scott. Thank you, Scott, for taking a look at this. Try to refactor it to make basically to prepare for BLE and also just kind of clean up the code a little bit. Because right now what we have is a bunch of places where there is effectively an if statement that's like if they're using web workflow, then do it this way. And I got a BRB one second. Just a cat climbing a window again. So there's a bunch of places where we kind of just end up with this logic if else, like if it is web workflow or if it is USB workflow, which is what I'll call it, you know, kind of normal mode via the storage drive. And one thing I do want to do is actually just take a look through the changes here, honestly, because I have not actually just gone through and looked at the changes, which we're going to, I'm going to need to wrap my head around what all's there in order to be able to make the next set of changes, which is refactoring everything I've done so far. I just kind of poked and prodded my way through it without necessarily looking at the overall set of changes for this PR. I did some testing, and obviously we looked at the parts of the code that I changed, but I have not really looked for the rest, so we're going to do is take a look at the old difference here. There should be a way to set a preference to keep tabs in memory if you close the browser window. I'll have to look into different settings. I ended up finding it because there was a, so inside history, there is reopen closed tabs or something like that, reopen closed windows actually is the one I wanted. The weird thing was though there was two things listed in there, I tried them both and they were different, different windows. I have a problem with browsers and tabs. So those were different windows that had tabs that I didn't care about as much, but what it also had is one more button down at the very bottom that said reopen all windows, and the thing I ended up clicking was that reopen all windows, and even though there was nothing else in the list, it was an empty list as in like there's no more recently closed windows, that reopen all actually opened the one I wanted with all the tabs. So yeah, thank you to Firefox for doing my tabs. And Christy's overted on that front. I think the CIRCUP last week was targeting the ESPV2. It's not necessarily targeting specific boards, it's targeting Web workflow, which that is one of the boards that can use Web workflow. So yeah, the specific change we're making is enabling CIRCUP to work with Web workflow, and that board is one of the boards that does use Web workflow. So one thing that's weird here, why is this a diff? I'm thinking we want to merge. Do we want to merge? When was the last time this merged? Wait, but there was no, there's no commits that exist in main, so we were already merged up. Why is that a difference then? That's weird. Nip-py, interesting. Is my main, maybe my local main is not up to date. Oh yeah, I think that might have been it. Let me close this and redo that. Okay, yeah, yeah, yeah, yeah, okay, there we go. So we do have some new commits in main that are not in this branch yet. Doesn't it look like there were conflicts? Let me double-check that for conflicts, then we'll go ahead and merge, otherwise I'll just leave it. Okay, yeah, there are not conflicts, otherwise it'd be red down here. Oh yeah, let me catch up on the chat too. So any boards will be able to use it, especially useful for V2. Yeah, NESP32 as well. Any of them that are web workflow, we do still show a difference there, which is, I find odd, because that's definitely unrelated to this actual change. That change came from something else. That's another change that happened recently, but it's not really related to this PR. I'm not sure why the diff shows up here, but that's fine. So Tomlfile requirements for... Yeah, this is different as well. Yeah, we kind of have a lot of... exists that exist in web workflow, but don't exist in main. Maybe I'm on the wrong side. Local code, I think this side actually is main, which I was looking at backwards, okay. I think we do want to merge main just to make this easier, actually, because we have Toml, we have all this stuff, which is now going to show as differences. I'm not sure, the versions for the 7 and the 8 there, I'm not sure why those are showing differences too, but we want to definitely... I think this will happen automatically, but just superstitious, let's fetch, and then update, and then merge main. Then that goes automatic, so I'm just going to push merge only. And now we're going to look at the differences one more time. See if hopefully this is down to stuff that's related to this change. No, not at all, okay. I don't know why that's counting as a difference. That's super weird to me, because that's unrelated, and in fact, now that's... to be here shouldn't even be a difference. You know what, I think we're going to skip the IDE for now and go back to GitHub, because I think GitHub diff is going to be... not as messed up. Well, it's not messed up, it just contains a bunch of irrelevant stuff, so it makes it harder to focus on the real stuff. We're inside of... But right, yeah, see, I don't really care for the GitHub way, either, because we can't see... I can't just easily see what function I'm inside of, which kind of bothers me, as I'm trying to look through this. The module init is one place where we have if the scheme is HTTP, or if it is... well, just else, so that's going to be, I guess, for files. Directories versus single files. Update HTTP, update the module. So this one, this file... this function is web workflow specific. Update file, update the module. What does update mean, really? Like upload, like install, copy the file, so this is doing what? If self.file, install file, HTTP, bundle path, path, else, if not file, then delete the directory recursive first. URL, face scoff, delete, brace for status, install their HTTP. If it's a file, then install file, else, if it's a directory, then delete the directory, and then install the directory. Which probably makes sense, because file will overwrite the individual file, you run no risk of there being anything left over from the old one, but directory, it could be the case that a file got deleted in the library. And if you didn't do this step of deleting the whole directory when circuit ran, then it could install the new version, and you could still have that file that's technically supposed to be deleted. Okay, so just delete everything, and then install the directory. Update file, this one is for the local storage. We have install file, HTTP, install directory, HTTP. So I think these two functions will belong to our new class, because basically, I don't know if I actually said the what we're aiming to refactor here, but we want to make these back-end classes. So we want like web back-end class, or web workflow back-end, and then USB back-end, or USB workflow back-end, and then ultimately could potentially branch into BLE as well. And what I'm thinking now is in this class, this class will be what owns install file, HTTP, install dirh, HTTP, and then the USB one will own, presumably there's install file, USB, install dirh, USB, or whatever is analogous. Although you would think it would be right there, I guess, so maybe there's not, I don't know, I'll have to look into that. Find modules, device, URL to path, module, module, getCircupythonVersion. So here's one place where we have a difference as well, getCircupythonVersion. So I think what we'll do is our objects, our back-end objects, our classes, they'll have getCircupythonVersion, and then they'll do the correct one of these based on whichever type they are, and then they'll also, they might as well, own these private methods as well with these leading underscores here. Oh, this is weird. Oh, well, it's just not an else, I guess. Okay, yeah, I don't know. I prefer explicit elses. So getDeviceVersions, I think that's another one. GetDeviceVersions. So it will no longer have the if statement, but what it will have is there, you know, there's two different ones of these back-end classes, and they'll both have this function, getDeviceVersions. One of them will be this logic and the other one will be this logic, getModules. Same thing with getModules. GetModules. What is this? GetModules. HTTP. GetModules. HTTP.dermods. What does that mean? Dermods? Dermodules? What is that? I'm sure with the name of that's four. So if it's a single library, a library.py file, replace, but if it's a directory folder then recursively delete. Yeah. I've never seen a library have folders. Yeah. I didn't catch if it was recursive or not, but it definitely was deleting the directory. I don't think it will matter if it's... If you delete the top-level one, I don't think it will matter if it's recursive, but I don't actually know. But to your other point, yeah, I don't... I'm not aware of any... I don't... Yeah, I don't think I'm aware of any... any libraries that have multi-layer hierarchies either. I think only one depth is the... I don't know if any of that go deeper than that. Dermodules.dermods. Directory.mods.dm. Request. Okay, so this is definitely... So, okay. Well, HTTP, of course, means that it's the web workflow. Directory.mods. One thing that would be nice to do is add a... a doc string to this. Get modules, HTTP.dermods. I'm going to put a to-do there right now. Okay, let's step through this, too, because just like the to-do says, I don't actually... Based on the name, I don't know exactly what this is meant to be doing. So, we're taking authentication, directory.mods, which is a list of modules, result, which is going to be a directionary... a dictionary result for the... a dictionary for the result. Which is weird. Why are we not returning the result, I wonder? URL is a URL of the device. For dm in directory.mods. So, we're looping over our list. We're creating a URL for each one. We're sending a request to the URL. If we got an error, raise an exception, MPY false for entry in JSON. So, we're getting... We're expecting to get back JSON, which either has a list or maybe this allows you to loop a dictionary, too. I'm not sure. Entry name? Okay, name. So, it's definitely a dictionary. So, for each entry in the JSON, get the name of the entry if... entry.get directory and... the name ends with pi or the name ends with MPY. If not... If it's not a directory. So, we're looping over entries if the current one we found is not a directory and it does end with pi or MPY. So, did we find a circuit Python file? Python or an MPY. And is it not a directory? So, if that is all true, we go here, we say... set this to true. If it's MPY, we send a request, we raise, we... entry name rfind... rfind. I don't know what that does. rfind. I mean, it sounds like find from the right side. I don't know. That's going to be an index, I guess, with tempfile named temporary file prefix dash delete blah blah as fp write content. So, content we got back from our in request here. I might need to add print statements. We write the content into a temporary file. We give... we set tempname equal to that metadata extract metadata from tempname. I don't even know. That does. I must not remove tempname. Is tempname a file? It must be a file name, but then I don't know what this is. If version equals... no, if version in metadata, metadata path dmurl result dmmetadata break now if any of the submodules has a bad format. And that part is simple enough, I guess. If result.getdm is none result dmpath impuy. Yeah, I don't know. I can't follow. I have no idea. We're looping over directory modules. But why? And what are... And then we're sending a request to each one. We're writing it into a temporary file. To what end though? I don't get... getmoduleshttpderbmods. I don't know. I'll have to put some prints in there. I think if I could see this, if I could see the value of this and maybe entry, see the value of whatever got returned here and see the value of each one of these, it will be easier to understand what this is supposed to do. Getmoduleshtpsingle. Okay, so the difference here is pretty straightforward. Der versus single. So whatever that's doing, this is doing it for single files instead of directories. Which is still creating these temp files and writing the content. Extract metadata. Is it building a list of modules that are currently installed on the device? That might be it. But then why are we passing in a list? So maybe we pass in a list and it's returning... it's returning the paths to all of them. That might be what it's doing. That might be it. This is awesome. I'm making various projects that will use web workflow. Nice. Yeah, web workflow is pretty sweet. Let me catch up here as well. See, do everything, which is nightmare if there's a ton of folders and folders. Yeah. Building a non-empty directory, maybe that's what I'm thinking of. In Linux, I'm gonna let you allow you to do it. I got you. Yeah, I don't know how it treats it on CircuitPython. That's my guess right now. That's my guess right now because we give it a list of, like, module names and then it's giving us back a list of paths. I mean, it's a dictionary so it's not a list anymore, but it's giving us back all those modules, the path to them if we found it. That's my guess. Get modules file, get a dictionary containing metadata about all the Python modules found in the referenced path. Okay, so that's probably the USB version of what one of those is doing pretty much. Okay, see, now we have one of these jumps, which makes it really hard to follow. Okay, so now we're inside install module and this is another function where it will exist on both ones. The different logic will exist in each back end. Install module MPY. This is gonna be another one for schemes right here. That's kind of our logic breakpoint install module Pi. Libraries from imports context is not none using web workflow. Libraries from imports. So what this one does is it reads a Python code file and it gives you a list of libraries that that Python code file requires based on the imports that it finds in the code. And this is another one where there's different logic for web workflow and USB. Okay, and I think we're close to the end here. Get device path. If path, if eleph host. I wonder if this will work on Windows. We should have somebody test this on Windows. If anybody who's watching uses Windows and is like interested in testing this out for me, I'd be super appreciative to have someone test this PR version in Windows. Could we find a VM to use somehow? I don't really have a Windows. Well, that's not true actually. I can reboot into Windows. Dual boot action. I haven't done it for a minute, but I guess we could. I mean, not on the stream here right now because I'm streaming from the Linux side of the dual boot, but... What is this? Get device path. Okay, so it does. Get device path. I mean, yeah, I think... I mean, this is another one where it's gonna... I mean, this one might be USB only or vice versa rather. It might be web only. Well, no, it can't be, right? Because this is USB. Yeah, this is another one where, yeah, it will belong to each of those back end classes and it's different logic. We'll live in the different... The two different ones. Same thing here. Basically everywhere where we have this if it's web workflow do it this way, if it's USB do it that way, we're trying to get rid of and put all of that into, you know, web workflow back end, USB workflow back end. Okay, and then these will go as well. Uninstall. Okay. I almost wanna like... I think I'm gonna start another new branch. I don't wanna just... Like, this is gonna be a relatively major refactor. I don't wanna just... I don't think I wanna just wreck this one in case it goes south. I think I wanna make a new branch and then we'll either merge it back to this or we will open a new PR with the new branch, one of the two. I'm gonna go new branch from web workflow. I'm gonna just go web workflow refactor. Refactor. Would it require a make build with the PR? I'm horrible at that. Would it... Oh, would it require a make... I don't think... I don't think that it would require to make... I don't think that it would require to make anything. The easiest way that I know how to test it is with a pip install. It would be basically cloning the branch that this is on and then going inside that directory that it's in and doing a pip install space dot or yeah, just dot I think for like current directory and then what it'll do is it will install it as a package but it'll install it from that repo from that local clone rather than from PyPy's web server where it would normally get it from and then afterwards you could always go pip uninstall circuit to get rid of that test version from this and do pip install circuit like normal to get up from PyPy. I think that's probably the easiest way. There's a couple of different ways that you could do it. I think that's the easiest way that I can think of. If you're interested in more specific steps let me know and I can show you. And while we're doing that we have refactors. We're going to finally start coding. There was a lot of reviewing looking at code but we are finally ready to jump in. One question is do we want to just put this in the same file or do we want to start really breaking this out? Right now we have bundle and module and then we just have a million helper functions. It's kind of our structure right now, right? There's like a collapse all but I don't know how to do it. I'm pretty sure there is a collapse all. I think that's our structure right now though. We pretty much have bundle. We have model, module and then we just have loads of helper functions. I'm like making a dent in this. I shouldn't do this. PyCharm, collapse all. PyCharm, collapse all. It unfolds the current code using a single method to expand all control. Gift. All right, doing that. I don't really have a numpad in the traditional sense. I don't use the numpad that I do have. What we can do is go class, space, bundle. Okay, only two. So we have our two classes and then we just have a boatload of functions. And they're both in here and there are no other... Oh, this is interesting. There's a JSON file with it. Does this get installed via pip? That does. That was a question somebody was asking about the other day. Over my head, the extent of Python using ESP... My Python is using ESP tool. Don't even have circuit. I got you. I'll show you here. I'll show you here a little bit. Make a new VM. Yeah, definitely good advice with that as well. New VM. Instead of doing it in your main one. Why don't we have all these? I don't know. I kind of get the sense that there's a reason why these are currently in the same file rather than separated. So I think I'm inclined to just add the new stuff to this file as well. So we're thinking class web back-end, which is not really going to extend anything. I'll figure out later. It's going to require some arguments like... I don't know for sure what, but the password, probably the host or whatever it's called for the URL to it. Probably a couple other things. Class USB back-end. Yeah, this is going to get really messy too in the refactor. So install file HTTP. That one is a web workflow. One thing that we want though is we don't... since they're going to live in their own back-ends, we don't actually want to keep the name, right? We don't want it to be called install file HTTP. We want it to just be called install file because now it's going to be like USB back-end dot install file. It's going to do the right thing for USB or web back-end dot install file. It's going to do the right thing for web. It would be redundant to have the HTTP or web or whatever, USB anything. It'd be redundant to have that in there. And furthermore, it would also make it so that we end up not having the exact same interface with both of these classes, right? They would have different function names to do the same thing, which would not be ideal, I don't think. So we want to gank that. We want to grab it and put it here. And then it's going to just be install file. And it's on a class now, so it's going to have to take self. And then for right now, the rest of that can hopefully just be the same. Install directory HTTP. Okay, same thing there. It's in a class. We want it to be self and we don't want it to be redundant with the name. Why is these red? Loose punctuation mark? I mean, it's part of the doc syntax, isn't it? So I don't know. Clean library name. That one, I don't think matters, right? That's just taking it from uppercase to lowercase, removing the middle circuit Python word. That one doesn't matter whether you USB or web. Almost exclusively use numpad, but my number one is broken. You are just because of that. I got you. I used to be more of a numpad user. The thing is the kinesis that I'm on now, it doesn't have a traditional numpad out to the side. It has a numpad that's kind of like a second layer on the right hand. You push another button and then that turns your right hand into a numpad. But I don't use it. I don't know which number is on which finger. I just don't use it ever really. So I don't even necessarily know which button turns it on and off. Also, one of the buttons that's up there prompts my computer to want to shut off or restart by trying not to push any of the buttons in the general area of the one that enables the number pad, unfortunately. Completion for install. What does that mean? Returns the list of available modules for the command line tab completion. Yeah, interesting. That's cool. That part does not matter, though. That's not going to be different for USB and web. Ensure the latest bundle, same thing. That's not going to matter. Don't care about that. Extract metadata. Given and file path, let's fix that. Given a file path return a dictionary containing metadata extracted from Dunder attributes found therein works with both Pi and MPY files. The Python source files, such metadata assignments should be simple and single line like version and repo. For byte compiled, a brute force backtrack approach is used to find version and version number in the file. Okay, so this is trying to find the version. That's pretty straightforward. Does it? Actually, hold on. This part doesn't support web workflow, does it? Because that's saying extract metadata path. If path ends with Pi, LF if it ends with MPY. Python files, MPY files. If it's a Python file, set this to false. Open the file. Open a path as a local file called source file, content equals source file. Read, close the file. Dunder key val, some kind of magic regex. For match, RE find all, magic regex, content, result match, blah, blah, blah. But definitely open file, right? Local open file, it definitely has nothing in here that's like web or HTTP. Where does this get used? Git, module, right, yes, I remember now. Okay, it doesn't need the extra support for the web. It doesn't need extra logic for web because inside of here, which is a function that's doing web workflow, it's pulling the file and then creating a temporary file and then calling extract metadata on the temporary file. So the logic difference for web versus USB is actually higher up than this. This function doesn't care. It's going to be the same for both actually, so that can stay. Find device, yeah, regex. Regex would get you like that sometimes. Return the location on the file system for the connected circuit Python device. This sounds like a USB only type one, right? If we're on Unix, do it this way. If we're on Windows, do it this way. I don't know what that means. Oh, nice. I think I've ever seen or considered the idea of just looping through all 26 drive letters before. That's kind of interesting. Yeah, this has different logic for Linux and Windows, but it doesn't look like you. It seems like USB only, right? This is USB only. This isn't doing anything with web anywhere. So one question will be, do we put this on USB back end or do we leave it in the base? Get device path main. I'll just get device path. Main's pretty important, I guess, right? Main, I think, is important. So get device path, if path file. So there is logic here for USB. Okay. So if a path was passed in, set our variable to the path that was passed in with our prefix, our protocol. Else, if a host was passed in, check to make sure a password is passed in to raise an error. Do some kind of web thing with the host? Verify the host name and address. Okay, make sure the IP actually answers or whatever, or it doesn't have to be an IP, but make sure the thing answers, I guess, and say invalid host if it doesn't. That's pretty cool. If it works, so if that succeeds, then we don't get the exception we go to here. We set it to HTTP. We put in our password and we get ready. Else, so this is down here. If the user did not pass a path or a host, which is actually, you know, most of the time that will actually be the case. If they're doing web workflow, then they have to pass host because there's no way to know what the IP will be. If they are doing USB, most of the time they're not going to pass this unless they just have multiple devices or something, or they're trying to install it locally. This is going to be what happens most of the time, and this is definitely a USB-only thing because the code we just saw would have handled web workflow. So the open question on Find Device is, do we want to put it on USB backend, or do we want to just leave it here? For right now, I'm going to leave it. Could get up from CircuitPython.local. I don't know if that works like that. I'm not, I don't want to change more, like I'm already going to be changing a crud load of code, a lot of code. I don't want to change more than I have to, so I don't want to mess with changing the URL to use .local. I've had trouble with the CircuitPython.local personally. I don't know if it's my network or if I always have trouble or if it's circuit certain devices or if I was just doing it wrong. I didn't have it set up, I'm not sure, but it's not set up to do it with that URL right now. Unless you pass it, that's the thing. Right now you have to pass it, right? If it works for you, then you can pass it with host. We could try to default to it, I suppose, but I don't want to change too much right now. I don't want to, I'm pretty sure we're going to break this thing pretty massively already, so I'm hoping to not have to try to fix it in too many different ways. It needs in DNS to work, but IP always works in DNS. Maybe that's what I don't have. Honestly, I'm not, the networking, I'm not so strong on networking. It's just being a mess, but... I took a networking class years and years ago, more than a decade ago. Honestly, before I took most of the programming classes I've taken, but I don't recall it, and I kind of didn't really go to the network or the IT router, anything like that. All of my knowledge is more so into programming specifically, computers. Find modules, device URL of bundles list, extracts metadata from the connected device and available bundles, and returns a list of module instances representing modules on the device. Okay, this is building module objects, but it's building them, so see here it says from connected device, but I'll get device versions. Okay, get device versions, this needs to exist on our back ends, and then I think we leave this here. I don't think we put this on, I don't think we put this on the two back ends. What we'll do is we'll have a back end. Somewhere inside a main or whatever, we're gonna decide once and for all whether we're using web or USB. We're gonna create the proper back end for the one we're using. We're gonna store that in a thing called back end or self.backend or whatever. Not self, because we're not in a class, but we'll save it in back end, and then here we'll say back end. We're not quite ready for it just yet. Should I go ahead and do it? I don't think I want to do it just yet, because then we're just gonna have broken code while we're trying to refactor it, although it looks like we got a bunch of broken code already. I guess because we started deleting functions, huh? Well, not deleting, but moving. Get bundle. So downloads, that doesn't matter for web versus USB, same. Same either way. In Mac, it's Bondure service. Let me catch up up here as well. You don't use it daily. You'll definitely forget what makes up different types of packets. Yeah, all the networking stuff has escaped me, for sure. Bondure service. Linux was AVI Tools package, I think, but MDNS is a local broadcast-based system. It doesn't always forward a long message and some routers, et cetera. Many OSs used to need it installed. I got you a broadcast network thing that can just send stuff out to everybody who's listening, essentially, sounds like. Returns a dictionary and metadata from modules in the latest known release of the library bundle. Uses Python version rather than the compiled version. This shouldn't matter for the difference, right? We don't care about web or USB for that. Get bundles.dict. Retrieve the dictionary from bundle.config file. JSON. Put the local dictionary in front. So it gets priority. It's a dictionary of bundle string identifiers. Return the combined dictionaries from the config files. I gotta be honest. I read that whole thing. I don't really know what it's talking about, but it doesn't sound like it should matter for USB versus web. And I don't see any code in here that's like if USB or if HTTP or anything like that. So I don't think that one matters. Get bundles local dictionary. Retrieve the local bundles. So this is similar, but it's local, I guess, instead of remote. Sounds like maybe this one is downloading. Get... Oh, well, no, this is just calling local, in fact. Bundle config items. Bundle config. Okay. It's basically names for the bundles, like prefixes or whatever. Okay. Yeah, that won't matter for USB. Web. Get bundles local. Retrieve locals. Yeah. Another one. Doesn't matter. Get bundles list. Retrieve a list of bundles from the config dictionary. No difference. Get circuit Python version. This one, I think, will have a difference. Returns the version number of circuit Python running on the board connected via device path along with the board ID. String device URL could either be a file or HTTP based. Return a tuple with the version, with the version string for circuit Python and the board ID string. Why are we so far indented? Okay. And then this one does not have... Well, okay, these are our only web, right? Because those had HTTP in the name. Okay. So this is our first one, actually, that's going to be on both one second. Always getting into something. Sith magic. That was talking about regex is my guess. Yeah, this is the first one that actually belongs on both. And then what we're going to be doing is we're going to be taking this, making one here as well. We need to fix this to be self. And inside of USB backend, we are going to assume that we are this one. So we just do this like that. And we won't have the possibility of this anymore. And then technically, the difference now is going to be USB backend device URL needs to actually only be a file. And it is weird to call it device URL at this point, but also don't want to change the name just yet. Okay. So we got a version HTTP. Got a version file. Those are easy, nice and easy. We're going to drag, whoops, cut. That was HTTP. We're going to take HTTP off. We're going to self paste self whoops. Mew it's because of watching your streams and workflow style. I figured out how to highlight and indeed it nice. Yeah. With like a tab and shift tab really comes in handy when the intention is part of the syntax for sure. Yeah. Totally agreed. Yeah. Happy to hear you picked up that trick or it's not really a trick exactly, but happy to happy to hear you picked that up. Yeah. For sure. Shift tab I use old. It just startled himself. Jumped over his own tail. Oh, MDMS. I see. Not the regex. DNS. EVHI 0 Conf. Device path. Okay. So then these the what I think probably we have read here now because this is actually self also, right? Yeah. Because now we actually have that there. Yeah. Yeah. Okay. Good. Very very good. Okay. Whoa. What is this? Okay. That was weird. I don't know what that was. Pop out box or something going on there. Are you getting into now? Please. What are you? Do you want to please? Sorry. And eat like a star. Liner thing. It's not really what cats should be doing. Contest on windows pulling the repo now. Nice. I appreciate it. Dexter. Those are fixed now. So we're doing good knocking out the red there. And then yeah, these ones actually can keep, I guess the these ones can keep the name since they are specific. And since this is like the public one right here, right? This is the one that outside code should call. And then it's going to turn around and call the private one. And I don't like the idea of having them have the exact same name save for the leading underscore. Right. I don't think that's the best. So in this case, I do think it makes sense actually to keep the somehow we managed to get back to this. I don't know what this is. Get circuit version. That's not going to be a difference. Right. It doesn't matter there. It's not actually, what are all these? Oh my goodness. What have I done? Why are these like this? That's how I was getting these boxes, but I don't know what this is for. Is it to do with this? That is weird. I don't know about those triangles. Get dependencies returns a list of other circuit Python libraries. I feel like there should be more here. There should be something right list of other circuit Python libraries. That are dependencies for these libraries. Yeah. Required by an in putted list of libraries from the input from the given from required by the given list of libraries. I don't know if input it is really a word. I don't really want to use it. If lib name RL skipping, this doesn't look like it does anything with, there's no difference for web versus USB because this is basically, this is looking at the bundles effectively, right? You're giving it a list of libraries and then it's looking inside the bundles to determine what do those list of libraries you gave it actually require. So it's building out the rest of the dependencies based on your list. That behaves the same whether it's web or USB. Now where the difference would be is wherever this list here gets generated, that would be different. What is the star? That's like destructor in the arguments there. I don't actually know what that does at the top of my head. Get circuit dependencies. Okay. This is getting the new, this is the new stuff I worked on this last week or the week before. This is getting circuit dependencies out of the Tommel file. Again, that's same though. There's no difference between web and USB. Get device versions. This one should have a difference, right? Yeah. Okay. So get device versions. We're going to take that, cut it. We're going to put one on each. Get device version. And then the USB backend will be this. And the web will be the top one. Get modules. Device URL. Get a dictionary containing metadata about all the Python modules found in the referenced path. And this does have a difference in the way it works. So we're going to cut that. And basically the same thing that we did on the last one, we're going to paste it into both and change it. So that each one is hard-coded for its own type. And we have read probably because we need self right there. And here, and here, a bunch more places. And then we're going to do this one next. Probably. No, actually it's not. I don't know where that is. It must be further. Oh, there's get modules. So it's probably under that. Let's do this. Let's keep going though. Get latest release from URL. Find the tag name of the latest release. Using HTTP head and decoding the redirect. I think this is for the bundle. The latest release from URL. I mean, I guess this could be used on any repo. But I think the thing that it does get used on is probably the bundle repos. Bundle, yeah. Bundle has the latest tag which calls. Shit, I mean, does this get used from more places too though? Oh, interesting. That's getting a version of CircuitPython. Okay. Yeah. I think there is no difference though for USB and web. So I think we just leave this at the root level. Let's do some cleanup here again. What is it on that I'm testing? Ideally, just to see if the USB workflow works, the normal workflow would be the part that I would say I'm least confident in on Windows specifically. So plug in a normal USB storage style device and then do like circuit install either dash dash auto or circuit install a specific, you know, display text or whatever library you want. Gonna take off brain start. Have a great night. Yep. Take it easy to do Devon. I will see you around next time. Thanks for hanging out. Hope you can get some rest. See. Get modules. Okay. So that one is web. Get a dictionary containing metadata about all the Python modules found using the referenced URL. Yeah. Basic auth. Anything that's using basic auth, like that's a definite telltale that it's hitting the web workflow, I think, because basic auth is not used by much in the sort of wider world in the internet anymore, but it is used by the web workflow. And that makes self here, which means this here is going to be self dot. So then we should have a get modules file somewhere. Have this one. Modules. So that one is a web one, but I'm going to leave it for a second. Where is our file? Want to find our get kind of want to find this one and move it in so that we are kind of like, there it is. So we're kind of like equally having the same stuff in each one as we go here. This feels like the right way to go about it to my brain. Let's see. So now that's going to be a self dot self comma than that's going to be a self dot right there. My turn. I'm like at this weird angle here. Okay. All right. And now we've got these two. All right. My apologies. I will be right back. I've got to run to the restroom. That will not make much longer. So these are both HTTP. Right. They both say HTTP. This is weird because there aren't USB analogous ones seemingly that I saw so far. Okay. Self onto both of those. New device is going to install it for a bit my phone. Nice. Thank you. Okay. That is weird. Why do we not have USB ones of those that part's weird. I guess because get modules file doesn't have those. It just has all this logic here. Whereas get module modules with an shdp that one actually does just call. Okay. That's why there's that's why those of us this cat I swear to goodness with this cat sometimes what he's getting into now but I hear him getting into something. Lord help us all. Don't know what it was but he knew he wasn't supposed to be because as soon as I got halfway to the place where he was he came shooting out of there. I'm not sure why that click through worked. I wasn't holding control which is usually what you how you click through in pyjarm is control and then click. I don't know why it actually went through. I've been no cover. I don't know what that's about finds a connected device installs a given module name if it's available in the current module bundle and it's not already installed on device. There's no check for version. Okay. I think that's telling you like if you already have an old version circuit doesn't install the new version is that true though I think does it. I don't know. I thought it did but I could be wrong. Okay. If no name see this is like install module it's not quite it's not quite main but this is very high level because we do have like we're actually in a click land instead of helper function land. So we're we're we're higher level than we have been most of this time we do have different logic here. So what we're going to have is install module is going to be on both this one is USB and this one is web and then in the logic here. So if name all that's the same. I will say one downside to the way this code is now structured is we do have a decent chunk of kind of copy pasted code honestly. It might be worth making a back end a class back end that contains some of the logic any of the logic that's the same for both and then it would call down into the I'm not exactly sure how to do that always though because like here we have some stuff to do before the split in logic. Here's our split in logic in this one. We have some stuff to do before it. We have some stuff to do after it fits a super class. It's kind of always going to come before after I don't know how to do like some super class before and then some super class after we'd have to then like split it into two other functions. Okay. For now I think we're going to leave it and just do the style we have. This is HTTP as we don't need else here. I actually don't need this check either. So we're just going to assume HTTP and do this stuff. How's it going Gary T. Thanks for hanging out my friend. Hope you're having a good evening or whatever time date is for you. Sorry it's evening for me but there we go. See this one is a lot of reused code honestly not necessarily think that that is the best. What do we do before if not name LF name in mod names device modules check if it's already installed if name in device modules echo already installed return. Okay. So the before part but we're still inside LF name in mod names. Okay. I'm going to leave this for now I think it might be worth a second pass at some point to try to make an extra layer in the hierarchy basically. So we would have a back end class that can hold the code that is the same for both but kind of back to what I was talking about before where I don't want to change I want to change as little as I can because I already know I'm going to be changing a lot. So I'd rather not do it all at once if I can I'd rather get to a halfway point and have the tool actually work as expected and then try to refactor further from there. This is another one where it's going to be on both but then when we're on the USB one we will no longer have this this one where you get rid of the else block entirely and then we don't need to if either will say self self and do a self here as well. Not supported or scheme. I did pip install dash e-circup HPR check out 163 dash e-circup pip install dash e dot maybe are you sure you did. Circup instead of dot if you did actual pip install dash e-circup I think it will still come from Pi Pi not the local one but I'm not sure HPR check out so that will that will get you to the right branch for sure and then Circup install it a fruit font and then it says no supported URL scheme okay. So yeah it's broken on Windows. Okay thank you for testing let me leave a comment or if you wouldn't mind will you leave a comment on the thing here with just a copy paste of that error. Not supported URL scheme most likely that's a bug in the way it's trying to find the path to the file. I'm surprised it doesn't have the file colon slash slash slash bug in it pip install okay. Okay dot slash circup and then that that will work because yeah circup is like that I do think you can leave the slash circup off I think you can do pip install dash e and then space dot and if you as long as you're in that directory which you must be for the dot slash circup to work as well it will work. That's what I was doing last week dash e was the new thing I learned last week which is super cool I'll report again to Scott for that. Why are these yellow unresolved. Okay doesn't exist yet. Oh I changed them that's right actually they do exist but they're no longer HTTP. Okay I think actually we should I think actually want to put this back because this is basically the same as like this other one here where it does not actually exist for the other one while it is redundant. The main concern I had was about the interface having the functions have the same names but the thing is there is not a analogous function to this. The analogous function is one level higher because in USB back end it doesn't call any other install anything it just does shu till to copy stuff whereas in web install module pi is actually calling these other ones so there is no analogous and since we did it that way keeping the HTTP in the name for the other one I think we should do it this one too. Post it thank you yeah thanks again for testing that out. I'm really glad that I didn't just blow through this and get it working on Linux and totally entirely forget to test windows and then break it for everyone who uses windows which is probably a lot of people. I think I copied instead of cut didn't I because this is install module pi and we just put that in both of these. Oh no no no NPY versus pi okay so yeah we'll do the same thing here we're going to have cut that this is USB yeah so we will be doing the else and expected types I think that's actually because of this right. I guess I haven't tested it since I merged it's possible my merge from main broke it I should have probably tested it before I started doing this refactoring everybody to start the evening watching Tim and his genius at work well that's very kind of you don't know that I would call it genius necessarily myself but doing some doing some refactoring I'm glad that's enjoyable to watch as well install module pi okay so that one's going to do our copies and then the other one which I did already cut this so we're just going to come up here and paste sort of strange jingle that I do not recognize must have only heard part of it wait it wasn't a twitch thing was it I certainly didn't pay attention to the last time no I don't think so I don't know which phone it would be and this is web we're going to do the if rather than the else and we are going to just get rid of the condition self it's mad about this also let's do this slowly but surely we're working our way through this I probably should have cut that with assuming that went on assuming that went on this but I don't actually know pylint is probably going to be so mad about this our code reuse libraries from imports first the given code pie file and return the imported libraries okay so yeah this does have a using web workflow usb first libraries from imports so we so we no longer need using web workflow as we're only caring about usb here so everything used to generate this I'm going to I'm going to comment this right now because we're going to want this logic somewhere inside main or somewhere we are going to want this logic because somewhere there has to be code to decide which back end to instantiate and use and that's where we will use this and I want to make sure I don't lose that chunk of code and after we write it yes we actually just delete all of this okay as long as context is not none I actually that can go away as well because we're no longer have this right just the only thing inside of there that's because of this we need to keep that's our usb one and then libraries from imports so this one we don't really need the condition anymore because we're hard coding to true so the way we're doing it is just this and that will set code pie equal to something and then the rest of this code will operate on code pie and so that's actually a part another part this whole chunk here we this would actually be very easy to refactor the reuse out of as opposed to the part before which was a little bit trickier looking but again I though I am going to save the cutting of the duplicates for a second pass instead of trying to do it all at once libraries from requirements cleanup supplied requirements text and turn it into a tuple of circ python libraries doesn't sound like it should matter for web and usb right and this is all we just got a requirements text from wherever we got it from really we got it from in the bundle not even the device so save local bundles that doesn't matter about usb and web tag tags data load load the list of version tags of the bundles on disk return a dictionary of tags indexed by bundle identifier slash keys okay I don't think that matters that should be the same for both right because we're using the same bundles for both so tags data save tag well let's read what it does my first inclination is I'd love a better name for this tags data save tag add or change the saved tag value for a bundle key and tag add or change the saved tag value for a bundle I don't know what that means I mean tag I'm pretty sure is like the version right simver like 3.1.2 or whatever but I don't really get what this is doing command line command definition cli okay main right here this is where we're gonna want our somewhere inside main here I think is where we're gonna want our if using like our using web workflow the setting up the Boolean variable and then if using web workflow create this back end you know web otherwise create usb back end and then we'll be calling get circuit python version on our back end once we have it how does it get into the command because the other thing we'll need to do somehow is we need to pass our back end need to pass it maybe we can just save it in a variable okay this was our one where we had back end dot so I guess it's click that's responsible for getting us from main into some other function like getting us into freeze for instance or into this one because in here is another place where we're gonna need to have the back end because amazing twisty passages at least me yeah I don't know if we can maybe add something to context I feel like that's probably a good way to go if it's possible if we could create our back end in main and then add it on to context then it would appear inside of the context that's in here and we'd be able to just use it with like context dot back end or whatever it ends up being stored as well because like a lot of these almost all of these where it's red here these are all going to be like back end dot oh oh wait where are these at okay they're the next two I was gonna say like why did we not pass those yet okay okay so yeah this it's gonna call this one we want to call back end dot uninstall and this will this one will actually be one where we're gonna drop the suffixes here the HTTP in the file we're just gonna have underscore uninstall it's the same arguments so it'll be the same name it'll be the same you know interface the same API and then inside of here it's just gonna call back end dot uninstall and it's gonna pass what it wants too far it's kind of weird that these are way down here intermixed with the click code I mean they're right next to uninstall which I guess makes sense but it's kind of weird because all the other helper functions were elsewhere way up above the CLI section I never actually tested freeze on the web workflow I guess it should work the same though because get modules is what it was calling should work the same so get bundles list expected type bundle got list bundle alright yeah I got it into the restroom again sorry PRB okay made it to the end we found main bundle add bundle remove bundle show all of those are for managing bundles they don't have any shouldn't be any difference update so update calls module dot update which calls either update HTTP or update file so that is tricky how so so the thing is these functions are on the back end the web back end this is the part where it's a bit tricky because the USB back end doesn't actually call other functions and we're inside the module object here buddy what is it about cats that makes them want to play with not cat toys but instead people things in your experience has refactoring ever been more time consuming than coding from scratch I guess it depends on the size of the program no good question in my experience has refactoring ever been more time consuming than rewriting from scratch um probably yes like if you mean in my entire experience have I ever spent more you know ever ever like even once I'm sure I probably have um yeah I don't like my instinct is that it's not very often my instinct is that typically when I do refactor if I choose to work from what I have instead of throwing it away and starting fresh I think it's more typical that I spend less time than I would if I threw it away but I think I'm also biased because the truth is I'm inclined I am you know my subconscious just what I will do by default is typically not throw it away and start from scratch I am biased towards keep what I have and try to make it into what I want even to the point where like maybe in the cases where it would be faster or better for other reasons to just throw it away sometimes I'm probably inclined not to just because it's how I am and how I operate and how I have always done it in this case in particular though I was I would not call myself familiar enough with the code to be able to write it from scratch so we had to basically Frankenstein what we had instead I wouldn't I wouldn't want to rewrite all of this I doubt I could do it in anywhere near the same amount of time but also I just I'm not sure I could even do it at all and make it work so I'm not sure what to do about the module problem I I'm kind of thinking we just maybe this is one place where we don't actually do it as a back-end thing instead like like these aren't used anywhere else are they hmm so I don't know what we do about this one actually so we have these functions on module module has update module dot update is then wanting to call either update HTTP or update file update file everything's groovy no problems we're good to go on update file update HTTP however is wanting to call install file HTTP and install Dury HTTP but we moved both of those to the back-end class the web back-end class but because we're inside a module here we don't have context so my earlier idea of trying to attach our back-end onto the context and then being able to access it by saying context dot back-end or whatever that is not gonna not gonna work here because we're not getting past the context currently at least so I don't know where to move these basically the trouble I'm in now is I don't know where to put these functions so that they can be called from inside here ultimately inside module dot update but also so that they can still be called inside of our back-ends because back-end dot install module NPY is trying to call this for the web back-end and the other one is probably inside of here and the DUR ones are right above in fact so we need to be able to call them from inside of the back-end inside of the web back-ends other functions we also need to be able to call them from inside of the module they really feel like they should belong to the back-end right because they are interacting with the web device or I mean there's one each right really they feel like they should be split it feels like it feels like update HTTP this should belong to the web back-end and update file should belong to the USB back-end but if we put them on the back-end how do we get access to them inside of an instance of module so inside of find modules we get a device URL and a bundles list truthfully I'm not sure what my plan to get the back-end here was either honestly because we don't have a context here either we could here we would have a context so we could pass context and then all the places where we use this we could go and update them to pass context and then we'd be able to get back-end from context I'm a little stuck on our module class problem though gets further dies trying to read the Python version from boot out okay okay you modified the code inside of find find device or whatever that function is it looks like okay and then that made it get a bit further but still died when it tried to go inside boot out and read the version this one is another function that feels a bit weird it's just randomly in the middle of all the click CLI stuff if I modify context right here will it when it gets when it when I'm inside of the other functions will the context that got passed to me contain the modifications that I made to it right here the problem is I can't run this right this code is all sorts of wrecked we can't run it we have read everywhere okay we're going to commit our problems those are to do is to do some problems necessarily go back to a workflow and I want to test this so inside of main if we modify context first of all can I paste this yeah if we modify context well let's make sure we're not going to clobber anything so mine I think I might already have it on the dash e thing so I think it will actually already be using this modified ones I need to get a device plugged in this device is USB must have been wrong okay there we go context is a click core dot context object and it has all of these things it does not have a thing called using web workflow meta maybe we should put it there let's do a quick search click plus add variables to context it's not really a variable though maybe I guess we're not I'm not talking about like arguments like command line arguments that's not what I'm talking about I'm talking about programmatically you know kind of adding a field to the object inside main and then having it in the other one yes we could try params we could put it in there I suppose is basically doing that so they're doing context object obj the dictionary so if we go CTX obj using web workflow equals using web workflow can we then access it from inside of install print does that work yes okay so we can modify object obj context obj we can put new stuff on there and it will get passed around to our other places so the the real stuff that we would want to put on there is the back end roll back all of this to get rid of those changes we're going to go back to our actual branch inside of main we're going to do that using web workflow and then we're going to say so we we don't really so what I tested it with was adding the boolean using web workflow actually out of coffee I shouldn't be drinking coffee this late anyway truthfully what we really want to do it with those with our back end so what we want to say is if we are using web workflow then context object back end is equal to a new web back end else is going to be a USB back end back and back end or recursive back end action and then inside of install we get context maybe we want to have that too maybe we want to have the boolean too I think we want to is there is actually that other that's a different thing it's doing there it's figuring out the auto file which does work differently and is not part of the back end but maybe it should be something to be said for maybe adding that to the back end I guess like maybe we could have back end dot get default auto file no that's wrong though this is not the default one this is whatever one they passed I don't know maybe something in here should be coming from the back end but I'm not sure honestly my brain starting to get a bit fried probably going to wrap it up here before too long this part I know though that's here libraries from imports so that one is on the back end so that's going to be context object back end same thing here honestly a bunch of these are going to be that install install module yeah probably this does need to be associated with a back end because when it comes time to do Bluetooth we would have logic for Bluetooth also probably unless it happens to be the same as one of the others which would actually be pretty convenient okay that's actually all the red that's in install not bad okay what is this this is inside main yeah and so here we could well we still need to access our back end like this because we don't save it any other way okay this is no longer like this is actually I have a piece that I think you should maybe make uninstall not private with the leading underscore get the vice versions we do still have that module problem I was worried about earlier oh here we have another weird one too because here we don't have a context I can get module file I do this this is on the USB back end returns a dictionary of the metadata from modules in the latest known release of the library bundle get modules file get a dictionary containing metadata about all Python modules found in the referenced file system path I feel like this should be called like get metadata from file baby or get file metadata or something it does it does it says get modules but it returns a dictionary containing metadata or get modules file but still it returns a dictionary containing metadata not a file or a module yeah misinterpreting it okay honestly a bunch of the code is starting to all run together to me so I think we are gonna I think I'm gonna stop for now there obviously it's not ready to like try it yet because we still have a bunch of syntax errors mostly it's pie it's like functions that don't exist like we either need to add self or we need to resolve that weirdness with the module and figure out how we're gonna access that function from there aren't too many spots but there are a few yeah so I need to work that out and then once that's done we'd be ready to actually try just be back end oh okay those needs I think those just need self actually this one's a tricky one we need to supply a back end but we don't have a context so let's figure that out same thing here the way it's written right now we would need a we would need a back end we don't have one or a context we need to figure out how to get one to hear this one just needs self I think yeah this is just self that's easy thank you Tim yeah for sure thanks for hanging out love the factory hope you have a nice night I'll take off for now for anybody interested in some more circuit python action I will be back in the morning tomorrow over on my own channel phoma guy underscore twitch and phoma guy on YouTube I'll drop links in the live broadcast chat when I'm about to get started the plan right now I think is tomorrow morning I'm going to work on a matrix portal project so if you're interested in matrix portal s3 I'm going to make a little scrolling project with some stuff that scrolls past the screen so if you're interested in a sort of less deep this is this is quite deep dive into circuit right here if you're interested in something not quite so deep and just for fun project on a matrix portal come back tomorrow morning and check out the stream then otherwise I'll just say have a good night thanks for hanging out have a good weekend and all that stuff hope you all have fun and yeah I'll catch you next time I will be back next week for deep dive as well that's the current plan is for me to do next week as well and I am not set on the following weeks but basically I'll keep going as long as Scott is unavailable on Fridays I don't have much coming up on Fridays so I'll be here until he is ready to return and I don't know when that will be so I'll see you next week for that I'll see you tomorrow morning if you want to join me for the other stream where we will play with the matrix portal and I will bid you farewell and say thanks again for hanging out to everybody else