 All right, welcome to the Microsoft 26th of August developer sync meeting. So last time we went over our sprint planning for this next two weeks, and today we're just going to go through and check in on progress and anything that people might be getting hung up on. So I think we'll just go through round table style here, and Cand is in the middle, at the top right there. So go ahead, Ken. All right, so it's been an interesting week, a couple of things. I read through your user stories, wiki page and commented, made some comments on it. I understand where you're coming from, I believe, which is you're trying to touch upon all of the things the data model should be able to speak to. I took it that way. At first I was confused and I was taking it like we want all this functionality and I realized you're probably speaking to the data model more than anything on that page. So I made some comments. I finished the auto train process. I am ready to integrate it with the new schema when it is ready. To that end I threw up a couple of wiki pages describing that process in depth and another one. So the one is called continuous model training, and it tells you what servers are running, what code and all of that good stuff, and why the limitations or the implementation is implemented the way it is or tried to justify it. I created another wiki page which describes some of the factors that affect the model's performance, just a placeholder. I'd like to get deeper on that at some point. I created a bunch of models. I've hit some interesting problems with hyperparameters and overtraining, so I'm researching that. In so doing it came to my attention and I actually commented on one of the tickets that you and Chris had commented on. I'm one of the wiki pages or I think it was a ticket or a wiki page, but the subject was around the concept of a model or the model of a model. So the continuous model training page now has a section called model object, which describes this artist's interpretation of the representation of a model. To that end, the automated process, when it's done it runs some tests. What I'm thinking of doing is creating a model database and I'll put the schema up. And after every new model is created and the tests are run, making entries in the model database so that in the future, somebody that's doing hyperparameter tuning can select them out in an organized manner and possibly start looking at clusters because I really do believe it's that kind of a problem, not a linear increase the RCUs and it gets better, add more epics and it gets better kind of thing. It's a relationship between the hyperparameters and the dataset size and possibly dataset quality. And so the only way you're going to get to the answers is to have the data and so I'm just thinking it would be helpful that every time we train the new model, we simply made an entry in the table, you know, broke out, you know, the hyperparameters as well so that you can select and get your datasets and then somebody better than me can come along and stand on my shoulders and use that data to do something with it. So that and I'm working on that a little bit and then I had an idea I want to run by everybody. I created a ticket, but I didn't sign it or put it anywhere because it's one to see if you thought it was worth following up on and it goes speaks to what you were actually saying to a certain extent a little bit, Michael, on that one page. But basically, what if we had a skill that you could say improve recognizer and it would play back the last 10 or 20 samples that it had captured, not only had it sent them up to the cloud, but it saved them locally. And it would say, you know, maybe four or five at a time, just very briefly. Was this a wake word? Was this a wake word? Was this a wake word? And then upload the results to the tag database. So now the user could not only help improve the quality, but much more importantly, after enough of those runs, I could then fire off a task that since all of those now tagged samples are in the database, select them out just his. Take the existing community model that he has running in his local server and do an incremental training session using his new sample tag data. Because when you do incremental training, which is actually the bane of machine language processing or machine learning, but it's great for this example, the newer samples get prioritized kind of higher by default in these algorithms. They fight to make that not happen, but drop out and all sorts of things. So by incrementally training on your last 20 or 30 samples you tagged, that model will specifically continue, even though it's a community model, it will start getting better for you personally or the people that are using that device. So I thought that might be a one way around that problem space as well. So I threw a ticket out there as well to see if I wanted to get any traction. And that's it. I'm getting ready to publish the model schema and start populating that model database, which is why I asked you, Chris, because I'll make it a separate model database in the Maria DB. And then later you can just bring it on over and it'll be in your Selene database as well. And then I'm not blocked, right? So anyway, that's what I was working on. Oh, and I've been trying to help keep up with the forums. I suspect, this is really weird, I suspect the latest release broke for size. And I know everybody's like, but you really can't test for that, because we do continuous integration and deployment in the cloud, if I'm not mistaken. And all of the VoIP comp stuff I did would have to actually run on a head server, not a headless server, because you can't do real loop feedback through a microphone speaker system when you have a headless server that doesn't support a microphone and a speaker. So there's no way to catch that. And I suspect what happened, and I'll tell you, it's a little complicated. I actually have an email out to Ake on this. When Ake was breaking me in, we discussed the concept of the next release. And he had mentioned that he had to work around in the new runner for problems that existed. And what I'm suspecting is happening is the runner and the engine are O2. In other words, the runner is O3, it's running the O2 engine. It's because that's O3, it thinks it's O3 and it's pulling the models down from an O3 model repository. And we know O3 models are not backward compatible with an O2 engine. So I have an email into Ake to say, hey, do you think this might be happening? And I'll follow up on it as well. But I've been trying to keep up with the people in the forums because there's been some matter most informed chatter about it. And so I'm working on looking at that as well. And that's it for me. OK, thanks. Yeah, that's a good, really good flag. I mean, one thing to note is that we've planned three phases in the BoiComp continuous integration system. And we've only implemented phase one at this point. The next phases will integrate just what you're talking about, incorporating the actual text-to-speech, speech-to-text and text-to-speech issues. And then ultimately, a hardware infrastructure as well, so that we can test it on the actual running hardware. But none of that's actually implemented right now. So that's a known problem. Yeah, I was just trying to give an excuse of why this wasn't caught. Yeah, no. And that's a good point. It should have been caught anyway, because we're supposed to test these releases on actual hardware before we send them out. I don't know about that. That's something that we'll, I'll learn more about in the future, I'm sure, by awesome. OK. So but you're, the status right now is that you've logged to the, you've created a log for a problem that people are reporting. Good to take care of that. Well, what I did is I saw that the chatter was heating up. You always see people complaining about custom wake words not working, right? That's common. And that's a really involved solution, so I try to not get into that hornet's nest. But what I have been seeing more recently was people saying it just, precise doesn't seem to be working. And I'm positive that my hardware is good. And I then started seeing more of this and thought back to what Ake had told me and said, well, gee, I could see where, if it thinks it's O3 and it's running an O3 engine, it will pull an O3 model. Or maybe it's running O2 because he had to do work around, I mean. But the wrapper says it's O3. So it's running an O2 engine, but it sees the wrapper as O3. So it's pulling it down an O3 model. And we know that's not going to work, is my point. So are they reporting what hardware they're running on? Are they running on Mark 1's or on PyCrafts or what? They're running on custom regular laptop running in Ubuntu 20 is the last one I saw. And it was something to do with the ARM architecture, that the O3 image wasn't ready for ARM during the O2 release or something. I mean, I'm not positive on all of that. And that's what I'm chasing down as I'm asking for more input. Give me the engine, give me the repository hash that you believe it's pulling from, that kind of stuff. And I'll step through it and work with them. And like I said, I have an email into Ake saying, hey, I suspect this could this be so. So he'll probably see that tonight. OK, we'll make sure to write up a ticket for that and try to figure out what version of the software and hardware they're using on all that. Yeah, I guess I was only apprehensive about writing up a ticket because I wasn't sure it was a real problem yet. But I'll write it up because worst case, it's not. And we can close it out. Well, exactly. Yeah, you're spending time on it. It's good to know. Maybe we'll learn something here. It's unfortunate that Giz is out right now. That's OK. Actually, I can hold up the board a little bit with one. OK, you mentioned a couple other things that I wanted to touch on there, but they've slipped my mind. So I'll just let it go for now. We'll go back and look at the notes. So how about Chris there? Yeah, so I spent a good part of Monday in the start of my Tuesday reorganizing JIRA a bit. That's just time consuming. So I think it's all it's in a pretty good place right now. And I moved on then to re-implementing the schema as we defined it in our meetings last week. So the DDL is written. And I wrote an SQL script that will take our current database. So I wrote the code so you can rebootstrap it, basically, some scratch if you want to. And I also wrote an SQL script for the test and production databases that will update them with the new schema stuff. So right now I am working on the code that uses the schema. My first step is to update any code that was broken by the new schema. And then I'm going to go in and add the stuff we need for the new collection mechanisms. Again, a lot of this code is written. I'm just looking at the code I've written right now and seeing how it needs to change for the schema as it changed last week. So that's where I am. I will probably be spending the rest of my week doing. No blockers. Everything's cruising along. OK, good. Derek? Yeah, so I don't really have any big blockers. Just continue to work. I was finishing up Kent's device. I'm going to get that set tomorrow. And I've also got the enclosure for the last thing I needed to do, forcing the enclosure to Kevin, was test the audio chamber. It was slightly different. I did that yesterday and it sounds pretty good. And I think I'm actually going to tweak the work a little bit. But it's plenty fine. So we'll go with that for now. And I've updated that design, that laser cut design that I'm seeing at Kevin, to be assembled with screws, instead of glue, I've got this kind of T-slot screw, a sibling, with the idea there that I could be easily shared. But I want to get it super easy. Actually, it could be something that we consider making available to those backers, or to anyone really, that just got the boards, because if you have a cheap and easy housing, it's not good. So let's see. Yeah, so then the other two big projects related that are just getting a version of that same enclosure that is an modified top to use the re-speaker microwave, so that we have an apples to apples re-speaker versus the SJ201. And then I'll get back onto the actual plastics design. There is one thing that I ran into yesterday while trying to test Ken's device that was troubling me. I did finally get past it. And that was this audio playback bug. I sent a clip to Josh and Michael. I consider it's you too, the rest of you guys. This is a known bug. We've known about it, but it just seems to be popping up so frequently now, not sure why versus a couple of weeks ago. But the problem with it is that, for example, I booted the device that I was testing for Ken with a fresh image four or five times I had to reboot it before the audio stuff actually worked. So as you can guess, that's kind of frustrating for me on the hardware side, because at what point I would say, Hey, Minecraft. OK, I know it's probably going to work, but is this hardware or is this software? So that one particularly is becoming troublesome for me. And like I said, Chris and I have talked about this, but it used to really not happen that much for some reason. Yeah, I mean the the mark to off-the-shelf unit that I've got here is the audio is fairly unreliable. And it has to do with boot up and stuff. And actually, I hear clicks and pops when I turn it on. So I think there might be something about a loose connection or something like that. I might have to open it up and take a look. Well, the clicks and pops, that would be coincidental that both of them had a loose connection. It's probably bring up timing issues. But presumably I will have experience. I'll be fortunate enough to experience this pain when I receive my device so I can look into it as well. Don't fix it. Yes, well, I think Gez was also looking at possibly something that, you know, in startup check, you know, a project he was looking on. Yeah, Ken's probably right, actually. We there are some known issues there. But you mentioned at the beginning of this, did you say something about the board Kevin's working on? You were saying this was? No, no. So I may have misspoken. OK. It seems like you were something about the audio that you thought was going to interfere with us doing the side-by-side comparison of the new versus the old design. Well, no, this is just being that it just becomes like, well, I mean, I guess I could see it becoming difficult for Kevin as well if he's seeing the same type of thing. He's testing new board and it's OK. This is the board issue or this known audio bug. It just becomes, you know, if you have to reboot it in so many times, you know, you just keep going to 10 or what point do you give up? And so usually in the past, I don't think I'd ever encounter where I had to reboot it four times. That's a good answer. Yesterday was the first time. So on the audio thing, one of the things that I wanted to kind of insert there was the big difference between this and our previous Mark II prototypes. There's a couple of them. One is we're running on an RPI-4. Another is we're running Buster instead of Jesse. And the third is that we're based on a high-croft image instead of based on a Mark I image. So those are the three major things that are different to me between this prototype and the prototype we had prior to this that did not have this issue. So I'm wondering if one of those three things has something into the drivers that the Raspberry Pi 4 or the drivers that the pie-croft supplies for the speaker array. I think so that some of that may be a good starting point as far as things to look into that might be causing the issue. Chris, was the pie-croft image based on Jesse or Buster? Buster. And this new image is also based on Buster then. Yeah. Yeah, Jesse image was for the old Mark II prototype we had. It was Pi 3 based. Well, what I was getting at is so we don't seem to have this problem with the Raspberry Pi 3 is running it. So that would probably take the operating system out of the equation. Now, it's not drivers but the OS, right? Look, if I'm wrong, the unit on my desk is the Pi 3. Yeah, but my point is we already run well. We know we run well on Buster. So we're not saying it's a Buster issue, right? We know we run well on Jesse. Well, but I thought the pie-croft was based on Buster. The newest pie-croft image is, but I don't know. Old pie-croft image. The newest pie-croft image is based on the old pie-croft image, correct? We shouldn't. We can't mix pie-croft and the Mark II image. The pie-croft is the basis of the Mark II image. Yeah. And the most recent pie-croft image is Buster-based. And I don't know enough to know if there's issues with it, like if other people are having issues with it, regarding audio or not. Well, let me ask a question this way. I just saw a video today that was actually excellent. You were right, Michael, that showed how to get pie-croft image running on your Pi 3, correct? Chris, right? Is that the one that Krista there shared on that list? Yeah, yeah. Is that image running Jesse or Buster? Depends on what's once you download it. It's probably running Buster. All right. OK, so let's turn this into a ticket, and if it's not already, and get it assigned to somebody to figure it out. Well, there was a ticket at one point, but I guess it should probably go under Mark II now in the organization of things. Since that's quite small. Yeah, it's definitely occurring on the Mark II. So yeah, if it's growing in other places. That's not the only place that's happening, right? Well, I don't know if it's occurring other places, whether it's on. Oh, yeah. Well, I haven't done yet a setup on Mycroft Lab. I suppose I'll have to retire to my garage and have a Raspberry Pi 3, and a Mark I, and a Mark II, and a laptop, and something running. Speaking of all, do we have any Mark I's that Ken can have? Oh, yeah, that's a good point. I should pack one of those up to try to send it with you tomorrow. OK, I can do that. Yeah, I don't want to use it. I've got a couple on the shelf behind me up here. If you guys need one, I got it. Derek's already sending my Mark II, so just send it together. Yeah, I've got one, or I've got a few of them. Yeah, Derek, go ahead and assign that audio system instability problem ticket to me as well. OK. Yeah, I mean, I'm going to be getting the hardware and bringing it up and experiencing the pain, and I'm going to fix it anyway. There you go. I will close the ticket. That's exactly why all developers need to have these on their desks. Right, right. All right, so excellent. So Derek, have you been in touch with Kevin today about the hardware and the status there? I haven't. I've just been kind of waiting for him to, he's been pretty good about sending updates, but nothing today so far. Yeah, we had some pretty exciting updates on Monday. But so the bad news is that I did have a chat with him this afternoon, and he seems to have hit a snag. Like, he got the entire board up and running except for the LEDs on the back, and that was just because there was a lot of stuff to install. So everything was working. But by the time he got the LEDs on board and was testing that last system, somehow the short had reappeared between one volt and ground. And so he's trying to figure out what's going on there. So he had gotten the XMOS chip up and running, and it was working fine. He actually recorded some audio samples and sent them over to us, and that was pretty cool. So we know all the subsystems work, but he's just having some problem with the manual soldering step. Something's going wrong there. So he's going to be working on that this afternoon. Hopefully he'll be able to resolve it. But at least we know that, in theory, all the parts work. It's just a matter of getting the assembly down. I got a generic question on the hardware. The mic array, is that one of the things that's controlled by the XMOS chip? Yeah, that sits into whole purpose. OK. And can you configure it to just use a single mic out of the entire array? Yeah, there's a lot that you can configure about it. You can turn off all the subcomponents of the echo cancellation. You can fine tune the parameters and the delay between the input and the output and all this kind of stuff. Do we have the, does anybody know the part number on the XMOS chip so I can dig up the specs? Or have you put the manuals on the chip to the website somewhere? Yeah, if you go to the GitHub repo for the Mark II hardware, everything's there. Oh, very cool. Yeah, no, the only reason I asked is I didn't know what a mic array was. And I started researching a little. And lo and behold, a lot of the results came back. Mic array sucks. So I'm like, oh goodness, I hope I can turn them off. Why ask? Well, it is kind of misleading to a certain degree. Because a lot of people in the community, they talk about, oh, have you used this mic array? And then they'll point to like a $10 re-speaker mic array or something. But it's just, and a lot of those are just for microphones that are just connected to the Pi with no DSP. So the XMOS can, in certain firmware versions, the firmware version we are using for the re-speaker combines all the input of the microphone for you. And just so the Pi only really sees it as one microphone. Because all the processing's already happened. So we're not doing that. We're not having to do that on the Pi. Whereas some of the other options, you have to do that in software on the Pi. You have to use some other software-based DSP. So we will want to use a firmware. I would assume you'll want to use the firmware that combines, or it does all the processing for us and gives us one audio stream, a clean-down, essentially clean, nice audio. That's kind of the problem. And the firmware documentation is on the GitHub repository as well? You might have to link it over to the XMOS set. What's that? XMOS has some very extensive documentation on their website, so it's probably a link there. OK, good. And so it's kind of a misnomer, just call it a microwave. It's really a microwave plus a DSP, plus a dedicated hardware DSP. Right, it's running some specific echo cancellation far fewer algorithms. So hopefully we'll get some news on that, hopefully today or tomorrow. I don't have any other updates right now. Is there anything else people want to talk about? Things seem to be moving along. So, all right then, I guess we'll call it a day. Yeah, just one thing, Michael, for your edification. Remember, I talked about a skill where you could say, improve recognizer accuracy and go through the whole process. In case you were wondering, it wasn't like I was planning on training on your machine. I was planning on simply sending a notification to the back end. As part of what I just finished, there's now a data pathway for requesting the training server to train. And so right now that is very specific for responding off a nightly trigger, but could be expanded to allow that to basically create the models for the device in the cloud and then send them back down if that makes sense. Right. Yeah, I think there's even a ticket for something like that in a system. Yeah, so by building this, just with a little bit of enhancement, I can expand that process flow to handle things like that, where we could allow people to train in our cloud, if you will, and then download the model. Yeah, so anyway, that was where I was going. I wasn't planning on doing it locally. So, just to bring up something sort of tangentially related to that, Derek and I had some pretty good conversations around how to close the loop on the intent processing side of this back in last December. So, we've got some ideas about that, that we can talk about once we get through the precise side of things. We're talking about a PDACIUS and that whole yeah, branch, that spaghetti junction code. Yeah, the idea of basically letting the user interactively correct the system and recording those corrections and learning from those mistakes. Yeah. So, that's okay. Well, that's in your own network, I don't believe it's based on TensorFlow, it's an interesting algorithm, it's an interesting tree search, but okay. And it could just have nothing to do with retraining, it just might be about collecting more data. But that's a topic for another. Yes, sir. All right. Yeah, okay. I was gonna bring up the other idea too around collecting, well, I can do it really briefly. We're gonna have the action button on the new device, right? And so we've talked about using that action button to possibly collect wake word mist utterances or mist, someone said the wake word doesn't get caught, right? So you might hit that action button and we could come up with something that says like, oh, did I not hear you say, hey, Minecraft, would you be all right with me uploading the last minute worth of audio or something to try and catch some of those mist wake words? Yeah, we could do that. Or that could be that action button could trigger the skill I'm thinking about, right? And which is the same thing, which would play back like the last five or 10 captured wake words and say, was this really the wake word? Was this really a wake word? Was this really a wake word? And then boom, send them up. Yeah, so that button could trigger that skill, sure. Yeah, yeah. So I think there's a lot of ideas we should think about for having, using that button that we really just haven't had on the market yet. Yeah, I think it's gonna be a thing that'll be something we wanna play around with and just figure out what is the real, what's a good user experience for that stuff. Right, yeah. Yeah, because it's gonna be better to have them help with that process manually than to try to put a bunch of code in a already tight loop that's listening and try to figure out if it was really a wake word or a mist wake word. That's not gonna happen there. I think the comment I made on your user stories was along the same thing, which was, no, I don't think it should send three samples. I think it should send one sample with two time stamps, right? Right, so, okay. Yeah. Well then. Another thing I think we're all making above the same thing. Yeah, about JIRA, I did, I don't know, you might have seen this, but you may not have the ID for the core project, no longer MC, it's core, C-O-R-E. And the ID for the Minecraft skills project is no longer NS, it's skill, S-K-I-L. So hopefully that will help some of the confusion as far as which projects, which goes where in the, of just the two letter acronym being used. Yeah, I saw you renamed the hardware to MK2 and MK1 as well, so that's good. Yeah. Always good to have TLS. Three letter acronym, recursive. All right, now I'm gonna stop it, we're done now. Talk to you all on Friday. All right, talk to you on Friday, have a good day. See you guys.