 Okay, welcome to Microsoft Developer Sync August 13th, so welcome everybody. So this will be the first of our sort of heterogeneous software meetings where we try to do things a little bit differently during the week than we do on the Monday planning meetings. So the idea here, I think, is to just discuss issues that we're having, things that we need to coordinate on, and maybe have some discussion around some of the bigger topics that have come up before. So I've got a couple of agenda items I'd like to talk about, one of them being this issue that has come up with, how do we handle architectural level features or contributions to core that originate in the community? So I'd like to have a discussion around that, and if there's anything that people have that they'd like to demo, hint hint, Ken, that would be great. But if not, that's fine. And does anybody else have any other topics that they'd like to discuss? I have something that relates, I think, which is kind of a large topic. It's more of a cultural or corporate culture topic, but it's going to touch upon the database schema. And so I have that issue. And then I also wanted to talk to you about the poll requests and bring that up. I think it might not be a bad idea regarding that for us to put together a periodic poll request committee meeting where these things are discussed amongst ourselves, and kind of the intent is initially to get through the backlog of poll requests that have been submitted and see which candidates we might want to promote and which ones we might not want to and get back to the authors. With the concept being that when user contributor or community members are writing code that they'd like to get into our code base, I think this is a golden opportunity. I think we want to keep them motivated and engaged. And I think we owe it to them to get back to them one way or the other in a timely manner and to consider all poll requests. It's not to say all poll requests will blindly make it in. It's just that I think we need to, rather than just paying lip service to it, actually put a stake in the ground and meet maybe monthly and have a poll request committee meeting. That was just my input on that. So that's the first one. If you wanted to follow up on that and go deeper, we could, or I could bring up the issues I have regarding the database schema and how we want to treat samples. Okay, so that sounds like two more topics there, right? The database and then the poll requests are two different issues. Yeah, yeah, two different topics, right? One of them is yours. And I also want to, following on from the sample thing, I also want to talk about how we're going to store. And my next thing I need to do is write the script that stores the samples on the NAS. So I want to talk through that a little bit. All right, so let me, let's talk upon that real quick. Just what I've noticed regarding data management, which is where I'm kind of at right now. The management of the data, once it's available, how it's balanced and how it can be selected and classified. And I wanted to bring up two points. And I just want to bring up one nomenclature issue real quick. There's a difference between automatic and manual classifying or tagging of data. And I was kind of looking at the documentation and realizing it can be confusing. So moving forward, if I refer to something as being tagged, and I don't explicitly say it, the assumption is that means it was manually classified. And if it's not manually classified, then it's automatically classified by a classifier. So that would be the difference I'm using a nomenclature between tagged, which would be manual, and classified, which would be automatic, because they both ultimately accomplished the same goal. But they are two different attributes, two different sets, and moving forward, I'd like to be able to select them from the database distinctly. So I could say give me all the data that's been manually tagged versus give me all the data that's been maybe manually and automatically classified, whatever. I think that's an excellent distinction. And in particular, I think it's important to make that distinction because the classification can always be regenerated, right? But that's probably not efficient. We'd like to be able to, effectively, the database then ends up as a cache of that information. And that can be really helpful in terms of using that data, or maybe even critical in terms of using that data in how we present it to the users, because this classification may end up being fed into the tagging system and the priority system for which samples we're asking users to later tag, and that sort of thing. So there's going to be some back and forth between those, and I think making that distinction is really important. So that's a good call. Okay. So the other issue is, well, there's two here, basically. Just wanted to point out, Chris, I emailed you a bunch of sample queries that touch upon some of these topics, and I explicitly tried to, like, in there, put manually tagged or manually classified versus automatically, so you could see that I was specifically trying to get at that with some of these issues. So the point is I emailed Chris a bunch of, here's kind of how I'm going to want to select from that database. Let's make sure we can get there from here. That being said, just to point out, it's very valuable in data for us to have the concept that was brought forward in the old database, which is the final tag classification, specifically the near wake word classification. And why this is so important is that if I'm training for Hey, Mycroft, there's a lot of things that could be not Hey, Mycroft, including a locomotive noise, right? But what's really valuable for being not Mycroft are near not wake words like Hey, Minecraft and Hey, Microsoft, because that will help the training process not pick up false positives because they're near. And getting that near data is difficult enough and when we do get it, we definitely need to make sure that the database gives us the ability to say, give me all the not wake words that are near this wake word so that we could have, maybe the new wake word is, I don't know, Hey, Jarvis. And so there'd be a lot of near not wake words like Jar and this and Hey, right? Those are valuable to have. And so if we can distinguish those in the database, that would be a huge win. And then the last one. Already done. Yeah, yeah. Good. What's that? Already done. Perfect. And then the I was hoping you would say that. And then the last thing, which is somewhere like there's a data type of that is wake word column that is wake word that is yes, no, and almost so almost would be your near. Yeah. Yeah. And I just wanted to point point this out because it's something I think we've been overlooking for a while. I just realized it as well. And it's kind of salient and I'm not sure how we're going to address it, but it has to do with the concept of a not wake word and what is a not wake word. So right now, I think a flaw in the existing data model that you're migrating, Chris, and I'm sure you won't have this problem, is it says, okay, this is not Hey, my craft, right? It's not the wake word. Hey, my craft, right? So it's Hey, my craft is the wake word and then it's got an indicator that says not wake word. That's wonderful. But notness. This is kind of tough, but it's it's salient. Notness is kind of related to the wake word you're saying it's not because Hey, Jarvis is not the wake word Hey, my craft, but it sure as hell is the wake word Hey, Jarvis. And so you don't want to confuse that and, you know, say, Oh, look, I'm building Jarvis. I've got this. It says it's not the wake word. They kind of meant that was not Hey, my craft, but it is Hey, Jarvis. And now it's in my negative data set as a not wake word. So the tagging of not wake words. I don't know that I have all the answers. It's just something I want to I wanted to surface that we could be cognizant of. Because one man's not wake word is another man's wake word. Does that make sense? Yeah, absolutely. And I have some thoughts about that. It occurred to me that when you were saying, you know, there's multiple different ways for things not to be the wake word, and it occurred to me that, well, it might it be important to us to have, to know which not wake word it is like Hey, my craft, you know, and Hey, Microsoft and Hey, Minecraft can be can sound similar superficially. But is it important to us to have, you know, a good distribution of Hey, Microsoft and Hey, Minecraft in our data set and to know that we have, you know, a certain distribution of those things. So saying that it's almost a wake word is useful in a certain sense. But I was thinking that maybe instead of saying what things aren't what we should just say what things are, and that then as a category we can say all the samples that are Hey, Microsoft, we can just say Oh, all of these sound like we can associate the categories the category of Hey, Microsoft sounds like Hey, Minecraft. And so all the all the samples in those those data sets will then be related in that way. And so, you know, we instead of saying what things aren't we just say what things are and then we can relate those categories that way you can be collecting a data set of, you know, Hey, Jarvis and Hey, Microsoft and Hey, whatever, all at the same time and put them in the right buckets and, you know, we could be collecting multiple data sets simultaneously and just, you know, creating a lot of positive data that we can use. Well, yeah, that's exactly where I was going. So right now everything so that was in on Hey, Microsoft. But the point is very rapidly here, we're going to be moving into the concept of user contributions for other wake words besides Hey, Microsoft. And that's where these sorts of classifying issues become more important. Sorry, Josh. Yeah, so I the mute the I just took the mute and the hang up button. Hey, the problem with tagging, especially false activation data with what it actually is, is that a lot of times it's unintelligible. So you don't really there's like two conversations going on in the background and you don't really know what the heck triggered the wake word. And it's hard to figure out what it is. So, you know, for those utterances, I think that they can hit the nail on the head. It really does have to associate its its notness with the with what it is not. So, you know, that that train sound in the background, right? You know, we don't have any idea what the sound might actually be, right? It might be a train. It might be an A380. It might be, you know, 2,750 tons of ammonium nitrate. But regardless, you know, we don't have the famous sniff what it is. But we do know that it inadvertently triggered on whatever the current wake word setting is. So the so I think Ken really nails it. We do need to associate the inadvertent activations with the the wake word that they inadvertently activated on. Now, one of the cool things, though, as as we. Oh, OK, well, then we're beating at that horse. But yes, we are. The one thing I will add is that as the once we add once we actually know what it is. So if we have a bunch of verified Hay Jarvis utterances, we can absolutely suck those into the Hay Microsoft training as what Hay Microsoft is not, because we know affirmatively that is a Hay Jarvis. And and so every time we get a validated wake word, that becomes another piece of data we can put into the wrong category for training all of the adjacent models. So I'm a little sorry about the dead horse. The schema now it's written because all these questions, if you look at the schema are answered by the scheme. So I'm a little concerned they were having these discussions about, you know, what the scheme should be when everything that we've asked, we've just talked about is already there. Yeah, Chris, I agree. Now, what would be good is if you could email me back the queries for those questions I sent you, that would demonstrate that. Yeah, this is definitively how you would select that set of samples. The only thing that's not in the schema right now is and I'm not sure off the top of my head is how would we identify a classification that is manual versus automatic? So is like, can pitch, for example, be manually tagged and automatically done? Absolutely, you know. Okay, I think I have an answer, but I'm not a five minute expert, but I think we need the concept in the data structure of tagging and classifying with them being effectively the same thing, but one represents the automated and one represents the manual and samples could even have both. Could you define the agent that did the classification? Right, exactly, that's what I was thinking. In one case, it's a user and could you define the agent that did the classification? So in one case, the agent is your classifying algorithm and in another case, it's a particular user ID. And so if at any time we find that an agent, whether it's a human or non-human agent, is corrupt in some way, we can then find all the samples that they tagged. Yeah, I hear what you're saying. So what you're saying to summarize, if I understand it correctly, is can we identify the person who tagged the sample somehow and maybe even, like Chris likes to do, in the absence of somebody doing that, it would be null, right? Yeah, I don't like null. I could make a type column on the classification table that could be used to tag or classify or whatever or not. Yeah, yeah, yeah. Or both. And yeah, I prefer not to have, like we saw Chris with nulls because then you have tough left inner joins that sometimes break and stuff. But if you have to do it that way, that's fine. But yeah, I think where Gez is coming from is important too, that if we, along with the tagging or even the classifying, right, could tie it to an entity, then if we had a bad entity, we could recover. Either that entity was a bad auto-classifier or a bad actor. So yeah, I did. I mean, I think you should have got a count, I think, right? That way, right? I thought that we were, we're not just, yeah, I thought that the database was designed such that we're logging actual tagging events in the database, right? We are. Right. So then as long as we're associating a source of that tag, then I think we're covered. Yeah, I agree. Then we just need to really, and then the, what the final values are or what the accepted values are for each, you know? And then the question becomes, what does final mean? Like, does this final mean final tag, final automated classification, you know? And you're getting pretty muddy there. Yeah, I think that's the final end of the piece. Yeah, I think that's going to be a derived set of data, right? There's the raw set of data, which is going to be, you know, the actual inputs that we get from the users or from a classifier. And, but then there's going to be derived data that we use and that's going to be probably under, you know, constant revision as we fine-tune the algorithms for selecting which data we want to use. And as we get more data, you know, how do we integrate that into the system? Like, I think that the attributes that we care about are going to be constantly evolving. And so they probably, if we need to summarize them in some way and say that, oh, this particular sample has been classified enough, I think that that belongs in its own space, right? Because that classified enough is going to change over time. And what the definition of that is, and maybe we even, I don't know, maybe that's, I don't know, I'm just spitballing here, but maybe that might even change with every iteration of the model, right? Well, all I'm saying is I don't have any easy answers. I just wanted to surface it for Chris V, as food for thought, because it's something I think we need to think about, but I don't have an easy answer for you, Chris, on how to structure it. And then just one last thing on the model. I tried to show it in those sample queries I gave you. Not all classifications are binary. So for example, right now, pitch classification is high low. It certainly could morph into high, medium, and low. Other classifications could have wider ranges. So I don't know how you're going to structure that either. I don't have any easy answers. Right now, they are, they're enumerated data types. So right now, like, is wake word, is yes known, or almost, pitch is high, medium, and low. And we can add to those numerations or subtract from them, however we need to. Yeah, that sounds about right. And I think once I start, once you have the data ported over and I start selecting from it, we can talk about some of the queries and understand maybe some things we overlook. I don't necessarily have to get everything. I haven't got the tagging stuff very much. Right now, I'm just trying to get them all at the loading part. So I haven't really dedicated a lot of time to the tagging part, but. Yeah, sure. I'll put information from what I did. From a product standpoint, like what we're actually producing at the end of this, I see for each individual piece of data, the ability to surface a web page that shows at the top of the page, the waveform and provides a player so they can play the sound. And then below it shows a log of everything we know about that piece of data. So it showed up here from this user ID, this automated tagging process, there's a transaction, right? It touched it and classified it as whatever, almost. That was validated by this user ID, so on and so forth. And so we have basically a transaction history for every single piece of data that we use for the training. And it's all shoved into a database, but if it's some future point in time, we wanna put a front end around that, that becomes a web page that we can click on any given piece of data and see its history, right? And then on the automated side of things, eventually some kind of an administrative construct would be able to go in and say, hey, user number 114936 is a known bad actor, right? This is somebody who sat their two-year-old in front of the system and had them randomly click buttons. So their classification data is garbage. We would be able to back out, and here are all the pieces of data that this malicious actor touched and put them back into the previous state, right? Like nuke off the tags. And this transaction stuff is actually an internal join. So it's not a particularly complicated data construct. It's just self-linking, right? It's all shoved into a single table. Yeah, from a product perspective, a page that shows that the transaction history for that particular sound or data object. Yeah, I don't wanna get too far into tagging right now because I wanna finish up this storage stuff and we'll have these discussions all over again when we start talking about tagging and designing that. So I'd rather hold off on some of those discussions. Make some high-level stuff. Agreed, Chris, agreed, Chris. The transaction history is wonderful and it could help with debugging, but there sure does sound like a phase two kind of thing for where you're at right now. But just wanted to mention that, just like I'm gonna be doing a lot of, just to give you a heads up, a lot of group buys and havings because I'm gonna start doing selects that limit by the contributor. So no more than 10 from each user and then group buy having. So there's gonna be a lot of that. Just wanna make sure we can get there from here. That's why I tried to send you some SQL queries. But yeah, you're right. Some of this stuff is obviously a phase two from where we're at now. Okay, yeah, it sounds like we've got the database structure is adequate for what we need and where we need to go. I think it is gonna be a separate discussion in terms of the system that ends up presenting these to the user for tagging and that sort of thing. That's kind of decoupled from the acquisition side. It's a whole separate problem to solve and it's gonna go hand in hand with the training algorithms and the sort of the research side of things there as well. So yeah. I don't know if you saw, but I cut a ticket for that. Yes, I signed it to you. I don't know if you're the right guy or the webpage for tagging. Well, I think that we should start that discussion actually with Derek would be my thought. It's as much of a user experience issue as it is implementation issue. Yeah. There you go, Derek. Sorry to sidetrack us on the data issues. I just wanted to give Chris a heads up. I didn't wanna come later and say, oh, well I need this and not try to expose everything. Yeah, and that's what I had on that other than the full request committee. That's all I had. Okay, well let's talk about that. Chris has an issue with deciding how we're physically going to store these samples, right? Is that the question? Yeah. Yes and no. So yeah, so the next thing I'm gonna do starting probably right after this meeting is writing a script that's gonna take the data that's stored on the API server of sample files and getting them onto long-term storage. So my current assumptions are that that long-term storage is our NAS in Lawrence. And so what I really need to know is, and there's a directory structure defined in Confluence. I wanna make sure everyone's okay with that or at least Ken's okay with that. And then the file naming convention so that basically I wanna do this right the first time. Yeah, you broke up a little bit there. I couldn't really hear you, but I was wondering, last time we had discussed this, you had mentioned maybe we could FCP those files over, but that seems a bit messy. I think what would probably be cleanest is just to have Josh have the guys expose the mount of the NAS to your Selene server. That would make it easier for you to move them over versus SCPing them to another machine. Is that sound about right or do you wanna SCP them? Well, what protocol is my question? I mean, I guess we could advertise an NFS mount like from Lawrence to digital oceans in New York and probably authenticate it based on like IP address or something, I guess, but it's not something that would be within that. And the SCP uses SSH, SSH is already exposed. On, you know, we'd have to have maybe the Persever over there in Lawrence, we could SSH too, something that's already got it mounted. And then- Yeah, you just, there's a lot of, I mean, Ken does have a point, there's a lot of processing overhead with that, but, you know, it's not, we're not dealing with Google levels of data at the moment, right? No, and it's a batch job. I mean, performance doesn't really matter. This is gonna happen overnight. You know, if it takes an hour, it takes an hour, but- Yeah, well, Chris, I really don't care. All I was getting at was that if you had a mount exposed, it would make your life a lot easier, but you can certainly accomplish it with SCP and I've forgotten, completely overlooked the fact that these servers were not brethren, that some of, I guess, Selene is running in DigitalOcean's cloud, is that correct? Yep. Yeah, so, all right, well then, you know what's best. So- So, over SSH, just after the mount and not as a local directory, it's not really difficult. I'm sorry, what was that you broke up there, Chris? So, you can mount file systems over SSH. So, you can actually mount it as a directory with SSH as the underlying protocol and I mean, I can hook you up with that if you want to. Okay. Yeah, that's Chris's call. He's writing the code, but yeah. I would think a mount would be easier, Chris, but if SCP is easier for you, then, you know, I think you have that already. Yeah, I think so too. So then, is it okay? Mind if I share my screen real quick when you show you the directory structure that is in Confluence, make sure. Yeah, yeah, that's fine. That's fine. I'm just turning off my video because everybody's breaking up to me for some reason. All right. Here's a second to bring up the document. Okay, can everybody see my Confluence document here? Yes. Yes. All right, so, I've done the sample collection part and it's ready to go. And this is actually wrong right now. The file name is just account ID dot timestamp because the wake word is gonna be the directory. So that's wrong, but so assuming an account ID dot timestamp dot wave file, this is the directory structure that I was considering. Basically, you have the name of the wake word whether or not it was classified, like classification is complete and that's a whole other thing about, you know, what does it mean if it's classified or not? Or, but, and then there's the unclassified. You took me aback with just account ID timestamp, only because looking forward, people may submit batches of them and I'm not sure that's gonna play out, but yeah, sure. We can reflect differently later, I guess. Right now, the only way we, as of right now, the only way we have to collect these is from devices and maybe from a single sign on supported application, so. So what I, the only thing I would say is that I see where you're going with this and based upon our conversation earlier, it's technically correct. All I would point out is that there is certainly a higher value to be placed, in my opinion, on samples that have been manually processed versus not. That to me is a discriminator, a huge discriminator, but I mean, all of that is actually technically available from the database, but I do agree that, you know, breaking it down to things that have been classified and things that haven't is a good breakout and then by the wakeward of me too. I have a concern actually about that because whether something is classified or unclassified can change over time and I think that this directory structure shouldn't be dependent on things that can change over time. But a manual processing or manual tagging of a sample cannot change over time, correct? Well, the tagging itself is an event that, well, we can't erase the event, but we could erase the sample that it applies to. Yes, in which case it wouldn't show up here and the value of the tagging could change, but the fact that a human had touched it, once that happened, you would think is sticky. So I am wondering, is it worth having classified versus unclassified here? I mean, if we have a pointer in the database to the directory this thing is in, we just need to have wakeward and then groups of 50,000 or whatever and use a database to figure out what's tagged and what's not rather than the directory structure. It's a slippery slope because it sure did seem like when I was looking at it, I'm not sure when you're looking at it that it was kind of assuring that all the new samples we just gathered today are obviously going into the untagged section of the file system, which is why I didn't break them out at wakeward at the highest level, but just that these have been tagged or manually classified and these still need attention. And the group that needs attention in my estimation is going to be huge. Orders of magnitude larger than the group that's had attention paid to it. And I was just thinking that there might be some value in partitioning the data that way. But again, it's a slippery slope, Chris. I don't know the right way. And certainly the things you pointed out are relevant here as well, but the classifications can change over time. Well, and it may be classified in multiple different ways, right? Like if you're using a wakeward, is it classified with respect to the schema we're using this year or is it classified with respect to the schema we're using next year? You know, we change what it means to have a classified sample. You know, we change our requirements from must be, you know, tagged in the same way by two different users to must be tagged in the same way by three different users. Then suddenly all of our, you know, not necessarily all of our data needs to change. It just depends on how many times it's been tagged, right? That's a database query, not a directory, you know, look up, I think. Because you'd have to end up moving your pointers in your directory around constantly as we change those definitions. Yeah. Yeah, I know, I know, and you're right there. Again, to me, it was just, has this been touched by a human or not? Because that data is more valuable than data that hasn't been. But again, that's information that could be derived technically from the schema. Yeah, I just don't know, I'm glad that we're having these discussions because this is kind of where I was stuck like a week or two ago, which is all these data related, you know, our data engineering related questions that I didn't have any easy answers to. And it's refreshing to see that people just don't go, oh, well, stupid, do this. So that's good. Yeah, no, I think that the group, I think just going by, you know, the wake word name, and it's really not wake word name, it's intended wake word name, right? Because it might be Hey Microsoft. So that's not the actual wake word. Yeah, it's considered wake word name. Be associated with the Hey Microsoft. Yeah, that's the wake word name that was set on the device or was set by the, you know, or whatever. Right, and then anything, you know, any distinctions or any kind of optimization we want to make in terms of the directory structure down the road, you know, if we start with this, which is the most generic way, then, you know, as we run into, I don't know whether it's gonna be performance problems or what, you know, this is sort of the simplest starting point. And I think that that'll make sense. Yeah, I mean, it'd be refreshing just to get this data into subdirectories that are smaller than a couple of million entries. Right, I mean, I guess, let me see here. You've got the, you've got the file name is based on the account ID and timestamp. I mean, that's another thing that won't change, right? The file, the account ID is a thing that might be useful to group things by, so that's something we could consider grouping things by. Because it looks like, you know, right now the default group one, two, three through N is gonna be based on the timestamp, right? If we just take the first 50,000 that the system collects, put them into group one, the next 50,000 going to group two, then you're de facto sorting by timestamp. But we could also consider sorting by account ID. So that might, you know, I don't know if there's any value to. My initial, I see, yeah, I mean, I don't like having arbitrarily named directories necessarily. So my initial thought was maybe this should, the second level should be an account ID or even a date, like a date we got it. Then we'd have to have, so the only problem with that would potentially be, you know, scalability if we have, you know, if we, if things blow up and we have a million people doing things and then there's a million accounts here and then we have the same problem. Well, yeah. This more generic solution, I think is more scalable. And I, you know, I do want to have a million people contributing data. I think our collection scheme will change when we have a million people submitting wake words every day because we won't probably need that much data, right? But, you know, we'll probably go through some sort of automated filtering process where it just checks to see if this contribution has any chance of improving the system and throws it away if it doesn't, you know, but, but I think the more generic it is at this level, the probably the more scalable it is. Yeah. I would recommend against organizing them by account ID sub-directories only because that that in and of itself can become problematic as the account is improved. But yeah, I don't know, Chris. I mean, I'm with you. Just arbitrary hashes on file names don't really bring a lot of value to the equation. So I don't, I guess the bottom line, Chris, is I'm going to be comfortable with anything you come up with and I don't have any easy answers for you. Okay. At some point, we're going to have to deal with the issue that we may end up with more than 50,000 directories. You know, if N gets to 50,000, now suddenly we're, you know, we're at the N squared problem, right? So we need to make sure that whatever system we come up with is going to scale in that sense too. You know, we have, we already have 50,000 user accounts. So if we were to sort things by user account, we'd are, we were already hit our limit, right? So we need to think about how this is going to scale with, you know, over time that way as well. If we go by date, well, there's probably, you know, if we get to 50,000 days in the system, then I think we'll probably be doing really great. But we can go date then group. Yeah, that might be useful. I don't know. Yeah, I think that might be a good way. Maybe we should try that because that way, you know, having that many groups in one day seems very unlikely, right? Yeah. Yeah, and, oh, by the way, isn't 50,000 times 50,000 something like two and a half gig or something? So it's a huge number. It is, but yeah. I guess eventually we may get to the point where more than a barrier and collect as much data as we can. I just don't know yet. I don't know. That's one of those hyper parameters that's tough to corner is one is enough samples enough and one is it not enough. And I'm still trying to, you know, answer that as I go through the, the thing is for now we're gonna have to do one. So everything goes in there. Yeah, this looks good to me. At the end of this, we're probably gonna end up moving into a NoSQL system. When we reach a certain amount of data, it's gonna make sense for us to move into Mongo or move into Neo4j or move into Cassandra and get out of the relational database business because as the datasets get bigger and bigger relational databases just can't handle it. So. And this is perfect for something like Mongo. Yeah. And the thing is you're absolutely right, Josh, because NoSQL databases handle variable or unknown attribute classification layering very easily. Whereas databases tend to have to be restructured. And if this database is, you know, I don't know, a couple of gigabytes of samples just restructuring and re-indexing could take weeks. So yeah, we may get there sooner rather than later. Yeah. Yeah. In the meantime, it would be the nearest gator to the boat as they say in Alabama. And, you know, get the haymicraft system up and running, get the training system up and running so that we can move forward with actual, per holding the company back at this point is, you know, that eight plus two project we launched, you know, 14 months ago, you know, being able to deliver a reliable experience on one smart speaker, you know, for just the haymicraft wake word, you know, in a way that is reliable and presentable. And, you know, once we're able to do that, a bunch of other doors kick open and then, you know, hopefully what we're talking about now is to a specialty developer who specializes in NoSQL databases. Because we have money to hire. Yeah, yeah, sure. Agreed. Okay, so let's shift gears then and talk about community contributions to core. First thing I'd like to interject here is that I was really excited by this idea of a plugin system where we can have, you know, significant changes to the core that don't have to be, you know, rolled up into the mainline database. So I don't, I think that'll be a really significant development and allow, you know, people to customize the system in ways that they want without bloating the core. So that's great. In terms of the particulars of how we want to interface with the community, you know, my background, you know, when I started off, it was very hierarchical approach. Like I would sit in my ivory tower and come up with designs and hand them down to the engineering managers and then they would, you know, we'd work hand in hand with the actual developers to build out the test system and the actual implementation and verification and all that kind of stuff. But information was, you know, very kind of one way. This is a different environment. And I think in order to continue to reap the benefits of this sort of bi-directional environment, I want to make sure that we don't, you know, lose the agility that comes with that. So while I'm a huge fan of documentation, like it makes me really nervous when I don't have a good spec for something before it starts even getting worked on, right? I see the value in working with the community in terms of, you know, exploring different ideas and spaces. And so ultimately what I'd like to do is, you know, find some way that we can, you know, keep working with the community in this interactive fashion. But ultimately, once something does get rolled in, especially into core, that we end up with really good documentation for it. And that before, and even before then, that we have some sort of process whereby before too much work gets done on it, you know, we make sure that we've vetted against our roadmap and what we can see coming. But even given that, you know, well, we're probably never gonna get the perfect solution on the first try. And if a user can contribute something to us, that's like an 80% solution to a problem we have, or that's 100% solution to a problem we have right now, but is only an 80% solution to the big picture problem that we have in six months or whatever, then I don't think that it's, you know, necessary for us to block that in terms of the perfect solution, you know, in terms of waiting for the perfect solution. The biggest problem with that statement is the backwards compatibility issue. Because we've already been stuck a few times saying, oh, well, we can't do X because it's gonna break Y and because Y came before X and then, you know, it really came back into a corner. So, you know, I'm all for moving fast and being agile, but if you get too far down somewhere and decide you wanna go a different direction, then you could really screw yourself. Well, yeah, no, I get that. And I think that the, but you know, on the flip side of that, if we're not able to do the work that we need to do to solve a particular problem today, but a community member can solve, you know, most of that problem today, then at some point we're gonna have to do that work anyway, right? So if it's a matter of introducing two changes that we have to go back through and do a whole bunch of compatibility fix ups for, well, you know, I guess we have to weigh the effort on our part of doing those fix ups versus the effort of, you know, engaging with that community member or team of people or whoever to get the solution that we'd prefer the first time around, right? I don't think it's just the effort of doing the compatibility. I think the more backward compatibility code you put in a system, the more complex it gets. So if we find ourselves doing a lot of, you know, you know, of workarounds to make things compatible, you know, because we had it the wrong way, then there's a lot of this in core right now, actually, you know, and for example, there's two, this upload wakeboard thing Chris just worked on. There's just two methods in the mic class to upload a wakeboard, one's old. I mean, this is the kind of stuff that happens if you're not careful. And right now it's all over our code base. And, you know, if we're not careful, it'll continue to be all over our code base and make it hard to maintain. Yeah, those are all valid points, but I'm looking at it a little bit differently. I'm looking at contributions from the user community as mana from heaven, you know, you can't beat the price. And how do we leverage that starting, you know, huge benefit such that we can balance it against kind of what you're speaking to Chris, but what I would call it is more testability. In other words, we have an existing system, right? We're going to allow a new pull request from the community to come in. And the question is, are our test systems robust enough to know if it broke something or not? And if not, and the assumption is that they all never be perfect. If not, is it worth our effort at that point to push them to do the testing or to build the testing to protect ourselves from it versus rejecting the pull request because it broke something. And my concern is that if we have a really good robust test system, any pull request could be immediately, it could be immediately determined on the merge, which I assume kicks off a Jenkins run, whether it broke anything or not. Now, you know, that's not the end all. I mean, there's subtle breakages that slip through the cracks. There's semantic changes. There's all sorts of stuff. And I think what you were speaking to Chris was more the cleanliness of the pull request because it carries the existing systems baggage and it has to work within that framework versus restructuring that. And that's why I think it's really important. We as a organization address this seriously with a periodic pull request committee meeting that we balance as a team and weigh the ultimate benefit or value of the pull request in our roadmap to the pain threshold. Some are gonna be no-braineries. Oh, this is way too complicated. It's gonna be too much aggravation. We're not gonna do it. Some are gonna be no-braineries the other way. It's just a modification of the skills. It gives skills an additional benefit and whatever. But some of them are gonna be tricky. Like the one we have, we've been discussing kind of back and forth where it's kind of significant. There's gonna be a lot of restructuring and maybe the design is not compatible or completely compatible with what we have. And again, that's why I keep harping back to or harping back to the issue that we're trying to balance a thin line here which is keeping the community engaged and motivated because nothing's more demotivating than to do a bunch of work and then have it be rejected out of hand. I wouldn't wanna keep contributing to that. But then the testability of said enhancements or additions to the code. And can we protect ourselves from it? Do we have the means to adequately protect ourselves so we know that we're guaranteed this isn't gonna break something. So that's the balance. And I think it's not a simple yes, no binary kind of answer. I think that's why we need a committee meeting with upper management in there to help us prioritize which of these would deserve our attention. And then as engineers, we can take a look at it and say, well, this is kind of what it's gonna cost and they can use that cost in their determination. So that's just my take on it. Again, I wanna encourage the community to contribute the more the merrier but how do we manage that process? Yeah, and I can see as our community grows and as our staff grows and our ability to interface with the community increases then we may end up with people on staff whose whole job it is to take valuable contributions that we get from the community and basically refactor them into the way we see things working down the road. So that may end up very well and it being like a valuable way to proceed. But I think we need to lay out the expectations with our community too as to what it is that we expect in terms of contributions. Like so we've, I see regular, I see pull requests regularly kicked back for things like, oh, hey, you forgot to run basically the lint system on this, right? And if these line numbers are too long, it's simple stuff that they can easily fix, right? And as long as the community knows that stuff ahead of time they can say, oh, yeah, that's my bad, right? And if we're gonna expect them to write a robust test suite to go along with their changes, depending on the extent of their changes, I think it'd be, if we make that note up front then it shouldn't be a surprise to them if they make an extensive set of improvements to core and then we kick it back because although the changes seem to work they didn't provide us with the testing framework that we need to verify that, right? Now we may decide to take that upon ourselves but at least establishing up front with the community what our expectations are for each sort of kind of scope of contribution, I think will be important. But we've been talking a lot about the abstract here. Maybe we could shift into talking about the specifics of this particular contribution and what about the particular pull requests that we're considering right now is not to Chris's liking and how can we address that with the community in a timely fashion that doesn't make people feel like they're fighting an uphill battle and that we're working together to get this thing done. Yeah, but Mike, the other thing just before we go to that is that our community of contributors is more of a customer than a vendor relationship and I think the sword cuts both ways. I think we kind of want to surface at some point or publish that we'll do our best to get back to you on your submission within a reasonable amount of time so that they have that expectation as well which is why I was saying if the committee meets monthly we could possibly tell them that we'll get back to you within 90 days or something. But just to give them a commitment as well because we want to keep them engaged we want to keep them motivated and we don't want to keep them hanging. I suspect we have some pull requests out there for years and I don't know of what value a two year old pull request is now. So I mean, if it was my pull request from two years ago and now you want to use it, I'm gonna tell you good go use it, don't bother me, I'm busy. I got bigger fish to fry, I moved on. And I'm trying to avoid that. So I think we owe the community also some commitments is all I was getting at. Yeah, I agree with that. I don't know that a monthly meeting is the right solution but it might be, I'm not saying it's not. In fact, my first thought was, wow, that seems like a lot of time between, to possibly let a lapse between something being submitted and then reviewed. But on the other hand, at this very moment in time we're a pretty small team. And so we can easily be overwhelmed. Yeah, I'm concerned about how it was an impact our velocity and the tasks that we have prioritized if we're spending half a day every day examining full requests and going back and forth with community members. Well, I think if we classify them in terms of things that, well, I think, yeah, I think if we can classify them in terms of things that are addressing current issues of ours that we're working on versus things that are maybe tangential to what we want or what are critical to us at this moment in time, then I think we could at least make it clear to the community that we're going to respond to things that are in our critical path in a timely fashion and things that are not as time permits. And as our team grows, I would love to be able to make more firm commitments about that review process. But at this point in time, I don't think we're in a position to promise anything with respect to pet features that are not necessarily gonna contribute to getting the mark two out there and making a great first user experience. Yeah, I think we've got to go through a few parts of that sort of stuff, like there's many bug fixes that are, as someone said, like really straightforward. And they should be handled quite quickly and there are things that are addressing current needs and so we need to handle those in some time fashion. And then there's the like, I want to kind of branch out and do this completely new thing, which is cool, but like not really what we're after. But I think ultimately we want to, I think, people that are contributing, they want to do things that are beneficial for Minecraft as well. And so however we can help people to know what it is that we are working on and what our direction is. And how their contribution ties into that is beneficial. And I think people are very happy to get feedback on how they can improve their code. We get a lot of people who are the entire spectrum of development experience. And yeah, how we want things structured and people are happy to make those changes most of the time. So it's just about that old time factor that we have. Okay. So with that in mind, do we have, what in particular is keeping us from going ahead and integrating these recent changes that I know, Chris wanted to get into before our 2008 release. At least that was the initial plan. I don't know if that's still a realistic goal or not. Or am I conflating two different issues here? Are you talking about the pull request in question that's been going around? Or are you talking about what Chris has going on? Cause you said Chris, I think you misspoke. Oh, I meant Chris Gessling. So. Ah, yeah. There we go. I think we're talking about two at the moment. There's the plugin system, the audio backend, TTS and STT. And so the benefit, you know, I think as you talked about earlier, Michael, like this is a really useful one in terms of stripping out stuff from core that doesn't really need to be there. So at the moment we have support for a whole range of TTS and STT systems in particular. And then a couple of audio backends. And, you know, the number of users that are using Yandex as their TTS is probably fairly small, but it's awesome that Russian users or, you know, users that are served by Yandex can use that. But by having it in core, it kind of indicates that A, we're supporting it. And B, it's just extra code that doesn't need to be on every single device, you know. So I guess the benefit of trying to get that in before 2008 is that, I mean, we want to kind of give it some really solid testing, but it means that we can strip out those extra services when we do a major release. But, you know, at the end of the day, it's not going to change the world if we have to hold that off until later on, you know. So in addition to improving the architecture of the system overall, it also addresses a particular problem we're having in terms of the boot sequence, right? Not for the plugin, that's separate. That's the product, that's the other PR. Yeah, so there's the plugin system, which is clearly about tidying things up and also making it easier for people to contribute new things. We've got a few PRs that are about adding new TTS services or modifying existing TTS services in a very particular way, like the insight team, they want to add a particular way of varying the rate of speech in Mimic. And with this plugin system, they can fork the Mimic TTS class and then use that themselves. And so then the other PR is specifically looking at how do we know the state of each of our services and therefore confirm whether or not the system is actually ready or not. And Chris, V, I believe you had concerns regarding the status service. Was that what you had surfaced, that there was a status service that we had to develop and perhaps this was not amenable to that or was going to preclude us from doing that or what were your concerns there? Yeah, mostly architectural. I mean, if we're going to have, what's the bigger picture as far as what we want to do with statuses? We talked about an prior meeting, I believe, having a new service that runs that knows the statuses or can query the statuses of each service and act on statuses that may not be ideal. And how does that, how would that interact with the work that's been done already, if at all? And maybe there are completely independent things. Maybe this is, and maybe what he's got has a way to interact with it that would be fine. I just, we haven't spec that out yet or really talked much about it or looked at it conceptually. So that's really my concern is that this, we introduce something like this that Pigeon holds us when they want to do that service and to do in something a certain way. Right, so this seems to be one of those cases where even if it doesn't do exactly what we want in the long term, I don't necessarily see that there's any downside to using it to solve the problem that it's intending to solve right now because... It's a backwards compatibility problem, right? So if we implement bus messages and stuff that use this and we want to do something different with that, with the new service, what, how much backwards compatibility issues are we gonna have when we do that? I mean, we don't really... And this is a more generic thing is, without a larger view of how we want things to look going forward. We implement little things, little smaller things like this and knock back ourselves into a corner. Yeah, I mean, the concern I have there is that we don't want to allow fear of the unknown and what may come in the future to keep us mired in quicksand today to a certain extent. If there's something in this particular pull request that we're concerned about, we should call it out, address it with the author and give them an opportunity to correct it. But if what I'm hearing is it's more, we think we have some other stuff coming down the line and this might not work with it, I would really just think we could drive off that bridge when we come to it, no? Well, I hear what Chris is saying in terms of, maintaining multiple levels of backwards compatibility. We're talking here about a system that's gonna impact the interface between code repositories that are distinct from each other, right? Because we're talking about precise and mimic interfacing with core, if I'm understanding it. No, this is currently the role in all the existing services that run as part of core. Currently, it doesn't take on the issue, really. It could, I don't think so. I do believe that the bus approach serves us well in that we have a bus where we can promote new messages and new functionality can latch on to those messages. It doesn't mean the old messages have to go away and the old functionality has to go away. That's why I'm kind of pushing to say, rather than saying do we wanna allow it in because we have these other issues, why don't we just say what is it that's precluding us from getting this pull request in there specifically and maybe attack it from that approach? So yeah, so is there anything in this pull request that goes outside of a single repository? No, it's all on core. Okay, so that seems to limit the danger, in my opinion, because we're not touching any interfaces that go across different versions of things that might get out of sync. Yeah, and the next question becomes in core, how confident are we that our existing testing methodology will be able to handle any breakages that this pull request engender? We'll need to have some more tests, probably for this, it's a new concept. So I don't know if it's behavioral or unit or both, but... Oh, no, no, no, I wasn't saying new tests for this. That comes, that goes without saying. I'm saying how confident are we that our existing systems that could be impacted by this are tested enough to know that they've been impacted by this? No, I mean, Boykamp is passing for this, right? Right now? Yeah, yeah, yeah. And so I know that this isn't a complete solution because we've already talked about how the fact that it's sort of, you know, it's asymmetrical with respect to Padatius and how it's linked into the system, right? Not every service is treated in the same way. Well, yeah, so I mean, Padatius is a part of the intent service, which is a part of the skill service. So with the skill service being ready, it assumes that the Padatius is services ready. And so it's up to the skill service to say, yes, I am ready. But this is more about consolidating, basically purely just asking each of the five services. So the bus, voice, audio, skills, and this one, oh, enclosure. Asking each of the services, yeah, what their status is. And then it's up to each of those services to actually define, you know, have I started, am I ready? Am I in an error state? Am I shutting down? Yeah, that kind of thing. So it doesn't actually have an opinionated kind of, it doesn't have an opinion on, you know, those lower level things. Does this, in your opinion, impact our ability to implement that supervisor service that Chris Vier was talking about down the road? Well, it depends on what that service looks like, I think. That's my point. I mean, I do totally get Chris's concerns around back of compatibility, you know, because there's also gonna be, you know, we have other projects that's built on top of Microsoft, right? So if we go down a particular path, then they're gonna say, okay, well, this is the way that I check in with Microsoft services. And so then if we make a change on that in the future, then it's not just our own code that we need to worry about. You know, it's the businesses that are relying on our platform and how they need to modify things. I mean, yeah, so there's a documentation that I linked. I'm not sure how many people had a chance to look at it, but it's a pretty, I think it's a fairly flexible structure. So I can't, I don't see major problems with an external status service, but, you know, Chris has a lot more experience with that for the stuff than I do. So, you know, I don't wanna make a final follow-up. That's gonna look like before we kind of pass it out. And I just wanted them to point out, I was pretty deep in the Padacius handoff code with Ake during the first week. It's been a couple of months, but in a sense was that it's intrinsically broken. It's just wrong. Padacius is a outlier to the way that that is supposed to work. It kind of works well with skills. And then it goes, oh, but if it's Padacius, then it's like this exception. And we looked at that and he said, yeah, I've been wanting to fix that for a long time now. So. Well, there's a whole separate PR that's like fully re-factoring the intent of this. That's another discussion. I've got bald under the, I'll get to it when I get to it, PRs. I think it is a really good change though. Like I've had quite a tape look at it. And it's so much clearer. Yeah. Okay. And this does, this is the change that does particularly address an issue that we have. No, I mean, it's a good change. I'm not, I'm not, I'm certainly not dinging the value of knowing the status of our services and knowing when we're actually ready to go. I think that's a very valuable piece of functionality. Again, my only real concern is how does it fit in with things we know are coming down the pipe? And if, you know, in this case, we've gotten far enough, maybe we just go ahead and let it go and we'll figure out how to deal with it later. I'm just going forward, you know, how do we not get ourselves in this position again? Well, I think to a certain extent, I mean, we'll always have these kinds of things cropping up. I think we'll have an increased capacity to deal with them when we've got a larger staff. But there's always going to be issues of, you know, people bringing up, you know, potentially major changes that don't quite necessarily fit in with what we're doing, you know, but might. So, you know, it's kind of, that's one of the things about this sort of creative work is that, you know, we don't know how much work it's going to be until we actually have done it, you know, so. I think it comes back to that categorization, like how do we, you know, define what's something that we can handle, you know, as an individual working on the code base and what's something that, you know, is larger and needs to come to the group. And, you know, looking at things like Python's whole, you know, pet process, you know, we don't want to get, we don't want to create too much bureaucracy to take a bureaucracy, but I think having a clear way that people can pitch changes and they're reviewed by the team and the community and, you know, developed into a final spec before anyone gets too carried away would be beneficial. Yeah. Well, I'm, you know, in light of the fact that there's no specific objections to the PR, I'm inclined to be agreeable towards including it. I think we should clearly lay out what our requirements are in terms of like, you know, documentation and that sort of thing and even share with, you know, the contributors what our current concerns are, right? I think we have, you know, but make sure that we have, you know, or even any vague concerns we have, you know, given the fact that we have certain concepts of what we want to do in the future and maybe they have good solutions for, oh, okay, yeah. You want to implement some sort of supervisory process down the road, here's how it would work with this particular system, you know, or at least give them the opportunity to maybe address our concerns. So, but, you know, as things stand now, it looks like it solves a real problem that we have and we don't have any specific objections to it. So, and it's occurring at a time, you know, on one of these boundaries where, you know, twice a year we do afford ourselves the opportunity to make breaking changes to the, you know, to the system in terms of backwards compatibility. So, you know, it seems to me that the worst case outcome is that, well, okay, now we've got to wait till 2102 to fix this if we don't like it, right? Yeah, I think the change, this change itself shouldn't really break anything, but taking it out, right. Yeah, I think something maybe a future discussion is, you know, about backwards compatibility and, you know, what things need to be more careful about, you know, what things really are impacted when we talk about backwards compatibility and what things are, you know, like a refactor of the intense service, you know, it probably isn't gonna, you know, you wouldn't think would hurt anything, but does it, you know, where do we have to be careful about this? Where don't we? Well, yeah, specifically if people are relying on bugs being in the system to work, get their software to work they want it to. I remember having to specifically implement incorrect addition in hardware because one company did it and they, and everyone else expected everyone else to do it the same way. So, yeah. Yeah, I mean, we don't know, we don't know. I don't know how many people are using, you know, working it and using it. Anyway, that's a different discussion for a different day. But yeah, oh, it's okay. So I haven't even actually reviewed the PR yet. I kind of started to and then I just came up. So I'll review that today for tomorrow and provide some feedback. Yeah. Yeah, so I think we definitely don't want to put it into a 20.2.5, I'm hoping that I can release that today. As if people had a look at their relationship, they're pretty massive. Well, it's been a while since we've done a release. Yeah, yeah, yeah. I think the one piece left is the PR on turning off the wake load upload where I literally just commented out the code. It sounds like Chris had a look at it because you saw the duplicate method as well. Why did you comment it out instead of removing it? We can remove it. I was just, I put a debug message in there that said, this has been deactivated for 20.2 because we're deprecating the API. It may serve as a good placeholder for re-implementing it with a new API. Yeah, well, I figured we would uncomment the code and the configuration file changes and the code actually will stay the same for the new API, I presume, because it is a simple, close request. But yeah, I can delete the code instead. Okay, I mean, if the intent was just that I'm gonna I'll replace that when I implement the new one, that's fine. We're gonna have to put a call new API in there somewhere. And that's probably, that's probably exactly where we'll put it, but you can probably go ahead and get rid of the old one. The one that's no longer used. Oh yeah, well, in the configuration you mean, or? Well, I mean, there's two methods in that class, right? They do the same thing and one of them is never not used at all. Oh yeah, so I deleted the duplicate method. So there is now only one, there's now only one method that uploads to the make. But yeah, the one that I thought we should keep, I commented out. Okay, okay, then I can just change that to do it. Well, I commented the logic within the method. So the method still gets called. It logs a debug message to let people know. So it was kind of the principle, I guess, was the least amount of touching as possible. Okay, so unless there's any other burning issues, which there shouldn't be at this point, we're gonna have another meeting tomorrow. So let's call it for today and we can follow up tomorrow with anything else that we need to do. It's all right, sounds good. Including I know that we didn't specifically address Ken's suggestion of regular PR review meetings. I think we should come up with a concrete answer to that. So let's make sure that's our first agenda item for tomorrow. All right, tomorrow at five o'clock, tomorrow at the same time? Yes, and going forward from this week, Monday, Wednesday, Friday at the same time. Okay, okay. I didn't realize we were gonna meet tomorrow too. Okay. All right, thanks, everybody. Have a good night. Bye. See you later.