 All right, welcome to Microsoft's DevSync on July 23rd. So let's get started again with the same format we tried earlier this week. And we'll start with the good and the bad and the ugly, starting with Chris Bayer today. Yeah, you're muted. That's the bad. So the good, I was able to put together a Confluence document with the options for Wi-Fi setup and how we wanna, some decision points on how we want to make that decision internally. I was also able to contribute to the document gets started about the GUI decision. And now we're gonna make that. And I started working on an issue with the Kivi UI before I was abruptly pulled away from my home and now my parents' house. So my productivity has been not great the last two days. That's the ugly. I'm here helping out my family, so I'm doing what I can, but it's not as much as I would do doing if I was at home. So after the documentation, I think I'm going to pull some Salini tickets since I don't have a device right now. I'll do some of Salini work through that. I can do that without the device, some of those bug fixes. And then I'll get back on the device when I get home. Okay, great. How long are you planning on being there? Or do you think you might be there? Right now it's open-ended. I'm hoping to leave though to go home Sunday, but it's not for sure yet. Okay. All right, thanks, Gez. Yeah. So as Chris mentioned, I started the GUI thing, but only that for a separate discussion. Also, we've shifted the TV image to be USB bootable, but I think Derek will probably go into the details there. To get it working, it was a little bit of messing around, but that's because we're working off an existing image. I think if we build future images specifically on a USB, then we won't have the same problem. It's just because the usable space on two different drives doesn't necessarily match up. Are they good things? Well, the other good thing is the boot stuff that I've been looking into, it turns out that sometimes some of our services just don't start up, but the Microsoft.ready message that reports, I guess this is not a good, it's good that I discovered it, but the Microsoft.ready message that gets sent out to say that Microsoft is ready doesn't actually check the majority of services that run Microsoft. It sort of assumes that the other things are running. It's really just saying that the skilled service is running and the message bus because it has to repeat the message. Doesn't that come from a predation or something too? Some place weird? Yeah, it comes from predation. It's a chain of events, and then based on that chain of events, you assume that everything in that chain is running. But it really begs to, well, suggest that a shift so that we're either having an external monitoring something that says that checks in with each of the services and waits for them all to be, properly booted and running, and then says Microsoft's ready or something internal that checks on that as well. So the good thing is that, okay, it's already been working on service hooks, so hooks within each of the services to say that things are booting, things are having that there's been an error on startup or that everything started successfully. It doesn't, some of them have watchdogs, some of them don't. Yeah, that all needs, I'm basically at the moment in the process of figuring out what that might look like from a boot perspective. So just FYI, I'd probably classify that as an ugly issue rather than a good one. Yeah, you've just discovered a giant spaghetti mess of work to do. Yeah, sorry, I'd like to say positive. The good part was that I found out that it's happening. Because it also highlighted why the news and play fields just randomly failed in Boitconk sometimes, because I was able to replicate that when the audio service doesn't start, that happens because there's nothing to stop, there's no playback to stop because there's no audio service. And so, finding out why these seemingly random events that are occurring is really good as well. Yeah, that is good. Again, it's ugly. Agreed. Yeah, no, that's good. That explains some things and getting a better definition on what ready means I think is a great step forward. Yeah. So is this something that you urgently need to work on? I mean, are you going to add it to the bug list and keep cranking on it? Or do you want to set it aside in favor of other things? Well, I feel like it's actually causing a kind of pull together maybe like four tickets, which I think is probably a probably related based on this. And that, you know, will at least be a huge step forward if they can be, they're all relying on knowing that Microsoft's ready to do something. And so, it was like a priority. Okay. All right. Anything else? Just the final thing is particularly in terms of bringing in community PRs and stuff. I think and I had a little bit of a chat with you about this. It would be really useful to document what we think does where and, you know, what should be in Microsoft Core, what should be in an enclosure, what should be in a separate plugin. And in my mind, ideally, we have core being as minimal as possible. So it is, it is literally the core. And then, you know, you can compose Microsoft instances, you know, based on a whole bunch of other stuff. So there isn't in its kind of direction with like a plugin system for the audio services and that sort of thing. But I think we probably need to wait until 20, 28 to actually put it into production because it's a big shift. Okay. Yeah, I'd want to review whatever, okay, it's coming up with two to do. Like, I know we've, I like the idea of having some sort of service running that monitors or other services. Yeah, I had that at a, another company that worked out really well as far as, you know, so that services themselves aren't self-aware but something's watching them. But I do want to I'm not going to say this, I appreciate Okay's help but I don't want him to necessarily peek off and doing things that may or may not be aligned with what we want to do long-term, architecturally, right? Yeah, and so that's why, yeah, I think we need to be able to articulate what we want to do architecturally so that we can help the community to know which direction we want them to go. But okay, hasn't worked on that external monitoring service at all. So this is a primetime take for us to get into that. So I think we can outline what we think that might look like that would be really helpful. So maybe Chris, you and I can chat about that and I want to just start straight into it. Yeah, I can lend some input from when we had that kind of thing at another job I was at and how it worked. Alright, cool. Alright, so I did want to add that and I got the volume issues fixed on the new image. So I was able to add that I2C set command into the boot set and I was able to make sure that the Mark II skill that's running does have volume controls on it. So the current image shouldn't have a problem anymore with potentially boring speakers. I did get to that before I left. Alright. So does that set like the max volume? Like the user can't turn it up past that number? Yes. Got it. I have some information on that bug you wrote up. Yeah. I didn't see that update, thanks. Alright, let's go to Ken next. Sure, so good work because I was going to tell you guys I didn't want to say anything because I'm busy right now I want to get pulled into it. But I'm monitoring the log files constantly on a Minecraft core and Minecraft and I'm constantly seeing the message come out and saying I'm done booting, right? And then for another 30 seconds to a minute I see a bunch of activity and I'm like well if you're done what are you doing? So I think you're on the right track there. Anyway, alright, so the good is the database represents the large data store of samples has been backed up. The large store of samples is in process it's a little over halfway through a 36 hour process it's all good. Analyze the data and row code to delete the offensive data when I get that marching order which I got so that'll be taken care of by end of day tomorrow probably. I started looking at how I'm going to partition that data and wrote some code to do so and I have some details on that and I'll update the wiki because that's usually the most efficient way to get that information communicated rather than spending an hour on the phone with everybody or on the call with everybody. I created a hyper parameters document and wiki I think that's good because I surfaced some stuff and was able to learn some industry standard ratios between some of those hyper parameters and what drives their usage and so that will help me reduce the tuning which could have been a combinatoric nightmare so that was good. I have some questions from Matt regarding some hyper parameters that are gathered in the later versions precise not the earlier versions but don't seem to be used and I have refactored the latest version of precise to include the hyper parameters from a command line so that the test suites can be implemented and generate CSV result files that can then be analyzed and possibly even make some inroads into auto tuning but that's still down the rotor piece and what is this change oh yeah and I'll change a bit based upon some of the stuff I've learned. The bad was and ugly was really just that we didn't have that data backed up and it's going to take quite a while to do so but other than that nothing really bad or ugly just a lot of work and a lot of work to do but you know starting to see the light at the end of the tunnel you know the bad is okay the bad is if you want to surface anything bad that I've discovered we've got three distinct versions of precise we have to be concerned about we have version 0.1 which is the version that produced our current production model we have version 0.2 which is what's installed in everybody's system and 0.3 which is what you get if you check it out from the code line and also I believe what is being deployed to the Mark II's so they are not backward compatible you know a model created by 0.3 cannot be read or worked with 0.2 or 0.1 how far up the chain that will work or propagate I'm not sure yet I'm pretty sure models created by 0.1 can be handled by 0.3 but that's ugly and I have to get a handle on that and figure out what we're going to do about that I have been reluctant to upgrade because it will affect the thousands of users that we have out there who are running 0.2 and if it's not necessary I won't do it so I'm just trying to get a handle on that ugliness and that's the only ugliness I guess I have to report okay Derek just quickly Mark II should be using 0.2 as well are you sure about that because last time I spoke with okay he had mentioned he was running 0.3 with a modified 0.2 runner I trust okay then yeah I don't know for sure either 0.2 when I get one it's installed I'll be able to tell you well we should be able to look at the images we're producing right yeah I'm not part of that process but yeah I can dig into it it depends on what Mark II you're talking about Mark II that okay working on in his house was different than the one we produced for project rollovers it's well there you go there's no use the one we did for project rollovers based off of piecraft is using as what project rollover is using we have to be careful what we say but to qualify because there's about 5 versions of a line around out there yeah and at some point and Michael if you want to cut a ticket for this I'm fine with that we can just make it a low priority I should probably put some sort of table together which defines default versions with which image and you know all that good stuff yeah I think we do need to look into that just in terms of what we can do I'm less concerned with the end flavors of prototypes out there that have different versions of precise or whatever in them but I am very concerned with the production stuff that's out there the Mark I can they even be upgraded to version III and that kind of stuff but yeah we should definitely create a ticket for that okay now Derek alright hey guys okay so ugly project rollover prototypes are a little behind schedule yeah we just we ran into a couple blockers I think Monday is probably doable to ship I think the most important one is just not really being able to test it until today as mentioned we got the USB boot going thank you guys and Chris too on pointing towards the rotation of the display so that's awesome that's working successfully but we really didn't get to test anything until today all was looking okay but I saw some of those audio issues that I want to figure out what's going on there and I haven't 100% determined that it is software yet so I need to swap out some parts and do some testing to make sure it's not the hardware that's causing the problem before I put my finger at you guys so yeah I think it would be too rushed to try a ship tomorrow with those concerns plus we still have a lot of assembly left to do the good news is we've got a lot of the parts as of today should be printed and Charlie's working hard to get everything all cleaned up and ready to go so there's really just going to be assembly and testing left to do so but I do plan on running if I get everything working well if we can determine what this I2C audio issue is then I plan to leave over the weekend to really make sure it's good just preliminary heat tests getting around 95 to 100 degrees when you're hitting the top of the device which seems kind of hot but it's really not that bad is that Celsius or Fahrenheit Fahrenheit just for reference in Amazon products is about 90 degrees on top surface as well so I think it's fine and the pie isn't shutting down or anything like it would do if it were overheating but I'm going to do some more rigorous stuff on that tomorrow yeah so yeah we probably I don't know if Josh we need to communicate that to the progress roll over guys for what we've actually promised so that's that and then on and that's really just been kind of taking up more time than I thought so I've made a little progress on the new SJ based SJ201 based enclosure but really not enough to make any noise about yet so just been mostly working on the project roll over bits yeah and there's some things I haven't mentioned about it they're kind of cool that you guys will probably wish you had on your speaker prototypes you can actually plug the ethernet cable in and use the device with an ethernet cable plugged into it that's why the display had to be rotated 180 because it's now exposed towards the bottom you have access to the USB ports and the one thing that I know you guys have been bugging me about forever is you can really truly remove the flash drive without any tools so the USB by switching to the USB stick you can actually grab it and remove it so unfortunately that's probably the last of those types of prototypes that we'll make but got a couple things mixed up for those guys that are better than and are known issues that we've had in the past so yeah that's it for me then okay nothing blocking any of your work going forward no just those kind of setbacks unless and then you kind of to be determined on this I2C issue see what's going on there okay how's notice Charlie's not here today is he everything is going fine I told him to jump on I saw him I was working with him up to about 3.30 and I don't know he's probably just heads down working on the prototype so everything is going fine there okay great well let's see I've only got a little bit of an update we're going to talk with Kevin this afternoon and finalize the order for the SJ201 prototypes so we'll get those underway I'll let you guys know what to expect in terms of timeline which brings me to the point of hey we need to figure out how we're going to test these things so I've created an epic in the ticket system to just list all the things we need to do to get a new enclosure up and running not terribly different from the last one but it is using a different chipset there are things that are going to be different so I passed that off to Chris to get started on just great tickets for all the things that need to be done on a new enclosure I'm going to work with Kevin to just create some test software for the device itself because it can work in a standalone format as a USB device so we'll test it that way to start with let's see there's a number of topics that we talked about last week that we need to put on our roadmap at some point the one of them we talked about the GUI decision matrix the wifi decision matrix coming up with we need to talk about the mark 2 update methodology I think we're going to have a decision matrix about that as well when I didn't get around to writing that's next alright cool and then the next thing I want to talk about is start to put together some metrics for precise and try to set an end goal in terms of how do we know when it's good enough I think we could there's a couple of ways we could do this but we could benchmark the Amazon Google products in terms of how accurate they are missed wake word just basically how often they recognize the wake word and come up with some crude targets and that might be one methodology we can also target it a system whereby we just make sure that it's better than the last model we released stuff like that we have some criteria for when to release it and how to release it I know that there's a fair amount of uncertainty regarding the new model versus the old model and the old model has been around for a couple of years now hasn't been updated so if we did roll out a new model it might change the way it behaves for some people so maybe we want to roll out a new model as the default hey-microft model but keep the other one around as one of those alternate models just call it hey-microft slash legacy or something like that in the Selene options where you can pick a new wake word so we need to come up with that methodology so that's something I'd like to talk to you about Ken or actually ideally just come up with a proposal okay, yeah I want to find out if anybody noticed the answer to this question before I ask Matt um we automatically download the latest version of Precise when mycroft core comes up so each time you bring it up it actually checks to make sure you have the latest version that's been released um if that process is in place for the model as well then this should be a relatively painless process so that's what I'm going to ask Matt if anybody else here knows the answer to that question which is is there a way to say in the cloud here's the new model and then all the devices will automatically update does anybody know if that's in there or not okay, I'll ask Matt I'm fairly sure it is I'll check the code and method you okay, thanks and yeah, Michael I'll work on the pseudo acceptance criteria probably it'd be multi-staged which will be internal analysis like you mentioned against competitive products and against our previous version and against a known dataset but then there should probably be some sort of staged rollout maybe just among us to say yeah, it seems better or it doesn't seem worse in other words, I'm anticipating a jump from about 85% recognition accuracy to 95% with the next model I don't know how you quantify to a human this is 10% better recognizing wake words I call bullshit on both of those numbers okay you're missing wake words that people are attempting to use and are not being recognized I would be stunned if our actual accuracy was more than 30% well I I'm not only quantifying a statement I'm not disagreeing with you a bit the statement is simply against a known test dataset that I have assembled that is balanced between male and female for high and low pitched voices against that known dataset is where those numbers are derived from and that dataset maybe as small as 500 samples so yes, in the wild the numbers are probably vastly different, Josh I just don't know how to quantify in the wild numbers I only know how to bring it into the lab and quantify right and that's actually another topic more things from the wild in right well that is one issue getting more samples from the wild could potentially help with that but also characterizing the nature of the samples that we're getting and that we have testing against if I'm using a headset my recordings are going to be a little bit cleaner than if I'm talking into the microphone on my webcam right and this is a big issue which is I don't know the answer by the way which is are we better served with clean data or are we better served with dirty data in other words is it better that the data has normal noise in the background because that's what the recognizer is going to be exposed to on a daily basis or is it better to deal with really tight clean data what's the SQ, Josh, which one dirty data it's actually one of the reasons we didn't get into fancy microphones with the Mark 1 the concept was we want to collect data in a dirty environment and teach the wakeward recognizer to recognize that environment and then solve the dirty in software so that we're not hobbled by a need to have super clean noise cancellation chips on every instance of Minecraft if it'll run on a Mark 1 it should run anywhere it was coming to thinking that's fair that's fair that we have no problem with most of we do have a lot of dirty data so I haven't cleaned it up which just lets me know that I shouldn't bother building a noise classifier and filtering those out thanks, Josh and as part of this what does precise look like when it's done I would argue that unless we have the entire machine learning loop closed and finished that the project's not done what I'd really like to avoid and my phone's about to die so if I leave that's why I'd really like to avoid a situation where we're back where we are today a year from now and Ken moves on and we need to train more models and we haven't automated this loop I'd love to, the completion for me would be the software stack is automatically improving the contributions from the community at which point we can close this item and it should improve by itself we shouldn't have to continue to develop additional software beyond tweaking how the models are trained and what not which you'd be surprised how many members of our community are willing to reach in and do that piece of it for us I think I agree with all of the above the only caveat would be that at some point and I suspect we're still a ways off from that point we'll have too much data and we'll start overfitting but I suspect that's not an issue and I hear where you're coming from Josh I am working actually on exactly that which is the entire loop from getting the new samples into the system getting them into a business process flow where they can be manually validated getting them into the training and testing data sets from there and periodically at some trigger level retraining models and then running the whole suite in this model better than the previous model and then yes if it is it automatically gets deployed yes I hear you and that's what I'm working on from a high level correct excellent and I'd like to abstract that one step above and that would be that we can reimplement that exact same loop for new records in some kind of automated fashion in other words if we want to stand up a wake word HRVS for example we can add that to Celine and anybody who selects it will automatically start feeding data into that exact same loop for the new wake word and so that would enable us as we have corporate customers step up and want to do custom wake words we could simply enable one in Celine let the community know hey we need help we have some prizes available for whoever tags in most of these queries and it would stand up and improve on its own right that's a very powerful position to be in on the wake word side kit AI sold to buy due for $50 million and they didn't have anything even close to this now you're mad after my own heart I agree so yeah that's the goal the only other piece I'd like to abstract up is I want to make sure that we are collecting the data from before the wake word we can start getting the missed ones right now the entire system is biased towards the ones we actually spotted we need to be collecting the attempts that don't work because without those our accuracy will always be sure that's fair enough Michael can you cut a ticket for me for that that would be one of the last things I do once I have the loop closed but yeah I absolutely agree with everything Josh said just to be clear Josh I ain't going nowhere this is my dream job I could become a watch on the side of the road though which is why I keep creating the wiki pages keep wearing the mask don't go to the grocery store yeah I'm not sure that like in terms of getting in data for people that were not successfully hitting it feels like the precise trainer skill is a much stronger way of doing that than messing with audio from before they've even spoken to the device like I feel like that in itself has privacy concerns and people will be yeah pretty concerned about that we're deliberately having people you know we're communicating what we're doing with it but unless we get those missed utterances we don't know what our accuracy is if my kid tries five times and on a sixth time it works but we don't know a lot of times if he tries five times I don't bother trying the sixth time and then we don't get it at all on the other one we can hit the button and on the mark two they should be able to hit the button too well let's just leave it at this that Chris's concerns are noted and we probably when we have our privacy policy discussions need to surface this and get it out there I agree with Josh that that information is a potential way for us outside of manual intervention to gather some of that data and improve in other words the value of that data is especially in the case where it was a missed wake word or in either case that data is very valuable in other words if I have samples that say hey mycroft it would be wonderful if I could have samples on the negative side that said hey minecraft because it allows the tighter discrimination during the training process so this gives us the opportunity to get some of that data in an automated fashion so if we can overcome the privacy concerns which I agree are existent I think this data is a bit valuable and not that hard to capture I don't view it as a privacy concern at all for me the privacy issue is binary like either the person opted out opted in or they did and so the you know on the privacy side of things the end user should you know we've informed them what we want to do with their data you know we've been up front about it they've consented at that point we need to stop being nuanced we collect what we need to make the software stack go and if they're uncomfortable with it they'll opt out we'll nuke it but you know we can't continue to have these nuanced discussions about it you know once somebody's consented let's go ahead and use the data because our competitors are using everything they're using everything they're collecting 10 seconds before 10 seconds after everything you say in the room they're collecting everything we can you know once we've disclosed to people what we're doing with it once we've asked their permission and they've affirmatively granted it let's go ahead and use the data we need it you know we don't want to be so far on the side of privacy that we can't build a technology stack that's competitive and works nobody wants that right and you know we can add tools we can add tools to the system that allow people to go in and selectively nuke samples or whatever yes that's what I would I would ask for because I think that people as I mean from personal experience seeing some of the data and seeing some stuff that I'm like yeah I don't think I'd want that out there if it were mine you know I don't that's a really sexy option yeah I like that no it is because it's a little bit difficult to implement but it gives the user fine-grained control so that we don't get gross opt-outs we can get just whack these 10 files of mine that that's awesome because there's this issue we are still a small enough community and I've been seeing stuff when looking at data we're like I know who this person is like you know and that's that's a little weird so and I could I mean I can obviously have it within the company but I think it could also happen within our close knit community and yeah there should be some thought into it and not everybody knows what it means when you say we're going to use your data to improve the wake word they might work okay cool I don't care how many times you listen to me say hey my crop I'll say hey my crop a million times for you they don't realize that it's actually more than that and so I just think we need to have some optics on it we understand like it could be a problem but but I like the idea of being able to listen to your own data and be able to nuke it if you don't like like what you hear right and you know to Josh's point I think you know they give consent then they've given consent right I think it needs to be informed consent and you know to Derek's point we should be very clear about what we're collecting with them and and as we change what we collect if we do we either need to get you know nuke a new consent or make it really clear to them the things have changed so but yeah all these issues are addressable and we should not be afraid to collect data when people want us to collect their data and use it to make the system better yeah I think I fully agree with all that I think my concern is that doing that without a way for people to actually see exactly what they're uploading will just say a huge drop-off in in opt-in because I mean I would opt in at that point in time and you know I imagine that a lot of other people wouldn't as well yeah I mean we'd even potentially implement a you know a system where you say like hey my craft you know forget everything I said for the last hour yeah then we don't need a GUI for it then we could even you know we can look at more granular opt-in controls potentially rather than just a single opt-in opt-out like some people might be happy for wait words and not utterances and you know you know like come from people as well you know they're like well I don't actually know what data you're collecting so therefore I opt out but if I could see the data before it goes up then I'd be happy to do that or if I could you know even if I could review it after it's already gone up then they'd be more happy with it so like there's a whole lot of nuance yeah we just we're not in a position where we can actually build that nuance in and what might be cool is having a skill that responds to stop collecting start collecting again and then what you have is a marketable feature that your competitors don't have you can't tell Google stop opting my data in or Amazon you can't say hey stop recording my data you know and then later you can say stop start again that would be a marketable thing you know that we could trump it out there regarding a benefit over our competitors and privacy issues so I think that's awesome I think we could also have a tool that allows people to that we can load the wakewood recognizer onto some sort of web interface where people can you don't have to have a mark one or a mark two to do it you can just go on there and say hey mycroft and if the wakewood recognizer says boop yep I recognize you that's a positive if you tried it and it didn't say that you know there's you know so that way and if you opt in if you do that then there's an agreement before you do it that says okay I agree that my samples are you know can be used for our data purposes but that gives people a way not only they make it they can almost self tag at that point too you know I mean I know we talked about give us 20 samples for example I know we talked about maybe having a cap on how useful it is for one person to have too many samples but you know if we put it behind our single sign on and people sign on and say okay I've given my 20 then you know and they're validated it would also help us with false positives too because they'll know right away that they tried to say and it didn't work so I'm really dig on the skill that hey you know turn off opt-in turn on opt-in I mean I can tell you several times during the day when my wife comes in and aggravates me and I would be using that skill and saying turn off opt-in and then I was done with her turn it back on again so yeah I'm liking that concept and then we just need the event we need the event driven database like the transactional database on that opt-in flag before we do that sort of thing yeah that's true I had some thoughts too on this going back on the conversation a little bit but on our kind of real-world testing aspect of it as soon as we get some of these SJ201s might be cool to write something that allows us to do some automated testing especially if you had two of them that you could run like wakeboard samples on one device at different decibel levels and you know move your other device back away from it and do some benchmarking with the real physical device we did something similar to that with the Mark 1 over a year ago versus an Amazon device and a Google device where we on a speaker played at different decibel levels we played wakewords and tested distances and volumes etc so obviously that's not going to get you the huge breadth of you know examples that you get from the community but it could give you some pretty useful real-world data pretty quickly a great up a really good point which is I don't know if we've been looking at this when we were getting ready to release the PC Junior at IBM I was working closely with the reactors guys at the time on a different project and they showed me an issue they had the PC Junior actually got recalled initially because the keyboards were infrared and nobody had thought to put two PC Juniors side by side and what happened was the keyboards interfered with each other and screwed everything up and had me redesigned have we taken and put two PC Juniors assistants maybe even a Siri in the same room and done a battle of the network speaker phones and see what happens we've done it I've got the 304 on my desk at the moment I mean in our Gladstone office we have a Google home and an Alexa and you know we usually leave them muted but I mean we've done we had a guy who was doing metrics for a while who actually sat in a room with all three of them collected some metrics on that kind of stuff around that wouldn't hurt to do it more often every now and again they can get in a conversation together well so that brings up another that actually brings up another thing metrics wise that could help us too with our Qt vs. Keeve going back to that one of those metrics was time to respond and that's like one of the big things that you want to be looking at with our product vs. other assistants that to me is like the time of response between the two GUIs that's the first step like okay if this if one is vastly superior than the other well then it's no brainer I don't know if is any of that stuff that we did for those I mean I know there was at one point one of our employees was sitting in a room you know stopwatch in hand doing that kind of stuff especially with the Amazon devices we don't have a way to test that other than that we do at one point have some tests like that for mycroft we have metrics for mycroft core that tell say how long it took to get from point A to point B and those metrics are up in Grafana what we don't have is what type of device were you using when you did that not that that correlation is not there yet so if we could add that correlation you know then we could look we could do apples that apples comparison of how a skill reacts on different device types well yeah at least on a short term if we could just do it on one like per one device you know if we just had this one instance of pv or this one instance of acute devices I mean again you can do this stopwatch in hand but I was just curious now if you if you were doing it yourself and you knew your account ID and the database and which device you were using that was the acute version which was the give you version I could get there you probably get it from the log files in the message box log file times are not necessarily the best way to view something like that because the files get written to the log not necessarily at the there's read write problems I've had problems before where looking at log file time stamps can really get you into trouble sure unless you actually have the log message as a crime print and the log message itself but I wouldn't at the time stamp the log message was written yeah you're right Chris we can put some code pretty quickly to run that test we have a metric in core it's just a matter of you know you have to be opted in and you have to be able to know when you were using a mark 2 acute device and when you were using a mark 2 kibby device which devices they are so we can identify them in the database once we have that then I think we can make that correlation I'll double check but I don't think this is make a lot of the tools in place for this I think we just need to make that final step is doing the correlation yeah all right so I don't think we're at the point yet where we need to make a decision on the acute kibby I think we should go through and make sure that everyone's had a chance to put in there two cents on the decision criteria and we should be talking about that soon um in terms of next week how are we looking in with our with the bag of issues that we're calling a sprint um so I did like I said I closed a couple of the volume ones um I mean you in particular I mean like as a team overall like do we need to is it time should we be looking at shifting focus to um you know to a new bigger project or we still have a lot of work to do in terms of debugging um so I think the reason we're doing this is is really project roll over right is to get them as best user experience as possible so I think one thing we haven't done is to go through the bugs that are not that haven't been started yet and say okay these would be really beneficial to get done you know quickly and these are maybe a lower priority bugs that we should be doing something else inside instead of fixing this bug right away right okay um that sounds like a useful exercise because the next two things that I want to talk about are in terms of overall um you know projects that the team can work on um I think that we can take the work that we're doing you know that Ken's primarily doing here on precise and expand that to be a team-wide effort because there are elements that everyone has some knowledge on that can really help move this forward for example if we're going to create a community uh you know a system that involves the community submitting samples or grading samples and verifying and that sort of thing then you know I think Chris can help do some of that work or get the community involved and helping out with you know improving that system you know and you know obviously there's stuff work to be done on the Selene side in terms of you know creating the infrastructure for that both for collecting samples and for you know the opt-in and all that kind of stuff that we were recently talking about so this this could be like a team-wide project that we could focus on you know get this wake word data system done robust and something that we can you know utilize going forward and I think that'd be a great project for us to work on that would have lasting value and you know potentially give us immediate results and improvements so so that's one thing uh the other thing is the the continuous revisit the continuous integration process and and look at turning that into a continuous deployment process because you know right now we don't have we don't even have a deployment process for the Mark II because we don't have a Mark II but we do have a snap package and we don't have a continuous deployment process for that really defined yet right I don't know how often we're updating the snap package but I installed the fresh copy yesterday and you know there were a couple of hiccups in getting it up and running but you know it's pretty easy to get installed right and so that's that's actually a system that we could be using as part of even the wake word data collection system right because it's you know it's right there in front of the user once you install it like it's a really easy way to to get people to to interact with the system and so like if you know if we had a skill for example that helped us collect wake words or grade wake words I mean even just spending you know a human spending five minutes doing that could do a ton of work for us and that would be just super valuable so so having a continuous deployment process I think you know would be really useful in that respect as well so those are some of the next two projects that are on my mind in terms of you know things we can get the team involved in so I I don't really hate to waste the momentum we have mine precise right now we've been we're giving it more attention than we have in a while and it's been an issue for a while so it would be really be nice to you know really put some nails in that coffin and say you know we've got this really cool wake word thing that is useful in many many different ways and you know be able to have that kind of feather in our cap might be nice okay yeah if we could get precise through to completion right so it's a production piece software it creates a lot of opportunities for us as a company and I would argue that precise combined with the initial setup and the wifi setup if we could get those two things squared away and then as icing on the cake get the exact to be stable on whatever hardware we choose we have a viable product that we can shoot and actually like have some customers and some revenue and stop being behind the eight ball of funders that's a very powerful thing I would love to see that and the other thing with the bug fixes is I mean I know the project rollover has a few prototype devices but really I mean we could have another bug fix sprint once we get more of the SGA 201 prototypes out there so I'm sure we'll have a whole new set of bugs and we'll show up with the new the new hardware you know and make sure that definitely you know at least don't send any of those out to anybody and so we're however happy with its you know performance from a UX perspective the hardware the SGA 201 hardware is really you know I consider it almost a Skunkworks project just to prove that we can do it and get something out there and so we can start to test the efficacy of actually having an audio front end and start to deal with some of these issues of clean data versus dirty data and what does that even mean if we're using you know these acoustic echo cancellation algorithms so so I think that's a nice to have but it's you know and obviously we want to fulfill the Kickstarter you know commitments there but yeah but you know we're not going to be able to do that until the software is ready anyway so I don't I don't want the hardware to become a distraction. Well I think prototype is essential for the Mach 2 as well right? Oh of course. Yeah. I think to Josh's point I even have the beginnings of a precise studio that I'm building but you can using the UI you can create your wake words you can assemble your data set you can test your models all that stuff just using a GUI so that would install when you checked out precise not that difficult. As much as possible I'd like to make sure that the precise stuff runs through Selene and verifying and adding feature or adding new models you know being open source is great and I you know I'm a thousand percent behind it but let's not build tools that put our stuff where we put ourselves out of business yeah. I haven't even thought about those issues that you guys will have to guide me on that but yeah for sure the process currently does not go through Selene we could make it go through Selene it wouldn't be that difficult I don't think I don't think the new wake words and all that stuff finding which ones you want to delete and all of that stuff can run through Selene and if a company like a major retailer wants to have that capability you know we can work with them for the community we can provide it for them but you know we are a company and not a charitable organization we do need to come up with the mechanisms where we add value in exchange for getting paid or you know we will be not a company anymore after a while agreed so to get started should we shape up a precise sprint well let's go through the current sprint look for anything that's critical that we need to get done either for rollover or for just you know the system in general are there any critical issues in the to-do column and make sure we close those out and in the meantime let's get started on defining a precise sprint around building this whole the whole system I'd like to be a precise epic and put all the tickets in there we can prioritize I think there's probably a lot of stuff that could be the same thing we're doing with the bug fixes now which maybe break it up into smaller sprints but it won't be a huge make precise worksprint yeah I agree yeah because we're going to have to define all these elements like the salini element the data collection you know storage element the training element the how often do we you know it can still work on the whole algorithm in terms of you know what's the best way to utilize the data all that kind of stuff so there's a lot of moving parts so I agree to identify which of these issues should we go through them as a group in a different meeting should we each take I mean I was thinking that there's a high priority flag on these issues we could use that and set the ones to high that we think are are the most important maybe we all do that between now and Monday are you talking about the current bug sprint or a new sprint current bug sprint and then we on Monday we can go through all the ones that have been marked as high and but then that would be hard to know if nobody missed one somebody else thinks as high but yeah I'm trying to think of the best way for all of us to you know have input on things that we think that really have to be fixed before we move on I think that's fine I mean the tools there why don't we use the priority just set it to high or set it to highest if we have to fix it before we move close out this bug sprint okay and that shouldn't take more than five minutes everybody can go through and if it's not already marked as high priority you think it should be then go ahead and change the priority to highest alright and then I think the way I'd like to proceed with creating the wakeward sprint the precise sprint I think Ken's got a pretty good handle on the overall system in terms of the training and the collecting the data and that sort of thing so I'd like to see him write down like he's been doing with the Confluence pages just an overview of the whole system how it behaves, how the user interacts with that sort of thing and that way we can all take a look at it and make sure we're all in agreement about what needs to be done and how it should work okay I'll take that there's a work item and I'll try to have that document available by meeting on Monday I don't think we should use the meeting on Monday to review the document so no need to do that as it's appropriate and we should have a separate we should have a separate meeting to review it in fact, actually what we should do is we should review it offline like when you're done send around to people to comment on then we have a meeting to go over the comments and you know resolve anything that's not obvious I agree, I guess I'll rephrase it I'm saying I'm going to try to have that page done and emailed out to everybody by Monday that sounds awesome, thanks okay any outstanding items that we need to talk about not here alright cool, well thanks everybody the meetings are not getting any shorter but I think they're getting more focused so I appreciate that hey guys I have a quick question for you before you go have you you say you have like two or three of them in the same room are they all at the same time? was that from me? yeah often they are, yeah it kind of changes pretty regularly so you're on the same time it doesn't create like an infinite battle of the speaker phone and automated calling system? it does sometimes yeah I thought I was right you constantly go into an infinite loop until somebody manually powers one down okay they don't often get into an infinite loop but you know you often get the same response from multiple places but it's also why we have multiple fake words so you know you can have an Ezra and a Jarvis and a Minecraft and that helps you pick one rather than the other that's what we do in the office is I use like a Jarvis on my desk so I don't when I'm testing I'm not activating every other device in the office you should be able to put you should be able to put them in an infinite loop by saying hey Minecraft waiting for it to activate and then say say hey Minecraft say hey Minecraft twice and that should put them that should put them all into an infinite loop and just by my eye of the data set I have already I do have other data other data for other wake words and hey Jarvis is one of them yeah we started some of that collection a while ago we just never got all the way through with the tagging and all that stuff so we just need to be able to I'd like to take whatever that Postgres database is and integrate it into Selene as well so we can take advantage of you know referential integrity and stuff yeah that's all I say we want to get it integrated with Selene not only that but it's MariaDB slash MySQL not even Postgres okay well yeah and then at the end of that that becomes the service that we can charge people money for and then we can pay our own bills which makes me very excited why are we worried about money hey if everybody on this call can stop worrying about money then we can stop worrying about money right I think Ken just volunteered his paycheck alright well I will focus on continuing to go get some money and make sure we keep the bellies full and we'll talk again oh no we're still recording we're still recording okay so the community knows it's no secret we're getting ready to go out and write some money so we can talk about that a little bit later alright we have multiple competing term sheets and anybody listening for this call who wants to get in they better get their check in now we're gonna get them on Monday