 Good morning. Welcome to this public meeting of the United States Consumer Product Safety Commission. We have one item on the agenda this morning, a public briefing to discuss the status of a commission-approved mid-year project related to our data analytics strategy. And I do have one word of logistics. We will begin this afternoon's hearing at one o'clock on the micro-mobility, so one o'clock if everybody can take that into account. This is a topic that's critically important to the work of our data-driven agency, but one that is better left to the experts to summarize and explain. For this reason, I'm going to turn things over to Jim Ralfus, CPSC's Chief Information Officer, to provide opening remarks and introduce our guest speakers. After the presentations, we'll turn to Commissioner Questions. I'll call on each commissioner in order of seniority. We'll start with five-minute rounds, but as always, I try to be flexible and we'll go as many rounds as needed. Thank you to everyone for being here this morning. Jim, you may proceed. Thank you very much. I'll make my introduction and context setting as brief as possible, really focusing on introducing our guests here today. Jennifer Rustami is here from the GSA Centers of Excellence, who has been here for working with us for the last six months to develop a recommendation for a data strategy and implementation plan and look for ways for us to improve our capabilities through improved use of data and analytics. We also have with us Mark Engel and Ryan Powell from Sapient, and they are also working with the agency in support of a pilot proof of concept, exploring artificial intelligence and machine learning for the complex processes associated with product categorization and incident severity categorization. So we have some theory and strategy and long-term planning perspectives for you this morning, and also some practical mission-focused kinds of activities trying to improve the work that we have. So both of them have worked very closely with staff, and we're excited to share that with you. The data strategy is an independent perspective, so that was something that was key. I'm pleased to say that much of what is included in the data strategy provided from the GSA Center of Excellence is in line, very much in line with the direction we've been heading. However, it is recommendation in nature, and so we may make adjustments to that as we move forward with implementation. Mark and Ryan also are representing a much larger team in terms of data scientists and engineers and experts who are supporting them as well. They'll be providing a status update today, but the work will continue to progress throughout the year. So I think with that I'll turn it over to Jennifer and let her provide a briefing on the strategy. Hello, commissioners. Thank you for having us today. I was going to talk with you guys a little bit about what the Centers of Excellence does, just to kind of give you an overview of the work we did, and then my goal is to just walk you through what we found and an overview of the target state and how you guys will get there. So Centers of Excellence is a part of GSA, and our whole mission is to accelerate IT modernization in the government. We partner with other government agencies. You can check us out online. We have different Centers of Excellence. This one was very focused on our one center, which is data and analytics, but our team includes experts in AI and all kinds of technology across the government, so we were able to leverage that in this work, which was nice. So the things we're going to cover today is the whole overview of this project. We started by saying, okay, we want to get to a target state. Target state for us is kind of like five to six years down the line. Where do we want to be? And so to get there, we said, okay, we've got to all get aligned on what is the current state, so that was our first step. Then we designed a target state together with CPSC, and I'm so happy to say your employees here were incredibly involved. We had no problems knocking on doors. Stakeholders were opening doors saying, oh, you should talk with so-and-so. We just really had a, I think, strong participation, so that was something commendable. And then after we did current state target state, we moved into the data strategy implementation plan, and then we did a bit of acquisition best practices to kind of help you guys get ready for the next steps. Three things I'd like to mention before we dive into everything. One was really that participation was exemplary here from the top to the bottom. We do this at other agencies like I mentioned, so I think it's important to say based on what we saw, we're going to go through what we found in the current state, where you guys want to be. I think you land, and my experts say the same, right in the middle. Some agencies are ahead where they're really already, they have these strong CDO offices set up for a couple of years now, and their whole mission revolves around data and analytics. So some agencies are ahead, and then there's about 50% that are behind. And actually an interesting takeaway is we would love to use this work at other agencies because the strategy that you've developed is something that can be leveraged across the government, and a lot of folks would benefit from. I've had a lot of people knocking on the door, interest in using this. So a lot of words, but I'll talk us through this. Some high-level things in our methodology and approach. Basically, we wanted to talk with everyone inside your ecosystem who's using all of your data and analytics. That includes we wanted to understand how does the public use it, how do you guys use it, how do executives use the data, what's important, what's important for mission fulfillment. So for that, we spent a number of months just talking with stakeholders and just learning what are your systems. We sat side by side with folks as they used their systems. We tried to figure out what's working well, where are the hiccups. And that was really our approach. So we were on the ground on the seventh floor. We might have passed you a couple of times doing that work. One of the big parts of working with us too is as we were observing those things, we're able to integrate what we see at other agencies and figure out really what are the best practices happening. And there are some great best practices happening in-house, which I'll highlight on the next slide, current state. And I just flipped right ahead because we can dive right into what the current state is. So our approach, like I said, we did stakeholder interviews. We did an artifact review and system findings. Some things that we learned, you guys have a great data governance that's just newly getting set up, which is fairly on par for federal agencies. The mandate is pretty recent. But we did notice a couple of other things, and you might have run into this in the past. There's a little bit of a lack of data traceability and lineage. So that was something we made note in the current state. There are some manual processes which now can be automated, which we'd love for the agency to take advantage of that technology. And then there's a bit of a need for improved analytics tools and platforms, methods alignment across the entire agency, not just in departments or program areas. Moving into the target state, some of the things in the target state, something I'd like to really focus on here, this was co-created. So the DMWG was very active in helping us form what the target state would look like, as those are going to be your champions who bring this forward. And of course, Jim, in all of EXIT, was participated a lot in creating this target state with us. We wanted to take what we listened to in the stakeholder interviews and really put it forward in how do we design this to work ideally to give everybody what they need from the data. So some of the things I'd like to highlight in the target state, and of course it's in the full reading if you'd like that afterwards. But one thing we wanted to make possible was self-service. This came up a lot. Folks didn't want to have to pick up the phone or write an email to get access to data if they were looking for insights quickly, and I'm sure you've seen this. So we wanted to make in the target state self-service a core part of the architecture. Next was transparency. We wanted to create transparency and lineage. I'll show you a little bit of what the target state architecture looks like, but it's all cloud-based. That stuff is very easy, transparency and lineage in a cloud environment. So that's part of why we made that recommendation. Folks were asking for more AI and machine learning applications. So that was something we factored into the target state. And of course, visualization and advanced analytics. So that is also possible in the target state. I'm going to show you a little bit. This is a great example of something we found, which is also in our target state and current state. If you take a look at the visual that says today, you may have experienced this. This is what it looks like to request data today. And you can see some program areas are really hot spots for data. The dash lines are informal communication, and the solid lines are formal communication. So formal could be, a request through a system. A dash line would be, oh, hey, in the hallway or a phone call. And frankly, it happens at all agencies that there are informal communications. But you want to try to minimize that so you have clear lineage of who owns what and when and how that data is being shared. So that's the current state. Now, if you go to the other visual, the target state, you see we have a centralized data repository. Things will exist in the cloud. Lineage will be very clear. And then you have less phone callings. So folks are able to focus on their day-to-day job and less of the doing favors to get data moved around in the agency. So on this slide, I'd just like to highlight a couple of things. We talked a little bit. This is the architecture. And you can see it in more detail in the full book we call it. But it is all cloud-based, which we are recommending cloud when possible. We also wanted to recommend common reusable enterprise solutions. This includes, for example, metadata tagging, which can happen across the organization and can give you a lot of just, you can leverage the commonality between program areas. So you're not dealing with data in a programmatic way. You're dealing with it in an enterprise way. So there's a common way to talk about all of the data. Data catalog would be something which is enterprise across the agency. You have one catalog to tell you about all of your data instead of programmatic level. Another thing, something that's happening that's really great in the agency, you've already started APIs, which allow for other, you know, inquiring bodies to plug in and just grab data without having to go through a cumbersome process. This is something that the government is really supporting, is open data. So it's already happening here. And in the target state, we build that out even further. And, you know, Jim is a really big proponent of this. So this is something great that's happening. An additional thing we put in is more and more open source that you can leverage things being done in program level and use them across other program levels when it's open source. Moving into the data strategy, a couple of things I'll highlight here. Again, cloud first. This was based on the foundations for evidence-based decision making, which is happening across the government. Everyone is moving to cloud. The data strategy we really wanted to structure, it is, it's a great read because, frankly, I learned a lot doing it. But you, this walks you through what are the phases of how do we go from where we currently are today to where that target state is in the future. We broke it into some big digestible phases, which are both progressive and incremental. And in order to make this data strategy real, I broke it into a couple of bullets here. There's centralization, there's self-service, adaptability, data management. And we wanted to kind of break it into, okay, what are the first things you want to do? So there's a six-month in a one-year, three years and an additional year, and that puts you at about the five-year mark for this, for accomplishing the target state. This is something that you can, you know, if you have more funding and you want to move faster on, you can accelerate this and move through it. If you do want to move slower, this can be, you know, still accomplished. The timelines are up to you how you want to accomplish it. But I will highlight something on this slide is six months, because that'll be very important, what you choose to do in the next six months. We are recommending that you keep strong momentum. I think I want to praise the commission for undertaking this work. It's not easy. It's not easy for any federal agency. That's why I said I have folks knocking on my door to try to understand what you've already done so that they can get a jumpstart on the work. So I think that was fantastic that you made that choice. We are recommending that you further strengthen your data governance, which Jim is an excellent, he's a fantastic advocate for this. You already have the pieces in place, but you're going to need even more, and you're going to need more participation from your employees. Something like Data Stewards is one of the next steps we're recommending. A Data Steward is somebody who has full ownership over a set of data. They say yay or nay. They say when things don't work, they are the absolute authority on one set of data. Once you have those folks set up, you really can make decisions about how you're going to manipulate that data. So that is one near-term thing we'd like to see in the next six months. Documentation, of course. I said setting up a data catalog is something really important, so that'll be something near-term that you'll want to accomplish. And of course, cloud keeps coming up. You'll want to start to evaluate and assess vendors that you'd want to partner with. Of course, we're the Center of Excellence, so we're vendor agnostic. We just make these recommendations so that you can choose and move forward. Moving into the implementation plan, this is the last thing that we delivered. Data strategy is quite a big thing, and it's something you guys are going to start undertaking. The implementation plan, we broke down into projects, and this is a little bit more fiscal-friendly, so these are projects that can be budgeted for, and I know that process may have already started. So I just wanted to give a light overview of this, just that it is, we broke it into projects. Each phase has a number of projects. The preliminary phase that six months is only six projects. So like I said, you can hit the gas and go faster, or you can slow down as you wish. And then finally, we have a couple of requirements for success and just some last thoughts before we move into Q&A. Something you're already doing, and actually you'll hear from the folks next to me, is pilots. They are a great way to figure out is this stuff going to work. And in fact, agencies are doing this across the government is starting a pilot, figure out if there's value, is this new technology like AIML, can you get something from it? Or maybe it's to immature at the moment, you have to wait a little, and that's fine. Pilots are a great low-risk way to make those choices, so we recommend to continue with that. Like I mentioned in the last slide, maturing your governance will be very important, and we want to emphasize how important this is to keep momentum on this that you've already started. And just commitment, so of course we recommend more funding, more resource for this as we do think this data and analytics will be key to the future of mission fulfillment for CPSC. And so with that, yeah. I think we should hear from Mark and Ryan before we take questions. Sure. Hi, how's it going? I just switched over. So can you guys see the slides up here on the screen? Cool. Okay. Yeah, so as Jim and Jennifer mentioned, I'm Mark, I'm with Sapient, the project manager for the pilot project that we're working on for AI machine learning. So yeah, we have a couple slides to just kind of walk you through like what we're doing and where we are. We are about halfway through or about a third of the way through the project. So we have some numbers and stuff and some metrics to actually share with you, so this will be a little bit more ground level and a little bit in the weeds, but feel free to ask questions and everything at the end. So yeah, just in terms of project goals and the objectives of the pilot project, so right now the CPSC is receiving incident reports from major online retailers and manufacturers, and those are coming in weekly. And at this point, the CPSC is at capacity in terms of what it can do. So let me back up. So each one of those reports and incidents has to be reviewed by somebody, right? I mean, these are descriptions of things that have happened. So someone has to go through that and make sense of what happened and then categorize these things accordingly. So that's what's happening and CPSC is sort of that capacity of what it can do right now with the people it has. So this pilot is to look at, you know, can machine learning help with that? Can it help with, you know, allow the CPSC to not only do what it's doing now more efficiently, but can it help expand that and allow it to take in more data to help achieve its mission? So that's what we're looking at. And specifically for this pilot, we're looking at categorizing these incidents into product codes and then severities. Oh, cool. Yeah. So the next slide just shows where we are in the project schedule. So we are breaking this down into different phases. We had a discovery phase up front, kind of get everything set up. The development is happening over three different phases, models one, two and three. We're moving, you know, through that in, you know, from less complex to more complex. If we can solve the problem with a, you know, a simpler solution, obviously that's ideal, you know, before we move on to the bigger, more expensive types of solutions. So we are, the results you're going to see today are for models 1.0, which we just wrapped up. We're about halfway through our second version models 2.0. So we'll be next. We can move to the next slide. Yeah. So we wanted to give a quick, you know, brief overview of what a model is, a machine learning model. And we've talked, you know, we talked about these things. So yeah. So what is a model? So the retailer reports, the metaphor I like to use actually is kind of like cooking. So if you think about it, the retailer report is like the raw ingredients, right? These are the raw data that's coming in. The data pipeline there is actually takes those raw ingredients, it's going to chop them up into different pieces, right? Before you actually like cook something, which would be, you know, the machine learning model part, which is kind of like a recipe. And then your final product are your predictions, right? So the model spits out predictions at the end. And so in this specific case for, you know, this problem, we're dealing with natural language, right? So we're using, you know, it's unstructured data. It's just a description of something that happens. So the model in order to interpret that, we have to turn that into basically ones and zero stuff that the computer can actually use to make a prediction. And that's what we call vectorizing. So there's a lot of work that happens that data pipeline to sort of, you know, break that, you know, decompose the text into stuff. And then the actual learning part happens during the machine learning model part. And that's where we feed in training data. So we give the model a bunch of past examples of incidents and the correct answer. And then it learns how to predict the correct answer from that. The next one. Actually, I'm going to turn this over to Ryan and he's going to talk you through our model building process and then some of our actual results. Hello. So I guess our model building process, just to give a little bit of background around comparing regular development where you have an in-state, a thing you're going to build. And it's a very linear process. You know what you've built, what you haven't built, and you just progress along and keep building stuff until you have the working solution. When we do model building, it's a little bit nonlinear. And when I say nonlinear, it's actually an experimentation process. And so we have a few different things we can try. A model itself is just an algorithm to make a guess at something. And in this case, we're using within machine learning just classification algorithms. How do we predict that something is severity one, two, and three? Or how do we predict that something is product X, Y, or Z? And so within that, what we are doing in each of these models or each of these versions we're creating is we're actually doing some experimentation. What happens if we, you know, basically change the weight of something during a training cycle? Or what happens if we split things up differently? Or what if we try to predict one before the other, that sort of thing? And how do we get results? And so those are the types of things we're testing. And then what we're doing is we're actually going through, we set up our experiments. We let them run that trains the model. And then at the end, we run that on a test set of data. And then we're actually comparing the results of a machine learning model versus what, you know, CPSC has previously classified this with to determine whether or not we got it right. And that is our score. And so if you can go to the next slide. When we talk about measuring our performance, and how did we get it right? Again, we're talking about classification models. So let's think of this as a multiple choice test. And you can probably ignore a lot of the numbers on the slide and just focus on the bottom part with me. But basically, if you're taking a multiple choice test, the accuracy is your score on the test. I got a 95 out of 100 right. The second part, which is the precision, is actually taking a little bit look more at the multiple choice answers. So if you think about it this way, precision is how many times did I guess A and did I get A right? If you look again, recall is actually looking a little bit more pessimistically, but how many times did I not guess A, but I should have guessed A? And then the last one, which is called F1, is really just the average of those. And so it, you know, accuracy looks at the overall picture, how well you did on the test. And then precision starts to focus in a little bit more on, am I getting my answers right? And recall looks at, when am I not getting my answers right? And so that actually goes into, we start to look at our performance. So coming out of Model 1.0, you know, what we've done is we've taken a set of historical data. And looking at data from between 2010 to 2018, we have partitioned that down a little bit. And for the first model to be simple, we're only looking at things where there are incidents with a single product and a single severity. So there's one right answer here. We're not trying to guess multiple answers yet. With that, we took the data, which ended up being, I think it was almost 200,000 records. And of those 200,000 records, we split that up. We said 80% we're going to use to train and learn on. So we're going to let a model run and look at those 80% and figure out how to guess each one. And then on the 20%, we're going to use that for our testing section. And so in doing so, our model for predicting severity, of which severity, remember, I think there were about a dozen possible answers, you know, we're getting about an 89 is our accuracy right now. So we're getting about 89% of the severity predictions right across those records. Something in terms of our precision, these numbers are going to be lower and they typically are, you know, how many times am I guessing right within each one? It's about 63%. You know, how many times am I not guessing the right answer within a very particular answer? And this is an average across categories. So while one category may be right 90% of the time, another category may be right 5% of the time, that sort of thing. We're about 56%. And then the average of those two is right about 59%. The real takeaway here is, you know, we're doing a good job in terms of overall predicting the right and the wrong, but there's certainly still some room for improvement. And that's what we're going to be looking to do in our models two and three is see how do we improve these algorithms and their performance. Key observations and takeaways in this is sometimes our models right now are getting confused and we look at where are we getting confused and why is the precision and recall not higher. This is on things like, for example, a hospital admission versus an emergency room visit and things that are very specific in the severity that our models getting confused on because it might think, oh, I saw the word hospital because you went to the hospital's emergency room and I predicted the hospital and that was incorrect. So we're sort of close there, but, you know, teasing out those differences is something that we're working on to get those a little bit higher. The other thing we took is this very much was it across the board categorization. One of the categories we're also trying to predict is unspecified and that was where, you know, when evaluated, we said we weren't sure where it fell within the other severity, so we're going to put as unspecified. Where our models trying to predict unspecified when really what we should be doing is basically trying to predict the other things and then when we aren't confident about where we want to put it, we mark as unspecified. And so if you want to give the next slide, that's actually where we're trying to go with our performance and to talk a little bit about the measure here. And so every time we're making a guess in our models, we have a probability that we're right. And for the sake of this, we're calling that model a confidence. How confident are we that when we guess this is severity one, that it actually was severity one? And so when you have a confidence, we decided, well, what about if we only start to look at answers where we are, you know, 55% sure we're right or we are 75% confident that our answers are right and going down? And what this does is it takes the whole test data set and starts to throw out answers. Well, we really weren't sure because we thought it could be one of three places. And so we're not classifying or making a guess on that. We're only making a guess when we're rather confident. And so what you'll see is that ultimately here, as our confidence goes up, our scores also go up. And you see we're seeing increases in accuracy, precision, recall across the board. But what's also going down is how many predictions we're actually making. So in this one thing we've talked about as conversing with the staff here at CPSC is what level of confidence do we want to set for these models? And then that's something that we can play with to get the scores to where they want to be. And the reason this is important is that it affects, you know, how much work is they left to do. If we can say, for example, at the 85% confidence line on the slide, if we're going to get a 95% accuracy, so we're guessing the right thing 95% of the time. And what that means is that we're able to make an accurate guess on about 83, 82, 83% of the overall data set. So 100 records come in, we can classify 83. Those remaining 17, we're going to say we aren't sure. And so then that might go to a second review by a person. But we're able to now eliminate about 80% of the work by using these models to make those predictions. Severity was our first case where we're trying to predict how severe an incident is. Our second case is trying to predict what product code belongs to each of these products. And I believe there are over 800 product codes that we're trying to predict against. So now we're taking multiple choice tests where we're not trying to guess 12 things, we're trying to guess over 800. And so in this case, what you're going to see is our score drops a lot. So we're getting about 68% of them right when trying to pick from about 800 answers. And so now we're starting to look at things in terms of, you know, of the overall data, how do we increase the performance here? Because we need to be better at predicting what a product is. And so looking at what we're getting, this starts to get into things like, for example, data quality and data quantity, consistency between answers. We're getting reports from multiple retailers and even within those retailers, say, for example, Amazon, Amazon might be giving us incidents that come from like subsellers below them. And so in those cases, if we see something and we're calling it a product name and we look at the product name field, sometimes those product names are inconsistent. So we don't necessarily know from just the name that this is one thing or the other. And so we might end up getting a really long thing that this is a 12 inch motorbike. And then there's what should be the product name there, but then it's got a UPC code with it and things like that. And so this goes into when we talk about data quality, it's doing some pre processing on the data to get it down to just that that model number in that field. And then we can probably do a lot better job of, oh, we've seen that model number before. So therefore we can instantly assign it a product code versus trying to guess based on other fields. We're also seeing that looking at these and actually doing a little bit more of a machine learning. So not necessarily just looking up a model number against what a product code should be, but actually including the narrative and then trying to understand what is going on with an incident, which gives you a little bit more context, is also starting to increase our performance. And this is something we're starting to see as we're getting into our model 2.0s. But as it stands right now, we just wanted to say this is where we are about 68% accuracy with the product code models that we have. And so that does get us into a little bit of what are we doing next? And where are we going from here? So really the whole point is we're going to continue, we've got two more of these model iterations to go through within our pilot program. So we're going to be trying to tweak different levers and go down different paths. As Mark said, there are more advanced models. We've tried the smaller, simpler stuff up front. We're now going to start going to a little bit more advanced algorithms or things that are newer in terms of the machine learning modeling and things like that to see if we can get better performance. Ultimately, that falls under the features and enhancement side on the right hand side. What we also are going to be working on is data quality and trying to get things a little bit more cleaned up. So our preprocessing that is going into these reports and records, is there more we can do there to clean these up that will help tease out better answers? And that also does go back to potentially looking at retailers. Is there a way we can help with formatting or consistency there too? And then also data quantity. We're looking at the historic records of 2010 to 2018. We also are going to start pulling in the 2019 records now because we've gotten access to those. So we are looking at increasing our data set as well. Having more records to train and learn from means our models are going to be a little bit more accurate as well. And so yeah, I think that concludes what we wanted to report on here. Yes, so thank you very much to the team. I'd like to thank the commission and executive director and other senior leadership here too. So I've been attending the CDO council meetings across government and the level of interest and commitment from our most senior leadership is by no means universal. So I hear from many of my colleagues and other agencies the challenges that they have with getting the interest and attention from from the senior leadership. And I want to thank you for your interest and engagement in this. So thank you to all the senior leadership. Thank you. So I will now start asking questions. And first of all, one of the things that strikes me is that we're expending a fair amount of money. And honest to God, Ms. Rostami, however, it's pronounced. I didn't understand much of what you said. It sounds really important and significant. But fortunately, we have Jim and his team to help us get our way through it. With respect to the retail reporting, I guess one of the questions I'd like to ask, trying to look at it from an overall perspective, is again, this is an expenditure of funds. Bottom line, is this going to turn out to be worth it? We've always felt that retail reporting was sort of garbage in and garbage out. We weren't quite clear what we were getting. And as you said, we really had reached capacity for analyzing it. So at some point, do you think this will enable us to do our job better, both with respect to writing safety standards and with respect to recalls? Maybe not a question for you guys, but I'm going to ask it anyway. Sure. Yeah, I think ultimately what the value and benefit is, is still TBD. So I think we're still part way through this. I think what we've seen so far is that there is potential here. I think what we're seeing initially from the models is that it is, in certain limited instances, like this tool, this machine learning, can be helpful to take some of that workload off. So at least in terms of making an impact to what you guys are currently doing now, yeah, I mean, in my opinion, yeah, I think there's value and benefit here, for sure. I think it's just a matter of where and how far that's going to take it. And I guess that's my question, is that, well, first of all, do we have, and Jim, you may know this, for the retailers that are reporting to us, it seems to me, and again, this goes to garbage in and garbage out, have we established with them a clear enough protocol for submitting information so that when these gentlemen get it, they will be able to analyze it and develop a good machine learning system? So I think that's a good point, that in terms of the model, the model is using the data in the manner in which it's received now, which can differ between and even within retailer reports. There is the opportunity to improve the performance of the model through improvements in data quality that aren't necessarily pre-processing, but are made at the point of collection. So enforcing validation standards or common approaches may be one way of improving that. Some of the use of the model, and these models may extend beyond this particular data set as well. So injury severity has implications beyond this and product categorization as well. So sort of bottom line, when we've got this model up and working and we're getting all this data coming in, am I going to be able to sit at the end of the process and say, oh, okay, here are five products we hadn't thought about or we have thought about, but we weren't quite sure what the dimension of the problem is. And therefore, these are things that we should pay attention to. Is that likely to improve what our emerging hazards capabilities are right now? And again, I'm not sure who's the proper person to answer that. So I probably defer to epidemiology for this authoritatively, but there is the- I'll ask Steve after the meeting. Yeah, there is the currently the level of accuracy and precision is not up to the level of the manual process. We don't think it's quite there. But there may be opportunities to be able to do reviews, even automated reviews, even if they're not quite up to that level of precision that we would not have been able to do at all, just simply because of lack of staff resources to do those. And my last question, do we- because we're always worried about cybersecurity if we're turning this over to the cloud, is part of what you're helping us develop the ability to defend ourselves against any cyber attacks to develop cybersecurity or is that a separate consideration? So it was certainly a consideration that we made and we were very careful as we were setting up the test data and granting access to the test data to make sure that it was in a fit ramp approved and secure area. Patrick Manley on my team and his whole team were deeply involved in that. We worked very closely with the contractors in this. And as we progress, that will continue, of course. Thank you very much. Commissioner Kay. Thank you, Mr. Chairman. And thank you, Mr. Office and to the folks who briefed us this morning. Mr. Rostami, if I can start with you and just try to get a general larger government perspective, please. Obviously you touched on that at the beginning. For instance, there was a piece in the Wall Street Journal this morning about how IRS and other government tax entities are using machine learning to basically identify non-filers and then going and knocking on the door, which sounds a little bit risky. In terms of where we sit, though, where are we relative to that and how much can we learn from those other agencies or are we able to learn from those other agencies about their experiences and then conversely, how much of what we do from your perspective is just so unique that even if IRS is doing it one way, that's just not that relevant to us. So to be curious if you contextualize that for us, please. Yeah, the IRS has made some waves with some of the experimental work they're doing and it's well known in the data analytics and AI space. So I would say they're a little bit on the first movers for the federal government. Yes, you land right in the middle where the, and no, the work is not unique enough that it needs a whole different recipe here than it does at different agencies. So I would say there's a lot that you've established in this data strategy, for example, and in the structure with which you will approach data in the future that other agencies can leverage, agencies I've seen and worked in. And then you have first movers where their mission truly, like in this example with IRS, this is mission critical for them. They're expanded on their, they cannot use any more bodies to do that work or they can't budget for it and it makes sense at that point in time to experiment. So yes, there's a lot you can learn from the first movers. Got it. And so then it's the substance of what you laid out as well as the sequencing in which we're doing it or proposing to do it, starting with these kinds of pilots. Is that entirely consistent with how the first movers have approached their work or had approached their work? Yes. Yes. Okay. You'll find that they did a lot of the structural work that we proposed in the strategy was done years ago and everybody has to start on the journey sometime. Everyone had to transition to cloud. You have to start doing those things at some point and they have to be done before you can get pilots like this up and running. They require certain environments that, you know, if you don't already have them established, you're going to spend the first couple months getting that environment up and running and you don't want to have to do that ad hoc every time. So we liked to put forward the foundation that you'll need to do more of this in the future. Got it. And that makes sense. And then in terms of the sequencing, so you obviously laid out certain steps that need to occur in terms of assessing the data, cataloging the data, cleansing the data, what have you. While that's going on, these gentlemen and Sapien are running a pilot which clearly is bumping up against some of those inefficiencies or tasks that have yet to be completed. And so how do we, if those tasks have not been completed and it sounds like some of what Sapien is doing, you guys can jump in is sort of doing those on the fly. Like you're saying, as you move from one model to the next, you're trying to clean up some of the data. How does that sequence and is that, how does it make sense to basically start jumping in with a pilot on machine learning if we haven't actually done some of these basic sort of data management tasks that you recommend? Yeah, excellent question. And it's something we consider. No agency has unlimited resources and unlimited time to do this work. So the frank of the reality is you're going to get benefits from pilots like this. And you're going to have to do the dirty work of setting up the catalogs and all that infrastructure stuff. So we do think that there is benefit to be had and quickly. It's also good to have those quick wins. And if you guys can achieve mission fulfillment, even if it's in specific areas, maybe not across the, you know, all program levels, it still is a step in progress. So we believe it's worth it. Got it. And then the last question was a staffing question. You had talked about data stewards, which sound like sort of magical people. Yes. And so you've obviously been here a while now and you've assessed our capabilities. Do you see that as something we have the capability of entirely in-house, partially in-house, and we'd have to hire from the outside? Or we would really need to look outside the agency for that. And what does a data steward actually bring in terms of qualifications, please? Yeah, another great question. We strongly recommend those are in-house individuals. It's extremely important that your stewards, you know, our employees of the federal government, that they know the work intimately and have strong accountability. And that's the most important part, because of course we talk about data stewards and all the great things they can do, but in the event that something not so great happens with a set of data, you do need somebody who is accountable to that, in addition to Jim, who's accountable to that. So we do recommend it's in-house. A lot of the governance that we recommend in those six months can be supported, of course, with industry as you're standing things up, but the main accountability does need to be core team members. Great. Thank you so much. Thank you, guys. Thank you, Commissioner Biacco. Thank you. And I'm so happy you guys are here that I could actually dance. I've been saying since I walked in, and this is a self-serving statement, but I can't help myself, since the day I walked in that this needs to be done. And I agree with you. Is it with straw me? For straw me. For straw me. I agree with you that this is a great place to do this. I'm not surprised that you've had the response from our people that you have, because they've been anxious to do it. They know it will help them do their jobs better. And I think the very first meeting I had with our previous acting chair, I recommended that we start doing pilot programs because we are a great place. It's a small place, and what we do is not that complicated as far as what you all do. And we would be a great place to demonstrate some pilots that could be used across the government. So thanks for validating that. I appreciate that. I would be interested in a couple things. First of all, I'm on the seventh floor, but none of you talked to me. How were the stakeholders that you interviewed selected? Great question. Yeah, we did focus on the operational side of the House for stakeholders. So we had all of the mission programs. I believe you also spoke with all of the mission support programs, but primarily focusing on the operational perspectives for the use of data. So who were they were talking to? They were internal. Yes. Oh, okay. So when you said stakeholders, I didn't know if we had some of the retailers or what. That's really what I was asking for. And I would love to sit down and talk with you because I do have a lot of ideas, one of which on the model and how to make this better, I've sat with the retailers that you have data from. And to me, it's a little bit, actually it's quite simple. It's not much different than electronic discovery and litigation. We need to have, if we're going to do a retail reporting setup or whatever data we're getting in, let's just all work from the same 100 words. I mean, that I think would help you a little bit get started. That way you're not dealing with this person saying that and this person saying that. And I would like to talk that through because I think it's easier than it sounds at least to get started on the pilot levels. Is that right or am I off? No, I think you're right. I do know that currently there has been structure put in place. So all the reports come to us in essentially the same spreadsheet format. So there are buckets of answers that we are getting consistently. And so what goes into the buckets? I don't know if it's just examples or things like that that can be provided, but it may be working with the retailers there to better define what those buckets mean. And they're ready to do that. Yes. So we should talk about how to structure that because it's easy for them to do that. And I think that would help us and I think it would help our accuracy go up. How much of this data that you use besides the retailer stuff was the NIST data? I mean, I don't know off the top of my head. I know we have that in the historical data set. For these specific models, we're using the section 15B data. So we're focused on that as sort of the initial problem, but the models could be expanded to other... Well, I'm not sure that I would want that personally because exactly what you said, a little confusing with between hospital visits and ER, and that did not surprise me when you said that, Ryan. So I want to make sure that we need to clean some of this up, which will help, and I may be using the term incorrectly, but it will help the accuracy. That's correct, the overall performance. Okay. I forget who said this, but the informal communications, that messy slide that you gave us, so I'm happy that we're going to get rid of that, is part of what you're doing, and it may not be, and it may be something we need to do, is setting up some type of best practice so that we can record how the requests come in for data and when they go out. Is that part of your rubric, or do we need to add that? That is part of our recommendations. So that is part of the data governance, is tracking how these things are happening. The centralized part is data lake. So when you do set up a data lake, it's a shared resource. Folks have to get authorization. So it's very regimented how it gets set up, and that automatically regulates some of those things, because you have traceability there. Okay, great. All right, I think that's it for now. Thank you, Commissioner Feldman. Guys, thank you. Thank you. I echo all of Commissioner Bianco's enthusiasm about you being here. It's clear that everyone at the table has put a lot of work into thought into something that I think is sorely lacking and overdue at the agency, so I appreciate all the time and attention and definitely for being here and giving the presentation. I have a number of questions. I'm not sure that I'm going to be able to get to all of them, so I may follow up in writing if that's okay. But, Mr. Rostami, I wanted to start with you. You mentioned early in your remarks that you found some real issues with traceability and lineage of CPSC's data. Can you talk a little bit about sort of what the mission impacts are about those deficiencies? Yes, they can, that's a lot of impact. There's two different ways to look at it. One is security. So there's a threat of security, and if something gets leaked, you don't know who did it, how it was done. That is a risk. We've experienced that. Correct. And then the other risk that you have is should incorrect data make it out or a poor analysis, there's no checks and balances and then at the end nobody's going to say, oh, so-and-so did it, they're going to say the agency did it. So it's risk to the reputation of the agency which of course you're here to shepherd. So I would say those two things both represent risk to mission. I appreciate that. You talked about the need for improved analytics tools and platforms. If we make those improvements, what's the return on investment that we would expect to see in terms of mission fulfillment? Yeah, so this actually, these folks are probably more qualified to tell you actually these specific benefits of different softwares, but I can give you a great overview of them. Different softwares have programmers like them for different reasons. So the ones that we noticed that could use more presence here, for example, can be used to cross-correlate better than the existing softwares. So the things you stand to benefit, first of all, young talent is graduating with Python, with R. Those are things that they already come out of college, fresh knowing, and they're having to relearn different software when they come in the door. So that's a risk to talent and retention. And then the other risk, of course, is that you're missing out on opportunities of seeing insights in the data. So yeah, we recommend, of course, when you bring in new software, it does require upskilling, which will be something the agency will have to resource for. But frankly, every agency sees that and takes these steps as the software has become more mature. I appreciate that. We're coming up on the one-year anniversary of the largest data breach in the agency's history. We had thousands of sensitive records that were compromised, and those included statutorily protected data. It included sensitive, confidential business information. It included, in many instances, consumer PII. The implementation plan that you're bringing to the commission today, does that take into account any of the open recommendations that have been made to the agency by either the CPSC Inspector General or Congressional Oversight Committees? I would say yes. We, of course, have a lens for security. Cloud is one of the most secure environments that you can get out there. When you go through the procurement process, you'll learn a lot about how vendors have made this extremely secure. You can go as far as to mandate where the servers are based, and we haven't experienced an agency that's had an issue with Cloud breach. And with respect to the traceability and lineage benefits that we talked about earlier, there would be a benefit when data leaves the agency system that there's some accountability and ease of identification of who the responsible party is. Correct. Okay. So what I'm hearing is that implementation of the plan would actually go a long way in improving our system's integrity and improving the agency's overall security posture. That's correct. Okay. Transitioning to the AI machine learning pilot, it sounds fascinating, everything that you're working on. I'm really excited that you're here. One of the complaints that I've heard in the past and I'm hearing on the dais today is that even though CPSC likes to talk about itself as a data-driven agency, that sometimes the data sets that the agency receives are too large to comprehend. When we're talking about model building, don't we want as large and rich a data set as possible to make sure that the predictive algorithms are sharp? Isn't that part of that iterative process? So I would agree. You want as much data as possible so that you can learn from all the instances of, yes, this was right, no, this wasn't. Some of the things that we're doing in addition to just having data sets is we also are looking at making sure that we're getting balanced records so that the model doesn't just learn one thing or the other. Things like, for example, if you have one outcome that is extremely rare versus another thing, something happens 95% of the time, another thing happens 0.2% of the time, we do try to, in that training data, upsample or create extra records of that smaller data set so that the model is seeing a more balanced influx of data. So there are some things that we can do from the data that we have, but of course the more diverse data that the model sees, the more it can learn from and the more it can infer. Even if that's just making a determination about what normal looks like. Yes, great. Thank you very much. My time's expired. I have no further questions. Commissioner Kaye, additional questions? Commissioner Biakko? And Commissioner Feldman, back to you. Okay, if not, thank you very much for an excellent presentation, some of which I understood, but I look forward to continuing to work with you and I'm sure our staff continues to want to work with you and perfect our systems. So thank you very much. This hearing is now closed. Thank you.