 Welcome. Good afternoon, everyone. This is presentation on data validation and quality assurance within your case management systems. My name is Donald Carter I'm a senior consultant that just tech I have a 20 to 25 years experience working with database design implementation and administration. You name it I've played with it broken it put it back together. My co host is David Johnson from the Legal Aid Society of Cleveland. David you want to introduce yourself. Sure. Yeah. I guess I've worked more in analyzing and finding sort of trying to show people what their data can actually tell them and how they can refine systems and methods of gathering information so that their analyses are more useful to them and helpful to them. And I've done that in a variety of areas and moved into the legal aid area about three years ago. So. Okay. So to kick it off. What is data validation and quality assurance. Well, if any of you all remember the a certain election the picture on the right there will tell you everything you need to know. Data validation and quality assurance is basically a series of procedural steps that you can do to make sure that the data that you collect from your clients and enter into your case management systems is both accurate and sensible. And the meet the standards meet any standards that you may have set for its intended use as an operational planning or decision making tool. By decision making tools we mean, you know, several different things you can use your data to plan to where your organization advances its services into new areas. Are there client needs that aren't being met by your current setup. Do you need to expand. Do you need to pull back from areas where the demand is currently dropped off. And more importantly though what data validation and quality assurance is for is to verify that the data that you submit to your funders is accurate accurately reflects what it is you do as a legal aid agency. So how can data validation and quality assurance benefit your organization. First and foremost, it gives you fat, it gives you the ability to have faster production of your case management reports. I don't think there's probably anybody who hasn't had an instance in their career where they've gotten to the last day before funding reporters do and notice that something in their data just doesn't make sense. The numbers just don't add up or they're lower than estimated values. By instituting a program of data validation and quality assurance you can kind of anticipate and preempt those problems before they occur. Another aspect of it is that you increase your accuracy. Again, reduce time correcting the errors. And your increase your ability to identify emerging data trends, which is something that David specializes in David you want to pick up here. Yeah, so to actually know that you're identifying something that is true you could say or valid you have to have ways that make clear how your data should be recorded and interpreted. So that the people entering the data know what they're entering and know what options to use and then the people analyzing and drawing conclusions from it are looking at that data with the same understanding as the people who entered it. And that's really obviously central to actually identifying a trend and then knowing that what you're looking at is something useful instead of just a collection of spurious bits of information that seem to indicate something. And then that also relates into a sound strategy for how you approach staffing or how you may approach deciding which cases are served, other aspects of organizational management, having a solid universal understanding of your data fields and appropriate controls on how things can be entered are sort of your central factors there that need to be addressed. And all right, so the methodology for instituting a program of data validation and quality assurance is basically primarily starts with the data itself. You need to sit back and identify the types of data that need review and the expect data expectations for what that data should show. Now this can be something as simple as saying okay our funders asked for a lot of demographic information we need to make sure that all our demographic questions are answered. But they can also go a little be a little more subtle than that like a good example would be when you go out in the community and you put on presentation some some organizations may classify the majority of that is outreach. But also a legal clinic could also count as a kind of an outreach type of that there's often some back and forth between the two sides as to which one constitutes which and what should be funding source. And so those those are the kinds of expectations that we're talking about you taking a look at your data and saying, this is how we want to interpret it as David mentioned on the previous slide. The second item here is to develop you need to develop systems and reports for monitoring the state of that data, not just at the date that the report is due but to run is precursors up to that report basically a monitoring system that can be in effect all year long. So that when you comes time for you to put in that end of the year grant request, you know, with a with a much higher sense of certainty that the data that you're providing that funder is correct from the get go. Again, there should be regularly checked regularly scheduled data checkups and evaluations. And my recommendation is the Institute of Regular Program or pre flight runs for critical funding reports. These will be things like your LSE or Iowa. VAWA funding. You want to make sure that your date your results are tracking with your planned estimates your projected estimates and give yourself some time for any corrections that may be needed in case you maybe got a little too ambitious one year and project a little more than you were actually able to do. Yeah, and yeah, so the the use of so like the systems and reports that it's sort of it's a broad intentionally because you can look at this in a in a couple ways. Part of this can be how the extent to which you can configure your case management system to accept only certain kinds of answers or certain ranges of answers or to leave fields blank. It's it's a question of thinking through what is appropriate for certain fields that you're trying to gather and then what information and what level of specificity with that information you may actually need. Then so something could be configured at the level of the what the field allows, or you could set things up to be reviewed in reports that would only pull records that are outside of certain parameters that you can specify. So you can you can approach it from the front end if you have a pretty good understanding of what you want to allow in the first place or from the back end and look for things that are outside of what you would consider normal bounds, but that you want to allow users to record in the system. For you may want to verify, for example, how any how any household size over 10, which of course is entirely possible, but they're going to be pretty infrequent in general or clients eight who are having age of over 100. They will certainly happen, but they're not they're not going to be so common that actually verifying that those ages are correct would be overly burdensome. Excellent point. So now we're going to talk about the core categories of data. And David's out of the point at the top that data may come in a variety of forms, but all of them basically end up in your in your case management system and can be extracted so that your funders your board and your managers and ways that your funders your board and your managers may find extremely useful. The first and foremost is probably client demographics almost every funding source that I've dealt with in the course of my 20 years doing this. Their primary demand is for client demographic. They want to know the racial makeup the gender makeup and the ages of the clients that are being served. The second item is for your staffing allocation. You know, are your are your staff adequately spread throughout your program to handle the legal demands that are within your areas, or do you instead choose to focus on a more narrow practice area say family law, family law specific timekeeping is another option you want to take this one can kind of tie back into certain types of grants. There are there are several types of grant applications that I've seen where you can either request a lump sum for a certain number of cases to be served during the course of the year. But there are also some that at that where you can promise a we will we guarantee say 10,000 hours of attorney work. And so timekeeping is included here for that latter category grant related and grant or specific information again ties back into all of the above. But you may again you may have some very fine line item. Issues that the grantors may be looking for. That you'll need to be able to track identify and report on. And then finally geopolitical information and this again all of these kind of interact with each other. But the geopolitical information let you know where your data is coming from specifically. So for client demographic information this is the meat and potatoes. As we mentioned, race, gender, income levels. Marital status, citizenship status, disabilities, living arrangements, whether they own a home or renewal. These are most some of the more common demographic information requested in most every report that I've ever seen and done. We're staffing in time. I'm going to let you pick this up. Actually, I was going to expand a bit on the client demographic information to certainly it is very common. And but the answers that you may have to provide are not necessarily common among all of those all of your funding entities. Everyone certainly seems to want information on say race and ethnicity. But they may have different funders may have different categories that they want you to report on. So being aware when you're when you're looking at how you set up possible responses to really all of these categories. It's you have to consider what exactly that reporter may be demanding in terms of sending information back to them if they have specified marital statuses that they want you to use or citizenship statuses. Some may have some may require some may some funders may be happy with citizen or non citizen. Others may certainly require greater detail on who is a non citizen or their particular legal status within the non citizen realm, you could say. So having designing your system to gather the appropriate responses and then ensuring that it can before you perhaps take on a grant or start the active period for a grant is really very necessary. And then, yeah, so staffing and timekeeping. There's really, you could see a considerable overlap in how these reports are approached. But you can one way that you could sort of distinguish their use is that the timekeeping reports may focus more on how you look at the work performed by an individual. While a staffing report can be more aggregate to see what a unit or even the entire organization is doing. So while they may have a lot of the same core data regarding time spent on certain activities, activity codes, counties where activities are recorded. They would aggregate and use present that data at different levels for different kinds of decisions that your organization may be making. So as our next slide has there. The staffing with it could involve looking at your overall payroll volunteer and pro bono work, looking at aggregates of time by where they happen or by physical office or sort or like unit within the organization. If you get for say family law or housing law. So it lets you see where your where the staff are working and what what sort of work is being done. The timekeeping is sort of then the subset within that that that would look more at what this individual may be doing in their week, how much how much and what kinds of work they're recording. How much is being billed by certain individuals to certain grants if you're expecting to fund if you're expecting to fund the given position with with one grant. How much of their work is actually eligible to be billed to the grant. And sort of giving you a window into well, what is this person doing are they doing what is expected. And then is this actually fundable perhaps by a specific grant. And does this person working so much on a certain case perhaps that they need assistance. You can you can identify if there's a particular concentration of time that may lead to them overlooking other things that they're currently assigned to be doing. And so, well, I go ahead. Oh, I just so sort of as we had mentioned that the staff in the timekeeping reports may draw on a lot of this same core information. But you could distinguish them conceptually by the purpose that you're trying to have the reports serve. And the things that I would point out here is, you know, this is a very this looks like a very simple list. Initially, you know, funding codes time spent activity types. What kinds of things you can look for in your funding codes is what kind of funding codes are being built, built on your time slots. Are they temporary funding codes. Are those temporary funding codes being replaced at case closed with a correct funding code. Is the funding code appropriate for that case or is it is the case work completely LSE eligible or completely Iowa or VAWA eligible. Is it a perfect match. These are things you need to look at from a kind of a higher level perspective to make sure that your funding codes that your time is being recorded to the funding codes that they need to be recorded to, or that they may possibly be recorded to in another manner. And the things that are spent in the activity types as David pointed out those could be indicators of a need where where you have some work, you see some work that could be overlooked because of an excessive amount of time being spent one area. Staff types and roles. So the things that you want to look for here is are who's doing the bulk of the work or is it is it falling on most of the time being recorded by your attorneys or by your intake staff. Volunteer status, a good thing to look for here is if the volunteers timekeeping is sluggish or and I know this is from from experience with a lot of pro bono programs that sometimes the pro bono act advocates do not always get back with updates on their time expense on those cases. Well, when that happens that can sometimes lead to the pro bono coordinator looking at them and seeing that it's looking at their record and seeing that there's a less or an overage of time commitment this case has been sitting with this pro bono advocate for X number of days, weeks or months. Then I'm going to be less likely to assign that pro bono advocate another case because it looks as if they're still busy with the other one. County of residence ties ties back in closely with county of dispute. You may have an advocate whose specialty is in another who is all the forms and all the practical experience to work in a county I know this is a big problem here in the southeast where each of the metro Atlanta counties has different procedural practices for divorce cases evictions. And so, if you have an advocate that's more familiar with the procedures that need to take place within that area, then maybe they, they need to be shifted over and assigned to the cases that deal with that area. And then of course physical and organizational office as David noted, if you notice that a lot of work is coming from a region that is outside your normal sphere of influence. Maybe that's an indicator that a new office needs to be looked at, or maybe an office that's more close close to that area could pick up some of the load. David you want to pick up for grant. Sure. So this actually, as you may know, a number of things sort of loop back to prior themes where a lot of legal service entities are funded by a lot of different. So your funders will very often present specific, specific reporting requirements that relate to how they want to hear about what you're doing with their funds. And so you have to ensure that your system is designed in such a way that the data that they require can be reported to them. And that even the options for data that you're already gathering permit you to be responsive to a new funder while remaining able to report to all of your prior funders. And so you may also, as an aside, like you may be able to say to a funder, well, we gather this information in this way and present to them a sample of the kinds of data that you can report to see if that is appropriate or acceptable for what they're doing. And the extent that they may allow you to use an existing report or fields obviously may vary. But if you're able to present in a clear way, this is data that we are used to reporting in this fashion that may be helpful to them in deciding what exactly they need to hear from you about the work that you're doing. Okay, geographical information just as, you know, legally, you guys are all over the place. All politics is local and geopolitical analysis shows you just how local your problems are. Which types are concentrated in which areas, how your client demographics are distributed within those areas. And so professional funders are generally really big on this kind of this kind of data because, again, they don't know your neighborhood as well as you do. So putting together reports that translate your knowledge and to something that they can understand and visualize is certainly useful. But it also helps you it helps you as a firm identify trends within your region. So be looking at your your geographical reports and notice that for, you know, the last six months of the previous year, your number of health law cases increased in a specific region of of your state, or, or even accounting. Is that an indication of a trend emerging. Do you need to devote resources to that trend. No, and the way that you can know that is again my monitoring the geographical information throughout the course of the year. David has another example here that I'm going to let him explain. Sure. So this was, we were trying to increasingly use GIS type analysis. And so one of one of the one of the things that we attempted to look at was where we receiving intakes from across our service area is roughly in line with the poverty, the percentage of eligible population within the area. So this is one of the kinds of ways that geographic analysis can help you understand what your organization is doing and where the clients you serve are coming from. And as a short explanation, so this map in the red areas were subdivisions of counties where we had fewer intakes than you may have expected, given the underlying eligible population. And then the white were around what you would expect and the blue were over what you may expect. So you can you can look at different you can combine different geographic levels. These are mainly county subdivisions, but then the city of Cleveland itself you can is divided into the wards of the city. It was all based on underlying census tract information to build in all of the population figures that were needed to conduct the analysis. Then obviously to do this, then you need all the correct addresses entered in a way that they can be geocoded. As you can see in some of the small font, there was an 86% success at geocoding addresses recorded at intake. When all this information was done, we were using pica and had a very basically all the address information was, for the most part, just free text input people, the intake staff could record whatever they needed to record in those fields. And there was relatively little validation that happened at the data entry point or data entry level. So 86%, it seemed relatively good for that. The it was that tens of thousands of contacts. So there wasn't really any correction that was done after the fact to try to make more matches. But it's just an illustration of how geographic information can show you interesting things about work your work. Okay, so the next slide here, this is where we start to begin talking about some of the things that you can do to improve your data quality and data validity. This is the, this is the how to section. And this, a lot of this will depend on your relationships with your database vendor your database designer. And also your own experiences with your case management system. But some of the typical steps that you can try and we're going to give some practical examples on on why these, why these should occur are prohibit no values on critical field what a no value means is a blank. A good example here would be again your demographic information. If your client is if you're not getting your clients demographic information, then your report demographics are going to be slightly off. Numerical fields or date fields restricting them to same values. The example here is will family side ever exceed more than two digits you're going to have a family, a family law case where you're the number of kids numbers more than 99. If not, then restrict the field size to two digits don't allow free form of more than four or five characters every now and then we call it keep you know cat keyboard typing somebody may rest a finger a little too long on a on a number key and add an extra digit. If you have a field restriction that extra digit can often be for him can be prevented ahead of time. Another same. Another example of the same value on a date field would be like setting restrictions on how far ahead, or how far back your timekeeping can occur. Keeping under most grant conditions is required to be done in a timely manner. We would ideally they would prefer that everybody get their timekeeping done on that day I know my boss certainly feels that way. But the fact of the matter is sometimes you just can't keep up and you need to go back and you need to get caught up with your timekeeping for a week for a month or whatever. Allowing that past date entry to go too long can leave you with serious holes in your timekeeping data. You may have a an attorney who who forgot to enter his timekeeping for months and months and if you're not restricting them or preventing them from from allowing it to go that long then then the chances are the problems with the date entry is not are not going to correct themselves. Another thing you want to do just for preventing things from being entered in the future. I do a lot of data migrations for firms going from one database to another and one of the more common aspects of that migration is a substantial amount of date cleanup where again it just these are just typos somebody types in 2019 and 2019 instead of 1999 or they type in 9999 or sometimes they drop it they didn't just do 201 and hit tab over to the next field and now we have somebody that looks like they were born in the second century. Setting restrictions on the date field you can also control the formatting of the date field these ways and and the numerical values. These are just a little way to kind of help correct for more what we call more the more common typos you're going to see within a case management system. To tie back in with the previous slide for addresses, particularly if you're going to be working with things like geocoding. The validity of the address is very important. Sometimes you will have attorneys that will enter in or intake workers that will enter in a client's address based on the apartment name because the client just assumes the apartment name is the street address. If the apartment resides on Elm Street but the apartment name is Elm Street Oaks in order for it to be geocoded the Elm Street is what needs to be what needs to be entered. And so for these types of things you may have to get with your intake staff and or your advocates and make sure they have an understanding of how the apartment addresses are being relayed both from the client perspective and entered into your system. We'll have another slide coming up to show some way that you can check check the quality of that data entry over time but it's just something to keep in mind. Another thing other things that you can do is you can set thresholds on timekeeping to prohibit a case from going on for too long. This also ties back in with monitors on the date open or date close field. Let's say you had a case that opened in 2006. It's gone through a number of appeals. It's still open but it's been two or three years since anybody's done any activity on that case. You can build up a report that monitors the last state of activity on that and then goes back to the manager and lets them know. Okay, it's been X number of weeks since attorney since the attorney revisited this case. Maybe it's time for a nudge. Grant billing is a little bit a little bit of a more tricky thing to do it depends on again on the systems that you have set up and how your accounting systems work. But a lot of this one of the things that you could do with grant bills you can prohibit the staff from entering time onto a grant that has already been used up say for example you have $100,000 grant. By February you've worked all the cases you can you've already booked 100,000 hours to that grant or you booked $100,000 worth of attorney time to that grant. From that point forward, no one else can bill any more time to that grant without triggering an alert. And then finally, you know, all of these design changes, all lead back to one thing and that is constant review review review. Your data doesn't remain static your client base doesn't remain static your client population moves your, your demographic shift, your caseload wither and die, or will increase and become more demanding. Only by keeping a review on this information can you keep ahead of where where you need to be as a business is your your legal service office is first and foremost a business, and it needs to have make sound business decisions on where it devotes its resources in order to serve its clients. So the following slides are going to be a number of we're going to show you a number of some sample reports that can set up. You can set up we're going to give examples of things to look for within the reports little red flags that you should notice. One of the things I want to one of the points that I want to make clear here is that when looking at reports to review or case management management reports to review. There is a tendency within within the community to look at these with a more personal interest. I mean, you're looking at this case, you're looking at this caseless and you're saying, does this meet my immediate grant needs. What what what a quality insurance and data validation is about is taking the personal out of it. You're going to step back a little bit from your data you want to look at it from a more a more disconnected, broader view. And take away your personal investment and what you want to see and train yourself to look for things that you didn't expect to see. All right, moving on to the examples. The first one here is a basic case reports. Sorry, that was my fault. A basic case report. These are, you know, the bread and butter of every case management system you can. It's generated fairly quickly, built up to capture the kind of basic information in this particular instance, describing predominantly demographic information. So you can track, you know how your funding codes are being distributed whether or not there are gaps in your funding code. Whether the close reasons make sense, or whether or not there's a stray code in there that doesn't. Again, looking for holes and data and gender. One good thing to point out here is there's a number of database systems that have fields for a number of people under 18, number of people over 18. And they also have a companion field for total household size. Some of the things you want to look for some disagreements between those fields. Do your, do your children and your adults add up to the overall household size or has some or the math gone wrong somewhere. The next one is user management report. Now there is a side, I'm going to, we're going to spin up on a little side joint here. There's a security aspect to the user management report. The primary purpose of a user account review is to make sure that your salary is entered, your staff are entered in correctly. You also keep a little, you can keep track of their, their case loads to see if they have an excessive number of open cases or closed cases, or too low. You can check there from a payroll and accounting perspective. You can check their exemption status, make sure their salary information is correct. You can even track, you know, payroll leave balances, sick time, vacation time. But one of the other things that you can do and I would, I would encourage you to do is to review the, the accounts that have access to your case management systems. And to do these on a very regular basis, no less than quarterly. One of the things you want to make sure of is should everybody that has access to your case management system, should they have that access. Have you had an employee leave in the last month that can still access your case management system, and all your client information was that departure hostile in any way. Those are some of the security concerns that I want to bring up with the user management review. You just want to make sure that your staff that are in there belong in there and the staff that aren't done. David, do you have anything to add here for user sign. Now with the users, I was going to say actually within the just a brief point. So your, your reports, if you all the reports should look at way things that need to be checked that you couldn't configure. Yeah, the, the, the appropriate values for at the, at the, at the data input stage when you with the example of people under 18 and over 18 and total household size. Your system may have that as three separate fields that require entry. And you could permit that or you could try to design your system so that you enter number of people at under 18 and number of people over 18 and then have the total household size be automatically calculated and not allow user input for that. Both the one obviously fixes the problem of the two not summon, but there's still the possibility that someone may enter a value incorrectly. So you would still want to review those numbers for their reason ability, but you could design the check on the total size either to occur at the time of entry or to occur when you review the data. In the report and that's, that's a decision that is really up to you on how your organization is going to enter and check data. Right. No, so our next example here is a timekeeping report that shows cases with no time or no time entered in next number of days. This is the one that we were mentioning that I mentioned earlier about cases that have gone stale or stagnant. These are actual numbers, believe it or not. They are, they have come from an amalgamation of client systems. The data has been anonymized and the client systems will not be identified but as you can see here there's a lot of range in the number of days that some of these cases have been worked. Now these may be valid time frames. You may have a case that's undergone multiple appeals over the course of the year. So what you want to do here is you want to compare your number of days since the last activity versus versus against the date of that last activity has it why has nobody touched this case since 2012 or did somebody just not update a case at that point. These are the kinds of case management aspects that are often overlooked in a legal aid agency. And a lot of this can also tie back in with the user review that we mentioned before. You may have if you don't have the proper protocols in place when an employee leaves your firm they may have left a handful of cases open and if you don't have the protocols in place to ensure that those cases get picked up by another advocate then you run the chance you run the risk of having those cases remain open indefinitely and turn up on a report such as this at which point you have to decide okay have you know how can we correct this problem you know are we running the risk of malpractice here by not speaking to this client since you know for X number of years. This is one way in which two separate reports can kind of enter interact with each other and that's one of the overriding themes of this whole talk is that you know your data your data is constantly lacing itself through other data. And by stepping back and taking a broader over overview, you can sometimes identify areas that may have escaped your notice before. David anything to add here. No. All right the next one here is just a sample grant goals report. As I mentioned earlier, grants come in multiple forms some of which are you know how many hours did we spend on the grant or how many or how much money how much attorney time and how does that translate into dollar amounts spent on the grant. I don't know why both these charts have dollar signed in them one is supposed to be a straight number. But basically this is a good way of visualizing how your how your grant expenses are going throughout the course of the year. Certain systems you can set these up manually other you may need to dump the data to excel and work this up yourself. But basically they're simple values where you take the the first chart for example shows that we were valor gave this firm $565,000 and as of this point in the year they we've only booked 148,000 of attorney time. Our attorney charges sorry. And the same thing applies on on the second chart but from a time book standpoint. Again should not say dollar signs. And this shows you how your case time is being spent with regard to that grant work grant related work. David anything down here. Yes, so one thing, of course, to be aware of if you have, if you have your staff selecting funding codes is also to review the cases or activities that have actually been assigned to those funding codes. So you may see that a report showing well this is all the time we have in this funding code and that's worth this amount of money. But you also need to look behind that to ensure that the activities for which that funding code was selected are actually eligible to be funded by that grant. So you have this overall view of how much has been accorded to the grant, and then you need a behind the scenes view of what exactly has been allocated to the grant. Okay, another, another report you can do is a simple, this is a very simple one. Just to let you know when your grant grant renewals are coming up. You can record your your the contract term start dates and end dates and a lot of your accounting and case management systems. You can build up you can set up. I know within the legal service system you can set up an automated email reminder, which is what the graphic there to the right that is delivered can be delivered on a schedule. You can say the first of the month middle of the month to let you know what what explorations are coming up, and then you can begin preparing preparing for the grant renewal process, or giving yourself room enough to get the time keeping caught up before that grant exploration comes to this second this next one is kind of a to build on David's comment about looking at the back. Again, this is another kind of tracking the grant expenditures over time, though, in this particular one we're looking at a DAP contract that has gone from 2014 to 2018. And we've got a historical tracking of how much that grant has been pulled in each year and how close the company has come to meeting the conditions of that grant there are a couple years in there that were the system was not tracking. This one could show you again, you know whether or not you're going over budget on a grant or, or maintaining staying on track with what you what you projected to the funding for it. David any comment here. All right, address validation reports. This is a simple one. This one just basically tracks this is from a legal server system, which has a GIS module that can be plugged in. And as one of the aspects of that module is that it does an automatic GIS validation it doesn't look up with the US Postal Service database and verifies the address that's being entered is registered within that database. And if so it gives it a mark of pass or fail on the back end of the database and this is just a simple report right here that pulls a list of cases that have passed or failed that data validation and geocoding check. You could build something similar to this. Again to go to refer back to David's map where he was talking about the 84% success rate on the geocode side. 84% is extremely high I think the best I have ever seen on a national level has been a 92% success rate on geocoding. So one thing to keep in mind is even though the ideal is 100 you're probably never going to see that. And the simple reason is is that a lot of this depends on government data from again from the post office primarily. And everybody is still building. You'll have new neighborhoods coming in once a week, almost within your area. And until those neighborhoods get picked up by the post office you may not have an address that validates at that point. So address validation reports become necessary because you can revisit them throughout the course of the year, what failed to check in January may pass again in March. So let's me know, go back revisit those cases rerun those check and see if they pass and then it becomes a self correcting report in a way and that as they as more and more get corrected that let's get smaller and smaller. Case activity report what this is is this is a report that's basically tracking the time expenses by hours on a case now have to switch my little window here out of the way for a while. And so what we've done here is we've broken the vertical columns down by zero to 20 hours all the way up to 200 hours or more. And you'll notice here that we have a couple, particularly down on I can't. There we go. Can everybody see the red dot. What I'm doing is I'm pointing at about the set the road the 75 SSI at the very end we have one hour. One unit of time booked under the 200 hour point and what that means is that that after after 200 hours. Whoops. Sorry, that the case has already exceeded the 200 hour threshold and they're still being time entered on it. Is that an indication of a case gone rogue or is that an indication of another appeal. Taking place. Only you will be able to know but this kind of monitoring right here let you know where the bulk of your case. Time entry is being focused on our this is basically another way of saying this is our case turnaround time. The majority of our work appears to be falling within the zero to 20 hour range, although we do have a few outliers outliers that exceed that. When you see those numbers you want to make you ask yourself as a firm, you know, do we need this indication that we need more people. Or do we need to spend more focus on these particular areas of law. David, I think you wanted to build on this a little bit. So, and I think this was partly the interpretation of this report in particular was showing the top four cases opened in a month, how much time has been recorded on them. So, this may also reveal errors that people may have in their time. If you see a case that has more than 200 hours recorded on it in the same month that it opened, someone may have submitted a time slip that has an incorrect number of hours on it. So, reports that look for outlier or suspect information are really your friend in a variety of contexts, contexts for trying to ensure that all of the data in your system is as accurate as it can be. From here on out, we're going to show you some reports from other systems. David is going to explain the majority of these. Sure. Yeah. So, the flags associated with. Yeah. So, this was these are all reports constructed in crystal that we used when we when we had pica as our as our CMS. We had the open open case aging sort of gave a variety of indicators for at the individual advocate level, which cases they had open the problems they were in, how long those cases had been open. How much time had been recorded, excuse me, in those cases, and then when that time has been recorded in the case's history and ultimately how long it's been since anything happened in that case. So it served a variety of purposes, all in one to see how much, how long have we spent on, say, this a given case. When was the bulk of that activity. Are we are we basically just waiting for the case to close. Has a case been left open without any activity. If this had been a different page as that day since last activity increased if it's over 30, then it highlights that in particular as you need to pay attention to this case. Why has it been 30 days since anything has been recorded for it. And that may be for a perfectly valid reason, but it's something that the manager of this employee would look at as these reports were automatically generated and sent periodically. And the next slide. So this was the case contradiction. So we had our, we, our implementation of PICA was fairly open in what we would allow people to do in terms of recording information in fields and a lot of what would be cleaned up was then checked on with check through reports. Like this particular report would highlight instances of cases where there was some sort of illogical or incorrect combination of fields that were recorded for that case. Like you can see rejected without a closed date. There was just missing a date or rejected with a closed code. We had the status of the case being rejected, but then we had a closed. We had a certain closing code that implied service. So obviously there'd be something wrong with that case as it cannot be both rejected and closed with service. And the next slide for this actually shows the, shows the definitions of what each of those categories specifically is looking for so that even a new person coming in and looking at this report knows exactly what fields have been set to what values to trigger this report pulling that case. And saying, hey, there's an error here that needs to be resolved. These report, the report could also, this one is not specifically configured to do it, but you could also look for a logical contradiction like a closed state that precedes an open date. So you can require that your closed state be filled out, but you may not be able to require that the closed state be later or the same date as the open date. So you'd want to be sure that someone doesn't accidentally type in the wrong year and close the case before they ever actually opened it. So designing reports that look for possible logical contradictions in your data can be very helpful, particularly for situations where you can't program or set up the fields at the input stage to only accept logical values even cross-checked against other values already recorded in the case. And this brings up another point just to reinforce the earlier point about the need for periodic review. You can run a report like this and see that you have lots of problems with, let's say a case can be rejected without a closed date. Okay, so you make an adjustment to your case management system to fix that. Does the problem go away? Or does the problem emerge somewhere else? Does the same problem emerge somewhere else? Only by doing these regular reviews and looking for these kinds of anomalies can you eventually get to the point where your quality assurance levels begin to increase and reach a higher and higher accuracy threshold. So that constant review is, I can't understand that enough, is necessary. Time consuming, but yes it's necessary. Yeah, and so the missing demographics report is very similar conceptually to the case contradiction report. It's just that in this case the data fields that we want to require on a case were null. And so this brings up an interesting point in that you could set all of these fields simply to be required and say, I believe all of these are gathered at intake for this report and simply not allow the intake to proceed until all of these values are filled out. That may result in you having to stop certain intakes depending on what information you're able to get from the client at the time that they're calling in. Or you could take the approach of allowing the intake to proceed and then reviewing after the case may have opened what information do we need to get to fully fill out the file for this client. So you may allow a field to be null initially, but you may ultimately want to get that information. And that this kind of report can help you in that regard to see, well, what is being missed and what should we try to get for this case to have a complete set of, in this case, mainly demographic information about the client. And so then our next slide is a sort of combined staffing and timekeeping report depending on the level at which the report was run. This report could be configured to pull for individual staff members or you could configure it for groups of different kinds, either all of your staff attorneys or all of your managing attorneys or everyone in the housing unit. Different aggregations could be looked at to see what categories of time they were recording, case matter, supporting time, vacation time, how much had been recorded in the year and then breakdowns of specific programs that have had time recorded in them or specific funding codes where that time had been allocated. And so it would give a perspective both if you're looking to see what an individual is doing or how a unit is according its time say to a funding code or two different programs. You may expect that one unit records a lot of time under bankruptcy. But if you see other case time recorded for that group, that may show that perhaps they're opening related cases for a client and they're being having time recorded in that. Or other anomalies or unexpected ways that people are working that a report like this could allow you to review. And then this is the last of sort of these examples. So resources, there's an awful, this is a highly technical topic. So the best we can offer as far as the resources are, you know, we've spoken here about a lot of the reports came from the legal server site. So there's a link here to the legal server health pages, which goes into the database design and the individual elements of the site. Intap has a number of resources available for some resources available for database design and structure. But primarily your, your, your majority of your technical resources are going to come from your database developers and your database managers, your, your IT specialist, your, your ultra geeks, who can get inside your access. Or your legal server or your SQL databases and can set these fields up for you in a manner that is best suited to your needs. That there's a there's too many different sources for me to link to here. But speak to your legal sort of speak to your database developer, your case management developer. Talk to them about some of the concerns that you may have that may have been raised by this discussion. And they will be more than happy to help you work, help work with you to figure out ways to deal with these problems within your case management system. Bear in mind that it is up to them to satisfy your needs, not the other way around, you know, you're paying for a product and part of the product is that it gives you the data you actually need. And yeah, go ahead then. Oh, I just want to say this was this is also an area. I mean, in a way it goes without saying, but your, your colleagues across the country, everyone is dealing with the same sorts of problems relating to data and funder requirements for information. And so consulting others who are in the same situation that may have decided on, well, we reasoned it out this way and it works very well for these applications. Can be very helpful in trying to decide how to design your system or what what data you gather where and how you allow it to be entered. But once you have made all of those decisions, which you've never really made all of them because it, you will change over time and the data you record, you should be certain that everyone knows how the system works, how to record data correctly and what all of the funding abbreviated funding codes and different parts of the system mean so that data is being recorded with a common understanding of what it means. And that the data is being analyzed with the same understanding as the people who were recording it. So creating a reference guide or cheat sheet to give an overview, particularly of fields that are extremely important, like closing codes, funding codes, what a program applies to, what the rules are for saying, does the case involve domestic violence, or what options you should select for citizenship attestation. These very sort of central data points is very important that everyone knows this is what you choose in this situation. So a reference guide like this page and then this this is just a front and back this this slide in the next slide to give a clear indication to staff. This is how you record these core aspects of our work so that everyone knows what is being done and where it's being done and for whom it's being done in a clear and consistent fashion. Right, the mantra of any any IT person that you'll talk to is documentation documentation documentation. And on this note is how we'd like to close close out this session. There are a lot of issues here to look at there are a lot of variables that are in play. Case management systems are huge. They're, they're complicated. But so is the practice of law and part of the function of a case management system is to operate as an appendage of the law to to serve the needs of the law and to serve you as its users in a way that helps you make sense of your clients and your activities. But there's a give and take and that give and take involves you translating to the case management system as much as the case management system translates back to you. And so be sure to document everything that you do. Try to keep it as clear and concrete as possible. And make sure that as David says everyone is everyone operates on the same page and then that's probably the greatest assist you're going to get to your data quality and data validation projects that we could ever offer. So on that note, it's like to close out. Thank you again for taking the time to listen. David in my contact information is here on this slide. And we look forward to any questions or comments in the future.