 Welcome everybody. We're about 30 seconds before the hour here. We're going to get going at about a minute or two after 10. We have a great group of presenters here. I just wanted to let you guys know about a few new resources while we're waiting for people to log on. Ellis NTAP has launched a new website. It is on the D-Law platform. We have transferred over our 100 most popular pieces of content from the old website. If there is anything that you can't find, just a second. Did you come back in 30, or sorry, 45? Thank you. Okay, I'm back. The hazards of doing things remotely, I'm actually out at the statewide legal training conference here in Washington State doing today's webinar from my room here. And I forgot to put up the do not to start, but we will be in good shape now. So we have a new website which has about all of our most popular content. If you're looking for any of the old content, please just send us an email and we will expedite transferring over those particular pieces of content. We're going through all of our old content. We have about 12 years of content dating back to 2006. Updating links and seeing if they are still relevant. Some of them definitely were not. Our YouTube channel is still prominently displayed here, and I highly recommend checking it out. We have our last three years worth of training all archived and almost 200 free videos for the legal services community over there. Today's training is being recorded. We will have a copy of it available on YouTube within a week, along with a blog post that summarizes it. And we will also share the slides at that time. Thank you all so much for coming out to this. I'm turning it over at this point to Laura Quinn, who is the organizer for this. Hey guys, it's great to be here with you. Let me, oh I just said something weird for sharing. Hang on just a second. I'm going to share my slides with you. This looks good. We can't see your screen. Luckily it defaulted to what it had done before. Fantastic. So we're going to be talking today about what we're calling baking evaluation into your technology projects. And we'll talk about what that means. So the idea of it in general is to instead of thinking of evaluation as its own kind of stream where you come around to it at the end of the project, to think of it as something that is designed into every project so it can be a continuous improvement process. So I am Laura Quinn. I am the principal of Laura Quinn Consulting, not a very descriptive definition. I am an independent consultant specializing in technology research and design for access to justice. I'm working with the Florida Justice Technology Center right now as well as a few organizations in Illinois doing assessments of the Illinois and Beagle-aid online triage system as well as working with an organization that you'll see a case study from the new Ohio Legal Health, so a completely redesigned Ohio Legal Health. And I'm really excited to have two other people with me on the line. I'm going to ask them to introduce themselves. First, Zach, you want to say a little about yourself? Sure, thanks. I'm the program director at the Illinois Equal Justice Foundation, which is a role I've had for going on four years now. And I work with legal aid organizations across Illinois. We're a funder, so we fund legal aid programs. And then occasionally we pick up extra kind of access to justice projects and initiatives including the Illinois Armed Forces Legal Aid Network, which I'll tell you about in a few minutes. Terrific. And Brandon Thomas is here with us from the Florida Justice Technology Center. Brandon, you want to introduce yourself? Absolutely. Thanks, Laura. So my name is Brandon, and I'm actually an analyst and developer for the Florida Justice Technology Center. And I'm involved in creating a lot of sites. You'll see one today in my presentation, Florida Name Change, a site that micro sites that kind of address certain legal issues. And then I'm also in charge of Google Analytics and a lot of the reporting that we create and develop in the organization. I'll show you guys some of that later today as well. Fantastic. All right. My esteemed group of panelists. What are we going to talk about today? So I'm going to give a bit of an intro to kind of the frame for this conversation. So kind of thinking through the term evaluation, which is actually a term I really don't like very much. And I try to push back against people who want to do kind of program evaluation of technology versus kind of the idea of baking metrics. And we'll all talk about kind of what that might look like. And then we'll do three fairly detailed case studies. So Florida Name Change, Illinois Armed Forces Legal Aid Network, and the brand new Ohio Legal Code. I mean, look at the metrics. So basically how all three of these sites are using those, and they in fact are all free websites. How would they are using metrics for a continuous improvement process? And we'll end up with some questions and answers. Certainly you can please feel free to ask questions at any time during the chat. And we will certainly take them as they come, especially as they apply to what we're talking about right in the moment. We'll also have, I think, a little bit of time for a bit of a panel discussion at the end where I will ask some questions of our presenters, or certainly I'm happy to ask your questions of our presenters and to kind of wrap up this whole theme. So let us dive in. So kind of thinking of this overall concept. So I think way too often in our day-to-day work, evaluation is something that you come to at the end of, you know, a year-long cycle, a two-year cycle, a three-year cycle, and you're like, crap, what am I supposed to be doing again? I need to provide a report as per this cartoon here, which I like. So, and this is, it's really ineffective for a lot of ways. And I would argue that one of the most important reasons that it's not effective is it doesn't actually help you to improve anything. So at this point, this kind of report that you can imagine that this woman in this program evaluator will be getting out of this exercise that they're about to do. It's going to be hard to actually make nearly as many improvements as one might like to see based on that. It's going to be more of a busy work exercise, unfortunately. So kind of in more detail what this might look like. So pretty much every proposal these days has a, asked for at least a quick overview of what you might measure in a proposal. And it's pretty easy to throw some things in there that will sound impressive. I've done this myself. And then maybe there's a more detailed plan for a funder, which is kind of think through, what do people do here? It's so early. I'm not quite sure what I'm going to do. And then you design and you build and you roll out your system. And then you get down to the end of the road. You get, you know, two years later and you're like, crap, what were we supposed to measure again? What are we doing? And it becomes now a really annoying exercise to try to put this whole thing together. So I would propose instead the idea of baking the metrics into the process. And this is certainly not a new idea from me, but kind of conceptually the idea that you're going to, you're still going to need a quick eval plan for your proposal and you're probably going to be early enough in the process that it's going to be hard to know precisely what metrics might make sense. But you can kind of take a stab at it for the proposal. And then you're going to design a chunk of the system, which includes not just the, not only the design for the system, but you're going to design the metrics to help you measure that piece of the system. You're then going to build this piece, you're going to measure that piece, and you're going to iterate. And each of these pieces might only be, you know, four months, six months. They're going to be a much smaller chunk than like a whole two-year project. And then at some point you're able to report, create a report for a funder out of already existing metrics. So the report becomes essentially a write-up of things you already know. So you've already done, you know, potentially three iterations, and you know how things are going. You write that up for the funder as opposed to that being, that report being the driver of everything you're learning. So this general idea is continuous improvement or performance management process. So it depends on which world you're coming out of. This is continuous improvement out of the corporate world, performance management out of the evaluation world. It also gets pretty close to kind of more of the idea of an agile, developing methodology out of the programming world. So like I said, this is not at all a unique concept to me. But it has a lot of benefits. So you don't need to scramble to figure out how the system is doing at the last minute. It also, there's no big one-time outlay of expense. So instead of thinking of evaluation as a, so either in my experience you have it as a big chunk of money, you know, like a $20,000 or a $40,000 outlay at the end of a two-year project, or you have it as, you know, a small outlay of money that isn't nearly enough money to actually know anything. So you have a $5,000 evaluation, but that's not enough to actually really understand the things you might want to understand. As you go through time, you can refine your metrics as well as your design. So you can say, all right, well, these things that we're measuring are not as useful as what we might measure. Let's refine those. You can look at the data that you're gathering and understand things about your data cleanliness or other things that really will inform upon how you're collecting your data. And you accumulate data before you have to report it. One of the actual really actually tricky problems coming from me as someone who has done a reasonable number of technology evaluations is at the end, so if we do a two-year project and roll out the project in August and we have to submit a evaluation report in October, there's very little data on how the system has actually done in the real world and it's very difficult to actually say anything because there's simply not enough time and enough volume of data to really know how it's doing. So there's a big benefit in doing the smaller chunks, which is that you have more data in hand to be able to say things that are more compelling in your final evaluation. All right, so that is my overview of kind of this whole kind of concept and thought process that we're going to be thinking through and kind of to focus in in each of the case studies. Any questions or thoughts on any of that? Just resonate, not resonate for anybody on the line into the chat. So just to remind people there's two ways to ask questions. One of those is raise your hand, which there's a little hand icon on your control panel. The other is there's a question area, if you type something into the questions we'll be monitoring it the whole time. I just also wanted to say that I strongly agree with that more agile like way of doing things, that iterative process that includes evaluation again and again can have really positive outcomes, especially when user testing is part of that. Great. All right, let's dive into a case study from the Florida Justice Technology Center, the Florida Name Change website. So Brandon, take it away. Great, thanks Laura. So before we kind of dive into this case study and look at the metrics and the reportings we developed, I wanted to kind of let you guys all know what Florida Name Change is. So Florida Name Change is a website that we developed to help Floridians illegally update their name and gender marker. And basically what we did when we were looking at developing this website is the transgender community was our cornerstone kind of our target user for the site because we figured that if we build something that can change both your name and your gender marker and if it worked well for them, we can also get the general name change users as well. So if we build for that one audience, we can catch everybody else as well because it kind of meets their needs also. And what this website basically does is it helps people generate legal forms, PDF forms that we email to them. And we do it for the Florida Name Change petition for the birth certificate updates for Florida and also for federal social security and U.S. passport documents. And as we said earlier, the question, why did we focus on the transgender community? One of the things we found in our research is there's over 100,000 transgender individuals that live in Florida. But also 68,000 of these individuals have never updated their identification documents. And that's basically 68% of them. So there's a large community there to work with. But also one of the things that kind of really spoke to us on this as well is the transgender community, there's a lot of vulnerability there. There are risks for discrimination and violence because if their IDs and their documentation show a gender that they do not identify as, there's a lot of people out there that have questions about that. So this really is also helping them solve a problem that keeps discrimination and violence from affecting someone's life. And that's kind of why we focused on them for this project. Okay, and then, so before we kind of dive in, I wanted to kind of give you guys also kind of just a quick look of kind of what our technology looks like on Florida name change. So what we've kind of got here is if you guys look at the left side of the screen, is we've kind of a screenshot of what one of our landing pages looks like for the site. Like in this particular case, it's the disclosure for a non-lawyer legal form for the name change position. And so kind of what this site does is it kind of puts everything into kind of a bite-size kind of info and instructional page for each form, and we kind of lead you, you know, one page at a time through the process. So it's very easy to understand. And then what you guys will see on the right hand side of the screen is, you know, once you click that fill out form on the left hand side, we actually are using a technology called type form, where we'll actually, you know, load up and bring you into another once again, you know, bite-sized, autofill, you know, form process. So it'll start them, you know, at the beginning of the form, and then if you follow the arrow to the bottom on the right side, you'll see we kind of will lead them through a question-by-question process of filling out their information. And then at the end of the day, we will send them an email with that PDF form, and they've got all their legal documents filled out from the site. So kind of with a kind of better understanding and visual there on how the site works, now we'll kind of start going into the reporting and how we're kind of measuring the effectiveness of it. Okay, so when we were developing, looking at developing reporting for Florida name change, we kind of asked ourselves some of these questions. And you'll kind of see in my standard items questions here, you know, these are the things that everybody wants to know, right? Well, how many visitors are coming to my site? How often are these visitors coming back to my site? You know, where are they coming to me from? Are they coming from Google? Are they coming from Facebook? Or are they coming from other referring sites? You know, where are we getting users? And then we're also wanting to know about, right, their demographics, their gender, their age group, those basic things. And then also, you know, are they coming to us from a desktop? Or are they coming from a mobile device? You know, kind of what technology are we building for? And then also at the bottom here, though, there was some enhanced items that we also wanted to know. And these are really the items, you know, where the top items kind of tell us our popularity, right? And kind of if we're drawing people, these bottom items actually tell us are we converting them? Are we actually doing some good? Those are some of the key things we really wanted to know. And so kind of, for example, on the first one, you know, how popular is each page on the site? You know, and are visitors actually interacting with them? You know, do we see them scrolling and, you know, looking at content? And if they're going from one page to the next, are they dropping off? You know, are we losing them somewhere? So those type of things were very important to us to know also. And then the second question on the enhanced items, you know, how many legal form packets are being completed by visitors? And that kind of goes for us to the heart of the site, right? So not only are we attracting X number of people to the site, are they actually completing paperwork, you know, that would actually let them file their legal documents, you know, and change their name or gender. So that question kind of goes to the heart of the site for us. And then kind of on the third item here, one of the things we wanted to know is how effective are our interview processes for each automated form? You know, are not only are we drawing people to the form, you know, are we getting people through them? And are there any actual, you know, issues there that we can improve upon? And so those are definitely all the things that we kind of asked ourselves going into this. And one of the things I'll kind of add for you guys as well too, is when we were developing out the reporting technology for Florida name change, we were actually kind of doing the same thing for our organization as a whole. So this slide here is kind of hopefully interesting to some of you guys, if you're looking to build reporting technology in a platform, this is kind of some of the technology we're using. So kind of starting on the left-hand side, you'll see we're using Google Analytics. We've got Google Analytics piped into all of our site. It's definitely the fact those standard, you know, for knowing what's going on with your site. We're using it with tag managers. So one piece of code goes on our site, and that's all we have to do with it. It's nice and easy. And then kind of looking at these three middle boxes here from Analytics Edge, the sheet go, is then what we're doing is we're actually using Analytics Edge, which is a small Excel plugin, to go against, you go out there and fetch information from Google Analytics on all of our sites on a daily basis. And then it's pulling that down for us, so we can actually consume it into a database. In our particular case, we're using Microsoft's SQL, you know, as your database. And my background, and I originally came from the corporate world, so I worked with SQL server a lot, so that was kind of the reason we chose that. But you know, you could use BigQuery, you could use Cloud SQL, you could use a lot of other databases as kind of a middle gathering place for this if you wanted to. And then lastly, we're also then, once we are using as your database to actually put our reports together, we're using a technology on here called sheet go that actually allows us to then take that information out of the Microsoft world and put it back into the Google world, so we can, at the end of the day, we're feeding, as you'll see on the right hand side, we're feeding Google Data Studio as kind of our visualization layer. And the nice thing about that is our organization uses G Suite for emails and a lot of other things. So using Google Data Studio, it's a free visualization platform. It's really easy to use and it's also really easy to administer and share because it just plugs right into what we do for G Suite already. And so I know this is kind of a world-worn overview of that, but the cool thing is we were able to build out a very comprehensive reporting platform for our 9 plus sites and we are basically only spending $280 a year in tools to do that. So I think it's a great example that organizations looking to build comprehensive reporting, you can do it and you can do it with a very realistic budget. Brandon, can you just say a little bit about why you're pulling data out of the Google World just to put it back into the Google World? Are there multiple data sources that you're combining? So the main reason why and actually there are multiple data sources we are combining. We are getting all the data from Google Analytics. We're also getting data from Typeform, which is our form technology we're using. So there are several pieces of data we're putting together, but then also it gives us a lot more flexibility because we're able to create certain views and certain slices of reporting very easily to create custom reports on our own to tell us different things that it is not as easy to do in the Google Analytics world. Great, thank you. Yeah. So the next thing we're going to do here for you guys is we're going to kind of take you through our standard reporting suite that we developed before the name change. And then this is also consistent with our other FJTC sites. So we're putting together a lot of consistent data for our different sites. And we're not going to show you every report that we do, but we're going to kind of cover probably some of the highlights here that are relevant to what we're talking about today. So this first report you guys are looking at, we're looking at the number of weekly visitors that come to the site. This is a fairly standard report. Kind of see how the site's popularity is, how it's growing, or how it's possibly declining over time. And then we're kind of seeing on here that for this particular case, new and returning visitors. This was a very interesting data point for us on this site because we had seven plus forms that fill out in the name change process, and some of them are quite large. So a lot of times we've got users kind of coming and going, and it's kind of good for us to kind of see what that interchanges. And this next report here, I think is a really kind of an interesting one we wanted to share because this kind of goes back to Laura's point about continuous improvement. This report actually lets us look at all the pages on our site. And if you're kind of looking at page views, you can see how many people view each one of those. And you can kind of see where they come in and where they fall off on the entrance and exits. And what this has kind of really allowed us to do is kind of say, okay, if I've got X number of people on one page, and if it falls to a certain amount of people on the next page, I potentially know there's hits in my process. There's something that we need to refine page or content wise that's causing our users to fall off. Okay. And this next slide, this kind of goes to the heart of what Florida name changes for us. This is actually a graph that's showing if you guys look at that name change petition line on the left side. You'll see like for our name change petition, our automated forms, you'll see how many opportunities we had, then also how many people started the process filling out the forms, and how many people finished the process. So this really kind of allows us on all of our form types, like our name change petition, our social security, our U.S. passport, this allows us to kind of monitor and actually make sure that people are using them. And then also we can say at the end of the day, hey, we've at this point helped almost 500 people complete name change packets and we can really kind of figure out what that real world value is to the legal aid world and to the user. And then this last report I wanted to show you guys here, this is basically a report that we built to kind of watch how our forms are actually being utilized and see how they're working. And I know it's a lot of data on here, but you'll see like on this each line on here is a report or a particular automated form that exists for one of our sites. And then kind of you go to that middle section or you can say, okay, here's the number of people that started this form. Here's the number of people that completed this form. And here's kind of that percentage, right? And ideally we'd love to see people, if somebody comes to a form, we'd love to see 90 plus percent of the people completing them. But if it's something significantly lower than that, we can use a report like this to identify issues that are going on with our process and then we can work to continuously approve that as Laura stated earlier. And then kind of the cool thing about this is if we do find the report that we want to drill into and actually look into, we've actually got a similar drill in our subreport that actually will show us the same information on a question-by-question level so we can actually really get in and see what's going on with the form and see where our form is breaking. Okay, and then the last thing that we kind of wanted to talk to you guys about is we're currently working, I know Laura primarily is working on a Drake equation formula that we're doing within Side Florida Justice Technology Center and this is kind of going to be a consistent methodology that we applied to all of our projects. And we'll kind of show you how it's working for Fort and Ancient here in a second. But the idea is that we want to say, okay, we've got a target audience for this site. Let's figure out how many people from that audience can realistically find our site and use it. And then we kind of track them to the process and say, okay, how many people actually came to the site and then how many people used our form. So it's kind of like a funnel. We're looking at our target users and we're trying to figure out how much people we help and how many people action the site. Okay, so this slide here, this is kind of the first slide here for kind of our little Drake equation we did for Fort and Ancient. And in this case, we asked a really simple question. We asked how many people that come to our site complete both a name change and a gender marker update. So this is going to be a certain segment of our site. And then if you kind of look at our chart down here at the bottom, you'll see that we started with, right, 100,000 transgender individuals live in Florida. Then on the next slide we're saying, well, okay, well, how many of those individuals are going to do some type of a transition surgery where we can actually help them change their gender marker and we came to the conclusion, well, that's going to be 25% based on research. And then further we said, okay, the percent of people that actually have had a surgery to transition to another gender, how many people have not changed their documentation yet? And we said, okay, that's 58%. So right off the bat here, we know that we've got about 14,500 people that right now in Florida, that's a potential audience for our site. And then at the bottom we ask some other simple questions like how many of them understand English well enough to use the site 93% and how many of them do we think will actually be able to find our site using a mobile phone or a computer library or something like that. And we came down to we've got about a 13,000 person target audience for the site. And then so kind of this the next piece here there's now that we've said, okay, we've got a 13,000 person target audience, now we're actually going back and we're seeing what's happened on the site and trying to figure out how many of those people actually reaching. So like in this particular case on this grid, we've had 8,200 individuals come to the site since we launched it. 36% of those on the next line are ones that actually have done people work that includes a gender marker change. So then in that particular case we know that we've had about 3,000 potential people right come to the site that actually is our target audience for the scenario. And then kind of looking at the bottom here we also know that of those 3,000 people, 177 people have fully gone through the process of actually doing a gender marker change. And then so this is a really easy thing for us to say that in this particular case 13,000 person target audience we know that 177 of them have actually gone through and completed the paperwork. We know that we've already helped potentially 1.3% of our target audience. And that's something that we can really look at and say yes, we are positively impacting the justice gap. And then this is the last thing I wanted to kind of show you guys to is the last kind of part of our Drake equation formula is trying to figure out and what those positive outcomes are from our site and to track them. And in this particular case this is a work in progress so these are not hard numbers that we're publishing at this point but this is kind of an in progress look to give you guys an idea kind of how we're measuring this. So the first thing we kind of wanted to show is much the potential amounts of hours and money are we actually saving legal aid organizations by being able to help people with a site like this. And what we kind of came to on here is you'll see 1.75 hours and a full case like this will potentially save a legal aid organization. And we're kind of also estimating that we'll save them 70 dollars an hour for that time. And so if you kind of look at the bottom here you'll see that we've potentially saved close to 25,000 dollars with a tool like this and even using kind of in progress estimates that we're still working on. But there's also some social factors here as well. So if you kind of look at those bottom three white lines on our grid 32 percent of the people transgender people are actually denied services or attacked based on their genders not matching in their IDs or people have been asked to leave establishments and things like that. So those are also things that we're tracking and you'll see that we've potentially saved at the bottom there in the blue 64 people from being denied services or attacked we've actually saved 18 people potentially being asked to leave us in establishment by actually helping people complete this process. So even though this is kind of a work in progress these are some very powerful statistics that we're starting to put together for this tool and other tools where we can really tell the story of what this actually looks like and how we're helping the legal gap. And that's pretty much all that I had for you guys on this scenario. I know it was kind of fast and I'm definitely there's a lot of material there and definitely definitely open to open answering questions and kind of revisiting this with anybody that you know wants to talk about it. Terrific, thanks so much Brandon and just a bit of mention for the Drake equation so that's a national effort that we are working with in combination with a bunch of both legal service organizations and research providers including Q and Rand and folks like that and there will be relatively soon within the next month or two some kind of more outcomes as to what it is and how you might be able to use it for things like this to kind of an early look as we experiment with what it's useful for but for those who are intrigued by this sneak peek there will be more coming soon. Any questions or thoughts for Brandon before we look at where we go to that case Eddie. Thank you so much Brandon that was so very interesting. Zach, certainly can we give Zach the control? Zach should have control. Okay. Is that coming up does everyone see the Illinois Armed Forces Legal Aid Network? Okay, great. So just to give you a bit of an overview of what this all is the Illinois Armed Forces Legal Aid Network was created by a state law that authorized a filing fee that's $2 that funds the project which works out to about $1.4 million a year on all civil filings. It created an oversight council appointed by a variety of state and legal aid agencies which I staff and it kind of sketched out the idea of what we were going to build which was a statewide hotline and network of providers to assist veterans active duty military national garden reserves with their civil legal problems. So what you're looking at now is our website. And so we were tasked with this mission as the Illinois Equal Justice Foundation and the first thing we thought we would do would be to figure out what the heck is going on. We did not know much about this so we conducted a needs assessment and this is part of evaluation and baking metrics because in addition to asking what's going on we also asked what's important, what's worth measuring why are we measuring it, what are you already measuring and who are constituencies. So we talked to hundreds of people across the state, organizations that were and were not working with veterans, organizations that were purposefully working with veterans and organizations just happened to be. We thought about our constituency this came from the state so we wanted to be mindful of our state legislatures. So we knew early on that something we want to keep track of for example would be where clients were coming from or the different legislative districts so we could show impact in those districts and make a case for continued funding. In the course of our needs assessment we also learned about the kinds of legal problems we expected to see and that also informed the kinds of questions and monitoring and evaluation you want to do. For example, we found out early on that we were likely to get a lot of requests for VA benefits, appeals and discharge upgrades which we do and discharge upgrade means you've exited the military and you have a characterization of service, most people get what's called honorable then there's dishonorable at the bottom and in the middle are a whole bunch of other things and depending on which one of those you get you may or may not be eligible for services. So we looked at those two that we were probably going to be handling a fair number of cases in those areas of law and we started thinking about what it meant to be doing that work. For example, they take years sometimes to complete so we want to know output and outcomes but we also want to know case load and capacity so we started wanting to keep track of how many of these cases are people on our network actively building because it takes a lot of time and effort, how many have they actually filed how many have they actually won of course and for some of these cases when you win you get a benefit you get cash so we want to keep track of that how much money do we earn for our clients or secure for our clients but with some of these cases you can actually lose but still win a benefit in the sense that you might not get money but you might get access to better healthcare so learning about this through the needs assessment process we realized we should keep track of those sorts of outcomes as well. So key lesson number one that we've learned was kind of thinking about this stuff early on we also had that oversight council and they made the great point that we should keep track of whether or not there is a nexus between the legal problem that's presenting and the military service because we're focused on that population. So we got a project we have a sketched out idea from the legislature and we've done a needs assessment but the next thing to do would be to get our partners members and affiliates together and try and figure out how we're going to standardize the data collection sort of setting the ground rules. You can see here the members who've received funding and then affiliates. We've actually had three organizations asked to join. Don't even ask for money they're just so excited about what we're doing which is a pretty cool thing. To give you the lay of the land we started out with nine organizations. Now we're at 13. Three of those organizations are funded by the legal services corporation so they come with those sort of restrictions and different requirements. Others are not legal services corporations funded. Some of them six I think use legal server as their case management tool. Three don't. Our hotline which is Carples uses Salesforce as its primary case management system and then a couple of our law school clinics actually use a product called Smokeball which nobody else used. So knowing all that we had to think how on earth are we going to standardize this and make it so that we're gathering relevant useful data in a standardized way. What you're looking at now is our basic intake document and we created this as a group we got together over several meetings and we really talked through what are we already collecting, what do we think we should be collecting, given the needs assessment and the experiences in the room, what would be worth collecting and we try to make it as minimal of an intrusion into the normal procedures as possible. So we had to add some stuff. We added things called military service to your basic intake and we set this as a floor. If an LSC funded program needs to collect more information they're allowed to go over but at a minimum they needed to collect this that we knew what was going on. So we have military specific questions to be answered and we also have an AFFLAN ID and that's a new tool that we created to help us track movement through our system. It's a unique identification number that we can use to deduplicate results and also look at how people move through our providers. So that took a lot of work and I'm not going to lie to you, we're not perfect. We're still working on training staff we're still working on getting everyone on the same page still working on the best methods to improve the ways in which people use their case management system to collect this information. But it's doing a lot. And when you get stuff like branches service early on in this process as Laura was mentioning you can start looking at trends earlier on as well. So when I crunch these numbers I can see in real time and I'll show you that in a minute that we're not reaching as many national guardsmen as I think we could be. So that's a nice way to see really quickly on what the value of the standard as intake is. In addition to talking intake and sort of basic data collection at the front end, which is sort of the start we also talked early on about the back end which are outcomes. What you're looking at now is the application for funding to be part of this project. So it's the actual application it's not the back end stuff it's before you even become part of the network. There's a fair warning here and it says you don't have to fill these out now but a year from now we're going to ask you to tell us what you learned. And these were also a product of collaborative conversations across all the different organizations that would likely be a part of the project. We really wanted to say what are you already doing what are you not doing what would be too much of a pain in the butt to do what's reasonable given the constraints. And this an example here that I mentioned earlier would be VA benefits. So I said that it takes a long time so we can keep track of how many are being investigated how many have been filed how many are at various stages in the appeals process and then we keep track of benefits that have been gained from a previously denied claim increased benefits and as I mentioned before you can get non-economic benefits so if you didn't necessarily win but you now get medical care that's something we want to keep track of. So this outcome conversation happened early on as well and it really allowed us to up everyone's game a couple programs did things like making financial outcome fields mandatory in their outcome measurements and legal server. So even if the answer is zero they still have to put something in and that greatly increased their capture of the economic impact that their work has. Others pre-formatted things so that they had a primary outcome and then optional secondary outcomes instead of having five primary outcomes that created a confusing report. So what you're going to see now is the kind of middle part of this process. We've talked about the intake we've talked a little bit about the outcomes and what you're looking at now is a platform, a portal built on the Salesforce platform and what this is is where our hotline lives. So when you call our hotline right now about 65% of the time an attorney who's staffing that hotline solves your problem completely over the phone. They provide brief advice, they provide assistance filling out forms, they have provide self-help materials and directions and they run down documents from the court all kinds of stuff. So 65% of the time it's done in one and done right there at the phone. About five minutes to talk to a lawyer right now given our current wait times and that's with about 800 calls a month that's inclusive of recurring calls. So in terms of that quick data again what you're looking at now is each program in the network and their accept or reject rate. So I can take a quick scan and see right away who's not accepting a lot of cases and who is accepting a lot of cases and who's an aberration given the kind of general status in the network. That allows us to get some pretty cool reports. So that data that standard basic intake comes through here, sorry hit the wrong button. And each member of the network has access to this and so they're able to take a look at their program itself which is what you're seeing load now. So on the left hand side is the program itself and on the right hand side is the region of the network in which they work. So this case Cook County. So CDLS, Chicago Volunteer Legal Services they can look on the right hand side and say okay almost 1200 cases in Cook and on the left hand side and they say we've received 161 referrals. Drilling down further I don't want to go through all this in too detail it's a lot of information taken but you can also see for example that in Cook County family law is about 20% of the cases that the network's receiving but for this program for CDLS it's nearly 45% so you can see right away that they're receiving a whole lot of the family law cases as a percentage of the case load and I'm going to go to a PDF, I'm sorry a PowerPoint now this is because what you're looking at is my version of that platform and since I'm not a service writer I can't receive referrals but this is what it looks like to receive referrals and this is again a moment where we standardize and without a whole lot of need to change standard procedures created a network based model that allows us to really analyze the data so these codes here are for when you receive a referral you've accepted a referral and you're closing out the case and those are standardized across funders in Illinois so everyone's familiar with them and they understand what they mean you know that's so you've closed it out, you've provided services, what level of service you weren't able to provide a complete service because the client went through or something else happens there's codes for that similarly we've denied service you know give a reason for that and all that gets fed into this analytics program and allows us to really see what's going on nearly in real time but also in a way that provides opportunities to look for trends and other issues so give this just a second as the analytics loads there we go so you're looking at now is anything that came from the hotline into a provider so you see 36% referral rate you can see the problem codes and categories here in the middle big bucket, middle bucket, little bucket veterans benefits is our biggest case and biggest bucket of that they're looking for an upgrade in military discharge so this allows us to really clearly look through right away and see what's going on with the network in the middle here I can see who didn't accept a referral so I can click on that and get a live view that shows me right away whether or not referrals are being accepted or rejected and we're now sorting just for rejects and I can see that through mostly veterans benefits second most are in divorce so looking down I can see which provider rejected those veterans benefits or divorce cases and why those codes that were part of the referral process I showed you in the PowerPoint so we got conflict of interest, insufficient merit and when you sort for insufficient merit drilling down even further and now it becomes clear that the reasons those rejections are coming through and are being marked insufficient merit and they're veterans benefits it's most likely or not people who really aren't eligible for an upgrade but they're desperate and they're given it a shot anyways and so one of the things the network provides is the peace of mind and the close out sort of understanding you've been running in circles for a long time you've heard of this mythical discharge upgrade it doesn't apply to you and here's why so that's another kind of outcome that we can track whether it doesn't feel like a traditional legal aid outcome but knowing that there's the peace of mind factor there is really important that was a lot of information very quickly to sort of recap and summarize I guess what I'd say is we really tried to think about what we wanted to collect and why what would be burdensome onerous or not burdensome on people how could we do that in a way that crosses different platforms and methodologies that are already in place and then how can we use a little bit of new technology which is the sales force platform to sort of tie it together and create a balanced product that isn't driving people nuts to be a part of and is actually providing them with valuable insight about how their network and the programs themselves work so that ill aft land in a nutshell I was hopeful we'd get some questions so I'll stop talking there and keep my fingers crossed I actually have at least one question for you Zach I'm curious as we're looking at this it's clear that there is just an enormous amount of thought and functional design that went into deciding which metrics to show and how to show them can you talk a little bit about the process as to kind of how you did that and how long that took and what kind of was that a hugely substantial lift is it beyond the realm of many organizations or not so much? Well it's either a good thing or a bad thing depending on how you look at it we were forced to use a variety of platforms and be aware of a variety of financial constraints so when you look at the sales force thing what was lucky here was that the hotline was already using it and so it was a minimal cost to add on to this and in terms of what we could add to this data reporting suite this visualization we're really just limited by screen real estate we can keep adding areas as long as we're collecting them and we're collecting them via that standardized intake and closeout codes so this can keep growing and growing on our end as the foundation as the funder we talked about some of the outcomes we put into the application and the eventual final reporting so people are being made aware of this early on so they can build that into legal server or in some cases keep tracking a separate Excel document so to sort of answer your question we focused on what we thought was important and we're learning we're learning because we're only a year into this we really just started our second year and as we move through this we've become aware of different aspects of this that really have an impact for example I'm looking at our senior section here we talked through people in the program about halfway through I said you know I'm doing a whole lot of this work and it's really not getting captured by what you're doing and given that almost everyone was already capturing that in some way it didn't really add a lot of inconvenience to anyone to add that into our existing collection of data to sort of drive home the fact that we turned out we were serving a lot of seniors a lot of Vietnam era veterans so adding that in wasn't a big deal and we did it again in consultation fantastic other questions for Zach I will save my other questions for our panel so your your data kind of dashboard that you have there how much of that is just pulled from Salesforce and how much of that is pulled from external sources like the legal server or other data sources that is a great question I'll show you the warts and all to this so what I'm going to pull up now is when you take everything from all sources and combine it and we are still working on getting our standardization down pat actually let me start with this sorry so this is a case management report when we started this out we envisioned it as being a hotline centric driven kind of operation it turns out that a lot of veterans like going to the veterans administration for their healthcare and other reasons and so we have medical legal partnerships at a few sources through us through the hotline and about 25% or so come to us as direct walk-ins so to manage that I showed you that standardized basic intake and then we created this Excel document which is a template which everyone puts all of their information and using these standardized values and then we feed that back into Salesforce we're working very hard at the moment to come up with some automated fixes for this because it is a little bit cumbersome but to again meet people halfway you'll see some things where the first list of choices like employment status those are the choices so everyone uses those and we have a standardized list but for some other things we said use your case management system codes it's not worth trying to dig in it's easy enough to convert the biggest and most challenging example of that is legal problem codes when someone closes out a case they use their own case management system code and we on the back end created a mapping tool to roughly calculate whose problem codes correlate with whose and then what that gives you then here is a full network report and so it's not quite as good as it could be yet because we just finished our first year and some people didn't quite use the templates right or there were minor errors and we're still cleaning the data but to answer your larger question we're pulling it in from everywhere with an eye towards eventually and not too long in the future really automating the process by which this information moves in there's a lot of opportunity with Salesforce with APIs we've been trying some stuff with legal server solutions there and then for some of those non-legal server, non-sales force providers of case management software trying to figure out if there's an API opportunity there as well Excellent. Fantastic. If you want to send control over to me I will talk a little bit about what we are doing now No problem you should be should have the option to be presented now Fantastic Great. Talk a little bit about the new Ohio legal help so we've been talking about it in this way because there is actually a site if you go to the Ohiolegalhelp.org but we are only somewhat affiliated with it so the project that I am working with I am working with actually it would be an important thing to put on the slide this site is being built by the Ohio legal assistance foundation which is the IOTA founder in Ohio so we are building something that is completely new will be a new statewide website we are in the midst of we are finishing design and we are beginning the development for a launch around the quarter of 2018 it's a pretty ambitious site and that's the timeline so we have been using all the way along the path a lot of data and research to create a something that is very user centered very mobile centric and we also really had an eye towards how we will gather data for future decision maker so I thought it might be useful for you guys to show this in process so rather than a fancy dashboard what I have sorry not to denigrate anybody else extremely fancy dashboard without really a reason so what we have essentially a set of requirements is what we are thinking through as we are so we have been saying you should think about this in the design phase as to what you should build in well this is what we are thinking about in the design phase as to what we should build in and in fact it's not yet finalized as to what we are building it's a set of requirements rather than a set of finalized features so in general just understanding what we are looking to track the site is the general model of the site is probably fairly familiar to those of you who are familiar with things like Illinois Legal Aid Online or Michigan Legal Health it is very what we are looking at here is a topic page and we get a little better look at it on the next page it is a very centralized around the idea of having about at launch probably about 200 different topic pages like get your security deposit back might be one and there are different sections that you can expand which with understanding the basics I'll show you the extra slides understanding the basics, letters informed, legal health and lawyers etc. so the whole site is kind of centered around this with the idea that there also is obviously a way for people to navigate to them but our assumption is that we'll be going directly here from both Google and we're going to be providing direct short links to appropriate topic pages to providers so this would be a slightly weird one but I could imagine like for instance we might have a topic page for veteran benefits or specific sub details of that and that might be something that we would be providing to veterans groups to get folks that they're working with straight there as opposed to making them navigate the home page so that's what the site is so in general as we're thinking about this we're thinking through alright what do we want to track we want to track who has visited the page so who's found it at all the percentage took the key action who took key actions that imply some kind of engagement while they were there this is really tricky on a website to try to understand whether somebody actually read or did anything on a page thinking through who took a next action who did anything that was useful after looking at it people who found it helpful is a little bit iffy in terms of metric it's actually unclear as to whether it'll make the cut to go in the site to do kind of a thumbs up thumbs down but we're somewhat interested in experimenting to see how useful it is and percentage took actual action down the road and for all of these things to try to find the right balance between everything that might possibly be useful which is I think all too often we kind of throw Google Analytics on a website and say alright well we'll figure out later what we want to track that boat it has a couple of downsides one, not everything is then feasible that you might want to do because of the way that you've implemented like there might be some ways that might be things that are easier to do up front that are then a pain down the road because you just threw it on without thinking about your metrics you then also have just a ton of raw statistics that may not really be very feasible to use so thinking through alright what's feasible to build but also we're not going to have very many staff we're going to have limited resources to actually look at this data so there's no point having just overwhelming quantities of data about everything on the site because we won't be able to take action on that so to think through alright what's this you know overlapping thing of everything in the universe but it's also feasible to build and feasible to use so what has this resulted as we have gone through this exercise so for the topic page itself so the number of visitors for each topic page is a fairly straightforward one that makes sense the referral path sorry this is all fairly jargonate because it was an internal document that I kind of captured here the referral path is where the visitor has come from when they get on their way to get to this topic page the quiz is a specific functionality on our site external search engine something else bounce rate so this is the number of people who come to the page and then very immediately read the page very quickly read the page so that's often a good very kind of reasonable not a great measure but a decent measure of people who for whom the page is not at all what they were looking for so it's basically doesn't really tell you whether it was useful for what it was intended for it tells you that it was not what they had intended to do the design of our site has a really big metrics benefit which is we have each of these there's in fact four what we're calling drawers with expandable headers so there's this understanding the basics obviously this is what you're looking at is not clearly a live text with Latin it is a design and this will be a small blurb to help you know what you might want to know you can click read more to read page less than the page to help you understand this topic or you can click read more to see letters or forms you can click read more to get legal help lawyers it has a huge benefit for metrics because we can track how many people click to read more so we can know who did some engaging with the page another one and I don't know whether this will be feasible this is not a trivial one to do through Google Analytics I'm sorry I skipped one but actually neither of these are trivial for Google Analytics we're interested in anybody who uses any of these buttons so who texts themselves or email lists or links this or sends themselves a link or people who click through on any content that isn't navigation so they click on a link understanding the basics click on a form or a lawyer percent to say it's helpful thumbs up or thumbs down and we are very hopeful to implement a SMS follow up to be able to capture in a key place in the site whether or not they're willing to tell us what happened down the road and to capture a phone number to be able to send them an SMS survey like two or three weeks after remain to be seeing whether that will be in page one but that is so those of you who are paying careful attention will notice that my list of things we're looking to track looks suspiciously similar to what we talked about for the Drake equation with name change it's not accidental that we actually took the Drake equation as kind of a model for how we might do that and the SMS is one of the only things we could come up with as a way to know what actually happens in the world after they are on the site so just two more pages for the homepage so here's the homepage for the site which is again we're not thinking of as nearly as critical a page on the site as the topic page and also honestly there's not that much on it there's not that much here other than simply this list of categories so the number of visitors each of these starts what we're calling a mini quiz so when you click on money taxes and debt it goes into a process of asking you some questions to try to determine what topic page you should go to and collect if you're willing to share information about your income and geography and balance rate so people will comment then to me and leave we're also interested in just the standard Google and analytics stuff like for instance where Google thinks they are so the geolocating new versus repeat so some of the things that Brandon showed you for the name change site and then forms so like we talked about for name change forms are a great opportunity to actually understand more for a couple of reasons one you have things like people who start and people who finish so you can actually know whether they've engaged with a form because they're actually moving through a form potentially again like the other way people say it's helpful to send to people click on any content link it's actually very different kind of form and here an SMS follow up might be particularly appropriate and useful because they're filling out information we can understand or we can collect from them potentially a phone number there's a good place to ask if we can follow up and collect their phone number so and maybe there's more than one to say they filed the form say they got a good outcome afterwards alright so that is Ohio legal health in all its glory or at least our requirements for the metrics of it are there any questions specific to Ohio legal help before we this would be also a great time to ask by a chat any questions to put to kind of this whole group potentially including myself in a kind of a panel discussion any questions on Ohio before I kick us off with some kind of overall questions maybe I what platform are you building Ohio legal health on off of Drupal okay and is it a custom Drupal or the D-Law template it's a custom Drupal why did you guys end up making that choice so a more complicated site could easily be supported on any of the templated site Drupal felt like it certainly was complicated it certainly was sophisticated enough to support our needs and it feels like there's a fair critical mass of developers in the legal aid space and outside the legal aid space in fact we have a developer who's not been doing a lot of work in legal aid there's a critical mass of developers who can help with it so it felt like something that we could get to work with excellent thank you alright so first question and maybe I'll start with you Zach so if you think about an organization with you know as we all do a pretty limited time in budget who is currently in kind of a project based build and then evaluate mode where might you suggest they start what might be small scouts to try to move to this more bacon metric model Zach what do you think I like to start with people working on the front lines like intake staff paralegals people like that they see trends and they can tell you trends that you're not seeing because it's not necessarily being reported already but it can really inform what you might want to think about more systematically measuring or what you're doing well but you should be formalizing because your program has a sense of it but you couldn't at the end of a year or something run a report and tell a funder this is what's happened so I think in terms of just an easy quick fix talking to some of the people we don't always necessarily talk to but who actually have quite a bit of a kind of bird's eye view if you will of how a project's working and so basically thinking of that as your kind of interim cycle so basically to say alright after we're going to roll something out and in two months we're going to ask the people on the front lines how it's working that's right yeah and we did that with Illinois Armed Forces Legal Aid Network we launched this portal that as you say is very fancy looking but it wasn't perfect right out of the gate right I mean we solicited and heard quite a bit of feedback and some of that was just basic you know website design and improvements around user interface and things like that but some of it was hey we talked long and hard about these closing codes for a case or these incomplete codes when we don't finish providing services to a client but now that we've done this for two months we're really missing this people for example we were capturing information about their discharge status and that's important for a variety of reasons but people found out that in practice you could tell that someone this is a bit of a double negative did not have a dishonorable discharge so because of the services they were getting you knew they weren't prohibited from them because they didn't have a dishonorable discharge but you didn't know what other characterization service they had because it was a family member calling on their behalf or their papers had long been lost but you knew they had at least above that level because they were getting some other services and so now we're capturing that in a way that's meaningful and they might otherwise accidentally get lumped in the wrong category. Yep, absolutely, great and Brandon, do you have any thoughts as to kind of what you would advise for an organization who's trying to think through just kind of a place to start? Yeah, I mean honestly I think we've kind of covered a lot of it. I do like the idea that if you're working with a project and you're gathering a requirement I think you do need to be as intentional about the reporting requirements as you are about everything else. I mean it really is just as important as your point earlier on it really shouldn't be an afterthought. You really should plan for this all throughout the process and measure whenever you can. One of the things that I kind of like coming from a kind of background in the corporate world in my previous life is I do like the idea of a lot of times you kind of have to force yourself to be intentional about baking things in where you act in it. I've always liked the idea of potentially using triggers or things like that or highlights and reports or you're saying that if something goes above or below a certain threshold that we find to be non-acceptable that it triggers to somebody and says hey you don't have to just go through reporting to figure these things out I'm going to bring it to your attention. And I do think for some small organizations those type of triggers or thresholds or highlighting lines can be very important to actually be able to proactively action a lot of these insights the type of reports and metrics will show them. Can I just say I like that a lot and I think you're completely right in my work as a funder we work with organizations that provide just legal information and kind of like know your right seminars for example and these are often small organizations focused on a very specific population and they have very small budgets. So if you look for those triggers as you put it we sometimes suggest just give people a quick one to five on the way out of the door from a know your right seminar and say like you know how much did you learn one to five it's not going to tell you a lot but if someone's at a one consistently one of your trainers everyone's hovering at two and a half and everybody else is at a three and a half you know something's gone wrong so at the very least it's a trigger and it costs next to nothing it's just those hash marks on a sheet of paper or print out a couple pages with one to five on it and have people circle it but as a broad sort of aberration noticing tool it really does a lot nice yeah and I would add on to that it seems this idea of a trigger sounds also very useful for the realm of deciding what to track and basically thinking through alright we're going to start out by saying alright we're going to have three data points that we're going to start by measuring for this particular project and to say alright what would be the data points that would trigger action you know what would be things that if we fell below or if that went this went over you know why would be things that would make us decide something or do something differently because logically it doesn't a collecting data about things that you have no intention to do anything about is mostly a waste of time they might invite you should know what you're looking to do with it and so basically starting by what is most important for us to do something about might be a really interesting way to start the question for the panel so as you think through I feel like for a lot of the organizations who have trouble transitioning to this kind of thinking some of the fundamental issue is trying to convince people who are have a somewhat less than rosy view of the value of data to begin with to basically you know there are potentially folks on the front lines who say you know no day they can capture the value of what I do every day all the way up to an executive director who says data is the necessary evil it's not useful to me it's just something that I'm collecting for a funder do you have any arguments that you found compelling in this particular realm to basically say you know here's a good reason why data is not just a necessary evil but is something that is very useful for you either one of you want to jump in? Sure yeah I mean speaking as a funder we we've you saw some of our outcome collection in that application page and we're starting to focus in on that pretty heavily and so I don't think it's a necessary evil but we're making it clear to people that you're living in an age where this stuff is becoming more expected and more standard so we look at our projects and our funded organizations in comparison to their peers you know we have limited funds and so in a world where it's more possible to collect this stuff it tells us a lot about relative performance across different projects and one organization in one funding category versus another organization in another funding category it may not be exactly apples to apples but it's pretty close and you know we've used that to show them the value of collecting this so they can be competitive for grants we also receive our money from the state of Illinois you know we're the state's commitment to legal aid funding so we take that data back to the state legislature and we show the impact that it has and we show that the local legislators from the areas where those legal aid organizations work the impact of their work and that really does a lot because they'll hear anecdotal things about you know their staff at the local office gets a phone call from someone who needs help with the legal problem and you know they just transferred them over to local LSE funded program they couldn't do it because of this and so they kind of have you know what they have in their head they're hearing from their local staff we can show them a customized report built to their general area showing them the impact that these programs are having with financial data and things like that and that's another one where we found that some organizations weren't really keen on that financial outcome measurement they thought it was a kind of a pain in the butt but when they look it in the aggregate the attorneys were actually really proud because no one had ever given them that snapshot before where you say look you guys in this housing practice group when you add up the value of all the homes you save from foreclosure or what have you it's in the millions of dollars that's an impressive feature of what you're working on so that kind of stuff really can be inspiring too I think. Absolutely. Yeah and I would love that so certainly data is really critical to communicate impact for funding purposes so I'm not sure any of us would argue with that but I would age to leave an impression that that is the primary reason for it because I feel like there is also a really so we have an important mandate to be able to improve what we are doing over time and whether that and that doesn't necessarily mean quantitative data maybe that means interview data or I'm actually a specialist in qualitative data so things like user testing or interviews so it doesn't necessarily mean gashboards and metrics but thinking through I kind of feel like for some of the more naysayer data people to kind of I've worked with in the past to try to think through what are meetings that you have that's where you make decisions where things might be informed by having actionable data what is that data and that's one or two things because it then starts to show its own work that once you are sitting around wondering how something is going and you actually have some data to be able to inform that conversation it can become a bit addictive so that can be a useful way to go and I would say too Laura coming from an organization previously I came from an organization that kind of looked backwards most of the time originally where we did something and then we tried to assess what was going on and I think experience with the data is honestly what really wins the way the naysayer is over over time once you can keep saying hey we made this decision in the past now we analyze it that we've seen some potential other ways we could have gone that would have been better so I think you can slowly win an organization over looking backwards and then kind of proving out alternate past and you can slowly get those people to turn to be more proactive over time could you look a little bit more about how you use this data in a larger communication strategy either externally with stakeholders or internally with individuals that you're funding like how is what are some of the real use points that you have for it Zach do you want to start with that sure as I was sort of saying before we do really let organizations see how they lay in a larger legal aid landscape so in terms of cost per service and cost per person serve and things like that you can show them when a program running a similar project is much more cost effective than them then you can start talking about why we've done stuff with programs where they really wanted to focus on a volunteer based project and when you look at it after the fact they served a good number of people but at a greater cost and more complexity than maybe if they just had more staff attorney time or resources devoted to it so we use it sort of to obviously to lobby for more funding from the state we go to state legislators and talk about all the great impacts but we also use it to refine some programming around the edges and sometimes at the core of the program where the model they've chosen isn't really the most effective one given the other options out there so that's where we answer that's where we answer your question yes and Brandon what about for Florida name change how are you guys using that data I mean we're definitely using it we're kind of still finding our methodology and kind of what we're doing with it but I think it's very useful from a communications standpoint a lot of things we're tracking to be able to say hey you know this site is like I said before this site has helped us stage 64 people from discrimination or you know from being expelled from a place you know there's a lot of those I think those social those quality of life aspects that once we kind of bake those type of things into the metrics you know those those things are updated on a daily basis and the people that are doing the communications for us you know with supporters or with stakeholders or other media outlets can then readily go to those type of statistics and really say hey this site is really doing some good things and you know these are the things that are doing and hey by the way here's the numbers in real time you know we're not just saying we're doing this we're we're providing you updates with it you know this is what's actually going on I think that's a really good point and I'd say that for a new project it was useful for us to be able to relatively quickly be able to get some good data up and out there to prove that we weren't just some fly-by-night who the hell are these guys who sent you kind of organization particularly given the population we're working with where there are a lot of you know less than reputable nonprofits they're trying to capitalize on an interest for veterans and things like that so coming out of nowhere fresh and using that data to really bolster our reputation and demonstrate the quality of the work was really important for us for getting trust in the community I guess as a follow-up what is the for each of you what was an example of one of the most surprising pieces of data that you got from collecting us this that you hadn't expected I think with us from like from from a Florida name change perspective it's honestly just been eye-opening for us to actually see the impact the site was having one of the things we looked at up front as I think there's normally like 6100 name changes that take place in Florida in a year and we saw that the transgender community was like 0.7% of the estimated population so we were thinking okay you know this maybe this site will do 50 name changes a year or something like that but we're helping a very vulnerable population was actually been kind of shocking an eye-opening to us with a lot of the metrics we've been looking at is I think that that community could potentially be a lot smaller than estimated because we're actually at the point now where we've since we've launched in March we've been nearly 500 name change packets so I mean there were just a lot of there's a lot of little thing that's a little thing but there's a lot of things honestly in our metrics that have actually been quite eye-opening to be monitoring those things as closely and I'll actually add so I don't have any for Ohio but I hope to do the drink analysis of name change because I'm also doing work with Florida so I'll just add in that multiplying through the even what is not very high numbers by the reports given by the percentages given by research on like the amount of the number of people attacked because they had an incongruous ID or denied service because their ID didn't match it really it just provided a level of insight on those numbers that I hadn't really thought about before that like you know these folks are every time they're presenting an ID like they you know they go to the grocery store and try to buy beer and it's you know gauntlet of like I'm presenting this IP for a man but I look like a woman so just kind of really brought that home to me so the idea of bringing it all the way through to those kind of those would be some of the core metrics that really are important to that type of analysis and really brought home for me I agree we I didn't show you our call volume metric page but we tracked that very carefully and so one of the first things that we weren't sure of is how many calls are we really going to get we were worried we get either completely swamped or have nothing right out of the gate and it started a little slow but it built but what we learned early on is that we done we could tell this because we track how long it takes someone for someone to hang up before they get to talk to a lawyer and we're seeing a relatively quick abandon rate they were they were hanging up really quick and we realized we hadn't done as good a job as we could have really good expectations and and with both the partners that were referring people to that and the way we communicated to our client population so we've done a much better job of expressing the people hey when their phones answered you're going to hear an automated thing but if you hang on it'll tell you what to do and usually in about five minutes you're talking to a lawyer having that out there has really improved our abandon rate dropped quite a bit you know it was first off the gate I think it was like in in the first two minutes or something when someone did abandon they abandoned within the first two minutes now it's up to close to four so it's nearly doubled the amount of time people are waiting and you know it's still a little frustrating I guess but it's pretty quick five minutes or so so that call volume stuff was a surprise for us we also had a law school clinic that's part of the network that has really good reputation with veterans but they were unfortunately because they had such a good reputation they're getting calls from all over about everything and they really don't do everything they do a couple things and they do them very well so they changed their voicemail sorry their phone systems messes around to say hey here are the things we do we'd love to help you with those if you need anything else go ahead and press one and we'll connect you to this hotline and they did that and their call volume that they actually had to handle dropped by two-thirds which meant that their student attorneys and all the people working on all that intake time for essentially clients they couldn't assist was freed up so it was a huge benefit to them and that's again just because we realized right away we need to communicate better who does what and what to expect and when you're calling and all that sort of thing one of the most frustrating things is to get a hotline and have to refer someone when that could have been something that they were listening to on the way in and already moving on to that service that's a great point the client feels better about it it's much quicker for them and then you know those student attorneys now they have hours of their day back where they're not doing essentially triaging improper referrals or less and we've kind of seen some similar things to it with some of the Florida name change reporting one of the things that we were kind of seeing in our data by tracking the page reporting and kind of following people through the flow we were seeing that we were potentially we were losing almost the sixth of our visitors on our disclaimer page and then we saw a case where we actually had a checklist page where we thought hey this is something really useful for people to print out and kind of use a checklist of all of the process and we saw we were losing another six of our visitors on that page so there was just little some very unexpected things that we were kind of seeing in the flow and we're kind of still working to tweak those kind of things out but yeah those type of things can really happen and I'm thinking well this is a great thing for the user experience but you might find that it's actually something that's tripping them up terrific and that's I think a nice note to end on the idea that this kind of idea of continual metric and not continual metrics which sounds scary but the idea of continual improvement can help you to refine things all the time to find things that people are tripping up on as opposed to just kind of one gigantic thing at the end which is hard to use to refine much of anything so fantastic thank you so much for attending this has been really fun thank you so much to Zach and Brandon for joining me it's been really great to hear more of what they're doing and thanks Art for having us thank you so much this was great we will be sharing the video along with slides here in the next week and on our new website please feel free to send us an email if you've got any questions follow-ups there'll be a short survey we're starting to put together our list of topics for next year's webinars for our request for proposals so if this is something that you found valuable please let us know thank you so much have a great afternoon thanks guys thanks everyone thank you