 Great, so let's get underway. Let me just introduce myself for those who don't know me. I am Laura Quinn. I am the Director of Partnerships and Learning for IdealWare, which is a nonprofit organization that provides information about technology for all sorts of folks within the nonprofit realm, including we've been doing a lot of work with the legal aid sector. And I'm excited to have with me John Greiner. John, you want to just introduce yourself? Sure. Hi, everybody. I'm John Greiner. I've worked in legal services for about 20 years, starting as a staff attorney and moving into management and back into technology, initially out of frustration and then ultimately out of sort of seeing all the wonderful opportunities of the collaborations that started years ago with the Department of Commerce grants and then certainly more recently with the law help type initiatives and online intake. Anyway, so my focus today is helping providers with their local enterprise technology and then with collaborations. Fantastic. And we're excited to be talking today about every kind of investment. John and I were just talking a little before the session and we both kind of geek out on return of investment, which is a little kind of a strange thing to geek out on. So we are getting the slides up and running here, but we're going to just kind of start with the obvious, what the first slides cover as to why you might care about this topic. So in general, this is a, it's a really critical thing that I think that people are really stymied by. They're really not sure where to start. They're trying to think through, all right, well, some of the, especially in the legal services area, some of these projects can be really expensive. And it can be hard to figure out, okay, what's the benefit in the short term and the long term? And what's the, you know, obviously the benefit is sometimes like great, we save a lot of money because we don't have to pay phone line expenses. But often it's more like things like there's more justice in the world or, you know, potentially something kind of mediumly measurable, like we're saving staff time, we're potentially serving more clients, things like that. So that stuff can get really tricky. But what we're going to pause it today is a methodology to help you kind of work through the idea of a plausible ROI. So something that isn't necessarily, it's not going to say, all right, the return is $842. But it's going to help you to say, all right, here's the plausible high and low numbers. And here's the plausible low and high numbers on benefit and the plausible low and high numbers for cost. And then you can compare those two. Brian, if you want to try passing to me again, I think I'm ready for that. And in the meantime, John, could you talk just a little bit about kind of how you've seen return on investment used in organizations? What are some of the real kind of core reasons you would want to have this? Sure. So I'd say that I got into return on investments initially when I was working with some vendors that were proposing technology for Indiana Legal Services when I was there. And so part of why I got into it was I really wanted to understand how they got to their numbers. But it was clear that this is a big part of what the private sector was concerned about because the vendors, the manufacturers, they all spent a lot of time thinking about it. And so my initial uses were to justify making transitions. It could be as simple as moving from one phone system to another. And there's this upfront cost of it. And as Laura was saying, the metrics are fairly simple. There's sort of the cost of building and implementing and buying. And then there's that monthly savings of maintenance of your phone service, stuff like that. And I basically had to use that to educate. I was very junior, educate the senior management about some of the various projects that I knew made sense. And frankly, I think at the time I didn't really have a good way to quantify those additional pieces that weren't so numerical, like making intake more user-friendly from folks calling into our offices that they wouldn't stay on hold as long. Like, how do I quantify that? So over time it evolved. But in the initial use it was very much following the private sector, learning how they did the ROI, and then translating that into what the Legal Services community cared about. Fantastic. Great. And I am just getting underway with this. Sorry, my computer is having a little excitement today. So this is a really critical thing. And we're going to be talking about, as I mentioned, both the plausible ROI, and we're going to be talking about kind of making chains of assumptions. So as you look at things that are a little less straightforward than a phone system, as John has said, and we are looking at kind of thinking through, like for instance the staff time, so what the cost savings are of being able to see more clients potentially, we're going to be looking kind of putting chains of logic together. John, can you just talk a little bit while I get underway here as to kind of what that, when you, when we did that together at the LSE session, kind of what that exercise of kind of tying things together was helpful with for you? Absolutely. I mean, I think that, well, so one thing is that, you know, that it's, I relate it to a lot of the outcome measurements that I think a lot of legal services programs have had to grapple with over the last decade, where you're making claims about the outcome for your client based on outputs that you've delivered, and then maybe taking it a step further and trying to quantify the impact on their lives going forward. Saving a family from homelessness isn't just saving the city or state the cost of shelters, but it's saving the society at some level from folks who maybe won't attain the same level of education, won't be able to sort of get to a level of income in their work and profession that they would have, if they would be able to kind of continue through in their housing, staying in a stable environment with a good school. And so in some ways I think it's the same thing with a lot of these projects. You're really, you're trying to, and I, you know, it's uncomfortable, I got to say, but you're trying to come up with an assumption that's plausible. I really love how Laura, you know, kind of uses that term. It makes sense. You don't have to feel 100% confident that there's a ream of data behind it that backs it up, but it's a logical extension. And again, as she gets into sort of that process, you know, you can discount the fact that you're not 100% certain that some of these assumptions are right on the money. And so you, at the end of the day, you come up with a level of confidence that I think your executive directors and litigation directors can appreciate and frankly then, you know, feel comfortable moving forward with your initiatives. Absolutely. Great. And it feels like the slides are up and running. Does that indeed, is that true, Brian? Yes, definitely. They look good. Great. Fantastic. So sorry for the delay there. So, so basically that's the over. That's what we're trying to get to. So how do we actually get there from here? So before we dive into that, I'd love to actually hear from you guys. We're going to try to have some kind of discussion going on in this session. And if it doesn't really work, then I will, I will abandon it. But I'm really interested in you just kind of typing into the chat, one or two projects that you're thinking about right now where you'd be interested in thinking about the return on investment. So basically, we, me and John can think about them, use them as examples and help you to think through that. So what projects are you thinking about right now? I mean, one of the projects that I'm looking at with a nonprofit that I'm working with is kind of a basic rights education campaign that is regarding terms of service and other things like that. Something that gives people an opportunity to know what their claims of action may be in regards to consumer fraud type issues. Yep. Great. All right. And that's a good example. We've also got straightforward examples like phone system. We've got viewer 17 says, I would like to demonstrate a plausible value for legal information webpage. Absolutely. The monetization of free information, that's an inch, I would say quantification rather than monetization implies you're going to make money out of it. But trying to quantify what the value of that is. Absolutely. Perfect. There's some great examples that we can work from. I would add if you want one other one, and this is maybe, again, I think some programs are starting, it's moving from file network file shares to a knowledge management or document management system. I think there are a number of programs that are grappling with that. It's complex. There are parts of it that make it sound very valuable to executive directors and the like, but there are so many unknowns. Absolutely. Absolutely. Great. That's another great one. All right. So let's start by thinking through what is probably the more complex side of the equation. Probably the one that most of us have the left handle on. So basically there's benefits and then there's costs. I think we as technology professionals often have a kind of sense of the cost, but it's the benefits that become really tricky. So let's start there. So there is some fairly straightforward ones that we can start to fairly easily itemize. So there's the obvious one of how much money will it literally save? So things like, all right, well, we don't need to pay for that, like the phone line we just mentioned. You don't need to pay as much for phone lines. You don't need to pay as much for software or hardware because you're moving to something that's cheaper. So that's an easy one. There's staff time saved. So something like a document management system, you're presumably looking to at least eventually save people time, whether that's simply time in finding a document or time in not having to create as many things or time in having more knowledge for the organization in total. So you forget less stuff. And we'll talk in just a couple of minutes about, so this is a fairly straightforward one to try to calculate out comparatively as to how do we quantify staff time saved? We go more and more down what we're going to look at, what we call a spectrum of tangibility. So we go from things that are very tangible like cost saved and staff time saved to things that are somewhat tangible, like more clients served. What's the value of having served more clients? And we'll talk about that. Then there's the idea that you may have served clients better. And what's the value of that better service, which gets pretty interesting. Outcomes achieved. So like homelessness averted. What is the value in dollars of the fact that you have averted homelessness? Things get a little tricky here. And ultimately we get all the way down to the end of the spectrum to say, well, we've increased justice in the world. And down at this end of the spectrum, we need to pretty clearly say, well, we're not going to be able to measure that, you know, we can potentially try to think around things like homelessness averted or literal money saved through data version. But we're not going to be able to say, all right, here's the monetary worth of justice in the world. So that's obviously a short list of what could be a lot of benefits. And this is a really valuable start to any process. So basically just thinking through, all right, let's brainstorm. Let's brainstorm all the possible things that we might be able to either quantify or just is a benefit for a project. John, we've got the example of a legal information web pages. Do you want to just brainstorm a little bit on that specific example as to what some benefits might be, whether quantifiable or not quantifiable? Sure. And I think one sort of maybe direct measurable benefit would be time saved as staff refer clients to or applicants to that page instead of walking them through it or giving them that page as a starting point. And so that they advance their conversation down the ways after they're done reviewing that material. So there's potential for direct clients, staff time savings. And then maybe I think to the extent that you have some web feedback tool set up on your site, you might even get clients to give you some sampling, some data on the information and how they used it. And again, whether they were able to avert homelessness or they maintained a good record of the payment on their rent or something like that. And then you can again calculate out how that would translate into saving the community or that individual resources. So you've got in there the idea as well of more clients served. You can probably more people will see information that is online and your advocates can actually pass on. So you've got the idea that you're serving more people. It can also be potentially a way for advocates in the community, so people at, you know, churches or community centers to find that information to be able to easily serve those folks or more easily be able to serve those folks without your direct help. So that's saving time and reaching more people. So we could probably go on. But the idea is that you're brainstorming and you're coming up with kind of everything, everything that you can think of that is a benefit. Obviously also let me just mention there's the benefit of people having better information. So there is the idea that more people are going to get to justice, you know, information being logically to all of us. Empowering people, yep. Yep, absolutely. And I would just say the other thing is that it might deflect. So it might reduce the number of clients served. It asks in saving time. But so, you know, to the extent that people know more and about whether, you know, they have something where they should contact a lawyer or not, or try to get representation or where to go instead of contacting you and waiting on hold or having that back and forth to get to the point where you recognize you can't serve them directly. It could actually, you know, sort of reduce the number of intakes or reduce the number of direct referrals. Which may end up saving you time on that screening process or something else. So you really want to seriously consider what metric you're looking at because if you're able to allow someone to refer themselves to one of your partner organizations, you've just saved a bunch of time that would have been spent screening and referring. Right. Yeah. Yeah, you've actually got staff savings in two different directions that as we've just brainstormed. You've got the idea that people are potentially self-serving with their direct questions. So instead of calling your helpline to ask you, they are looking it up. And then there's also the idea that they are not trying to get more detailed services when potentially they don't, you know, they don't, they're not eligible for it or it's not appropriate at all. So there's kind of two different directions there. All right. So we are, so we're going to continue down the spectrum here and I'll define what I mean by that. You can see that we've taken the example we've got and pretty expanded pretty much on it. So certainly we're still taking examples if you have kind of a project that you're thinking of that you want us to go down. So when you look at kind of corporate methodologies or literature around ROI analysis, there's a big emphasis on what's called intangibles. And I really dislike the way that people use the word intangibles often when you look at this because it implies that they are a thing. You know, all right, well, this thing is clearly tangible and this thing is clearly intangible. But in fact, it's really, it's a fuzzy line. It's a spectrum from things that are easy to measure in dollars. So like for instance, all right, we're reducing mailing costs because we're, if you are mailing things, then there's another advantage of potentially putting web information up that is opposed to actually sending out postcards to inform people of their rights on this particularly very important thing. You are serving that through a web page. So that's an obvious, all right, we're literally not, we don't have that expense. You've got advocate time saved, measured in dollars, which is not too hard. You've got homeless, less averted, which is, we'll actually look at a potential calculation for that is doable, but more complicated. And then there you get to things that are practically impossible. And basically at some point, you just, and we'll look at this in more detail, you just basically say, I decree this stuff at the end intangible. I am done with attempting to quantify and everything to the left of this point, I'm going to define as intangible. So it's not like there is some, you know, platonic borderline that there is some state in the world where things, some things are tangible and some things are intangible, but rather there is a, just a spectrum there. And Laura, would you say that almost any initiative, any change is going to have a fairly broad range of what's measurable and not measurable. I mean, so even like if we move from mailing postcards to mailing to emailing or to texting, I can see some significant intangibles like, you know, what's the value of being able to inform people immediately, you know, that to have that instant urgent communication. I mean, that's going to be a little bit harder than knowing those direct, you know, postcards that you don't have to print and the postage you don't have to pay. Yep. Absolutely. And that's an excellent point. As we brainstormed around the web pages, there's not going to be a single benefit per project. There's going to be lots of benefits per project, some of which are hopefully quantifiable in some way and many of which will not be very quantifiable, or at least not practically quantifiable. You could make the argument that anything could be measured. Like you could attempt to, through like a million dollar university study, attempt to look at the amount of justice created in your community. But that's not a practical approach. So you basically say, all right, I'm not doing that. So yes, so there's going to be a whole host of benefits for any one of these things. Fantastic. Brian, anything to add about kind of this idea of a spectrum and things that might be along it? No, I guess in putting together this spectrum, it's often a good idea to get together a set of stakeholders to brainstorm things that may be on here, because your partners or other stakeholders may come up with some things on here that aren't already there or aren't apparent to you inside your organization. Yeah. We actually, so when we did this live at the TIG conference and we actually walked through an exercise, which I think is a pretty plausible exercise for you guys to do in individual organizations. So we basically, we brainstorm projects, actually we grouped people by potential projects. So we just assigned out projects that they were thinking about each group brainstorms, benefits onto post-it notes. And at this point in the process, we asked them to arrange them approximately into a spectrum from most to least measurable. John, as you were doing this exercise, so John was a participant in this workshop, what did you find as you arranged that spectrum? Was that a helpful thing for you kind of as a mental exercise? It was, and I think that it was also, I mean for one thing, actually it was actually kind of fun, because how everybody views what's measurable and frankly, how they would measure different elements of different benefits, I found intriguing. And again, I think Brian, your point's really important one. It's kind of like how you approach an ethics issue. You never do ethics on your own. Maybe never do ROI on your own. Because the exercise is so valuable, more valuable I think as part of a team. But yeah, I mean I think that to the extent that there were folks who hadn't gone through similar sort of exercises before, as they started to think about it a little bit more, both the benefits and sort of how we would measure it, I think that they, people pretty quickly picked it up and they sort of got to a level sophistication where I think they were comfortable to go back to their programs and run through this process on their own projects. Because even some of the harder ones, frankly together, we figured out ways that we would measure them, so it was a lot of fun. Great, and let's move on to that, so kind of thinking through, so how would we measure some of these more sophisticated ones? Sorry, Brian, were you chipping in? Oh, sorry. Sorry, I'm jumping in. Great, so let's just look at this idea of kind of chains of assumptions which are going to lead you to be able to quantify benefits that are potentially not immediately, you know, measures. So something can be measured directly. So you have things like, we talked about mailing costs saved, toners saved, decreased temp hours, you know, like if you're changing the way that you're doing data entry or things like that, increased fee for service, so logically you're actually getting more money. But more typically, you need to actually link things together. So basically to say, all right, we want to figure out how much advocates at time is going to be saved. So for instance, let's think about like a document assembly. So basically if we implement a online assembly for this particular form, about how much time do we think it would take a lawyer to create that form for one client? And this is where we start to get to low and high numbers. So what's the lowest amount of time, you know, well, there's no way it would take them less than five minutes to create this form. And well, okay, they're doing something wrong if it takes them more than 20 minutes to do this form. So there you have a low and a high number for the average amount of time per client. You've got then the hourly rate of an attorney, which is hopefully a fairly knowable number to you. And the average number of forms you do or the forms that you think that you could replace with this, the document assembly process per year. So and what that comes to is now the amount of advocate time saved measured in dollars. So let's look at another example here. So here basically here we just have the low and the high. So basically we can say, all right, we've got anywhere from two to 10 minutes, we've got 800 to 1200 clients. So here's our low and here's our high and they're fairly far apart. So we've got a low of 2000 and a high of 16,000. And that's okay. That is, we just acknowledge that that is our level of certainty. And we'll talk a lot about what to do with your level of certainty later. But if this thing is going to be a $9,000 investment, like basically smacked out in the middle, it's unclear whether you're going to get a return on it. And that's okay, because sometimes it's just not clear. So in this, John was talking about this, John, you want to talk a little bit about how this is more of an art than a science and going more towards the plausible than the specific? Well, and exactly, I mean, or maybe it's, there's parts of it that are science or that are math, but those are the easier parts. But I would say even there, you know, what how like, for instance, the hourly rate of an attorney, I mean, should you take the hourly rate that you actually pay them on their paycheck or should you take what some people call the fully loaded cost? So what's the cost of a percentage of their supervisor, their receptionist, the health care, the office space, you know, so like so there are there are a lot of decisions to make even on pieces that are fairly cut and dry, but certainly moving to, you know, the more esoteric as as Laura was talking about, you know, like the the averting, not esoteric really, but but you know, quantifiable like there are a number of measures that you could look at for averting homelessness in terms of the impact on client lives on the on their family on on the next generation on the neighborhood on their community, you know, so it's and I think part of it is also sort of figuring out who you're talking to with and what you know who your funders are and what they're interested in and also, you know, to the extent that you're you're trying to make the case with your with your own colleagues in your organization or with your collaboration. I mean, I guess it could be also useful to look at something like a unified communication system or a new phone system and what are the different ROIs that you would suggest an organization start to look at for a project like that because that's a very common one that many organizations are going through. Right and well, so I mean, I think that that you know, so again, the to the extent we were talking about some of the savings, you know, one of the things I think that that folks sometimes don't think about are some of the costs and we'll get to some of that. But, you know, for instance, on the phone system, there is this sort of initial cost to transition to it, whether it's cloud based or premise based. But then, you know, a lot of folks, I think have a hard time projecting budgeting for the ongoing upkeep management maintenance support of that system. So they're think they're costs like that. But on the benefit side, I think with phone systems, you know, there's the level of frustration where folks, you know, basically sort of drop the call, they call abandonment because they can't either have to wait too long or they can't navigate some of these ACD systems where you press one for housing and two for family and so forth. And so if you're looking at a new phone system with programming options that make it more user friendly, you know, I mean, I think you can, again, what does that do for your relationship with the client or the applicant at that point? Do you start that call when you actually, when a live person picks up with a caller who isn't frustrated and upset and maybe thinks less of your organization to start with? Why do you have such a crazy phone system? Or another functionality that a lot of folks are taking advantage of, you know, are using phone systems to improve language access. Are you going in the opposite direction? Are you being very friendly to folks who are not principally English speakers and making them feel more comfortable so that when you do the, ultimately the interview or even if you bring on an interpreter, you're getting sort of to that point where they have a level of trust and belief in your organization that you really care that you do want to serve them. What impact does that have on their representation, on your ability to represent, on the quality of that representation because they're more likely to share the information you need to make the right determination? I don't know, Brian, what would you add to that? One of the big things to really think about when you're doing a new tech project like that is not just what the technology currently does and how you would do that better, but also new features, new ways to integrate people into that. So for example, when we were doing ours here at Northwest Justice Project, previously to be a member of the call center, you had to physically be here in Seattle. The new system that we put in allowed individuals from across the state and also potentially outside pro bono attorneys during their lunch, other things like that. So it gave us a way to take volunteers or other individuals and possibly link them into that system. So you really have to look at what the technology that you're putting in place is capable of not just the business case of what you're currently replacing. And just to quickly go through just one last example here of things that get really very fuzzy, so the idea of a verting homelessness. And the idea here is that I don't expect that any of us really know any of these numbers, that we are even doing just wild guesstimates, but wild guesstimates are useful. You can see whether it's plausible that something will pay for itself. So basically if we say, all right, if we do this technology, we'll be able to increase the number of eviction cases that we take on. But we think it's got to be at least 15 cases that we'll take on. But these are really time consuming cases to take on. They're probably we're not going to do more than increasing it by 30. And even if you say, I don't know, maybe five we could take on or maybe it's as many as 30, that still is potentially worth doing to basically do a kind of back of the napkin calculation here. Lacklihoods in evicted client will be homeless. And similarly, we might be taking just wild guesstimates or we might have some anecdotal data from our own client pool, which we can then kind of wildly extrapolate. You're hearing a lot of wild from the wild guesstimates, wild extrapolation. And I think that that can feel really scary when you're putting numbers on things. But the idea is if you say, all right, this is the least I could possibly imagine, and not just this is the most I can possibly imagine, then at worst what you'll get to is, I don't have enough data to know anything. In this case, you've wildly guesstimated 12 to 40,000. We might actually be able to find some research for this particular number of the cost to the state of homeless person. So what we know here is that, all right, well, this seems plausibly somewhere between $9,000 and $60,000 in costs averted to the state. So the billers equivalent of adverting homelessness. And even if this we knew that this is somewhere between $2,000 and $100,000, that probably becomes not useful. That's such a wide range. But it's interesting to run through and to say, all right, do I know enough that I could say, well, even at my lowest estimates, this has got to be like $20,000 or whatever. It's worth it doing the wild guesstimates. Laura, could I just add to Brian's point earlier that, again, working together. So here is sort of an opportunity to work with the courts because they have data and, frankly, the local state government. And I think even engaging those conversations with them can lead to interesting new ideas and opportunities. Oh, you're interested in this stuff, or maybe they're not thinking about it. So maybe even funding opportunities. The fact that you're being systematic about it, and you're trying to establish more rigor in your planning, I think will, it can only lead to positive things. And certainly, if there's data out there, you may not be able to afford, as Laura mentioned, that $1 million academic study. But when you reach out and talk to people and they have data that maybe is, again, a little bit more detailed, maybe a little bit more reliable, and you use that, it's both a benefit for your ROI calculation, but it's also now another person who's more likely interested in what you're trying to do. Yeah, I strongly agree with that, especially on the funding opportunity side. Once you engage courts as partners or city government as partners or local schools or university as partners, it does open you up to some interesting forms of funding. Also, there are particular grants that are aimed at those particular partners that can be used to fund this type of stuff. Northwest Justice Project has went after both city grants and the top broadband grants. And then the ATJ Board that I work with has went after federal ABA money targeted at improving court services using the administrative office of the courts as a partner. Right, absolutely. So continuing down kind of our path here, at some point, so you've got a list of benefits for your particular technologies, you've got them arranged from most and least measurable, and at some point you simply stop, either because things become super impractical to have any ideas or any numbers around, or because in fact you have enough information. So you might, and we'll be looking at the cost side in just a few minutes here, so you might have enough information on the cost side to say, well, I don't need to go any further. This has already, I've already made a good case that this will pay for itself. And then you can basically say, you know, in addition to these measurable things, we get all this great intangible that we get to create more justice in the world. And obviously that's also useful for something, you know, that also has value. We don't even have to measure that. If you are finding that most of the project is things that you can't measure, or in fact the things that you can't measure are just so important comparatively, you may find yourself with a different kind of mental weighing, which is basically saying, is it plausible these intangibles will make this project worthwhile even though the quantifiable aspects will not. So basically to say, all right, well, it looks like our expenses are going to run somewhere in the $20,000 to $30,000 range, we can show that there's going to be probably somewhere between $10,000 to $20,000 worth of benefit is what we haven't measured, how what we haven't put numbers on, is it plausible that that makes up that additional $10,000 to $20,000? And that is a discussion to have as an organization, you know, if we're talking about a document management system, is it plausible that the, like for instance, the morale boost of not having to slog through a million documents and not having to recreate things that you feel like you might have already created, is that morale boost worth this differential if you're finding that there's a differential. So it's a thought, kind of a thought project, it's something to be as a discussion point for your organization. John? Or I'll just add, I'm going to get that. Well, I'll put this out there to see if, if anybody wants to take us up on this offer to put things out there that they think might be measurable and see if we have thoughts on them. And sorry, John, I cut you off. I was just going to say that, you know, and there's been, you know, increasing talk of this, and certainly the TIG is about taking risks, but there's, you know, there's a level of risk taking involved, because in part it may be that your calculations are off, or that you're not quantifying the things that are, that maybe would be quantifiable if you had thought about more of the, you know, sort of the potential costs or benefits. But I think, you know, I think as you sort of said in Texas, I mean, there's, you know, there's a, there's a sort of a point at which you've got, you feel like you've got enough. Maybe it's not, if there's not a, you know, the ROI in, in what you can sort of say are hard dollars isn't there. But, but that the, you know, those intangibles, you know, that let's, let's, let's do it. And then frankly, when we, when we get to that other side of that project and we're, and we're sort of getting the feedback from the staff or from our clients or seeing the impact on our work and the outcomes, you know, then we'll know to continue it or, or we'll, we'll stop and we'll, we'll, we'll change tracks or we'll, we'll shut it down. Yeah. So, anyway. Absolutely. Let's take an example from, so Brian offered early up that he's working on some kind of advocacy information to help people think through terms of service and stuff like that. And so logically for a project like that, you're going to be looking at at least one of the core returns is going to be basically keeping consumers from getting sucked into terms of service that they shouldn't be into that they shouldn't have to commit to. So let's think through how one could potentially form a chain there. So you'll logically start with the number of people that you think you are affecting. So how many people are likely to sign up for a term of service, that terms of service that is, you know, not what it should be. And then thinking through, all right, how would we measure the monetary impact of that? So is it a, is there a monthly, like they are paying more per month? Is there some percentage of people who are going to get sucked into something that, you know, like, like a payday lending, that it's not, not everybody that will be affected, but some people will be affected. And then to think through, all right, of the people who are affected, what's the dollar amount on average, a higher, high and low. So again, just kind of forming our, our chain of assumptions around this particular area. And basically we get to at the end, the realm of, okay, here is the amount that the consumers are saving by doing this advocacy effort. All right. Questions or comments about kind of this whole idea of benefits before we're going to talk relatively quickly through costs, which I think are just a little more straightforward. And then we're going to wind up by kind of looking through some scenarios of what this might look like in terms of, okay, we've got the benefits minus the costs, high and lows, how might you use that to make decisions? Questions or thoughts on benefits before we move on to costs? I think with the terms of service example, looking at one other potential benefit for that, what we were looking at is the ability to reach out to potential clients that that educational information may also allow us to find clients that would work for a test case for particular theories that we were looking at. So it has this kind of marketing benefit as part of it. So we're educating consumers, but also letting them know if you've been harmed in this way, we can also help you. Yeah, and that's interesting. And that sounds like something that you might think about measuring potentially in fundraising dollars. So I would assume that you're testing out this case with the idea that, well, obviously you're going to increase the justice in the world, but that also you might be able to fundraise around the idea that this is going to really important thing to do. So you could think through how much money might you raise? What's the likelihood you're going to raise it? What's the likelihood of clients you're going to get for it? Yeah, so kind of tying that together into a chain of assumptions. Awesome. Yep. Fantastic. So let's think quickly through costs here. So some of these are really straightforward. Like I suspect that none of us are going to try to think through their costs without thinking about the hardware and software costs. So that's a fairly obvious one. And then you've also got the staff time around it. So both and two really important sides to this, you've got both the time and the cost to plan it to make sure that you're going to effectively design it for success, and then to actually implement it, to roll it out, to train. These are, in my mind, kind of two halves of the project, which are often fairly similar in size. It's interesting, some people tend to really overestimate the planning part and underestimate how long it'll actually take to make it happen. And then some people do the flip side, that they underestimate how long it will take to plan. They're just kind of like, well, you know, we're going to do a little something, but really we need to just get rolling on building this thing. So it's important to think about these as almost equal partners in your project. You want to think about your processes around this particular thing. Some types of, it's going to depend some on the type of project that you're working on, but a lot of the types that we've talked about here involve, you know, staff members, lawyers, advocates, even clients doing things in different ways. And if you're going to change how things work, you need to make sure you think about not just the technology, but you think through the process. John, you want to give us an example of something you've worked on, where you thought about kind of the process as well as the technology? Sure. I mean, and I think that probably most folks on the call have dealt with processes with case management change, case management system changes. But it's certainly, you know, the opportunities around any of these initiatives are, you know, and as Brian was pointing out, are very significant. It's not just replacing one set of technologies with another that's identical. Typically, they offer new opportunities or you have an opportunity to train people to do things differently. But certainly, I think the whole intake process, you know, what's the data we really need? How do we, you know, get the sort of that essential data, build that relationship or not impede that relationship by asking a lot of what seems like bureaucratic and very private personal information, you know, from clients. So when do you ask for certain data? How do you ask for it? And then building that into the system so that it flows the way that sort of the group, the experts who are really the program folks believe it should. And frankly, allowing for, on all these projects, allowing for the fact that whatever, you know, your best thinking is not going to be perfect. And so maybe this is, again, part of the planning is planning for that month or two down the road where you go back, evaluate, and adjust. Yep. Absolutely. So there's some other costs to keep in mind. Consultant costs that if you're going to need to bring in outside time, you obviously need to pay for that. Training costs, really important. If you're changing processes, then regardless of whether they're actually using your end users or using technology or not, you're going to need to train them in the processes. You want to make sure that you think through training, not just as a system, a technology thing, but something about help that also helps them understand how to do the things that they need to do from a process perspective. Thinking about just staff time in general, so thinking through the steps on your project and how much staff time is going to be involved. For instance, some projects may be really intensive with the amount of advocate hours that need to be spent. Like if you're doing a document assembly project, there's a little bit of technology time and a lot of lawyer time in thinking through kind of that whole decision tree and things like that. And then last, but certainly not least, to think through not only this year's cost, the cost in getting it up and running, but the cost to maintain it down the road, to basically remember that basically nothing is going to support itself. So this could be an area in which your maintenance costs go down. You have something that's actually easier to support than whatever you're replacing or if you are replacing something. But it could well be that this is you're increasing your ongoing costs and you want to make sure you take that to account. You have more intangible costs as well. You've got the idea of morale. Change is hard. Even if everybody agrees that the old thing sucks, having a new thing is still hard. It's still a distraction. It's still something that people need to take time out of their obviously really busy day to deal with, which leads to the idea of opportunity costs. So if you weren't doing this project, so any project where you're asking people to commit any kind of mental bandwidth to it is somewhat distracting from other things they might be doing. And so thinking through what is it that you're not doing and is there a potential, a much bigger return on things that you'd be doing instead? And that could include anything. That could include technology projects or that could just simply mean serving more clients. So to make sure that you're thinking through the time that there's obviously a limited amount of time in the day and hundreds of things that people could be doing with that time, what actually makes sense. And logically, some of these may be intangibles in the same way. It should be the same spectrum of tangibility to intangibility that we looked at on the benefit side. Can we just feel that people tend to run into trouble when estimating costs? I just wanted to add something to your last point. Sorry. I think that these initiatives that we sometimes take on in a silo, there's this great idea and frankly there's great ROI. The challenge I think for a lot of agencies is really doing the planning work more broadly. And so if you have a plan, if you're evaluating your program overall and you're trying to come up with the strategic priorities, what are you going to do? What we really want to do is improve our legal work supervision. Or we really want to improve our resource development with fundraising. And so that may then lead into what are maybe the technology projects that would support social media, like better networking to help you raise awareness of your organization. I kind of find that frankly that there are innovators within the agencies, but I think who have great ideas and get a lot of great stuff done, but I think one of the jobs is to get the rest of your agency on board with the notion that we really need to rethink a lot of what we're doing. And then in terms of the opportunity cost, then we're all on the same page in terms of we're going to do this instead of that, or we think this is worth evaluating first. So you're not blindsided. You don't go down that road on some massive project like a phone system. And frankly, the impact is significant, but it's not as significant or maybe as critical as some of the other needs of the agency. So that's maybe more out of frustration. And so again, the planning should be in a vacuum. Technology folks, even legal technology folks really need to bring people in to think about these matters. And that may be hard to just get them to make the time to sit down over a series of weeks to, you know, you don't need to necessarily bring in a consultant for this, but what are the issues? What are the needs? So I think that will help improve, I think, the quality of the decisions on what you pursue and what you don't. Absolutely. Fantastic. All right. So let's kind of put this all together with a few kind of closing kind of thought pieces here, and then we'll come back around for any questions or comments that you guys have. So basically, you then, you take all of your benefits and you add them together and you take all your costs and add them together. So here's our benefit of if we assume that all of these things are, so this looks like, it looks like a document assembly system except for the mailing cost. I'm trying to imagine what mailing cost is there. I don't think there is an actual project behind this list. So we basically say for this particular project, we've got a low of 16,000 and a high of 83,000. One of the things that's important as you go through this exercise is you definitely don't want to present these numbers as 16,383. That's far too exact for what we've actually done. So that's basically, that's about, so we've got a low of about 16,000 and a high of about 80,000 is what we have here. And you want to just, the more exact you present them, the more authoritative they seem, and the less, so either people will start to really question, you know, all right, where did this number come from that it's so very exact instead of thinking of it as a kind of an opener for discussion. So you've got the benefits versus the cost. So let's say this is what we've come to for this particular project. So the benefits as we just added up, we've got 16,000 to 83,000 dollars. It's somewhere in there, plus and tangibles. So those mean to us. And we've got the costs are somewhere between 20 to 40 plus and tangibles. John, what would this mean to you if you were looking at this? So what do you just kind of a stream of consciousness? What's your thought on whether this project might be worth it? Things to think about? I mean, so one of it is one of the thoughts is again, is this sort of an important, is it strategic? Is there some value here that sort of advances the agency's mission? I think it's certainly worth proposing it in part just to get, again, if you've done this somewhat in a small group and you want to expand that conversation, propose it, share this out that there are these, potentially there's this sort of real hard dollar benefit so that we will be ahead of the game, so to speak, after we finish the project. And then frankly get folks to weighed in and critique it. But again, I think your point's a really good one about don't try to give people the misimpression that all these numbers are so exact. I mean, let them understand your thinking, but that these are estimations and they may have wildly higher estimations on both sides. So I think this is worth consideration in my view. Yeah, so basically this is a back of an Afghan analysis, which I would agree. This says this isn't a ludicrous project. This is also a slam dunk project. So as a lot, I think John's immediate reaction kind of framed in a different way, was that it comes down to a lot of intangibles here. So basically thinking through does it how does that position us for other things compared to like the cost of change morale. So this is basically an ROI that tells you that this is a plausible project, but it's not an obvious project. So you need to think this through and think about those intangibles. Here's some other things to look at. Here is one at the top here. You've got, okay, the benefits, 16,000 to 25,000, and the costs are 30,000 to 40,000. So unless you've got some hell of the intangibles over here and very little intangibles on this side, this seems like maybe not the right project, or to say that this is a project that is all about, you know, increasing access to justice in some unquantifiable way, or it's all about advancing the brand awareness of the organization, or basically to get buy-in that this is not about the quantitative value of this project, that it's all about the intangibles. Here's one on the bottom here where basically what we calculated is we don't know. This is, we don't know what the benefits are going to be. The costs are pretty substantial. It could pay off big. It could not pay off at all. Basically what we would, what we look at in this analysis is we say, all right, this is a relatively high-risk project where we don't actually have very much information. And so we come back around to, is there a way to get more information, or does this make sense to take this knowing that we're not really sure what the benefits are going to be? And that is an equally affected, I mean, so it's basically a quantification of what is known. And in that case, it's not much. Well, we know the costs actually reasonably, to a reasonably finite degree. It's the benefits that have this just huge range. And that is a totally reasonable thing to present. You asked me to do an ROI on this. Here's what we know, which is maybe not enough to make a decision based on that. So we're going to need to make a decision based on something other than an ROI. All right, Brian, what do you think about this whole laying out, as we've laid out this whole process, I think this is just a very specific model and methodology here is potentially new to you. Does this seem like something that resonates? What seems like it might be most useful in this process? No, I think the most useful piece is really looking at the different types of return on investment, that scale that goes all the way from the intangibles, for lack of a better word for it, to the concrete. And being able to really divide each of those also allows you to successfully pitch the project to different stakeholders because there are going to be people who care about different pieces on that spectrum. Yep, that makes sense to me. John, if there is one thing that you're hoping people will take, by the way, this would be a great time if you have questions to put questions into the chat. We'll certainly take all commerce here. What would be your kind of one piece of advice, John, on this topic, if people are only taking one thing away with them? Gosh, well, I guess my one hope is that people would take this back and run through this exercise. I think with some colleagues, maybe some colleagues, if you're thinking about, again, a project that crosses programs in particular, those are really interesting. And I know a number of states are doing a lot of interesting collaborations. Run through that exercise and see for yourself, because I think it just can be a real eye-opener. And frankly, it may reinforce the decisions that may, so you might do it with the project that you're already engaged in, or it might point out potential weaknesses, things that you might want to adjust. So one of my other thoughts on one of the prior slides about where the costs and the benefits are close or where it's sort of a margin call, whether we really should do it or not, it might also lead me to go back to think about, so what can I do? Can I pick different technologies? Can I reduce the likely cost of this initiative? Can I, for instance, and again, I don't think necessarily that interns are the solution for a lot of these projects, but is there some way to get lower-cost labor and how can I improve the certainty of, because a lot of these projects go up in cost as the length of time it takes to implement them, can I somehow think about carving out more of my staff attorney's time to work on this now instead of getting just a couple hours a week over six months. So if I've got that gut feeling that this is really worth doing, what can I do to now rework it? And then frankly, again, going back to those benefits, maybe take another stab at the intangibles, in part by expanding that conversation with others who will sort of expand your perspective. Right, absolutely. Yeah, and another just, that made me think of, as you were saying that, John, the idea of also doing kind of pilot, if we don't really know, or in the realm in which both costs and benefits, I just have this really high range, and we have no, basically we have no idea what the RLI is to do something that's going to add to our information about that, like doing some small project with the goal of understanding better what this will cost and what value it will bring. And talking with Brian and others who, you know, because like to your point, so you're like, who's done it? And it may not be that they've done your thing, but who's done something similar, and can you learn from that? And again, that's sort of expanding the conversation, benefiting from the fact that you're not alone really in some ways. You've got a community of folks out there that are typically very eager to share what they know. So yeah, do that as well. Yeah, if you're doing a project, you are probably not the first one doing at least something similar. I would ask on the LS Tech email list, figure out who is already engaged in the project, and what their practical outcomes were. Yep, absolutely. Fantastic. And that's kind of shout out to the LSNTAP community, makes a nice segue to Brian, are there future webinars that you want to talk about or other things coming up on the LSNTAP side? Definitely. We have put together our webinar schedule for the year. It is up at alasntap.org slash training. It is linked on our front page. We've got a webinar coming up on April 12th on using Google Analytics to understand web traffic. And then we've got one coming up on privacy, encryption and anonymity, and the use of expert systems and predictive analytics. That second one is a brand new topic for us out of the ATJ Technology Center from Florida. So that's the first time we're going to be really working with them on a project. All of these are going to be recorded and put up on our YouTube channel, but we have over 12, and I think we're maybe up to 16 by the end of the year free webinars available to the community. Fantastic. And just to put in a plug for that analytics seminar, which is one that we are also conducting over here at IdealWare, that is in fact a fairly different webinar than other analytics sessions we've conducted in the past. It's a little bit more about 201 deep dive into not just kind of the basics of analytics, but how to really think about tracking kind of referral paths through your websites, thinking about the campaign functionality and how that might help you understand where traffic is coming from, really with a kind of a deep dive into understanding where your traffic is coming from, and how you might be able to drive more to it. Fantastic. Thank you so much. Thank you so much, John, for all of your insight, and thank you for helping me to fill the time and a little bit of technical issues there. So I really appreciate all your wisdom. My pleasure. Yes, it's been fun. Yeah, thank you so much, John. I look forward to having you on for other topics in the future, and thank you once again, Laura. We're very happy to be hosting IdealWare again for our webinar series this year. Thank you. Fantastic. Thanks so much, guys. Thank you. Bye-bye.