 All right, it is two o'clock, so we're going to get started. Thank you so much for joining us today. I am Shelly Reed, the manager of Legal Services National Technology Assistance Project with LSNTAP for short. And I am thankful that you're here today. We're here to hear from a great panel about software selection. And I'm going to turn it over to the lead of the panel, John Greiner, to introduce all the speakers and get us started. Good afternoon and good morning to some perhaps. Thank you for joining. I just want to first let folks know that unfortunately one of our panelists is stuck mid-air and so she won't be joining us. We're hoping that Hama gets home. Eventually she's been traveling or trying to get back for two days. And so she regrets not being able to join us. And we are going to do our best to sort of fill in the content that she was going to provide. But of course, you know, I think after the webinar, if she's up for it, we'll maybe share a little bit more in terms of the resources about some of what she was going to present. But with me today in addition to Shelly, Reed is Jay Singleton, who has been in the Legal Services community for, you know, quite a few years. He's won a number of awards, but has been relevant to this webinar, sort of a focal point in Minnesota, helping. Well, broadly speaking, helping with a lot of various projects, but you know, helping with a lot of technology related initiatives, including working collaboratively around software selection. And, and, and Stacey Hover, sorry, has sort of brought the executive director's role into focus of, sorry, she's someone who has a, I would say a very tech forward perspective on the delivery legal services and has been for many years has been part of a number of California statewide partnerships. That's how Stacey and I first met. And, and of course has led her organization through the, you know, the COVID transitions and, and really appreciate Stacey actually during the height of the pandemic helping with other webinars to help share some of her experience and pivoting and helping everyone continue the delivery of services. So, I guess, to kick off, I'm going to be providing a little bit of context. The panel actually five of us got together to sort of work on what's, you know, is involved in software selection and some of the, the lessons that that we've learned and, and also some of what we've learned from, you know, webinars that we've gone to or, or resources that we've found. And, before we do that I guess what we would love to hear from folks attending today is is a little bit, I'm sorry, my here we go is a little bit of in chat a little bit about sort of your successes and, and let's say misses in the software selection or system selection process. We'd like to make sure that this is sorry I'm not sure why my, let me pause there. We want to make sure that that we're you know towering the conversation. You know to the, you know to the folks in attendance in attendance but also I think provide a little bit more context. You know software selection as we'll get into can be as simple, well, maybe not so simple as selecting a new anti malware pieces software or as complex as as a case management system. We'd like to get a sense of, of who's joining us today, what your role is and experience has been with software selection and, and, and maybe what you, you might share so if you wouldn't mind dropping those experiences in the chat. Shelly and I and and and Jane Stacy will be looking at those and trying to weave that into our discussion. So, you know, we, and hopefully my screen will not auto advance now I hit the pause button. So, in, in preparing for this webinar we were. No, it's doing that again. I am sorry, I haven't had that happen before also hopefully it won't do that quite so much. I've, it may have been some paper that I have on my desk. We were talking about sort of how, you know, a smart software selection process is important. And, you know, they're, you know, typically they're in any sort of product area any software system area they're multiple solutions, but picking the right system can can make, you know, a significant difference in how well it's adopted how well, you know, people can obviously use the software whether it integrates fully, whether it you went over some of your staff whether they're, whether they feel like their needs have been addressed. It certainly can help you support your mission. You know, in terms of, you know, for instance like an online intake system, you know, how do we increase access to our services how do we make access more equitable. And, and it's also typically something that that is a significant investment. We don't always, we don't change our software systems very regularly or we try not to. It can be very disruptive. And we, you know, ultimately want to have enough support to continue that process because it's usually there's, you know, as James going to talk about with with a, you know, collaborative software selection process. Once, once we, you know, generate some success it helps us build momentum to, to, you know, move forward with additional software additional system advancements. So it lays the groundwork for for future projects, you know, success built on success. So, and you may have additional I see their, you know, Sarah's offered some comments you may have some additional reasons why you know it's important to have a smart software selection process. But what follows basically are some of the things that we think are key to that process in actually doing the work and and certainly, you know, big part of that is defining what the business needs is not not really a technical piece of it at first. But for instance, you know, we just talked about with online intake. We have a hard, hard time reaching certain populations, maybe language access is an issue. So we're trying to define it in terms of the work we do in terms of our mission in terms of how it will impact our staff or our volunteers. And that there's a compelling a benefit to making the right software selection and that people will understand why we're investing this time in the process and this time and this money in the software and the implementation project management and so forth. And again, it's not one size fits all I mentioned, you know, like if you're buying new antivirus or anti malware software. It's a it's a different scale. But you still need to explain to probably your ED your your director of operations, why you're spending this additional money that it's going to help protect your system so that your staff will be more reliably able to get their work done that your client data will be better protected. In other cases, a case management system, you know, it, it, you know, it's going to be probably a little bit more apparent when you have folks involved who are actually using the system in helping to define the need that they're typically coming to you with with specific complaints about the current system or they've identified opportunities with other systems that they'd like to take advantage of. So developing that list of requirements and, and again that that sort of understanding of what you're trying to accomplish edit edit in a plain language way, not, you know, you don't necessarily need to have the techies in the room at that point is really important to sort of set the right objectives for the project. And, and get the buy-in or the initial buy-in and then help you in your more technical process that follows. Just a quick note, one of the things that I think has come up over the years for most programs, if you go back far enough is this notion of building tools, building software used to have more programs that have built their own homebrewed case management system. I think people newer to the community probably would find that hard to believe. But, but generally I think the strong consensus among everybody on the panel, and I think Shelley as well is really try to avoid getting into the software development business. If you can buy it, that's definitely preferable. If it might need tailoring, it starts to get a little dodgy because then there you know those costs might not be so certain or the timeline for that development work that the vendor might need to put into it. And we'll talk to some of those pieces later in the J and Stacy are going to go over. A governance process again for software and we were going back and forth between software and system selection because sometimes it's a little blurry. But, but understand what your decision making process is going to be. And, and again, if it's a very technical piece of software, it's probably going to be your IT lead and maybe your finance director. But, if it's going to affect your, your advocates your legal work supervision, you're going to probably want to make sure that the decision making process is established so that folks understand who's going to be involved, what their opportunities are for and what the, the feedback mechanisms will be to help resolve issues maybe after that selection process or, or you know the trade offs that you're most likely going to have to manage. Another sort of, we thought sort of important point was, you know, how do you resource it and again, sort of goes with governance if it's a very small technical pieces software. You don't need to have as many folks involved you may not need to spend as much time doing a lot of testing and demoing of like, you know, again, maybe going back to the security software. On the flip side, there may be pieces, you know, for instance, you know, there's, you know, VPN clients that help folks get into their offices remotely you may want to think about well how is this going to affect the user community the user base in your organization. And so having some users involved so that they can weigh in at least in terms of the user experience on the selection of a VPN client. And that's actually from our experience been a really important piece of software for users that's a very technical tool, but can lead to a great deal of frustration so thinking about how you're resourcing the project is important. And that goes along with time how much you know time are you going to allow for it. And, you know, committing, you know, folks is, you know, the most folks are going to be, you know, wearing at least a couple hats already and so this is wearing an additional hat. So being realistic in terms of timeline is important, and also being really clear about everybody's role. And the budgeting, you know, piece, we weren't talking so much about budgeting for the selection process but, but developing a more detailed budget, both for the initial procurement implementation project management. Training launch, but also for that ongoing maintenance. Management enhancement that you're ready to continue to get the most out of that software that you're selecting. And, you know, it again it varies based on the type of software that you're looking at so in some cases there isn't really a lot of extra enhancement or configuration. Let's say on some PDF software, you sort of get, you know, the functionality that that the Adobe may have provided, for instance, but with case management systems we all know that there's sort of your initial implementation. And the program continues to evolve your needs continue to evolve and so you need to be budgeting for that ongoing costs. And, you know, I guess one other area and this goes back to some of the work that else and tap and, and others have done around security is really sort of thinking about. It's doing that again. I've got a haunted computer today is really thinking about sort of what the security implications are, including many compliance requirements that you may need to be cognizant of like you know if you're going to be dealing with private medical information. And you might be covered by HIPAA, you know, so you basically making sure that you're. This goes some extent back to your requirements but that you're thinking through all the various security issues, including, you know, the practices of your vendor. The vendors that are involved or consultants that might be involved with the project. Obviously legal aid generally deals with a lot of confidential privilege information and a lot of these projects touch on that data and so it's really important that you make a point of doing an evaluation, both of sort of what your needs are and what the vendors are doing. And, you know, again, this sort of ties into the broader vendor valuation, you know, their commitment for instance to the community if it's a very specialized piece of software like a case management system. You know their their knowledge of your needs what their development cycles are so are they going to continue to enhance and advance the platform that you're you're buying into. Again, partly because we it's in most cases it's a little challenging to, you know, keep flipping between pieces of software or again like a telephone white system for instance, because of all the training and the upheaval. You want to make sure that you're you're you're you're sort of building a relationship with a vendor that that is going to be aligned with your needs going forward not just on those today. And again, I think we're going to get into a little bit more of the selection but getting, you know, getting into the sort of the selection process once you have several different solutions from different vendors, you know, having a more formalized process now that can be through an RFP and in some instances the bigger projects it definitely I think I think we'd all agree strongly makes sense to build that RP, but even in in in sort of smaller projects having some sort of formalized approach to selecting your vendor whether it's making sure that everybody comes in does the demo answers a set of questions that you that you have maybe some user testing you you deploy it to a few machines, especially if there's some flexibility, so that that you're not, you know, fully committed until you know you you basically sort of done that that full evaluation. One additional sort of piece that came up, which I thought, again, kind of lesson learned is where where you're buying licenses for software or service, you know, a cloud service. So really think about what your rollout is going to be and whether you need to purchase all those licenses right away or whether it's something that you can do progressively as you roll out that software because a lot of cloud services start charging pretty darn quickly, and those fees may be reasonable when you're actually using that software or service across the board but if you know 10% of your users are working on a cloud platform, while they're essentially sort of configuring the client or, or, or getting it ready for the rest of the organization but you're paying for everybody. That could be a lot of additional expense with with very little value and obviously legally that's critical that we, you know that we're mindful of all those expenses and, and that the calculus makes sense. So, we didn't talk, you know, much in terms of getting into the project about change management but how we get into that project defining that you know the business needs, being really clear about your governance, being clear about your requirements certainly helps with change management. And project management I would add sort of, you know, how we communicate how we work with folks how we make sure that the training that we develop for that software fits that we are not leaving people guessing or, or maybe even fearful that we're going to interrupt their work, because we went there, pretty much everyone's working on deadline, and, and we need to sort of acknowledge the challenges of doing, you know, the day to day client work while also changing these systems. I think the, again, the, the, the level of engagement the level of focus that you know you need to start to, you know, think about when selecting a product. Again, varies by how broadly it's, it's going to be used so you know if it's a specialized piece of software for bankruptcy for instance it will affect you know and effects like 5% or 10% of users. It's going to be a lot easier than something like a document management system would or a phone system would. So the level of investment of your time and effort to support your users is going to vary. So that's it in a nutshell and I want to, you know, you know, again, thank Kamra for her involvement in helping us come up with some of these sort of high points and and turn right now to more of the specific sort of experiences from, from Jay and Stacy who are dealing with manage software selection on the frontline for a number of years. Sorry, and, and before we do that. Again, I think if you wouldn't mind sharing out your thoughts on on what the value would be for your organization in terms of having a better software selection process that would that would help us in responding to your, your, your current issues. Jay, if you wouldn't mind taking away. Yeah, certainly. Thanks john. Just a little background first on my organization's role within Minnesota. We're legal services state support and so we're kind of the state support agency in Minnesota and we work very collaboratively with the legal aid and other access to justice organizations in Minnesota. So we're not providing direct service, but we're, you know, doing a lot of that background work. And the, the story I wanted to share which we, Stacy and I wanted to share some stories to try to illustrate some of the points, the high points that that john had shared the give a little bit more context to what this can look like when it's actually playing out in real life. And so the story that I wanted to share is, is a process that few organizations in Minnesota undertook recently in the last year or two. So my office state support was approached by one of the legal aid organizations in Minnesota, who was interested in looking into other immigration forms software, they were using a software. They weren't, they were unhappy with some of the, the features or lack of features offered by that software. And so they wanted to see what was out there and since my office works on a lot of technology projects they asked if you had any insight and then they also had a really just great suggestion, and they said hey, we've heard from other partner organizations that nobody's really happy with their various immigration form software so maybe we could all get together and tackle this project as collaboration. So we had eventually five organizations joined together to look at what the landscape was of immigration form software out there, and undertake that software selection process collaboratively. So what that looked like kind of, and a bit more detail is, we had five organizations and we had a couple of initial meetings with representatives from each of those organizations, where we really hammered out what are the requirements that we absolutely need to have in order to make it a solution that will work for the various use cases at those organizations, and will also, you know, make it worth making that switch from the existing systems that various organizations were currently using. And then we also had our list of what are the nice to have like this would really enhance the, the, the process of filling out forms and make it easier so we can devote more time to clients. So it's not going to be a make or break. And one of the great things about having the five different organizations present was that we had a lot of different perspectives, not just from the different roles that those organizations played so we had one organization or two organizations that do only immigration we had three organizations that do general legal immigration was one so that we had organizations that were primarily staff attorney organizations, and then we had a couple who were pretty extensively with volunteers. So they wanted to know how will this work. If we have volunteer attorneys that need to use the system. And then beyond just the different niches that the organizations play and their different perspectives. Because we had the different organizations they also sent different representatives who played different roles with those organizations. So we had from one of the largest organization, they sent one of their it managers. And that was really helpful to have that very specific it perspective. And then we also had no management, helping out we had staff attorneys volunteer coordinators. So we had all of these perspectives, but each organization was only sending one to three representatives to help us through this process. So it wasn't a huge as huge of a commitment for those organization. So we had our list of requirements and nice to have my office, and I'll give a shout out to Amanda Sauber who was on this webinar. She researched the field of different vendors who might be able to provide a solution, and then also reached out to Ellis and Pat. So thank you, Ellis and Pat to see what other folks were using. And from that we put together a list of the potential vendors and kind of the, the pluses and minuses of the various vendors and then we came back together as a group to select finalists. So we looked at how the different vendors matched up for our requirements and nice to have. And then looked at pricing, of course, to see which ones would be the best bit. From that point, we set up demos and interviews, which with each of our top four finalists. So we had everybody come together, get a little demo of how the different software solutions worked, got to ask questions of the sales reps. And then another really cool aspect of having all the orgs work together is that each of the organizations took one or two of the finalists to take back to their organizations and demo we got a demo account for, I think all of the solution. And so everybody went back and they did their assignments, like testing out how the software actually worked in real life. And then a couple weeks after that we came back together, compared to notes. Throughout all of this, it was really helpful to have Amanda as a centralized project manager. She was the one who was trying to schedule 15 people. Find time for a meeting. She was the one who was making sure that we were taking good notes and then sharing them out so everybody stayed on the same page. And then she was also the one who was coordinating with the vendor to set up demos and gather questions from everybody and then batch those to the different vendors that we were talking with. So once we had established our requirements, done our basic research, done our testing demos, we then collaboratively made our final selection. And the group was able to agree upon one of the vendors who they felt would best meet the needs that we all had. This was a vendor who was a little bit newer to the legal aid space in terms of immigration form software. And because they were really trying to break into this field, they were willing to do some of the integration work with our state's case management system. We all use the same case management system. They were able to comp some of that work, because they saw viability and using those same integrations throughout the country, if other legal aid organizations decided to hop on board. In addition, when we were doing negotiating contracts, we still had separate contracts for each organization that ultimately decided to contract with this vendor, but we are able to do a little bit of collaborative negotiation, which I think helped us as well, because we had several orgs trying to get a better price, which I think we were able to get a little bit of a bulk discount. You never know how well you're doing in negotiation, right? But I think that it was helpful. One of the pieces I wanted to touch on, which is one of the things we wish we would have done is that while we did have that IT person helping out from one of the organizations, and they did make sure that we had certain protocols like multi-factor authentication and accounting for those kinds of security issues, we wish that we had worked with a security consultant who could have helped us create a list of questions that we would want to know to make sure that the company security practices were up to as good as they could be, and to help us interpret the responses that we got from those questions, because none of us were security experts at that level, and you just don't know what you don't know. So you don't necessarily know which questions to ask, and you don't necessarily know when it gets to a certain level of technicality if the responses are meaningful and if they are satisfactory in terms of security best practices. So that's a process that we're currently working with JustTech on right now, just to make sure. And one of the reasons I think this is kind of assumed, but just to state it explicitly, just with the kind of information and the sensitive information that gets collected in immigration forms, we really felt like we needed to make sure that security was like absolutely best practices because that info is so sensitive. So if it were a different project where we didn't have like personally identifying information, we probably wouldn't go to those links, but taking a step back and realizing the use case we felt like it was really important that and it would have been great to do that upfront so that we could enter our evaluation in terms of selecting which vendor to go with. So those are the high points. Anything else you'd like me to touch on John. Okay, I think it's great and you kind of touched on almost everything in the outline I you know I would say that in terms of some questions the CIS controls which are available for free. So if you look at that this has its own security standards that you know that that vendors who you know are trying to sell the meatball to the Pentagon but certainly the federal government as well need to meet. I think one of the big frustrations with you know the range of cloud service providers and you know again that's where we're getting a lot of our software a lot of our integrations are happening cloud to cloud is is that it is really difficult to to you know qualify and quantify the security of each vendor, especially if they're not all really large organizations that have met like talk to type two sort of certification standards right and so I think again this is I mean Shelly's has against really sort of push the collaboration and working together, you know negotiating together on pricing I think it's, you know, like a buyers club I think we need to continue as a community to move forward in that sort of evaluation that security evaluation because it is expensive it's it's time intensive. And some of these, you know companies inherently are doing that because they're selling to a very broad range of customers like the Pentagon and so they've got you know they're like document management systems out there that say we're DoD compliant it's okay good. So that's, that's that's sort of one thing we can check off, but a lot of a lot of our vendors are not and and maybe don't need to be either and so it really sort of depends on that role but I think it's, it's, I mean, you know the collaborating I we didn't we didn't do this before but, you know, in New York when I was at legal services NYC when we were looking case management systems we invited the other providers in New York City to join the discussion join the requirements gathering process and the evaluation process so to the extent you can do that collectively you know even if you don't save time yourself you are helping the community and advancing the awareness of your, you know, sisters and brothers in the community about what's out there so I would just say that even if you're going to make a selection, just for your own organization, if it's okay with your leadership to involve others who might be interested. And generally I think you're going to you're going to benefit you're going to understand a different perspective and that's going to help you in your requirements or in your selection. But thank you Jay I think it's, I love, I mean to the extent that more and more states are working kind of more collectively I think you know Minnesota has certainly been a leader that way and with our case management system implementations and their online portals and everything else it's I think the quality and the value has been a lot higher. So can we turn the mic over to you. Sure. Thanks john thanks Jay. I am the executive director of the Legal Aid Society of San Mateo County which is a direct services program so we're kind of the opposite of Jay's perspective. It's a single county program, kind of old school legal services, and we do work with partners in the county and that's one of the two stories that I wanted to, to touch on is related to that where. So there's two projects that I want to talk about. And the first one was an, we're trying to construct an eviction database that us and of our kind of primary partner in the housing space in the county would be able to make sure that we would both it's really just a really just a dashboard that's all we're trying to create was it was a visualization so that we could see each other's cases evictions are very fast paced as I'm sure all of you know. So getting them out and making sure that you've done a very quick prioritization is a sustainable sustainable tendency is at a high risk, or a highly vulnerable client, you know, is this one of the ones that we really want to get full representation to now. Before you know as early in the eviction process as possible. So that was the idea was to grab case data from our case management database and from our partners case management database, and just really just show it. So that we could look at it and then we could kind of grab and quickly assign cases out. There was a third partner that was just going to be kind of a view only but they were going to be grab some cases as well. So that would be a very, not very detailed information. We do have HIPAA constraints so we wanted to make sure that we weren't sharing any kind of health data so we're really trying to minimize personally identifiable information. We obviously needed addresses so that you could tell who you were talking about but trying to minimize PII and make it all very quick, very accessible, very visual. And I made lots of mistakes in this process and I'll just touch on a few of them. The main thing was not really drilling down with the partner I was dealing with somebody the partner who was probably more in the accidental techie space rather than a true IT person because they have an IT person. They had people who knew how to use their database and that's who I was working with. But it turned out that he was not the person who was the contract person with the vendor for their database. And so when we got to the point of like, okay, how are we going to access your API like we need to get into your case management so we can pull the data out. No one knew how to do that and then when they got to the point of actually talking to the vendor the vendor said no. So really just the internet's going to come right back. Well, so while we're waiting for Stacy I mean I think one of the things that Stacy shared with us was, you know that again you have these these these plans for a collaborative project and selecting, you know, sort of for data visualization and selecting sort of a platform with the understanding that, you know, pieces or responsibilities are going to be met by, by all the partners and I think it's sort of like you really have to bet, you know, down in the depths to really understand what the potential sort of challenges are with the software integrations are again more and more common. And maybe we also have an expectation that that the the integration is just going to work and it's not going to require additional development. But I think somewhat, you know, Stacy was sharing with us was was you know that that that those challenges are not insignificant still, and certainly with older platforms in particular it can be more challenging, or with vendors who are overloaded maybe with their own development work and they don't really have time to address an integration need that you have. So when you're selecting a system when you're with what that's going to be used by multiple partners and California has a number of examples of that. It is inherently more complex. So your, your, your, you know, so the scale of your requirements gathering and your evaluation need to go up accordingly. So if you want to add anything to, again, hopefully Stacy will be able to rejoin but from our past. I think I'm back. Oh, you're back. Okay. Am I back. Yes. Sorry about that. I apparently Comcast shows today to work in our neighborhood. I was hoping that wouldn't happen but sorry about that. So, yeah, so that was one issue was really drilling down on the partners technical requirements and access. Someone was a budgeting issue that you know that John mentioned earlier on really drilling down into the issues of licensing that would that would be, you know, ongoing from, you know, after the project was done so we had, you know, we have done an RP process we were very clear on what the development process was going to cost us. And we selected a vendor who wanted to use Tableau for the visualization part of it which made total sense to me. I don't know if you can hear us but your audio just cut off. I apologize everyone. I guess, Jay, while we're waiting for Stacy's audio to come back do you want to share sort of, you know, kind of relevant experience or you know to what you know Stacy was talking about with hidden costs and I think the, I think you know Stacy was going to say wasn't she wasn't aware of the ongoing Tableau license fees that they would have to pay. We've had kind of a similar situation recently actually where we had some data visualization software as well that that Minnesota went with, and then we recently discovered the monthly cost is going up like double. So even if you are as prepared as you can be in like understanding the ongoing can always have surprises and have have those costs increase. Yeah. And I think in some projects that we've been involved in we have gotten vendors to agree that you know what's the cap what's the most of Stacy's back but like just to make sure that you know that the inflator can't be like 30% or 40% because once you're involved in this software, it's really hard to switch out there's all the all those transactional costs Stacy I'll be quiet and pass the mic back to you. Sorry again and Jay hit it right it was the Tableau licensing that I hadn't really looked into I hadn't. I'd use Tableau, just, you know, a little bit but I didn't realize that you had to have kind of a minimum number of licenses that you were committed to I didn't know So just didn't do my due diligence on the licensing model and figuring out what the ongoing cost would be. So that was one issue and then the other issue on Tableau was the vendor had opposed using that as a standard visualization visualization platform rather than building something custom which made total sense to us, but it turned out over time that that wasn't really a core expertise for them. So they were prepared to set it up, but then a year later when we came down to kind of sometime later when we were wanted some tweaks or wanted to figure something out. There one Tableau person had left and they hadn't replaced that expertise and so we just didn't weren't able to get help from them on that so that was a, again, and a kind of a somewhere where I should have talked to them more about how how well integrated the Tableau piece was in their core offerings because it turns out they're really, you know, very strong development shop but Tableau part was a little bit of an overlay that one person had expertise in. So lots of mistakes in that project. The other project that I wanted to talk about was much more successful and is, you know, being used as an important part of our work today and that's an online intake project, and this goes more into the management pieces that Homrose was going to talk about. And that I think it was pretty a pretty good process in many aspects of the set up of it and that our vendor, you know, really made it very easy for anyone to test it and access it and in development and also to put in their comments you know he created a spreadsheet that everyone could add to and he would actually respond directly to each comment in there. So it was a good involvement of multiple staff people, some who are kind of formal testers and others who were just invited you know anybody who wanted to could participate in testing the product as it went along. And so I think all of the development pieces and the identifying bugs and just user interface issues that didn't work like people thought they should. All of those pieces I thought went pretty well. So our biggest issue in the change management wasn't was really not talking through with people. Why are we doing this and what does it mean we got the why are we doing this piece which was, it was a it was really a coven response we know we moved from almost exclusively an in person model where clients were on the whole filling out their intake sheets in the office sitting in the reception area with a little bit of help from receptionist, especially if they were language or literacy issues. They were filling it in themselves and then it was getting input later on, again, by typically by the admin staff, and we would move to, you know, essentially a phone based intake system, where the program frontline staff were the ones who were suddenly doing the intakes, and they said it was taking them an hour or so to just get the basic intake information, you know, previously the client was just doing it sitting in the waiting room. So it was a huge time suck for our intake staff. And that that was really what we're trying to address and they got that part of it we communicated that clearly like this is we're addressing a need that you've identified for us. But the piece that we missed was the piece around just because the client is now entering the information directly. It doesn't mean that you just hit accept when it shows up in the case management system as a new intake, right, we didn't talk through with people. You're still you as the intake person are still responsible for the data quality. Just as if you had taken the information on the phone and the client had said well my monthly income is $35,000, you would have said are you sure you don't mean yearly income. The same thing if you're if you're hitting accept for the data into the database, and it's creating a new case and all of that information is in there. You're responsible for doing some kind of a basic common sense check does this make sense is it consistent with this client who's told you they're on section eight. And is it, you know, is the birth date does it make sense if you sound like you're talking to somebody who's you know well into their 80s, and their birth date they gave makes them 10. Does that make sense. And we didn't talk through that with people I think there, there was an assumption that while the client typed it in so it's fine that's the basic data. And that was something we should have just done a much better job with just talking about, and I think just abstracting a little bit to why does data quality matter right why does it matter what's in the database. And a lot of that is client based right you need to know who you're working with and if you're doing especially if you're doing an eviction case, it's actually important to know what their real income is, as well as you know grant reporting and other funding. You know a lot of the data that we collect is does map to funder requirements or deliverables. We do a good job of that overall thinking through, what is this going to mean who's responsible for the data quality what our expectations for the line staff as they're reviewing. So we did a good job of saying this is what it's going to look like this is what the screen is going to be like this is how you hit the button to push it in. This is how we're going to eliminate duplicates and sort of the technical aspects of it, but we didn't really talk into what you're responsible for with it and what our expectations are, what you will do with that data. And that was and that was a mistake on our part. So it's so it's like you got the right solution in a sense you've got the the the software selection that the implementation, but you anticipate that that we're changing our workflow and and and are not just a flow but our process right like we had another step that quality assurance that you didn't have to do quite the same way before, where it was implicit going from paper you see the client in front of you and you, you now start entering it and you can ask those follow up questions more naturally. Yeah, again it's like these are these hidden sort of challenges. And so, sorry, Stacy, any other points before we kind of move a little bit more on to to homers. I mean, I think we got we've addressed some of the, the change management, you know, planning and anticipation, but anything else you want to add before we move. Just to follow up on what you just said you're absolutely right it was a workflow mapping or a process mapping issue where we should have done a better job of sort of mapping what the different steps are so not just like hit go here but there's actually building in that formal step of, you know, look at the data here are the things where clients typically make errors when they're doing this kind of an input data input, make sure you're reviewing those key parts, then you hit go or, you know, whatever that is, so it's really making sure that your mapping is really very is very clear. That's good point. Great. So, I, again, we'll, we'll ask, if she wants to share some, some of her comments and the resources when we publish the webinar with with Shelly's help, but it's once seems like whenever I advance that slide that next slide it goes has a mind of its own to go on to the but I don't know Jay anything else or Stacy you want to add on change management or project management. I think you both you both brought that into the into the discussion but Yeah, I think Stacy's story made me think of one one story from my past kind of related to that change management piece and kind of what you may or may not anticipate so when I first started at state support almost 10 years ago, it was in charge of our online systems in Minnesota and we had a previous version. And I remember I was talking to one of the intake staff, one of our partner organizations, and I learned that for like a couple of years, our online intake system had been hadn't been importing into their case management system like it had been designed and so these poor intake folks had been grabbing this XML file and then transcribing the relevant information into their from this like code into their case management system and you know needless to say they did not like online intake did not think that it was saving time at all because it wasn't but it just made me realize and I don't think we've gotten this perfect by any means but like the importance of also thinking about like, as this is implemented, that people know what to do when there's a problem. They know who to reach out to because it was like a centralized system that was sending data out to different organizations case management systems and every at that time we were all in different case management system so everything was a little different and folks just weren't sure like do they reach out to their manager do they reach out to their case management system vendor do they reach out to asset state support. And so a lot of things were breaking down without anybody ever knowing and then, of course, it makes for a terrible for experience on the intake staff worker side so just one other kind of issue to think through as you're thinking about what are the processes that are in place, as you're preparing to implement a new software. Yeah. Stacy anything else you'd like to add. The one thing I wanted to add I think hammer was going to talk about security considerations, it's to some extent so I want to just add one thing that you know one thing we found is that sometimes you end up losing some functionality or maybe you know not choosing the cleanest solution sometimes you're giving something up in exchange for in our case you know HIPAA business associate agreement or something like that, especially being you know relatively small program. It's not everybody's willing to kind of negotiate with us on a kind of a, you know, on a one off basis. And if we need that business associate agreement, and we need something else from thinking of our, you know, this isn't even software right it's our phone system. And we know we can't, we know they can't do a business associate agreement or SMS and we said well can you just exclude it from the agreement right we'll do a business associate agreement around the phone stuff. And then can you exclude the SMS, because we need our clients to be able to text us we understand that's not going to be covered, and there's a very limited, you know group of vendors who are willing to do that. And so we ended up with a solution that we don't love, but that does allow clients to text documents to us, and also allows us to have that HIPAA protection on the phone side so sometimes you do give stuff up in exchange for that those security considerations are that's really compliance I guess. And knowingly doing so is a lot better than finding out after the fact and so again like, I think part of what we've been working on this topic I think part of what we found is we needed to both dive deep into, you know, like the particular environment compliance the vendor analysis and research and demos, and also broad we need to think about sort of what are all the other factors around change management around the context our partners that we're going to work with our vendor staff sometimes in their capacities. How we do the project management that these are that you can't to be successful you really have to be considering all these other environmental, I guess factors in addition to selecting a really good tool. So I think Stacey your point about building out, you know, a good online intakes, you know, tool was great and, and, and the thing that could have been better was how you thought about these other factors of that quality assurance that's going to be related to that workflow or that or that business process that you needed to change. So they're inexorably linked I think you know we've got we've got to be sort of thinking about this holistically, maybe the term used too much but, but, but how holistic how broad and how deep is there's not a one size fits all I think what we what we I think what we need to read on is that it depends really on the project on what you're trying to accomplish. But, but to do that analysis that the, at the, at the beginning, and make knowing decisions and document it. Whether you're using an Excel, I think Stacey your source, you know, or, or even it was a Google doc or Google spreadsheet, you know, like a very basic tool to sort of come up with your list of requirements. And I think things I would, and, and I think they're, there's a fair bit of this going on already in the community but I would suggest is that if you are selecting some particular software system is to ask others for their selection process documents or their RFPs and I think Ellis NTAP has some of that. Shelly is that fair. Some of those resources. I'm not sure. Be happy to check it but we certainly have the archives from the lists are there's certainly RFPs available there. As far as I've, I've never seen a vendor selection worksheet. But that is something that I'm always looking for things that we can provide for the community. And I made note a few moments ago about a list of security questions to ask vendors. Those two resources would be something that we will start considering putting together for the community. And I think if you like I just did some googling before this around, you know that vendor selection workbooks and stuff like and so there are some tools out there they're, you know, I think people typically want you to give them that your name and email address before they share that resource and so you know but but but having a more formalized process and then we want to share a few resources and as you can see on the screen that we think it would be worth again, not just going to Ellis NTAP.org but certainly emailing out to the list if you're starting a process we see that pretty routinely people are saying what they're about to do and and so you get some peer support. I think even I guess sort of tech support if they reach out to Ellis NTAP for some tech support American Bar Association has had, you know, a big focus on legal tech for years and so they've got some, I think some pretty decent resources. You know, each past, you know, to grants, and so on so, and then again going to tech conferences state or national and, and, you know, learning from others and, and I think, you know, again, one of the biggest collaborative efforts I was a part of years ago was that that's again how Stacy and I met the health consumer alliance in California and, and part of what I think you were able to do say see together was a lot more than you would have been able to you know like because you could resource pool the dollars together do that selection process, have done that integration, and you ended up with a solution I think that was much better than the, than the prior iteration right that that, you know, I just remember people waiting hours for a basic report or maybe that's exaggeration but way too long to get some basic data that they needed every time you know there was a change in, in a requirement it was, it was a real pain to, you know, push out those changes so so again working collaboratively being very you know methodical generally leads to a better result better economies. Any other. And so we're going to publish. Sorry, I just want to. So, if you have questions for any of us. Please, I guess feel free to reach out. Everybody agreed to share their email addresses. And again, one click and it just keeps going. We're again going to post the. We're going to post the PowerPoint deck and some additional resources after the webinar. If you do have some, I saw some some great comments in the chat said if you have some other questions or thoughts you'd like to add please. Please do.