 Great, welcome everybody we are talking today about understanding document assembly so this is a relatively entry level look at what document assembly is how it can be useful for you and then thinking about the idea of how to actually go ahead and implement it so what the issues are potentially in the realm of the human side of things and that type of stuff. So I am Laura Quinn the voice that you're hearing out of the ether. I am the director of partnerships and knowledge at ideal where over here at ideal where we've been doing a huge amount of work with legal aid organizations to understand kind of their needs and to look at the projects being done in the in the TIG world to then with LSE to figure out how best to support legal aid organizations. And I'm excited to have with me Chris Alfano Chris you want to just quickly introduce yourself. Sure. Can you hear me. Yes. Okay, we're having some trouble with audio earlier so my name is Chris Alfano I am the automated document producer at Illinois legal aid online. I've been in that position for about two years. So I am responsible for all of our automated documents. And yes knows a ton about especially about kind of the process and the people part of document assembly. All right. So let's just start with what is document assembly. What does this mean. So to start with this is if you are thinking about technology investments for your organization. Generally the experts say that this is one of the best investments that is likely to have a good return on investment for you. So in a bunch of different ways it saves you time and money in assembling forms. It can improve outcomes through the accuracy of form and especially by allowing folks who might not otherwise get representation. So might be over the threshold for you to represent them or might be more simply more people than you can see can help them get a good outcome for themselves. So the overall concept of it and we'll go through in more detail is that a somebody who needs help with a form can go through a user interface like what we're looking at right here is a to J. Access to just this firm pro bono net to fill out a bunch of questions and it generates a form for them. So this process. So the idea of document assembly in general. I think we often think about pro say litigants and folks who are otherwise not represented when we think of these forms but they can also be really useful to more professional staff. So your actual attorneys your paralegals people outside your organization but the people who might be able to be trained up like navigators or social service agencies. So it's a really interesting tool to think about not only can we make this incredibly easy to use for someone who doesn't know the law or this matter at all. But also can we make this something that speeds up the work of somebody with some at least some background in this area or potentially a ton of background in this area like an attorney. But in general a lot of people are thinking of this as a great way to help people who otherwise can't be helped by their own legal services. So the idea of helping pro say litigants. Chris can you just talk a little bit about kind of how how you guys over at Illinois Legal Aid Online think about kind of the helping of pro say litigants here. Sure. Yeah. Most of our interviews are targeted at pro says. So our main objective with those is to make it easy for them to understand and easy for them to get through. So everything has to be broken down as much as possible. We try to keep them at a fifth or sixth grade reading level and we try to educate them a little bit about the issue as they go through the interview. And we've had a lot of positive feedback from judges who think it's great that the forms are filled out correctly and litigants know a little bit about the issue before they're appearing in court. So I think it's worked out pretty well. Great. And in the whole scheme. I'm still getting a little bit of feedback for when you're not talking with my meeting and see if that deals with it. But we're clearly getting a little bit of feedback still. Great. So Chris when you are dealing with so when you are thinking about pro say litigants in general how is how does document assembly fit in with other techniques. So you're obviously providing a lot of information online. You're obviously doing other things. Is this a big part of your strategy towards pro say litigants. Yeah I would say so. It's definitely something that people associate really strongly with our organization and get a lot of positive feedback on it. But we do have the website where they go and they learn about their issue. And from there they go and fill out the form and their self help centers throughout the state of Illinois where they can go and get assistance from a law librarian or a law student. Some volunteer who can help them get through the form if they maybe don't have really good computer skills. Fantastic. Great. All right. So let's think about just kind of a few other issues here. This is also a really useful step in maximizing attorney time. So with the idea that it doesn't make sense for your attorneys or necessarily even your your paralegals to be filling out the same form over and over answering the same questions over and over to instead to maximize their time to you know create a brief to appear in court to do all the things that really only an attorney can do. Another reason to think about document assembly is it allows you to save and reuse the documents. So the idea of being able to if folks are entering information into a form and creating a document you can potentially depending on what you're doing save that document into a case management system or onto a server. You could potentially save the information that the litigant has entered in order to be able to reuse that again in a different form later potentially. So there's a lot of potential kind of economies of scale of doing this. It can also be a step towards court standardization. So obviously we've got a ton of local courts which think about things somewhat differently in having standardized forms and working with some courts or many courts to make sure that they will be accepted. It takes a small step towards the standardization of what the courts are asking for and hopefully weaning down the courts off of having completely different things that they require from every person. All right. So that's kind of what document assembly is. Can you. Sorry. Do you have any questions in regard to kind of how it is or why you might want to use it or anybody have thoughts as to just kind of the overall return on investment into the chat or start six if you're on the phone to unmute your line. Chris would you mind. Sorry. I think it's definitely important to point out that any variables that are put into one of these can also be put directly into many of the case management systems. So A2J author which was highlighted earlier from Chicago Kent. Those things can go into a form via hot docs or they can go directly into a case management system or into something else. So there can be a lot of time saving if that would normally need to be entered or collected by somebody that might give to you in advance. Absolutely. Great. There are questions or thoughts about just kind of what document document assembly is or why you might or might not think about using it. Let's actually take a moment to see if we can get some thoughts for from the folks on the line. Can you just put in a quick thought as to either what's the core thing that document management is helping you with. Or one of the main things that comes to mind that it might be able to help you with. So just in 10 words or less kind of put what this what resonates to you into the chat. Well we're waiting for people to type in. Chris do you want to just kind of say what comes to mind for you. What is the primary. What would be the primary thing that you were to lose if other than your job. If if Illinois like late online were to say we're no longer doing this. This is no longer a strategy or technology that we're using. Well I guess if we were no longer doing it then all of the individual legal aid organizations throughout the state would have to develop their own interviews. They wanted process to be able to use those. Are you talking about what we as an organization would do differently if we didn't have automated documents. Automated documents. Sure. So I think we'd be a lot more reliant on volunteers to help litigants fill out the forms. This past year I think we had something like 50,000 assemblies on law help interactive. So that's 50,000 people that we help that would be a huge amount of work for volunteers. I mean they help the percentage of those people but if people were not able to do this on on their own at home. I think it would create a huge burden on volunteers that we have. Absolutely. Great. And we have a thought from Melissa. So creating resources for follow up from walk in clinics is one of the things that really comes to her mind that this is either is or could be useful for. Also the ability to create standardized forms across the state. And Michelle says at L.A. Superior Court we are planning to use guide and file to integrate into our new case management system and ease the burden on our self help centers. Absolutely. So instead of someone coming to a self help center sometimes they can help themselves. Jessica says New York state courts have survey results from 14,000 pro se users of their do it yourself form made an A to J author over the past five years about 90% of those survey results are positive and say things like it saves time it saves dollars very thankful show that these really work for pro se litigants fantastic thanks so much for sharing that that's really useful. Yeah, something Michelle's point really brings up that a lot of different court systems and the vendors for case management systems are seriously looking at this on large scale. So if legal services organization can get involved with those processes as they're going through kind of their administrative officer the courts or their clerks that type of thing. They need to be able to leverage those communities to do this type of work. Instead of doing it in themselves. Great. Fantastic. All right, so let's move onward and talk a little bit about how it document assembly works. So I'm going to run through this fairly fast because I suspect that most of you have some background here but if you if I'm going too fast certainly ask questions or let me know that. So for core parts of a document assembly system and we're going to go through this in more detail. So there's the user interface, which is where the information goes in to create the form. There's the document template so the template that is the kind of the standard of the form that things are then added into. There's a data file which is created out of the data enter the data entered into the user file and that data file is then merged with the document template to create this individual document that can then be filed. So just going back through this one by one, you've got the idea of a user interface that potentially wouldn't have to like it could have some less easy to use but faster interface for lawyers, but often it's a very easy to use for average people to actually kind of interview them to create the forms. So we were looking before here at law help interactive you can see it's basically giving you a pathway and helping you to to navigate this child support modification plan. So then moving on from the user interface there's the document template. So this is the template itself that is going to essentially be what things are merged into. And this could literally be or at least is conceptually similar to merging into a Word document so you've got a bunch of fields that need to be customized to a person and they they go somewhere in a particular form. You've then got the actual so coming out of the user interface you have data which is stored prepared for assembly. So here for instance we have the answers from a particular constituents to all of the questions for the forms. And as Brian was saying this is certainly we're going to in next in this step merge it into a form, but you could also do other things with this data you could store it in a case management system you could create multiple forms. This is in fact a data file as opposed to just a just a document. And then there's a step in which the document template and the data file are merged to get the actual documents and so the document is typically presented so it is if you're doing it online it's given to the constituent as a file. Sometimes it can be files depending on exactly what it is or what you're doing to finish up and to wind up the process. So that's the overall process that of course begs the question of what are the software options to do this any questions on kind of just those particular steps in the in the process before we move on to what kinds of software can help you support it. Great. Certainly feel free to enter stuff in the chat at any moment. If you have questions. Alright, so software. So a to J and hot docs have kind of our first to mind in this realm and we will talk a little we'll talk a fair amount about them, but we wanted to talk about what else is out there especially if you're looking to get started with less of an investment. So for the user interface basically anything that allows you to ask questions will work as long as it allows you to support data to export data. Like building it into your content management system like Drupal so it basically will take information and create a data file. You would then need to actually merge that information in with something like word to create a document so something like a form and word might make sense if you are if you're asking folks to for instance fill out some questions and have a form ready for review with a lawyer before coming in. So there's a divorce case that they want to talk about a lawyer with you could ask them to go through the process. The form could be merged people the attorney and the constituent could look at it together and save a lot of time as opposed to showing up with with nothing. So this is probably not the idea of a form and and word is not really very practical for the idea that it's going to be completely self serve that it's going to be able to be presented online or something like that but can be a really interesting option for people who are going to be coming in and talking to a lawyer or to the help center anyway. Brian we were chatting about this scenario at the beginning of the kind of before the class started. Do you have thoughts to add as to kind of when this might make sense as opposed to a all out much more expensive investments in document assembly. There are several programs who have done some customization on Drupal and Drupal is very powerful. The ability to take forms in and then put that out into an XML or something else that is then pulled into hot box or something is definitely there. I would also anything smaller scale if you just want to give it a try and see what kind of feedback you can mock up a very simple form that pulls some information with Google Docs and then have that information available when talking to a client or someone else. It's just the ability to give them some little self help references that are next to questions so they kind of prepared on the way in can help save time. Fantastic. Great. So moving on to so we actually so Brian talked a little bit about Drupal. So this would be similar to the idea of creating it in a Google form. But you can have more branching and more complexity in most content management systems. So something like so this is a Drupal is a system that will help you to manage your website. So Drupal or WordPress or most of the systems that you might be using will help you create more sophisticated forms. Document assembly is often where it sometimes bundled into case management systems as well. So it's certainly worth looking at what is available within your case management system or what could be what could be an additional module that is is available at extra cost in the case management system that you're using. All right. But as I mentioned in this space a to J author has become a really strong option for a lot of different people. Chris can you talk so you are certainly vastly more familiar with this to all than I and possibly then most people in the world. Can you talk a little bit about why a to J author makes sense for you and why it might make sense for other folks and maybe just a little bit about it. Sure. So a to J author was developed by Cali at the at Chicago Kent Law School. It's a great tool for pro se litigants for any process you want to break down really step by step. The interface is really comfortable for people who don't have really great computer skills really great reading comprehension. It's very easy to use. You can learn it in a matter of weeks. It's very intuitive. I don't know I can't say enough good things about it and I'm sure. Jessica was unmuted. She could say even more but it's also free. So that's what makes it attractive for legal aid organizations. And it works with hot docs to merge those answers into your document when you're all done. Great. Yeah. I don't know this much more I can say. Can you talk a little bit about kind of the hot docs licensing so a to J is free but hot docs is not. So if you want to so a to J is essentially the interface portion of this. And if you want to assemble documents you will you will need hot docs with it. Can you talk a little about kind of the ballpark price that you are paying for for hot docs. So I don't know exactly what it is. I know that you're probably looking at close to a thousand dollars for an individual license for nonprofits and government organizations. They do offer a discount. I know they have a free trial available as well. But I don't know the specific cost of it. So in general the idea is that a lot of organizations will be paying several thousand dollars for hot docs that you potentially can get started less. But it's not it's not a trivial investment but it is probably a very worthwhile investment for you know if you're going to take the effort to do all of the you know the rules and the work to get this up and running which we'll be talking about in a little bit the investment for hot docs is unlikely to be you know that much compared to the investment of lawyer time and staff time to get it up and running. Fantastic. There's also things like so you could use hot docs without a to J. And there's other things like it like rapid docs or deal builder etc that basically will have less of a detailed user user interface. But have potentially so pretty sophisticated and potentially more sophisticated logic for forms though hot docs is pretty darn sophisticated in of itself. Chris can you talk a little bit about so I know you use sometimes you use the interviews through a to J. And sometimes you use the interviews through hot docs. Can you talk about when you find it useful to use hot docs alone and when you find it useful to use it with a to J. Sure. So hot docs does have its own interface component. You can do the entire interview in hot docs. It looks a little bit more plain than a to J. It's kind of just to you know it looks like a fillable form. It's like a gray background with fields that they fill in and we use it for all of our interviews that are directed at legal aid advocates. So it allows them to get through it a lot more quickly than click through a bunch of screens and a to J author because they don't need all that extra information. They already know the law. I just fill out a form. And some are more complex per se interviews like our uncompestive divorce interview are done in hot docs because you just have a little bit more control over the logic and maybe a little bit easier to do more complex things. I know in the new version of a to J author they have revamped the way you do logic so maybe better suited to those things than it was in the past. Fantastic. And Jessica through the chat has mentioned so Jessica who is with a to J author that they are working on their own document assembly back end right now with the plan to deploy it in 2016, and that will be free as well. So it isn't going to. So she says it isn't going to completely replace hot docs for everything, but we plan to cover many forms that legal aid or legal aid or generally automate. So the idea of kind of moving off the hot docs license model to allow that to be free for organizations as well. So that's it's terrifically exciting news. And I believe Jessica am I right that that is supported through LSE. And is is one of their in fact they're great take innovation grants yeah she she agrees with that. Fantastic. Great so kind of. So this is the. So I think I had one last kind of one last thought of, are there other things so if you are using other things like Tyler technologies, expert systems, things like that. So basically if you're using even more complicated systems already, is there a way to adapt them to support document assembly because they may well be. However, for most of us document assembly would be a technology that comes before most of these more complicated technologies. And so it doesn't really necessarily make sense to to start at this end of things. That was mentioned. There's a question here about it. That was the retail last time that I looked for hot docs hot docs is went through three different iterations of ownership. They used to give it away for free to legal services organizations. I would contact them directly let them know that you're a nonprofit legal services and try and negotiate them down. I know organizations that have gotten a license for significantly less than that. They just don't want to put that up publicly which I so you got to try with them. Yep. Yes, our experience was the same so they are not publishing nonprofit pricing, but it does get lower thousand dollars. Hi, Liz. Hi. This is Michelle from LA Superior. So this is kind of you're looking right now at Tyler Technologies, but they have released a product called guidance file, which is basically an interview function to do document assembly and we are rolling it out in 25 counties in California. So that that'll be an interesting kind of trial pilot project with guidance file though it already exists in South Dakota too. Great. And can you tell me tell us a little why you've chosen guidance file compared to other things? Well, it's because most of the courts or 25 of the courts in California are using to the Tyler case management system. So guidance file naturally integrates with file and serve the E fun E file function of the CMS. So it'll be kind of a seamless project where you can use guidance file to fill out your documents E file them and then proceed with your case that way. One of the other things that we're doing here is we're also working with hot docs to make sure we have standard questions, whether you're using hot docs or guidance file. So that's kind of exciting that we're standardizing both of the systems so that they kind of match no matter what county you're in. You'll get the same type of question and it's very much standardized in that regard. There is the ability to build a way for hot docs to also E file into Tyler, but that requires an API development. And that's about I think last time I heard was about $75,000. Yeah, yeah, for hot docs. Yes. But for guidance file is completely free. And if you have a fee waiver, you don't pay fees to E file or anything else. Great. And do you have a sense as to what the price point for guidance file is guide guide and file is free to any county and any, any, any organization within that county that's using Tyler. Okay, they have not developed a price point yet for counties that aren't using Tyler for a self represented to go on fill out their documents using guidance file and print them. But they're looking at making that available for free. Fantastic. Thanks so much for sharing that that's super useful. Anybody else have thoughts of so either questions about any of these technologies or thoughts of other technologies that they've heard folks using or and or using themselves. This is a very emerging area in terms of the software that is that is up and coming into the chat or if you're on the phone line you can hit start six to unmute yourself. And a couple of things that I've seen is willful forms. Some programs have been using willful forms for a little while. And other people are you mentioned the innate case management system forms. I know that some programs love using the legal server. For example, intake interview Mario and to do online intake directly into their CMS. So those are things that I have seen out there on the guidance file piece and it would be good to point out that the courts purchase the Odyssey Tyler system. And then they get the guidance file so it's an item to the original license. Yep. And on the discount the discount if anybody there was the discount there was in 2010 hot dogs made a commitment to make the hot dogs developer licenses to create forms available to the legal nonprofit community. And so if anybody wants that original letter that explains the discount of 70%. Let me know and I can send those to you. And so the discount can be a 70% a deep and generally you only need one hot dogs developer license per legal aid group. Unless you're going to have two developers in house and if you hire a contractor most contractors have their own license. So you don't need to worry about that. Are they still honoring that agreement through the ownership changes my understanding was that they weren't. It's your understanding differently. Yeah, if you read the letter carefully they gave about $2 million worth of free licenses that lasted up to hot dogs 10.2. That is about five editions behind. They said that after hot dogs 10 they were going to provide a 70% discount. And as far as I know they're still honoring the 70% discount. Fantastic. The market value is 1000 the license is going to be 70% of that people need to call and requested. But as far as I know they are honoring the 70% discount. It's not free. It's a discount. But still absolutely that is fantastic knowledge because they are. If they are honoring it they are tight lipped about it. So that's great to know that if you if you have that inside knowledge potentially you can get that 70% discount. That's great information. If you're willing to send the letter to me I'd be happy to write something up about it because I'm all for holding them to the. Fantastic. All right, so let us move onwards and talk about kind of the organizational stuff around this so obviously there is software that is useful. But it's not going to magically make things happen. You know it's obviously not going to write your interviews and create your documents for you. So you want to think through what else is going to be of importance before you say okay great we're going to get this free and low cost stuff to make it happen. Starting with the idea of thinking through who's going to use these forms. So they, we as we talked about they can be useful kind of to many different people, but different groups have quite different needs. So, for instance, if you're looking to support your attorneys or your skilled folks, though, then you really want to think through speed auto fill stuff like that. So the idea of having a, like for instance, a wizard that's going to easily walk you through the questions that need to be answered is probably going to be really annoying to someone who really knows what they're doing. They want to fill things out in mass in order to create forms. Or I know that Chris has mentioned that this has been a tricky area for them in general. Chris, you want to talk a little bit about supporting kind of skilled staff or volunteers with these forms. Yeah, the top goal is just getting them through that as quickly as possible. The whole point of this is to cut down on the amount of time they're spending preparing these forms. So the biggest problem for us is we provide these interviews to weekly organizations throughout the state of Illinois. And they all have different documents that they use. They all have a different way of doing things. So it's really hard to please everyone with these interviews. So that has been the struggle for us. Yeah, so you find that the, it's hard to get the lawyer buy in to actually standardize that they don't want to, they want to kind of have their own particular thing rather than using the standardized stuff. Yeah, so that's right. I think if you were doing this as a legal aid organization and using it, your own, your own attorneys would be using it. I think that it'd be a lot easier to get them to buy into it because they'd be able to just do things the way they've been doing it. Yeah, and certainly either way, the buy-in becomes super important. That's even if you're feeling like, oh, well, this is going to save tons of people tons of time. Really important to get people's buy-in. And then it's actually going to save them time and it's what they want as opposed to foisting it on people. Or it could well, it could well, one, not actually save them time. Or two, it could, even though it would save them time, they'll decline to use it. Michelle is mentioning, sorry, go ahead. Oh, sorry. Just real quick, one advantage of it is that you get the entire form packet, so you're not going to forget to do any sort of little procedural forms or addendum or anything like that. So it has an advantage in that respect. Great. And Michelle is mentioning that when she was with LASOC, we were using smart forms for a diesel for advocates to complete forms. Michelle, you want to say a little more about that, so what your experience was there and tell us a little bit more about smart forms. Sure. We, as part of a project that was funded by the Orange County Court, we had developed a smart forms packet for Dissolution Parts 1-2. We didn't have the three at the time I was there, but I believe they have three now. So what we did was we used those in our workshops when we had assistance there or paralegals there to help people complete their forms. But we also had our advocates using those forms so that they could easily e-file without having to pay an intermediary e-filing provider. So those are just the forms where you fill in the answers and then it does an auto check at the end. It was very useful for our advocates to use that just because it was just completing a form and doing an error check. Great. And did you have any issues with buy-in with that or were they generally feeling that it was definitely a time-saving thing for them? Well, honestly, people think a lot about Orange County as being an advocate for technology, but we did have a lot of problems with buy-in because people liked using legal server or the other products that they had before. So yeah, we did have a little bit of problem with buy-in, but I think once we had enough of our technical staff really walk attorneys through doing that, then the buy-in was much higher. And we did really take their feedback on how we can make it easier and incorporate that. And it was really good. It actually worked out really well after about a year of investment in getting people on board. Great. Perfect. All right. So let's talk a little bit about self-service users. So these are folks who really do need, and we've talked a little bit about these guys before, so they are really going to need help to navigate, to know what to do. And in fact, one of the advantages of a, like, for instance, an online document assembly system for pro se folks is that they can't or even just folks prior to when they see a lawyer is that there can be help tools along with the document assembly tools. So there's not only, there can be attached to the form in more information, so to go read an article about this particular things to choose which of these options you want. There could be live chat to actually ask questions about what you should do while you're filling it out. So really the more you can do to help them to navigate the system, the better. And Chris was mentioning having things at a fifth grade reading level and really thinking that stuff through. Chris, just out of curiosity, do you tend to user test your forms with constituents before you put them out in the world? Not always before. We do try to do it at some point. Usually we'll do usability testing maybe a couple times a year and we'll bring in a group of students or other volunteers and they'll test some of our newer forms and give us their feedback. We do have feedback from the self help centers throughout the state, so that's coming in all the time and it's great to get their feedback as well. And what types of things are the most common types of feedback do you get? Is it the clarity of the terminology? Is it kind of branching? What typically are the things that come up that need really careful attention? It's usually difficulty understanding the process and sometimes difficulty understanding what the question might be asking. So we do try to educate the user a little bit about the process so that they know what to do after they get the form. One thing that we've been doing now is we're trying to include a short instruction page with the forms telling them how to file, what to do when they get to court, like how to check in and how to behave in court and stuff like that. Fantastic. Great. So it's also in the thinking through what you need to do as an organization. One of the actual biggest pieces of implementing any document assembly process is thinking through the logic. So thinking through what are the questions that you need to ask and where is their branching and if so where does it branch in order to get all of the information you need. So there's often at least a thought process behind it and for a more complex form you might actually have a whole set of essentially logic rules behind your system. So I think it's really important to allocate the time for that piece which is substantial. And very related to that you want to think through all right what documents are you going to start with. So you probably don't want to start with some super complicated form that is going to be particularly hard to understand that instead you want to start with a straightforward form that is is used a lot. And potentially in one of your areas of case law where your attorneys are more excited about the process. Chris, can you talk a little bit about how you guys decide which forms are the right kind of the next priority to get into a document assembly process. Sure. So our help centers often will request that specific forms be automated. It's a little bit difficult because we don't have too many standard forms for the whole state so each county has their own little forms they use for each different process so we generally focus on the larger counties because we can help more people that in that way the state is starting to create standardized forms that are specifically targeted a process so each new packet of forms that comes out of that process we will automate it as soon as it's released. Fantastic and what types of criteria are the the self help centers using you think to define which are the most useful forms for them. And this is what they get asked for the most. So maybe they get asked for eviction complaints or termination notice and we don't have that automated and they'd like. It's so much easier for them if it is automated because they can just park the pro say in front of the computer. They start filling it out and then they don't have to do the whole thing with them so. Fantastic. And then you want to think through whether someone will actually check the document so is there going to be a paralegal or a legal lawyer actually reviewing these documents and this is a question that. It tends to be potentially somewhat controversial in organizations as to there's obviously a lot of time for someone to check them but sometimes forms are such that it is worth a double check or to ask somebody to potentially if they have any concerns to bring it into the self help center. So this can be an interesting process and things that sometimes can be part of the attorney buy in process that you may find that you're assuming that no one is going to check it check check them it's just basically a pro say process. And you may find that you have attorneys who are horrified with that idea and we're assuming that everything would be looked at by a lawyer before it goes anywhere. Let me think about this one I know that you guys don't don't review any of your forms or is there any word that was anyone who was advocating that any of these forms should be checked. And not that I'm aware of that we really make it difficult for us to help as many people that we do. I think it also starts to bring some of the unauthorized practice question. If you're reviewing it then you've seen it and it's like a step closer to representing that person that you were before so I don't think we have any plan to start doing that. Great. Brian do you have any thoughts in this particular area as to kind of how to think about this idea of checking or not checking. No it's a great question. No it's a great question. We've worked some with the housing clinics here that are at the courthouse and also with the law libraries. I think letting people opt in and have a resource as part of the document assembly for checking is definitely important and it's a great way to get intake and to also let people self screen out who don't actually need help who just needed access to the form so I would put it in part of the help or intro as you're putting together a form. Or if there's a more complicated question that you know is going to be difficult give them the option some information there when they get to that question. Great. Thanks guys. All right. And then last but not least, thinking about the always interesting working with the courts. So as Michelle has been talking to us about sometimes this is driven by the courts. And that can be useful pretty useful, especially if it's a big court system. When you are thinking about, for instance, automating documents statewide, you have, you need to make sure that you're going to get at least most buy in from the biggest court systems that they will actually take that document because otherwise you've obviously got a document that in fact is not good for anything. So this is an issue that a lot of people face in thinking through there really needs to be at least some partnership and buy in with the courts in order to make this a successful process. Chris, again, you want to just tell us your experience with with court buying. Sure. For the most part, it's been overwhelmingly positive. So the statewide forms that Illinois is developing that was actually a commission that was started by our state Supreme Court. So they are fully behind it and one of the requirements for those forms is that they are automated by a layout after they're completed. And just our existing forms, my personal experience talking to judges is they love them. They really appreciate that these forms are filled out correctly, that the litigant is at least prepared when they come into court and they don't have to send them out to the law librarian and have to see them again and drag on the process even longer. Often judges say positive things about our forms before I even tell them that I am the person who is responsible for creating them. So I think they have a pretty good opinion on them. Great. You have mentioned that you've heard a few isolated incidents of courts not accepting the forms. So what's as to kind of what is often the stated reason for not accepting them or what kinds of courts are less likely to take them? Usually it's some of the smaller counties. So every county has their own way of doing things. And if they don't like the way that your form looks, even though it has all the same information as the one they use, they might be hesitant to accept it. You can usually bring them around eventually, but sometimes it's just a matter of getting them used to it. And once they become accustomed to it, they see the advantages usually. Terrific. And Michelle, I'll put you on the spot because you've been so generous in sharing your thoughts. From the court perspective, so obviously you're doing a ton of standardization yourself, but if someone approached you to say, All right, we are looking to do this particular kind of standardized form. What would be the things that you'd want to make sure that they have your, have your buying and understanding on what would be the types of things that might irritate you and make you less likely to buy into that process? Well, I've been on both sides. So I was on the legal services side and then on the court side and working with the Judicial Council, we have found that because we have statewide forms in California, it's a lot easier to get buy-in. So we haven't really come across that problem. I think that where the line may get a little blurry is when the information that you're providing is additional information can kind of lean to the side of advice more so than just information. So I think that's a really difficult line to kind of draw in the sand as to when you're providing that additional information, how you do so. So for the California Project, being that I've come from both sides of this, that we are trying to really work with legal services and that announcement will be made in January, but if anyone's on the line from California, you can hear that now. We are going to work with legal services to get that additional information and then make sure that it's appropriate for rolling out two statewide forms. And then with the one issue we have here in California is there's a few local forms. So we're having to train up people at the courts to create those local forms and that's being done through court creation. So I anticipate there will be some partnership grants come out of this, but we don't know yet. So it's still fairly brand new. We just started it in June. Fantastic. Thanks for sharing. And actually back to the review question as to whether lawyers should review the documents that have been assembled. A Claudia with pro bono net mentions that many groups who are using pro bono net or sorry using a to J can use the built in remote review using the native email capacity and law help and send the documents to remote supervisors or pro bono lawyers for review. So that's actually a really another really interesting option is you don't necessarily need to take it in person. You could send it off to someone like a pro bono lawyer who is potentially someplace else in the state or in the county than where your your litigant is to review the form. So let us make our way into the idea of actually setting up the system. So you say great. I want to do this. I want to move forward. What are the steps involved in actually making this happen. So first of all you want to think through project management as a key element of this. So somebody who's going to think through timelines the scope the budget making sure that everybody is working together and communicate. Making sure the files are being passed around making sure stakeholders are included and they have reviews and they have buy in this could be the person who is actually creating the rules building the forms or this could be somebody completely different. There it's important to think of this as a really a central role which is about kind of the human side and the buy in as well as just making sure that things happen technically. And this is the role that Chris is in. Chris you want to talk a little bit about kind of what your job tends to look like when you look for any particular document. Like if you were saying you're only working on one at a time. What are the types of things that you do and what are the types of things that are that you get other people included in. So some of our documents I automate on my own and on others I will use a contract developer. So the two processes are kind of different. So for if I'm doing it on my own I sort of maybe create a quick outline and then just go dive in and do it. And maybe ask some of our interns or other staff members to test it once I am in that phase. If I'm working with a contract developer I'll create a scope document tells them exactly what I want out of the project. Depending on my previous relationship with that developer I might give them the question text that I want if I know that they kind of know the sort of thing that I want then I will let them draft that on their own. And then when we get the project back from them we still go through the testing and usually we just use interns for that because we don't want a lawyer's opinion as much as someone who's maybe still in law school or not quite even there who is a little bit closer to our target audience. And where do the questions and the kind of the knowledge of the legal form come from? So a lot of it comes from legal aid attorneys in Illinois who we reach out to for support and advice on the project. We of course do our own research. Most of us here at LAO are lawyers so we're capable of doing that on our own if necessary. Great. So just pointing out that there is also coordination kind of there to reach out to get the right legal help and yeah. And logically you are also coordinating with the counties that are going to use it to make sure that that is what they wanted. Yeah. And usually we don't necessarily reach out to them beforehand if it's just a form that they have available online and they might have a fillable PDF. They are happy to have us create an interview for that that breaks down and makes it easier for the prosa. I really haven't heard any negative feedback of it. If we have procedural questions from that. Yeah, I might reach out to the clerk in that county. Great. And it's important to think of this as a time consuming process to actually think about these creation of 10 questionnaires and templates. So it sounds like a lot of what Chris and I are working off of are are already existing kind of fillable PDFs and things like that. So if you've already invested the time in creating the fillable PDF, then that is part of your time invested. But if you're starting from scratch, you want to think through this is going to be not just kind of a quick developer whips it out type process, but rather something that is a reasonable amount of time for someone to invest with the idea that it's going to be returned many fold down the road. And as we've talked about making sure that you're getting the input from folks that will help you design the right thing and then also make sure that you've got people bought in down the road if you just kind of slam someone something in front of a bunch of people and you've made a bunch of judgment calls, then there is a significant danger that they're going to say well I don't agree with these these judgment calls and I'm not going to use it. And you then you need to either go back and reconsider or you need to say goodbye to that particular stakeholder. Perfect. Brian, anything that you would add here just in terms of places where people might go wrong potential pitfalls in the kind of the rollout of this kind of system. I would definitely make sure to have iterative user testing as you're going through the process seen individuals create forms on their own, have a user look at it and it really doesn't embody kind of the plain language usability best practices that really are essential to make this work. Absolutely. I actually just a couple thoughts on user testing I actually have a background in user testing way back when I was doing it for a living. It sounds very scary. It sounds very official and in fact it doesn't have to be at all it's basically saying alright we're going to sit down with this form, and with someone as close to our target audience is possible. And in fact if you, you know, cultivate relationships across organizations it's often pretty straightforward to get someone reasonably within the target audience, but honestly anybody is is better than no one. And basically say alright let's sit down why don't you think through this process as, as you would if you were actually filling out this form let's either put your so put yourself in this particular scenario your landlord has said that you have, you know, five days, hopefully five days to pack up your stuff and get out, but you have a lease that says that you're going you you can stay. So that's your scenario let's go through this form to to fill it out and you basically just ask them to do it, and ask them to talk a little bit as they're walking through it, you want to make sure that you are not really stepping them through that process which can allow you to really miss important things, but to kind of turn questions back on them for at least for the beginning so if they say I don't know what to put here what should I put here to say well what would you do if I wasn't here, or if they say I don't know what this means, what would you interpret this as you know what does this mean to you. So asking them to to think about it kind of essentially on their own with you there, and then you can of course at the end of the process go back through and actually answer their question so it's not all just like a mystery process. All right, let's just do I think we have about another five or 10 minutes till close here and then we definitely have time for your questions if you have other questions. So let's think through after installation so this is not something that's going to be just kind of a set it and forget it type of process. One of the things you definitely want to do is to think through your metrics so to make sure that you have a plan to know what's being used. People are using it where they go after they use it where they've come from and Google Analytics is a very common thing to use here so you if you have forms on your website, you can see some usage of those forms through Google Analytics. Chris, what do you guys track and what do you do with that information when you track it. We use Google Analytics to track a lot of information on our website. All of our automated documents are on LHI so we use their quarterly statistics that they put out to determine which interviews are being used the most which aren't really being used at all and then we can try and figure out why that is. So our most popular interviews are going to receive the most attention from us because they're receiving the most use and we can see also how many people are starting and not assembling a document which maybe not doesn't necessarily mean that they're getting stuck or the interview doesn't work and I mean they just don't qualify. It's also useful statistic for us. Fantastic. And are there statistics out of I think you were telling me that there's not that much out of LHI or A to J or hot docs. Yeah, are there things to get from those. Once they, and Claudia can provide money of, they provide everything you'd want to know about how many times the interview is launched, how many times it's assembled, what we can't track is maybe like what questions they were when they gave up or whether they were kicked out because they didn't qualify or whether they just got frustrated and quit. We don't know that based on the statistics so we we do have a there's the option for them to provide feedback afterwards. There's live help, there's usability testing, there are other ways to determine that but not everything can be reflected in the statistics. Great. We provide quarterly reports that will give people a sense of how their content is being used. And we have a length of interview report now that it's really interesting and because length of interview is something to keep in mind. Most people can't sit through 45 minutes of an interview session so and then we have a project that is with tick funding where we're going to see if we can figure out where people are dropping off. But it would be more complicated on once on the ones that are on flash, because we may not be able to track inside that so we're working on a project with central Minnesota to see where we can figure out exactly where people are ending stopping or coming back. So that's hopefully something that we will be able to share. And in the fall of 2016. Great. Great. So more more stats coming from from a to J in the fall of 2016. That's exciting. Thanks, Claudia. I want to think through ongoing maintenance. So you've set this form up you've put it out in the wild. You want to make sure that you have some process for coming back to it to making sure that it is up to date with both the most league of most up to date legal courts are expecting. And if you have a number of forms that could be a an actual a process that needs a reasonable amount of management to basically say alright what cycle should this form be on should we review it every six months should we review it every year. And and how can we ensure that there's not things that just kind of fall out of sight but are still on the on the site and people are using it although it is out of date. With the goal of obviously increasing access to justice so as we talked about way back at the beginning it this isn't a trivial process so this isn't something that I want to say like okay everybody should do this immediately because it's going to take you five minutes and bring all you know enormous returns it is I believe going to bring you enormous returns but it's going to take some time. But it is something that in general of all the technology investments kind of past the core you know infrastructure and making sure you have file sharing and stuff like that it's basically one of the first steps past that in most legal aid technology folks mind that is the best return investment this is not really any more an innovative let's go explore the wilds and see if there is return. No it's known that there is really good return on the time and the money that you'll invest in this in in terms of providing more access to justice so more effective help for more people. All right and then is the official end actually so let me just check in before we wrap up so thoughts from Brian and thoughts from Chris on things that are just important for people to keep in mind and as we're checking in with them this would be a great time for any questions that you have either to enter into the chat or if you're on the phone hit star seven to unmute if you're not on the phone then unfortunately you're going to have to enter into the chat because there's that way to unmute Chris just any parting thoughts for us as to things so people who are thinking about doing this themselves or expanding what they're doing parting words for them. I would say treat it like software development it is not like writing a piece of legal content it takes time you have to test it. It's a long development cycle. Even a simple form packet could be maybe 25 to 50 hours of developer time and then probably at least that much over again spent by you reviewing and testing that so it takes time but I think it's worth it the amount of money that we put into it we it probably costs us a few bucks for every assembly that we get so and that doesn't even count the people who got information and figured out that this wasn't the right process for them so it is very cost effective and even though it's just a small part of what we do at Illinois legal aid online I think it's one of the things that people most strongly associate with us because it has such a positive impact. Terrific. And Brian closing thoughts in this area. Dedicate at least a half time FTE to giving us a shot over a six month to year period doing your first form is difficult doing your second one is easier the more that are done easier it gets but if you don't put the resources towards it and you only try one or two forms it's going to feel like a nightmare. Putting in that time and getting through the learning curve is going to help so much and then you get the benefits that come from this. Great. And I think Brian if I'm interpreting correctly I don't think you're saying that it's going to take you have time for six months to a year to do your first form. But rather you should think of it as that kind of investment in doing a number of forms because you'll get up the learning curve and it will really be a much better bang for the buck. Yeah. Correct. Yes. Yeah. If somebody tries to do this four hours a week over a six month period of time it's going to be a nightmare but they could do several forms over several months develop this skill and it could then be part of your program. Great. Perfect questions or additional thoughts so via voice I will pause or into the chat. I just also definitely want to let people know that we've got some more trainings coming up including one on virtual remote pro bono legal services on the 14th of October and then on the 20th we've got a nonprofit that specifically works with connecting students or recent graduates with legal service or nonprofit organizations to help do mobile development of apps. Those are our next two. Fantastic. And I think that there's a survey that goes out afterwards. Yes, we will have to get the link out there here. Fantastic. All right so I think that we are out of clothes. Thank you so much for your time. Thank you to Chris and Brian for their expertise and special guest star expertise from Michelle and Claudia and everybody else who provided their their thoughts and knowledge here. Terrific. With that I will bid you good day and hope to see you at another seminar soon.