 Hello, this is Brian Rowe with LS NTAP. Thank you guys for coming out today. This is a brand new training for us. It is with Transcend, who does some of the absolute best plain language work in the legal sector. Very happy to have this as a new webinar for us. As with all of our webinars, this webinar is going to be recorded and posted on our YouTube channel along with the documents. There is a sample test document that is under the handout section that we're going to be using during the webinar. We've got a small group today. So if individuals want to talk, ask questions, allow that type of thing, there's a little hand button there. If you raise that, I will unmute your mic so that you can directly ask questions. Additionally, there is a section called questions. Anything that you type in there, I will definitely be monitoring and get that over to our speaker. We're going to get started here in just a minute. To let people know, our training schedule is pretty much up to date through the end of the year. We've got a training coming up next on 614 that has just been added and yet to be publicized, it is over using GitHub and Bitbucket for sharing source code. We've got introduction to business process mapping in early July and then we're going to do a voice over IP unified communication system webinar probably in late July. Along with that, we've actually got about six more webinars. We're doing almost 20 webinars this year. I'd like to turn it over to Mariah at Transcend. Thank you so much for covering this new topic and I'm looking forward to it myself. Thank you so much, Brian, and thank you for inviting me to give this seminar. Today we're going to be talking about field testing, which really closes the loop on plain language. This is where you get to get some feedback from the actual users of your documents so you get to see how they're working. As Brian said, please feel free to raise your hand and jump in at any time if you have questions or comments. If I understand correctly, this is a 45-minute seminar so I'll do my best to keep track of time and keep a few minutes open at the end for any questions that you may have. Just a little bit of background about me. I've been with Transcend for 25 years and been doing plain language and certified translation for roughly that period of time. It's hard to believe that so far we have done the only comparative readability study of plain language court forms in the United States, which is a little bit sad. I think it shows that we need to have more research and testing going on on our forms to really evaluate what type of writing works best for consumers. We've been very fortunate in that we have done a lot of writing for different court systems in the country. I have been very lucky to have worked with Professor Richard Weidich. Many of you may know he was the author of Plain English for Lawyers and really was the guru for plain English, having started this movement some 30 years ago, and also edited portions of Brian Garner's red book, The Portion on Visual Accessibility. Today we're going to talk about field testing and of course the question is why? Why should we bother doing this? It's a question that many people don't ask. They finish writing something, they put their pen down and that's that. In my opinion, that's not enough. We need to ask this question because we have to find out whether or not what we've written is really reaching our audience. Field testing will tell us if our audience really will read, number one, what it is that we've written. In many cases, when we did that article that was published in Scribes, we found out that they won't even read it. That's important to know. In other cases, we find out that they don't understand it. That's very useful feedback to get from field testing. When we get that information, that can really influence the way that we write future documents. Of course, this is another important guru in the field, Orwell. Who's the one who tells us to write for our audience, write only for our audience? When we field test and when we focus group test, it's really a chance to listen to our audience and to collect data. The idea when we do this field testing is to listen to what they say and listen as opposed to talk or instruct or explain, tell, and listen to the words that they use, the words that they find easy, what they find difficult, the words that they prefer to say. That can help us explain things in the future using words that are both legally sufficient and understandable for them. Today, there's, of course, many types of field testing. Today, we're going to be talking about one type of qualitative field testing. There are other options. We're going to be talking about focus groups. There's also observations and interviews and chats. But we like to use mostly focus groups because they're quick, they're relatively inexpensive. For those of you that are part of court systems, you are very lucky because you may have access to jury pools that makes them very inexpensive. And, of course, it's very instructive for you. And the findings that you get are, can be very rich. And how rich they are, I think, is based on how competent you become at developing an instrument and how skilled you become at being able to hear what the focus group participants say and how you can translate that into what you're able to write in the future. Okay. If there's any questions so far, if you would raise your hand and otherwise we'll just jump right in and continue. Okay. As we field test, one of the things that we do at Transcend, and this is so helpful as we move forward and we do this and we share this with the people that we write with, with the different court systems that we work for, is we develop glossaries of the sort of the official legal, difficult, perhaps terse term and what a legally sufficient, more understandable term might be. And so we develop these lovely glossaries over time of difficult legalese words and easier plain language words that can be understood. So it would just be so cool if all of us working in this field could share these wonderful glossaries or dictionaries or whatever you want to call them of easier ways of saying things and could pool them together. And just to give you a quick example, even for the most wonderful writers, one of the things that I see so often in so many court documents, well, not court documents, court info sheets that are written with a lot of love and a lot of care is that to seek legal assistance. And there you have, you know, why are we saying seek and why are we saying assistance? So just two unnecessarily difficult words that could easily be replaced with easier words. And, you know, if we could start tracking these words and just get in the habit of saying, you know, to find, to get, to have, and assistance would be help. And we can just get in the habit of saying easier words as we go along. And there's something about writing those words down and keeping a log of them that helps us to translate and get in the habit of using easier words. So where do you suggest finding these less thesaurus or whatever you want to call them for lack of a better term? Oh, thank you for asking. So, Brian, one of the projects that Law New York is doing this year on their, we clearly write clearly that we've been working with them on is they have on their, I think it's un-bright clearly, is developing a glossary that takes more difficult words and you can, you know, you put your cursor over the word and it translates it into, it offers you an option of an easier word. So that would be one way of doing it. And... Yeah, and I put the URL for that in the chat. It's writeclearly.org and that redirects to the Google site where it has all of those and some particular tools including a tool that will look at your document and suggest easier versions. Thank you so much. Great resource developed with a TIG. Right, and then the read clearly will also, if you access any part of your website, so for instance for the Judicial Council if you wanted to look at how difficult the reading level of any portion of your website and I'm not sure if this will do it for your form posted to your website. It will also give you your readability stats. Okay, and so as you do this, as you complete the circle and get your feedback from the field test, you start to become guided by your reader's preferences and by their reading proficiency. So you start hearing again and again what your readers find difficult and what their level of understanding is and then it's less of you add your computer in your office feeling like well this is quite good and I think this will do as opposed to going out there and realizing oh my goodness this is not going to cut it because this is really not being understood and then you realize you have to push yourself a little bit more to get it to a level where genuinely more people are going to understand it. And this is my favorite quote ever and it's from Glenn Rodden of LSE who is the most wonderful person in the world. Brian, have you heard this quote before? Yes. It's in our readability book and says I used to know how to write and then I went to law school. It's just so accurate. We teach people to write in a whole different language that our clients never understand so it's right on. It is and unfortunately we teach people to write poorly at many levels of education and we're sort of always teaching people to get a little puffed up and to make long sentences and long paragraphs and to not really get to the point and I think unless you're a communications or a marketing major we don't really teach people to write for communication and so when we write for plain language we have to teach people to undo a lot of the things that they learned about writing. So let's carry on. Okay, if you want to spend $15 that will be very, very well spent and to have some support please get this little set of books. It's absolutely wonderful. We spent a lot more money than this and we have lots of collections of testing books but this is really the best and it will give you some wonderful support and you can get a use set on Amazon for about $15. Okay, so in today's class I think we only have 45 minutes so you won't learn the details of everything but you will learn how to prepare an instrument and of course the instrument is the basic script that you take with you when you go to test your document. You're going to learn how to recruit participants how to train the people that you do your test with how to conduct your focus group test and how to report your data. Okay, and the things that you need to understand what we're going to talk about today are the sample test document that Brian explained you can download and it is a language access language rights notice and the instrument is actually on this presentation so you'll have it here. Okay, so think of this template of the instrument as a starting point. I want you to have a template so you won't have to start from scratch and the most important thing to understand is that each template will be different and you have to make sure start with the fact that you have to figure out what you most want your readers to understand about your document and then adapt it to the template and ask open questions so that you can see if your readers are understanding that and the good thing about a particular organization or perhaps a larger group of people using this sort of template which is really nice is that we can all start talking about things in the same way did they understand the content was the format appropriate what were their likes, what were their dislikes that sort of thing it's like all using the same readability instrument it just makes it easier for people and also one of the biggest problems that people have in using an instrument is they make it too long and that's not good and it will help you to see what an open question is and that's really one of the hardest things in the beginning so I'm going to just move forward on that the questions there should be no more than 12 and you don't want people to get tired you want to keep a good pace keep moving forward make sure that everybody's involved we have a limited amount of time and it also helps you keep focus on the main questions if you have a document you have to know what your main points are you do not focus on having people understand the tiny minutiae of the document so if we look at our document which we will right now we're going to try and figure out what is it that we really want most people to understand so if we look at this which is about language access rights I would like to ask you to take a minute and tell me in your own words what is it that we would want people to understand and you could tell me in a chat or raise your hand and tell me what it is you think is the most important point that we want people to understand and of course this is already pretty plain language and pretty short, pretty easy to understand it seems like you've got two focuses there first letting people know what their rights are and then second how to actually execute or use those okay and can I nudge you a little bit what are their rights the services must be made available in a way that they can understand and interact with that it is not acceptable to deny service because of a difference in language okay alright good enough and so I would just say maybe we could take a poll do you think this is a pretty easy piece of text to understand would most people understand what their rights are and what to do if they're not able to have access to their rights I actually think this is a little bit on the complicated side that you have language rights people don't necessarily know what that means I don't know I think how do you get help portion is great if you need an interpreter these are the areas it should be free the emphasis on those I like a lot okay alright so just to show you how hard sometimes easy things are this is written at the I think it's like the fifth grade level although it's not formatted yet it's pretty chunked it's got good use of bold and so on but it it's still quite challenging to understand okay so we're going to take it out for testing and I'm going to move this over here and all tests start like this and I'm sort of going backwards but this is really important before you start a test you have to tell people some really basic things and this to get people situated that who you are and when you go there I think it's important to really dress down no fancy clothes or jewelry or anything like that and that you're here today and thank you so much and you want their help because you have a document or a website or whatever it is and you really want their opinion so that you can make it better because it has important information that will help other people and you need their input so that we can make it better to help more people so anything that you say will help us to make it better for other people so even if you really don't like it and if you say something that's really negative that's really great because it will help us to make it better so does everyone feel comfortable about saying anything even if it's really something critical and so you sort of just go on like that and then get everyone really comfortable and that's how you start and so one quick comment in doing field testing like this I've noticed that just emphasizing to people that you are not testing them that you are testing the document if you don't understand something from the document it's the document's fault and we're here to improve the document but just make it clear that people cannot give you wrong answers here there are only right answers coming from people that help us make the document better yeah and I think one way to say that is to not even to bring it up just to talk about their opinions and how everything they say is really helpful and because we want it to be helpful for other people so what they're doing is really an important service and the other way that you can convey that and this is over in the right hand column when we work with people in the right hand column of the instrument we always explain why we're doing this and maybe what to say and so sometimes when when people say things like when they say something that's completely wrong or weird or something we just say oh ok good alright and then you move on so no matter what people say you always accept it, repeat it acknowledge it and you know just very warm, very professional very accepting and then you move on to the next thing so we're never correcting we're never admonishing any of those things ok so for those of you maybe that have studied readability a little bit this will sound familiar the first thing that we say is what is this whatever flyer, poster, website about and here we don't allow very much time and the idea is that if you can't determine what something is about in a very limited amount of time you're not going to read it so we want to see if we can get that person if there's going to be any reading interest immediately and here along with the facilitator we'll talk a little bit more there's two other people in the room the observer and the note taker we're going to have people write exactly what the focus group participants say and this allows us to keep track of the type of words that they use to talk about this subject matter so we'll hear what they say so if someone said I don't know write to an interpreter or language rights or write to talk in court or whatever they say we'll be writing those words down we'll be repeating it thank you and moving on so it goes very quickly we acknowledge, we accept and then we move on so that's your reading interest part who is this for? this is your audience and this is for reading persistence and here in terms of readability there's the idea that if something is not for you if you're not the natural audience someone you care for is not the natural audience you're not likely to read it so we want them to say who it might be for and then we write down what they say we acknowledge it and we move on and here we start pushing a little bit deeper do you think this flyer would be helpful for people who don't speak English well and then we talk about the overall level of reading interest and we start exploring and this isn't an example of an open question because we want to see what aspects of the flyer they actually were able to read if you're looking at the sample test document you can see we talked about lots of services and what they were supposed to do and where they were supposed to go and where it was they were going to get help so this will tell us which parts of the document in fact they actually did read they did see so we'll write down those things and that will tell us more about what they read and what they understood and of course when they say no that will also tell us some very interesting things I'll give you an example when we tested this one of the first versions of this one of the people said no we said well why and they said well if we were to ask for translations it's our experience that we always get some like Castilian Spanish or Argentinian translations and they say wow and words that we don't use and we don't understand it's not really worth it because they don't use the same type of language that we do and we thought wow that's a really interesting comment so you'll always learn things from their comments it's always a deepened understanding of what their experience is and here again this is we're learning to ask a question that's open with out giving them an answer so the opposite of an open question of course would be yes no and here we're just putting them in a particular situation and we're seeing little check boxes aren't something that we would say out loud we would just check them if they happen to say one of those things and it lets us know how they envision that they would use these particular services if they are understanding what it says and what it means to them and so this would let us gather information about what it means to this particular group and this is not a good question and I put this question in because this is this is a very typical question that sometimes we get from our clients that this is a very specific question that they want to know and so here they would say okay is it clear to them meaning the focus group that if they call or they text this number that they're only reporting it that they're not going to get an interpreter and if it's not clear how can we make it more clear so in the first part of the question it's yes no so clearly that's not an open question right and as you can see and I think this is a little bit what you were saying Brian law school kind of questions it's a long complicated question and then the second part of the question makes it a little bit longer and more complicated and so you have to think okay this is what we want to know how do we turn this into an open question and any ideas and this one here it's hard to say without this in front of us and I'm sorry I don't have the document in front of us probably not fair but let me show you what we did so we said what is this phone number for and this was the phone number that they were supposed to call if they had a problem so it's really a different way of thinking right you have to think okay what can I ask that will tell me what they understand and it often is a very small minimalist sort of thing and and then of course it's often something that the focus group may not understand especially if your document isn't well written they may not understand it so then too the facilitator has to be very gracious and very accepting so let's say you have a document that's not very well written and you're asking well what is this phone number for and no one understands and do you have to say yeah that is you know boy that is really difficult to figure out let's you know just Florencia do you have any comments on this and or does anyone else want to add what you know have any idea what this is about and that's really difficult to understand let's go on to the next one and if someone says well this is how you make an appointment with an interpreter it'll say exactly the opposite of what it's supposed to be and you say okay does anyone else have an answer and someone says yeah that's the number that you call for an interpreter okay does anyone else agree with that or think something different and if other people say they agree then you can say okay and so again don't correct don't anything just write down what they say and move on alright and this is the best part one of and this is where we get to develop our glossaries one of the things that we always do is to ask them about language and we have yellow highlighters in the middle of the table and we ask everybody to grab one and say okay let's look at the flyer again take your yellow highlighter and highlight any words that you think might be hard for some people so not for you but you know that might be hard for other people so even if it's a word that's easy for you some people might find that think that it's difficult and we always get a lovely list of words and then we then we talk about what are easier ways of saying those words and then we go over them as a group and sometimes say yeah did anyone else think that that word might be hard and then you know other people will chime in and is there an easier way of saying that and it's always produces a very rich conversation and a very interesting group of words and that's what we keep in our glossaries okay and this is we ask people what they like and dislike and of course they don't see the instrument this is something that the focus group team has okay let's look at the flyer again tell me anything absolutely anything remember this is we really want to know we want to make this flyer the absolute best that it can be so anything you like or dislike and you know so once they start saying they really get into it and then of course you learn with experience that some people are just like really good at this and in fact we have a couple of people we do our best to invite them to every focus group because they just have a knack for finding these little things that are so powerful and if so people tell you things and then if they you know if they don't for whatever reason if they don't have a hard time getting started we might say well how about the colors do you like the colors so if you get them started then they're fine and going to the next one the title and if you've studied readability we know the title is everything the first thing that people read is the title a title is very important on the Spanish version for language access language rights in Spanish this was derechos lingüísticos which is a mouthful and do you like the title do you think most people will understand it no one liked the title they thought some people would understand it but that it was very academic and sort of a little bit snobby for lack of a better word and they thought idioma would be better or interpreter would be better they thought it had more to do with interpreting actually than lingüístico and so they had very interesting things to say about the title and so that's very rich you know if they can help you to make a good title that's meaningful for them that will get them to look at the document which is the most important sort of key piece of text that will get someone to look at the document then that's a very important contribution and then the last thing we asked them as you can see these are only 10 little things but boy we talked about a lot of important things okay let's take a step back and just look at the whole thing again and if you could change anything just anything you wanted what would you change to make this better and they always have very interesting ideas and you know by this time they know the document backwards and forwards and they have wonderful things to say here and not only wonderful things to say but you as a writer you really start appreciating the way they look at this information and what they see and what they don't see so it's a wonderful step into the seeing the way your information is read by your audience so that's the instrument and I think if we can go back you can see that these 10 little questions are easily adapted to the way can be easily adapted to your documents there's the content questions have to be adapted to the content to make it work for you this one question five the first question five remember that that's not a proper question that's an example of a question that's not good you'll have to take that one and make it an open question but other than that all of the questions you can adapt for anything that you're writing okay I want to keep track of time alright focus group size small or large we tend to do unless it's a jewelry pool because of the cost we tend to do small people we think it's very important to pay people we want them to take it seriously we want it to be a job we think it's part of valuing them and knowing that it's important that they take it as a job but I think many of you are with court systems and they will take it seriously and they'll be happy not to have to be in the jewelry pool so you may not those issues who to recruit so how much do you pay people on average we pay people between $30 and $50 depending on how many documents and how long we're going to go we pay them enough that it should be we want them to feel that it's important and they're going to be there we want them to show up we want them to show up on time and we want them to feel that it's important and that's typically for a two two and a half hour total time show up intro testing and thank you we don't we don't go to two and a half hours unless it's extraordinary we feel that after we tell them it's going to take two hours but we're really efficient we set up before they get there we give them their name tags we're ready we want them to sit down and we go and we're like an hour an hour and 15 minutes and we we think people get tired and don't do well after an hour and 15 minutes an hour and a half excellent thank you I'm not sure I'm going to get through everything today but let me get through this the recruits should match the profile of your consumers in our readability article at the Transcend website at our library on demographics you should know about your own state, county demographics in terms of language ethnicity and so on so that's really important if you shoot for eight people you'll probably at least get five or six which is good you want people of different ages different education backgrounds different literacy levels gender and that will give you a nice mix it takes time to build a database of participants now we can easily pull together people you know in the day or two because we have enough people that we've worked with but in the beginning it takes some time we tend to go to waiting rooms at health clinics, social services offices to have a big pool of people a jury pool is a good place to go and when you do go to the health clinic and social services of course you always have to let the people know that you're going to go there and no one has ever told us no so and if someone says they won't go you can ask them if they'll go at another time and usually they're quite happy to do it but they really like participating in these things and it makes them feel valued and important in a way that they're usually not asked to be valued so it's good and collect their demographic information okay Brian do I have time to do a few more slides yes definitely okay facilitator and for those of you who are attorneys I think the one thing is that you can't explain the law or tell them what's right or wrong and you can't really help them and you really have to lay back so smile be calm anything they say you repeat it okay great so you think that part's not clear and this part seems too long be flexible if someone arrives late say just okay have a seat we're on this part call everybody by their name you should have the name tags ready if someone says something just summarize what they say and do other people agree keep a good pace if someone gets off track you know you have a difficult person say okay we'll talk about that after it's over let's move on and if you wrote the document and you can't handle someone saying something negative about it don't be the facilitator because it's not going to work okay and let's see for the note taker and the observer you really need a couple at least one other person make sure they read the document first and the instrument you have to have to be familiar with it stay in the background don't say a word but they have to sit close enough so they can hear what everyone says and they also have to help you write the focus group report and they have to be able to take it all in and then cross edit as you write the report at transcend one thing that we do that really helps a lot is that we pre-test every time we do a focus group test and you know it seems silly that instrument that I went over with you you think it's so easy the questions just go in there and do it but it's really not the moderator has to feel comfortable with the questions and you have to kind of decide what your pace is and if you're not really situated it doesn't really go well and so you just grab three people from the office during lunch or whatever you want to do and you practice it and also the moderator they should have read it and they should know if they have any questions if something seems weird to them and by that time too you know do you have all your materials do you know where everyone's going to sit do you have your timing was there something wrong with the document are there typos in it sometimes when we're pre-testing you think oh that question's weird we better fix that you know just something's wrong and it really lets you fix something before you're like right in the middle of this focus group test and you realize that you've spent this money and you range this time and you know there's something wrong so pre-testing is a really good idea it doesn't take that much time and it really makes the focus group test go faster and it gives you more data that you can look at your space chairs tables a mock-up of your document highlighters pens the name tag we have the tiniest little conference room in the world but that's fine we always have toys and extra chairs very often people show up with their kids and we say we just give them a little toy and you know just go with it and make sure there's a clock and have everybody's pay ready so that you know at the end of the test you give them their checker, their cash or whatever and just send them out the door and we have clipboards because there's usually not enough space so sometimes the facilitator note taker and observer have to stand up and having a clipboard will make that easier and we have an instrument we always leave enough white space that we can write our notes very easily and for quiet people we can call them by name and ask them to raise their hand you know who else agrees with that if they don't really talk you can take quick polls so varying the type of questions that you ask also help and we've talked about difficult people just say okay let's talk about that offline and that usually works and you know being positive but you know shutting them down right away okay and just really quickly I want to show you that how we sort of this document got better it got better it got even a little bit better and at the end you can see the side by side the one on the right it got the one on the left got even better you know it got to be that they told us that it was really an interpreter that you know what the heck does language rights mean anyway you know a language rights is something it means something to lawyers it doesn't really mean something to people who are not lawyers what it meant to people who are going to an office was that they could get an interpreter and this oh well but they could get translations too it wasn't really the main point you know the main point was they could get an interpreter and you know the little icons about where they could get those services were really helpful and the names underneath them got more simple and the colors you know to point them out helped people see that more clearly and the text on the card got shorter and simpler so everything got simpler and more direct and that was all as a result of input from the focus groups and that's it well I would like to thank you so much for doing this training this is highly important to what we're doing here at Northwest Justice Project we've got a tag where we're doing some testing with videos that include focus groups and this has a lot of practical advice in it thank you for taking your time to cover this today and just to remind people our training schedule for LSM TAP is up for the entire year although we are always adding to it if individuals have other trainings or TIG grants or other things that they want to talk about the link is there in the chat our next training is on GitHub ways to share your source code and Drupal projects there's actually even a Drupal group sharing session coming up on Thursday that's being done by Urban Insight and then we've got our what is very popular 50 tech tips in June hosted by pro bononet on June 22 so check out our training calendar and thank you Maria from transcends this has definitely been a very useful educational process here and I hope to see more testing like this going on you're very welcome I guess one thing I'd like to say that if anyone wants any free feedback on their instrument I'd be very happy to provide it so just shoot me an email and I love doing things like this so let me know that's a wonderful resource we will add that into the blog post and to the post in the video also thank you bye