 All right Hey, welcome to our two o'clock session two o'clock central time session for big talk from small libraries 2018 I am Krista Porter your your host here from the Drasca Library Commission Next up we have small data can still be smart data how outcomes can help your library And with us is first up Samantha Lopez. Hello, Samantha. Hello. She is a program officer at PLA Public Library Association up in Chicago and along with her is Pam Bonnie. Hi Pam. Hi And she is library director at Winslow Public Library in Winslow, Maine way over on the east coast for us and Together they're gonna talk about a la or PLA is project outcome This is a such a program that we've heard about I've heard about before elsewhere I'm gonna share it with you all today. So I'll just hand it over to you to take it away All right. Thank you so much Krista And thank you everyone for joining For today's presentation on how outcomes can help small libraries Krista already did some introductions, but I'll say it again. My name is Samantha Lopez I'm with the Public Library Association And I'm so happy to be joined by Pam today who's going to talk about her library's experience using project outcome Here's just a quick look at what we'll be covering today I'll go over the importance of measuring outcomes and why you want why we want you to do it What project outcome is and the benefits to libraries and how libraries are using their outcome data? And as I said Pam will also share her library's experience using project outcomes So first I want to take a few moments to explain what we mean by outcomes and why we are helping libraries measure them So for years libraries have gauged success through their patron stories and the core metric has been how often they came back for more But as you know times have changed and so have libraries Intuition anecdotes and attendance counts are no longer enough To demonstrate the value in the community libraries, especially small libraries need more data and evidence There are many data pieces that help demonstrate your library's value You may already be conducting some needs assessments in your community and answering the question What is our community need? You may be doing some patron satisfaction surveys in your library to answer? What should we do better? And you're most likely collecting outputs like attendance and door counts to answer the question. How much did we do? Collecting outcomes is a little bit different. It helps you answer the key question. What good did we do and At PLA we define an outcome as a specific benefit that a patron receives from a library program or service Outcomes can be quantitative or they could be qualitative They can be those stories that you hear and they're often expressed as changes that individuals perceive in themselves And again, they answer that key question. What good did we do? So an outcome is a way to determine the extent to which a program achieved its intended results by Measuring outcomes you're learning what changes effects or impacts your programs are having in key areas such as knowledge Skills attitudes opinions behaviors actions or even status So you're learning did the program positively impact the lives of the patrons? Did it make a difference and how are the lives of the patrons better off as a result? During this presentation I ask that you think about a problem your library needs to solve or is currently trying to solve and how Measuring outcomes can help here are a few examples that we've heard in the past when I've asked this question Libraries want to know if patrons are finding value in their programs if they're actually helping people achieve their goals if they're actually helping people get jobs for example and if there's a way to better show funders that they're being successful and When I do present this in person This is about the time where I see some hesitant faces in the audience But since we're virtual today, I'll just assume that they're there, especially since I'm talking to a small library audiences You may be thinking yes, that's great I would love to measure outcomes and collect all kinds of data for my library But I just don't have the time I'm way too busy and I know small libraries in particular are strapped for time staff capacity and resources Well, the good news is that PLA's got your back with project outcome A majority of the work is already done for you to make measuring outcomes as easy and simple and painless as possible So now I'm going to go into Highlighting what's actually included in the online toolkit and what the benefits of using project outcome are Project outcome if you don't know is a free I'll repeat free online toolkit designed to help public libraries understand and share the impact of their programs and services By providing simple surveys and an easy to use process for measuring and analyzing outcomes Project outcome also provides libraries with the resources and training support needed to apply their results and confidently advocate for their libraries future We currently have over a hundred thousand patron surveys collected into our system So far from hundreds of public libraries across the US and Canada and to put that into perspective We officially launch launched the project outcome website in June of 2015 So we already have over a hundred thousand patron surveys from hundreds of libraries Which is really exciting our goals with project outcome are To make measuring outcomes business as usual for all public libraries It's also to supply libraries with standardized outcome measures that any library no matter the size or capacity can use and learn from and Also provide a user-friendly tool that anyone can use at any capacity the project outcome toolkit gives libraries free access to quick and simple surveys and easy to use survey tool to collect and enter all of your survey data custom data reports and interactive data dashboards for analyzing the results and All of the resources and training you need to help you throughout the outcome measurement process and What we've heard from library users is that the combination of the ready-to-go surveys and the easy to use tools really help library staff Save time and energy in their planning so that it leaves more time for decision-making and advocacy once the outcome results are in And again, it is free free free I try to repeat that as much as possible because I'll still get that question at the end But really how much does it cost no I promise you it's free and everything I'm going to show you today is available to you on the project outcome website at no cost Project outcome surveys are designed to capture four key patron outcomes We focus on capturing an increase in knowledge and increase in confidence Patrons intent to change a behavior or an actual behavior change and their increased awareness in library resources We also help you collect some of that qualitative data We ask two open-ended questions and all of the standardized surveys that are similar to What did the patron like most about the program and what can the library do to improve? And it's these open-ended questions that have been invaluable to libraries Here's where we hear that they gain the most insight into how to help their patrons And they gather that qualitative evidence to demonstrate their value in the community So they're sharing out that qualitative data There are seven survey topic areas to help libraries measure a broad range of programs and services So we have surveys for early childhood literacy digital learning Education and lifelong learning civic and community engagement job skills economic development and summer rating There are several survey types that libraries can also use to measure their outcomes We have immediate surveys that measure patron-reported learning. We also have follow-up surveys that measure patron-reported behavior change And we have outcome measurement guidelines that help you design your own outcomes and data gathering methods to capture longer-term benefits And with project outcome, there's no specific pathway you need to follow. It's really a pick-and-choose type tool You can use all three any combination or just one It's really up to you and your needs and the learning outcomes of the program that you want to capture The immediate surveys that we provide are designed to capture that patron-reported learning and the immediate impact to the program These surveys are the easiest for libraries to use because you just hand them out at the end of a program or series and then you Automatically have their results. So these are the most popular surveys that we see a majority of our data That's collected in the system do come from the immediate surveys because they are so easy for libraries to implement And in this example, this is our digital learning survey example You see that patrons are asked if they feel more knowledgeable about using digital resources If they feel more confident when using digital resources If they intend to apply what they just learned and if they're more aware of resources and services provided by the library And of course those two open-ended questions The follow-up surveys that we provide are designed to capture patron-reported behavior change Change once a patron actually leaves the library These require the library to follow up with the patrons. So they're more time consuming. Like I said They're less less used in the system because they do take time to follow up And you will have a lower response rate since you're kind of narrowing the pool of responses But they will provide some hardier data for you if you do decide to use follow-up surveys Because instead of asking patrons how they feel follow-up surveys ask whether or not they actually applied what they learned So in this example, this is our early childhood literacy survey our follow-up survey For example, it's asking caregivers if they spend more time doing what they learned with their children as As a result of the library program So think of how powerful that data piece could be if you did collect even a handful of surveys that said Yes, I have been spending more time with my child ever since attending the story time. That could be a really powerful message The outcome measurement guidelines that I touched on they live as a resource on the website And they help libraries that want to expand beyond the standardized surveys that we provide So they help you think about if you want to maybe write your own outcomes If you want to think about measuring data over time how you would go about doing that If you want to do something other than surveys if you want to do say a focus group or an interview And it also helps you think through strategies of working with partners And how either you could measure outcomes and share them with a partner or if how you could measure outcomes together with a partner So thinking back to the program that I just kind of listened in on before this one thinking about how you could work together with those Fitness partners and who would collect outcomes and think about how you would maybe Provide the advocacy for your partnership using your outcome results Everything for project outcome can be found on the website like I said the url is project outcome.org really easy to remember As I said registration is free and it only takes a few simple steps to begin Once you are logged in this is what your home page will look like as you can see You have quick and easy access to all of the tools and resources We have our survey management tool our data dashboards. We have all of our training resources We have a peer discussion board and our news and events page I'm going to quickly walk through these tools, but there is more information for you once you are logged on There's a ton of tutorial videos and FAQs to help you learn how to actually use project outcome So this is the survey management tool. It's where you will create and access all of your surveys It's also where you'll customize your program information You can track attendance and response rates Enter paper survey responses download your raw data and view all of your past surveys And we have filters so you can easily manage and sort through all of your surveys and data Here's a snapshot of how you create a new survey You'll walk through five easy steps to create your standardized survey And then you'll customize it with your specific needs and hopefully what you see in the screen In this screenshot shows how user friendly and really simple the tool is The survey tool also allows you to add your own logo So whether you have a library logo or a program logo you want to put If it's something like summer reading that has its own logo. You can put that on the survey You'll include some specific program information like the name and date You could add any intro or thank you messages to your survey And you can also add up to three additional questions of your own So if you already have something that you're required to collect in your library You could simply add it to the project outcome standardized surveys And once you're done, it's really easy. You'll have a pdf ready and available to you You'll also have an online url that is specific to that survey so you could collect Surveys online if you plan to do that And surveys are all available in English and Spanish as well Once the survey is created, you'll be able to enter your paper responses Or view the online responses that have already come in You'll be able to track your attendance and response rates and download an excel sheet of all of your raw data So that's one of the nice things about project outcome is that you don't have to wait for your results to be in You could access them immediately once they're entered into the system You can download and see your raw data The data dashboard is where you'll go to view and interact with your results It's also where you can get your summary reports And here's a snapshot of the report builder tool and different dashboard views for analyzing your results We have this report builder tool it acts similarly to the survey creation tool So it'll walk you through a few steps to choose your report criteria For example, you can run a report by system or single branch You can also choose to roll up all your data by survey topic Or you could narrow it down by a specific program name or even a single survey Once you do select your criteria A report is automatically generated for you. It's already designed and ready to go Here's an example of what it looks like the reports provide snapshot percentages of your outcome scores And comparison averages at the state and national levels As well as program detail information About which programs are included in the report what your attendance was response rate for the survey We also have some community talking points So these are really great communication tools for libraries because they're ready to go like I said And they can easily be shared internally with staff or externally with your library board or community stakeholders And once you have data in the system everything is automatic including your data dashboards So here's some snapshots of what the different dashboards look like And I'll briefly go over them But trust me it's more fun to kind of go in there start a survey enter data and then go in and play with the dashboards yourselves Screenshots really don't do it The overview dashboard provides aggregate scores in each survey topic and outcome type So whether you're interested in just increased knowledge or if you really want to be focusing on Your summer reading results. You can look at your overview dashboard You'll also see state and national comparisons and you'll also see the total number of surveys The matrix dashboard allows you to interact with your results So you can filter soar and scroll over them to see where your strengths and weaknesses are in each of those categories We also have a detail dashboard which shows your results by each survey question. So you can analyze them at a deeper level You can also access your open-ended responses in the dashboard and add them to your reports So if there's that qualitative data that you really want Included in your report, let's say, you know, two really good quotes about From caregivers about how they're spending more time with their kids You can add that and you can view that here in the dashboards as well There's also a library info dashboard. So you can incorporate some of those library outputs with your outcome data And we also have a map dashboard that lets you see how each branch that's participating is doing And if you click a certain library Population size the map will automatically show demographic data. So you could talk about your community impact in those areas And like I said, we have a ton of resources on the website to help you There's no official training Needed to get started with project outcome We have everything from getting started to best practices for data collection to taking action using your results We have a news and event page that will help you Stay up to date with our latest trainings and any announcements that we have We also hold regular webinars to help libraries continue moving forward with their outcome measurement We also have a peer discussion board where you can connect with other project outcome users and you can ask questions and share experiences And that was a very very short overview of project outcome And the online toolkit, but like I said, you can go on there and there's more in-depth tutorials or you could view past webinars There's definitely more training on there If you want to learn more But hopefully I've highlighted some of the benefits of using project outcome, especially for busy small libraries This is not a tool for rigorous data collection And random sampling in your library What you're doing is you're targeting your users and finding out what they're gaining And what else they need from the library to help support them in their learning This allows you to get snapshot data And the short simple surveys help you get more responses from patrons So as I said before the open-ended questions of the surveys are a gold mine for libraries This is where they're gaining the most insight into their community's wants and their needs Plus we've done all the heavy lifting for you. We have the standardized surveys easy data entry and ready to go visualizations and the reports And the key benefit of using project outcome for your evaluation is that you're using standardized outcome measures across time branches And a variety of programs. So while your programs and audiences may be unique Such as, you know, who's attending a story time and the learning outcomes of that versus who's attending an excel 101 class By standardizing the outcomes you're able to talk about your impact across the board Which then allows your results to be aggregated in our system with state and national datasets For your comparison's sake, but also so we can talk about library impact at a national level So now, um, I'm going to hand things off to Pam who will talk about her library's experience using project outcome Great. Thanks, Samantha. All right, Pam. I'm going to give you presenter control now Correct. Am I presenting from this screen that I see or do I need to go to the slide? I'm going to be switching it over to you right now. Hold on just a sec You shouldn't see a little pop-up letting you share your screen And Krista, I'll be taking it back once Pam is done. Okay First timer for this I'm tempted to title my part of this presentation project outcome for dummies We're just To gather outcome data about our programs and have just scratched the surface of the resources available on the website We've done a few basic surveys, but we're wowed by the great information. We've gotten from getting these using these resources Oops, here we go Oh, I'm sorry. I'm lost in space Oh dear. I was afraid of this There we go. Thank you A newbie at a lot of things here. Okay. So this is our library. I want to tell you a little bit about it We're a small public library in Maine, but Nationally, but in Maine were considered a medium-sized public library really with a town of just under 8 000 people We have a staff of two full-time professional librarians and one full-time and two part-time paraprofessionals We're open six days a week for a total of 46 hours We're a municipal department and and with an advisory board of trustees And we have the distinction of being the only library in Maine that is a former happy wheels roller rink And actually it works great. It's a big wide open floor plan very accessible. We really like it I've been the director for the past five and a half years I must admit that I was intimidated by the thought of joining project outcome I had heard about it for several years But I felt it was just going to be too difficult for a small busy library with a small staff to tackle I was afraid it would be too much work that only libraries with large staffs could handle the work involved And I feared it would be a commitment a big commitment of time and that by joining We'd be agreeing to survey every program we did I was also afraid our patrons would be reluctant to participate and would tire of being surveyed But the main state library hosted a day-long program on project outcome last spring And after attending it our youth services library and I decided to give it a try Summer reading was coming up It would be so great to get some real feedback about the program that we've been doing for so many years And to be able to show the impact the program has on participants Like all libraries we've been using statistics to tell our library's story for years But mere numbers don't begin to tell enough We need to show our stakeholders the impact our programs have on the community I was motivated to investigate participating because we need patron feedback to improve programming in general And we're not getting much of it And we need to make a strategic plan and need compelling information to help create it We also need to show the impact of what we do to the stakeholders the public the town council the trustees To ensure continued adequate funding and to get support for a needed increase To be used to create more services and provide the staff a living wage Well, it turns out my fears about participating in project outcome were unfounded the project outcome surveys are so easy to use They're already created for you and can be customized if necessary We've been fine with using the surveys as they exist So you spend zero time creating a survey you just have to choose a ready-made one from the categories offered and off you go We can choose how many programs you want to survey. There's no commitment to survey all or even a certain number of them It's entirely voluntary And as yet we've had no patron complaints about doing the surveys project outcome gathers and analyzes the survey results for you, which is an amazing time saver They create beautiful reports from the survey results at the push of a button. This is our report on summer reading You get graphs the national comparisons of results and a narrative that explains the survey and the outcomes of the program being surveyed Something to be used with our stakeholders for sure They do all the number crunching and the written explanations So in august after summer reading 2017 came to an end We decided to dive in and do an online follow-up survey of the children's caregivers slash parents And the adult summer reading program participants We went to the project outcome website and created new surveys for each group and chose the type of survey Summer reading is a category. So it was an easy choice The surveys are ready to use. We were able to add a logo and introduction to personalize them We thought emailing the survey to participants would be easy since we had gathered email addresses when they registered A link to the online survey you create is provided and can be just pasted into an email However, this is the one spot. We found a lot of work was necessary Typing each of the hundred email addresses in to create an email group was time consuming Maybe next time we'll recruit a really good volunteer We didn't get a high rate of response, which is a little disappointing 25 out of 100 for the children's program and six out of 40 for the adult program But the responses were really interesting and next year. We're going to promote the surveys before the end of the program All the responses are collated by the software and the reports are a thing of beauty Here's an example of an excerpt from our summer reading program This beautiful essay about the effects of our summer reading program is just something I can't wait to share with the town council at the budget meeting this year I'll read just a little bit to you wherever public libraries are working possibility lives People who work in public libraries know that library surface services open new opportunities for anyone who enters And then it goes on to talk about The effect we have and the patron benefits are in high demand patrons most commonly suggest improvement for the live For the library or new more and frequent classes and programs Then it goes on to say that the percentage of parents or caregivers surveyed Who either agreed or strongly agreed? They benefited It gives you the great percentages 83 percent report the child maintained or increased reading skills 68 percent report the child reads more often and so on and all we did was Typing the surveys send it out and project outcome gathered all the responses and created this beautiful report They also create collect the responses to the open-ended or qualitative questions in a spreadsheet for you Which is a great time saver as you can see Here's an example I pulled them out of the spreadsheet and put them in a slide But what you like most about the program or service and right here they are From our small survey of adult participants very easy to look at and analyze on your own This past december I decided to survey another long-standing program our adult book discussion group I decided this would be a good group to try out an immediate paper survey with Since it would be a small number of surveys completed and would be easy to enter into the project outcome software by hand Attendance at our book group usually ranges from six to twelve And I wanted to get some feedback about this successful program I chose the lifelong learning category of surveys Easy because it included book groups as an example of programs to use the survey with I handed out the survey at the end of the meeting of one of the book groups And I had a collection box for the surveys right there in the room And again, I customized our survey with a logo on a brief introduction I was really pleasantly surprised at how easy it was to hand enter the survey responses into the website And I was again thrilled with the coalition of responses and the beautiful report instantly created And I've decided it would be pretty easy to delegate typing this Of the responses in with the next time I do one of these to another staff member or a trusted volunteer So in conclusion, I like to encourage you to register on the project outcome site and give it a try We're just getting started. We have much more to learn and there's so many resources on the site still to explore But already as newbies we feel it's extremely worthwhile We'll definitely survey summer reading again this summer And we'll consider handing out paper surveys at the program finale as well as emailing them to participants In order to see if that will increase the response rate by giving the participants a choice I also want to survey more programs in the near future. We plan on doing a survey of our weekly Weekly story time program this spring One of the great things about the project outcome reports is that they connect our survey results to a national database of library outcomes So they give a lot of weight to our local survey outcomes By contributing our surveys to this national database We're also able to work with libraries across the country to support and justify The importance of public libraries to the culture of our nation and to the fabric of democracy I'll hand it back over to you net folks now Okay Uh, so men thumb. I'm gonna make you a presenter again. Correct. All right. Great. Thank you You should see that to be able to share your screen again And are we seeing the correct screen? Um, it says wins the public libraries experience Okay. Yep. All right. Thank you. Pam. It's so great to hear Your perspective and to hear that getting started perspective and also what we hear from so many libraries Which is just start small try it out. See what happens So now i'm going to try to quickly touch on what we've heard from participants and how libraries are using their outcome data This is a snapshot of our registered us libraries by size Not surprisingly, we see a lower number of smaller libraries registered I think this speaks um to what pam touched on about small libraries fearing the time and the work commitment with measuring outcomes But we do have a pretty good variety of registered users thus far We have over 116,000 patron survey responses collected in our system overall across a broad geographic range, which you can see here on the map And what we've learned from Some external evaluations of our project is that participating in project outcome does help increase library Use of outcome data, especially among smaller libraries We've seen an increase in use of outcome data and regular use of outcome data among participating libraries And as you can see here out of 275 small libraries, there was a 10 percent increase in regular outcome data use after joining project outcome How libraries use their data differs as well. We Find that smaller libraries are especially using their data to inform or measure progress on their strategic plans And what you might find most interesting is that libraries are reporting increased benefits to the library Including having a bigger impact and reaching more people And over a third of libraries reported either getting more funding or non monetary resources by using their outcome data 45 percent What we've heard over and over is what is that getting started is the hardest part something that pam touched on We know you have so many other priorities and responsibilities that this might get pushed aside You might want to do it, but you might just feel you don't have the time But what this figure shows is that once libraries get started once you survey just one program And actually use the results 97 of libraries plan to continue to measure outcomes They see the benefits and they want to continue those benefits And as we've seen small libraries are slow to adopt project outcome But here are some ways that libraries have overcome that getting started barrier You can adapt the surveys into other tools that you might already be using Or you can also pick and choose when you want or need to survey You don't have to do everything all at once start small with one program Actively promoting the surveys to the patrons also helps increase your response rate and create more useful feedback than just we love the library Talk to your patrons about why you're asking them these questions and how their feedback will help improve the library Building staff capacity and buy-in is another common thing we hear you need to get other staff on board and make sure they understand You're not evaluating them as instructors or practitioners You're assessing program and service impact to see if there's ways it can be improved and magnified So now i'm going to walk through another case study we have from a small library in west virginia burnsville public library Bpl has used project outcome to better understand the impact of its programs develop new partnerships design new programs based on community input and increase funding Project outcome surveys showed that caregivers of young participants in bpl's summer reading program wanted tutoring and extra help for their children Equipped with this information and evidence of program impact the library worked with the local school district to have two teachers Offered tutoring at the library the following summer A library staff person whose two children participated in the program Shared that the children work on reading math. They get on the computer It's really benefiting them and some of the kids are here because they need extra help And some of them are here just to try to beat the summer slide and it's working very well Bpl also started a new after-school program because surveys showed an appetite for additional programming for children On the surveys children shared that they wanted to learn more about science and technology So one of the after-school classes that bpl added will include science hands-on science technology engineering arts and mathematics activities Here's a quote from uh bpl board member The reason we do it is we are trying to find out the interests of the community So we could serve the community better any library can use it. I love that quote because any library can use it Um digital learning surveys also showed that basic digital literacy skills Classes helped adults feel more confident using technology and become more active email users So they were changing that behavior Bpl shared the results in a grant application and as a result they received funding from the west virginia library commission To upgrade and expand their computer lab. So they could continue to help those lifelong learners in the community Evidence shows that project outcome helps libraries with programming decisions advocacy funding requests and grant work and building partnerships Since libraries see their results immediately, they're able to analyze their data and make quick and effective improvements to their programs and services This is the most common way libraries are using their project outcome data to improve or add programming In this example example Tulsa city county library used project outcome surveys To make an improvement to their story time So they measured story time across several branches, but only one branch had Designated playtime and toys for after story time and what they found Was that the open-ended comments really highlighted the post story time Toys and activities almost as much as the story time itself so the The library was able to use that data and share it with their friends groups So they got funding for the toys for the other branches. So they could provide the same post story time activities Libraries have also reported being able to use their results in a variety of funding requests Like I just showed including grant applications easier lsta reporting Helping pass referendum and requesting money from their friends groups For example, the cleveland heights university heights public library used project outcome to apply for a dollar general grant for their early readers program And by using project outcome and the summer reading caregiver survey as their evaluation method The library was able to set quantitative targets in their grant application They ended up being awarded the three thousand dollar grant and their early readers results exceeded their initial grant goals They got 85 percent of survey respondents Reported their child maintained or increased their reading skills Many project outcome libraries have also been able to use the positive results to reinforce their value in the community They've been able to use the summary reports and the data dashboards to give a clear and concise message about their patron impact So they're increasing that transparency and changing the way they talk to patrons They talk to staff and externally with their library boards and community stakeholders For example, the atlanta-fulton public library shared their summer reading survey results with their friends group their board And to the businesses that donated summer reading prizes and the businesses What they did was share photos of the kids thanking the businesses and they included some of the quotes from the surveys And the businesses just ate this up. They ended up sharing them through social media And because of this the library anticipates doubling their local business prize donations next summer One parent survey comment also made it into the library foundation newsletter newsletter, which goes out to all of the library's donors Collecting outcome data also helps start partnership conversations. It can create new and strengthen existing partnerships For example, workforce development and small business creation and growth is a high priority for pima county, arizona In the library's quarterly updates to the county as well as their annual report. They included some of their project outcome survey results And when the career center for pima county Asked for more money instead the county administrator suggested that they partner with the library Because from those results, they already knew that the library was doing great work in this area So what resulted was a new collaborative effort between the career center and the library to provide youth with work experience As well as college and career readiness and programming And this year round program addresses all aspects of teen development from academic goals to social and personal interests And combined with the career center's efforts to provide employability skills training It helps create pathways to college career and entrepreneurship for this youth group And you can find more examples like I just showed under our from the field resources on the project outcome website Which include case studies or speaker wall of fame Which includes all of the presentations that people have done alongside us We have over 60 people who have presented with us over the past three years And all of our on-demand webinars as well So now I'd like to circle back on my earlier question What's a problem you need to solve and how could measuring outcomes help? Hopefully seeing a little how the toolkit works and hearing from Pam's experience and some other library examples has helped you Think through how project outcome could help you collect the data you need in order to help improve your impact On those problems you're trying to solve or the community needs you're trying to address I'm going to wrap things up with a few announcements and then we can open it up for questions So we've recently released our second annual report and final project outcome or project evaluation results And you can find these under our about us page on project outcome.org. I suggest you check them out If any of you are attending PLA conference in march in philadelphia, we have two programs featuring libraries using project outcome So, uh, please join us for those if you're able to Uh, we also offer a full day in-person workshop training for state or regional groups who are interested in training up to 60 Librarians in using project outcome So if you're part of a larger group or just know someone, please share that information Um, we're happy to work with them to provide an uh regional training I recommend checking out our tutorial videos and case studies if you're just getting started And to learn more about um, how to use the tools and more about how other libraries are using project outcomes And of course sign up for free at project outcome.org Review our resources that I talked about Get started go in create a survey run it run it through a program and just view your results see what happens Um, also encourage you to engage and ask questions on our peer discussion board Okay, christa, I think I'm through do we have questions? Yeah, okay. Um, great. Thank you very much. Samantha and Pam. Um Yes, we do have some questions. Uh, let's see. Um There's one first question came in up, um earlier in the session and I can't I'm not I don't remember honestly I probably exactly if you'd mentioned this um, so let's know if um, can we link the surveys that you Provide to our social media pages to share them out there and um, so everything Like Pam said every survey will automatically generate A unique url for that survey So if you're confident that if you can share it out on a page and be really specific if you attended this Storytime at this time Um, and people will complete it then you can share that url. Yes, sure. All right um all right then um The question about uh for it was for when um, pan when you were talking about your surveys Um about the return rate of them What the percentage of the uh return rate of your surveys? Um, was it very low? Um compared to other other um Ways of reaching out to your to your community. Um, this person says so in direct mail You're lucky to have a three percent return rate. How does how do you feel that compares to what you got from? The surveys and what you use through project outcome. Is it higher or lower? Well, that's interesting. I've never heard that statistic before. Um Well, I look we sent out about a hundred surveys for the children's program and got 25 back. So that's 25 percent. That's awesome Six out of 40. Oh my mental math isn't good, but that's I don't know 20 so I guess it's not bad um We were a little disappointed with the with the summer reading one But we also think because it was just kind of a okay. Let's do it decision kind of last minute We didn't promote it all summer long and I think next summer we'll definitely do that and talk it up and I think we'll do better Just needs more a little bit more promotion So did you think it was an accurate representation of who attended this time or? Ah, that's a really good question. It's hard. I know these are hard to The the the immediate survey we did with the with the book group, of course was was 100 of the people attending and So of course it was very accurate, but uh with the larger group. That's hard to say It is yeah until you like look at them and then really evaluate what you What they say it's hard. It is there is a difficult question I also wonder uh, if because we only did online surveys with email that some of our patrons Low-income people don't have an internet connection at home. Don't do email. It might not have gotten to them So we realize after the fact that we'll definitely give the paper surveys as an option to people too Different different varying ways of getting that getting it reaching out to them. Yes Yeah, we do know overall that the paper surveys definitely yield your highest response rate So it really just depends on your capacity or your audience You know the the online surveys are good if you're doing some Uh digital literacy classes and people are already at their computers You could already have it up on the web browser and say hey, this is here. Just click complete But it is while the paper surveys do take the time to enter the results into the system You will get more responses if you use the paper surveys. Sure. Sure. Okay All right, anybody have any other questions type them into the questions section in your go-to webinar interface We do have one comment from one of our librarians here and up in Valentine here in Nebraska Project outcome seems like such a valuable tool. I was thinking I could use some of the reports from my annual reporting to the city and county They always like to see outcomes and visual assessments visual stuff. Yes. That's always awesome That would be great. Please do that Um, we tried to like I said in the presentation make everything as user friendly and simple to use as possible So if you do have to enter a couple paper surveys, you know, the benefit is that everything's automatic You get those results. Um, you can analyze them really quickly share them really quickly We know those reports are really useful to small libraries in particular because you know If you don't have the time to create data visualizations, I mean, I don't even know how to create data visualizations Something that can do it for me from the numbers Absolutely and someone else says totally agree regarding any report. Yes. Um, the more, uh, Interesting just interesting you looking you can make them the more people are going to actually read them potentially and um And you know what you actually have said in the in the dense text potentially Yeah, and you know the more you can share the more you can say about your programs and services other than We reached a hundred people You know just more data more data. You're building that story. Yeah Great. All right. Well, it doesn't look like anybody else unless something somebody's typing a really long long question Um, it looks like we are good for this session today Thank you very much. Samantha and Pam for talking to us about um project outcome today