 Thank you so much. I appreciate all of you joining us today for another amazing episode of the nonprofit show today We have Shari Smith with us talking with us about turning data into action So we're excited to dive deep with you into this evaluation conversation Thanks to Julia Patrick CEO of the American nonprofit Academy. We continue to move and groove with the at these episodes well into our 1960s plural now well above that so really excited I'm Sharon Ransom with a frog in my throat. I'm also the non-profit nerd CEO of the Raven group And always join Julia or mostly join Julia here for these shows and really excited to continue to be of service So extremely grateful to each and every one of our presenting sponsors without you We would not be moving and grooving with these amazing conversations You know when we started these Julia March of last year I think we both said it's really a labor of love to get up and do this But these sponsors allow us each and every day to have these high-level conversations and truly Put our finger on the pulse of the nonprofit sector. So thank you to our sponsors and back to Shari again Welcome so much as Shari Smith is the president and founder of Evaluation into action. Welcome. Thank you so much for having me It's a pleasure to be here Okay, Shari. I gotta say You are involved in one of the more I'm just gonna say it less sexy aspects of what we need to be doing But I think you're probably at the core of Decision-making for so many for funders for collaborators for you know, all these different things and yet you're so Misunderstood within this profession and I'm really interested to know Like how you came about this because this is not a typical drill down within the nonprofit sector So how I came into program evaluation and what it's all about. I think you're right I think it's a very misunderstood. I think it stems from the reasons of why people dive into doing program evaluation at all Is usually it feels and it or it is mandated by funders, right? So when you're told you have to do something You know think about it like you have to eat your vegetables, but I really want to you don't understand the why you don't I can you tell I have kids you don't understand the why right? So it's really important to shift the mindset of why we do program evaluation because at its core It's a learning opportunity It's an opportunity to gather data in a systematic formal way to understand if you're truly making the difference You intend on making it at its core What it's all about is what is going well so we can demonstrate the impact we're making and where our Improvements needed so we can use data for continuous ongoing program improvement. Oh My gosh, so for a subject that is not sexy, right? Quickly not sexy. I don't know I mean that sounds really robust and We're also going to talk a little bit today about a newly released book I think you published a book and it came out in January. Is that correct? That's correct nonprofit program evaluation made simple Get your data show your impact and improve your programs It's all about making program evaluation accessible to the non-evaluator because I'm really passionate about getting it out There so people can do this work because too often people feel Overwhelmed with where to start or if it's mandated by funders like what do I even need to be doing and this book? Really drills down the process in a way that makes it accessible to the non-evaluator So let me ask you this question. Are you finding that funders foundations? You know people that are looking to make investments are Looking at your information and wanting to work with you so that you could then Get nonprofits up to speed or is it going the other way because that's a great question, Julia It kind of goes both ways You know, I've worked with both funders and foundations as well as nonprofit organizations in both capacities and one of the things I seek to do is to help with that power dynamic right of Putting nonprofits in the driver seat of the data that they're collecting Rather than funders mandating it and the foundations and that I've worked with are very receptive to that Let's make it a collaborative process, you know, if you get my book a nonprofit program evaluation made simple I talk so much about collaboration because it is so important to Collaborate organization-wide to define the difference you expect to be making I think that's the real stumbling block that people get to right we have to define or impact But what does that mean? You know in fundraising the grant writers put that into the proposals, right? They put those outcomes in the proposals. Here's how we're measuring success or defining success But sometimes and you guys please please let me know your experience I think sometimes my observation is they put those in to secure funds Without necessarily talking to program staff About gathering the data that they're putting Promising into the grant proposal super common in my experience. What have you guys seen? I Yeah, I agree with you. It's very common My two main wheel houses Fundraising and strategy and so I am by no way shape or form like a specialist in program evaluation However, I do see the deep deep importance of using it to help tell the story and to help share the impact, right? And I've worked with so many organizations some new some with newer programs that don't have the evaluation Or they don't have the metrics to even evaluate quite yet. So I'm curious how your book might talk about You know an organization based off of a program that is established and one's brand new and there there is no You know set KPIs if you will So but yes back to your question I would agree it's often kind of done in this vacuum You know filling in the blanks for the funder, but not really eliciting or collaborating back to that word With the program team to ensure that the data that you know The program team is tracking is what you're also reporting on So I would say that there's there's a misconnect there. Yes, you know, I think that there's you know We all know chasing money. I mean, I think so many organizations are so, you know They're so desperate for the money. They'll be like, well, yeah, how hard can that be? We'll add that on or what we'll do this and then it just creates a lot of problems because you can't meet the expectations That's right. That's right. So you said something really interesting Okay, that is going back and maybe I heard you incorrectly But it's okay for a nonprofit to go back to that funder and just say, you know what? I don't think we can do this or we don't have the bandwidth or This this might be a challenge But could we do this and I think that's what's missing with with this relationship between funders and nonprofits they're so fearful of Coming forward and being less than Positive or capable or addressing shortcomings that they have That they dig a hole for themselves and ultimately Create an environment that damages those relationships with those funders because they'll never be able to Manifest all of the demands that these funders set forth. Oh my goodness, Julia I have like just a competing thoughts to come out. So I'm gonna just talk and see what happens. Okay? So in terms of the funder and putting the nonprofit in the in in the driver's seat, right? So when you put the nonprofit in the driver's seat and you collaboratively create a plan internally, right? So that everybody is on board with hey, here's how we're all defining impacts So grant writers you put these outcomes in and you know with confidence Program staff are gathering data around those outcomes. You have to have alignment and agreement with everybody, right? So By doing that nonprofits are standing an expert, right? Because they are the experts, right on what they're doing And then you know instead of running from that fear. I really encourage you to run to it, right? Be brave be vulnerable. All right. I love brunette brown I'm gonna quote her be vulnerable get into it get curious because at the end of the day Don't you want to actually make sure you are making the difference you intend to be making? So the the way you get over things like the fear or the I don't want to report if we're not like a straight A student I don't want to share it To go the way to get over that is to look at what's not working create a written improvement plan, right? Here's what we discovered Here's what we're going to do differently and here's what we're going to measure again When you do that and you share that with your funder that is again Okay, I'm not going to guarantee it. So don't come back to me and say sorry said But I've never had a funder or a or a client or a nonprofit have that go badly, right? It's always been like we have our learning. We're a thriving and learning organization You have to fail in order to learn. I don't think that's a new concept It's been said a million different ways by the million different people, right? That's what program evaluation allows to you to do It helps you define the impact you expect to make right collaboratively gather data around those outcomes And then ultimately report out honestly and transparently. Here's what went well So you're demonstrating impact and here's where we need to improve and here's our improvement plan to get there, right? But the key I think is to have that plan in place. So when the funder is asking for the data They have those questions the grant writer can turn around and say we have this Evaluation plan in place will this fulfill your needs because my experience has been foundations are like fantastic That you have a plan sure because they're not expecting that But I think the more nonprofits that can understand the value that program evaluation brings to them The better that they're going to be able to sit in that driver's seat Right stand an expert and say this is what we do. This is how we're going to measure our success And we'll share it back with you Right that is so critical and for something as we said is not sexy a written improvement plan sounds really good to me And really getting ahead of the problem Um and again to you know quote brunet brown many of us are in the arena We are you know, the nonprofits the leaders that are in the arena and so when covet hit Many clients um, maybe saw a decrease in the amounts of individuals They were able to serve not all but some saw a decrease in the quantity, right? But they went deeper with those individuals which actually shifted Their evaluation and their impact numbers And I've heard that on a couple occasions shari from You know clients of mine here here in my community How does your book or how does your expertise really talk about when something like covet happens? And there's this complete pivot and shift in how we do what we do How what are your best practices, you know to talk about that? Right? So that's a great question. So when something happens What I like to do is help to build capacity and doing program evaluation work, right? So I'm all about being an evaluation guide and the people I work with are content experts, right? So I build capacity so they eventually don't need me at all, right? So in terms of like when covet hit my experience was when I reached out to um former clients just to check in see how they're doing They had built that evaluative thinking muscle So they were right away creating a survey or creating focus groups or creating data collection opportunities To get feedback from their participants How do we pivot so they're not in a vacuum and they're not in crisis? Oh my goodness. What should we be doing? It's we need to get right away like the reflex reaction I was so excited to hear this one is we need to gather data We need to gather data and know what to do next. I actually have On my website evaluation into action.com. There's a blog and I featured in for all, which is one of my clients They're also one of the clients that are included in my book I'm just going to give them a shout out because there are four of my clients That generously gave permission to share their internal processes and documents and that's what's in my book non-profit program evaluation made simple So I spotlighted on them on how they responded to the pandemic and it's that story I just shared of okay. What data do we need? What do we need to collect to understand how to adapt our program for this new environment? So it takes the panic out of panicking That's so no, that's so good Which then you can't spell panicking if you I guess so I said that I'm like shari. That's not working as it came out of My mouth. I was hoping I could grab the words back like come back. It doesn't work I totally get it. No, it's good And we should share but you know before we went live both shari and I Shared this commonality that we're both lesbians. So we have theatrical, you know experience In our life. So we're really just kind of you know impromptu There's no script for any of us. So it's really a fun conversation Um, I love that because I feel too often shari many of us regardless of sector We are so reactive as opposed to Predictive and so these elements and best practices of your book really help us to identify Okay, there is a shift. What action do I need to take to get ahead of that? So I know I myself can't wait to get my hands on your book. Oh, thank you Sorry, I'm really interested Just in general. Do you think that nonprofits are actually Working towards understanding data and making that a part Of their ecosystem. I mean you you said something really interesting to me that you know when the pandemic occurred And all these shifts were going about you could see a delineation between those That already got the practice and had the mindset To use these tools versus those that didn't I mean jared and I in the very beginning And this whole show started because of covet. I mean, yeah It was something that Was generated from this situation But the number of people and institutions some very large that told us we're hitting the pause button We're just going to sit tight And then when it's all over we'll come back Versus those that were like hell. No, we got to lean in and you know, it was such a divergent thing And so I'm kind of curious if if you could track some of this back to the use of data Well, I think that Understanding how to use data, right? I think if you have a formal program evaluation system in place Then you are set up for using data to make decisions as well as to demonstrate impact, right? I think if you don't have that formal system in place I think a lot of people like we talked about earlier are intimidated or afraid or feel like it's going to be a Burden to set up such a system, but I can't tell you how many times I've gone through that process And there's generally at least one or two staff members that are like I Creaky like this, you know, they're like not interested at all But once they realize it's a collaborative process their voice matters at the table We're making decisions together on what data to collect and why to gather them That is an absolute game changer, right? um, so I think that really Really when people are committed to setting up that system and willing to set it up That is when you really change the game, right? But if they're not willing there are you know, I see that a lot And that's where a big stumbling block you have to be willing to shift your mindset But I think it comes from back to what we talked about people are afraid They don't think that they need it and so on and so on But there's so much empowerment to have from having a formal program evaluation system in place, right? What are some of the systems that you recommend um, because You know, I know there's the good old pen and paper and then there's an evolution into an excel document or Different systems. Uh, did your book go into that or can you perhaps go in? Absolutely, it goes into that and so by system. I mean not just the database and the physical technology system I mean the entire system of the people and who's gathering the data and what they're gathering and when and All of those components, right? Um system the whole system, right? So that's so many. So thanks for clarifying that word I do have an entire chapter. They're dedicated to how to manage your program data, right? There are a lot. There's so much out there for managing your donors and your fundraising, right? There's not as much out there on managing your program data in a way So you're not up at 2 a.m. Trying to pull data from an excel spreadsheet. Well, how many I mean raise your hand if that's been you so I mean So that's easy That's easy avoided. I do go into that and I talk about that how to avoid the 2 a.m. Report writing In my book, you know, nonprofit program evaluation made simple. It lays it all out One of the I just want to share this real quick about the book because it was exciting to me I met with a professor dr. Shana Parker and she is at lander university and she's going to use my book in her undergraduate course So any any alma mater serious? Yes, that's fantastic So she's going to use it as a textbook because she wanted something more user friendly And what she said to me was that my book takes the fear out of doing program evaluation And I would my heart just melted when she said that because I'm like success I mean, that's how I measure the success of my book, right? If people feel like they can access it and understand how to do it Then I feel like I've been successful in that So but to get to your question about the systems and whatnot, you know, I want to share this little example I do have it written out because I don't have what everything my clients have done memorized But I do want to share this with you We've been talking a lot about the alignment of setting up a system of like, you know Collaborating and I want to give you a very brief example. This example is in a lot more detail in my book But this is again within for all and they have a stem connect program Okay, and what this program does is bring stem volunteers from local businesses to do a specific curriculum in fourth and fifth grade classrooms You know get kids excited about stem So here's the process three steps to find your outcome, right, which people are like, I don't know how to do that collaboratively I'm going to tell you Gather your data Right and then report it out. Okay. We've got to align those three steps So like whatever you're putting in your grant proposal, that's what you're actually gathering and reporting out So one of their outcomes is to increase the interest in stem related careers To okay the data we gathered we did a pre and post survey with fourth grade students We asked questions like strongly agree to strongly disagree A job in science is interesting to me. I want to learn about different jobs in science A job in science is boring like like i'm going super fast. This is just a little excerpt from the survey So don't email me later. Be like, well, why didn't you ask? Okay, this is a little bit about the survey Third step is the impact reporting it out. So they were able to report out the numbers and the stories So you have the quantitative data and the qualitative data together, right? So you have a mixed method, right? So one piece of data 94 students that had a negative or neutral interest in science in the pre survey Increased their interest in science related careers Stories a couple of quotes when I grow up. I want to be a vet that includes math and science Or I would love to have a job in science when I grow up and those statements came from Because of participating in this program What has changed for you kind of thing? So that was so fast. I know but basically I wanted to show you the alignment I mean if you want to see that full report it is on the portfolio page of my website So you can check it out there But that gives you like your outcome define your impact collaboratively, right? And then gather the data around that outcome and then report it out So those three steps and that I go into so much more detail in my book Specifically, I have a formula in there on how to create measurable outcomes So let me I love what you're saying and it's really pulled a lot of this journey together for us But I want to ask the the deep You know question and that is how expensive is this because in the nonprofit world We're always worried about that bottom line and something that you can't physically see at first A daunting aspect and I I know with with grant review Situations, you know a lot of times people are looking at this and saying It's going to cost me so much money To abide by the terms of this grant In the long run We won't recruit this. What do you what do you say to that? How do you navigate? Well, I think I actually think that um a couple of different things first is a common guideline And I just want to say disclaimer. I did not make this up It comes from the evaluation community But 10 to 15 of your operating budget for your program should be dedicated to program evaluation work now That means um part of the fte of someone internally they are charged Usually the program director is charged with doing the evaluation work You could also hire someone externally like myself to come in and do the evaluation work But it can be anywhere in between you can pull someone in like myself and I do this a lot Of creating the program evaluation plan But then the implementation of that plan and ongoing gathering of the data is done internally So it can be a combination of internal and external But at the end of the day you want to make sure you've set aside some money in your budget So it's a part of the program not an add-on of like oops It's you know, we have a fund report doing two weeks. How do we get two years worth of data? So you want to make sure that you have that planned out? But I do I do want to do another little reading for you I um I have this quote from a client who shared with me how Setting up a program evaluation system really impacted their fundraising capacity So to your question Julia about making this investment and is it going to really be worth it in the long run? And I honestly come to program evaluation with the mindset of it's a learning opportunity, right? It's not a marketing activity. It's not a pr activity. It's not a fundraising activity But it can complement and some and support those things. So here's what this is brandy tech She is with portland homeless family solutions. She's their executive director Also one of the four organizations included in my book. So a little little story time Here's what she said because of having this program evaluation system in place Are annual fundraising from foundations increased by 677 percent? I know jaw drop, right? That's what I did an annual fundraising from individual donors increased by 753 percent So if you're wondering, what did they do when I met them in 2013? They were only tracking outputs or just being counting how many people are being served which historically was Okay to do for funders not anymore, right? So we worked a lot on Understanding the impact they were making on everybody's lives like it with the goal of ending homelessness with these families But what happens leading up to that goal? What is the impact that occurs? Once they were able to really articulate that right? We went through that collaborative process and gather data around those outcomes and then report them out That's what changed their capacity and fundraising Okay, so I know I wasn't the only one whose eyes got like enormous minded too when she told me that I was like Yeah, well that that is unheard of I have a sneaking suspicion Shari that so many people are gonna want to know how they can implement some of these practices Um, I you mentioned so of course you have a book and that is available for all of you watching That will also be added to our book of list Is at our book club on the american nonprofit academy dot com and you also have a workshop coming up Could you tell us about that you bet? So on thursday october 21st at 9 a.m. Pacific standard time I'm doing a workshop program evaluation made simple where i'm gonna really go through in about an hour Like the basics of how to get started on doing program evaluation work in your organization So please join me you can go to my website the link to register is right there on the home page at evaluation into action dot com What is the date again? It is thursday october 21st 9 a.m. Pacific standard time Okay, so if any of you miss our show, we will assume that you're at shari's workshop and we'll be watching our archive our recording This is fantastic and the book So again, it'll be added, uh, correct julia to the website. Yeah, probably up today. Um, yeah So you'll be able to access that Sorry, this has been really great. I have to say that My sense of it is from talking to funders is that they're going to really start To implement more of these reporting techniques and they're not looking for You know rainbows and unicorns and it's all good But the problem has been to communicate this to the nonprofits is that don't work from fear Work with us so that we can navigate So some solutions and we're not going to always hit a home run Right now that will be times when we're like wow that didn't work. Okay Fail fail fast move on let's go. That's right, you know, so I think that this is really a valuable discussion to have and um Really really thrilled that we could have you on today and talk about it's been a pleasure to be here. Thanks for having me sorry remind me um What was that? It was the written something report. I want to write it down Oh, if people want to go check out the report that I was talking about earlier Well, I was thinking of um, so if we don't have the numbers It's the corrective kind of action of like here's what we are doing to take steps towards Achieving the results and you said it was like a written. Oh improvement plan improvement plan. Yeah, right an improvement plan And I have a lot more detail about doing that in my book too. Of course awesome Well, it's been great. You know, um, jerry and I um are always so inspired by the guests we have on I gotta say you're you're ranking up there pretty high sister. Oh, thank you so much. It's been wonderful I think I feel like we could talk for a couple hours about program evaluation. It's so much fun Well, we were not until now really had a dedicated episode on program evaluation Um, and I really think it's important. I know you think it's important too, julia So this is really why you know, um, we really wanted to have you on we'd love to get you on again because As everything continues to change, you know, really looking at how we are Telling our um our impact and you had said earlier, you know, it's not just numbers It's the story so how we can best do that is really really critical during these times. So thank you for Yeah, thank you so much for having me here today. It's been a pleasure and I look forward to connecting again soon in the future Well, we will definitely um Have some more questions and I know our viewers well as well as we end up today's episode We certainly want to make sure that we thank all of our presenting sponsors without you We would not be here having this robust discussion And so we express our tremendous gratitude For allowing us to do this each and every day and as jarrett mentioned this we're into our 360 something episode count So 364 Wow, congratulations. Thanks. Yeah. So um never a dull moment as we like to say Hey everybody, you know, this has been another great episode We're so honored that you would take um your valuable time and spend it with us. We want to remind everyone To stay well So you can do well. We'll see you back here tomorrow everyone