 Should we, should we sally forth and continue to find out what's going on? Yeah, do you have any idea? You might be asking yourself, what did I just buy? An actual fireman? Video three out of four, last one. All right. 75% done. Yes. All time. Just, just close enough. When we get close to finishing, we just add more. Well, we found out nothing else had values. We just dropped it off at 75. It's just fluff. So that became 100%? Yeah. So how to make 75% out of time? So I think the last thing in the series here that we want to remind you of is to evaluate your own effects. I know we kind of hinted at some other areas here, but. Oh, but my policy and procedures are perfect. They don't need to be evaluated. No, they don't. They don't, not at all. Are you a baby analyst? Yeah, yeah. Did you take your research methods classes? Yeah. Yeah. Don't blame me. You're the one I attended yours. That's scary. No wonder we don't measure things well. Yeah. Neither one of us are probably trained. Yeah. It's all good. We're just out there with measuring tapes. I just want group behavior. That's all I want. Just average. Percent. We'll get the percents. Averages. We'll do some enovas and some t-tests and throw in some structural equation modeling and life's golden. There you go and do it all. That's structural equation modeling, but it is now. That's how I feel about it. I'm going to monitor it all indirectly. It would be great. Yeah. Ooh, indirect assessment. I don't want that. Yeah. No. No. No. Please try again. Something. What was it you were doing? You were recording something with people? No. It's from the recent kind of promotion, if you will, and or what's going on. Yeah. Now I'm in charge of people and trying to monitor what they're doing by caseloads. Okay. Yeah. So I got a series of people. Yeah. And they all have a certain level of product, i.e. billable requirements. A series of people. I'm like, what's model? Yeah. Here at Sidecore, we use the science to teach the science. From Skinner's program instruction to show and tell. With a little bit of entertainment along the way. Anyway. Well, we do get into that, but that's way too detailed. Yeah. But the idea being we have certain metrics, those targets they're supposed to be hitting. And I have gotten into the role where my responsibility is to make sure that that was at least hit and investigate what's in the way of why is that achievable or not. Sure. Right? And then there's the new BCBAs coming through and building up their caseload. Well, they're not hitting certain billable targets. Why? Well, they don't have enough kids. Why don't they have enough kids if we start looking at wait lists and all that fun garbage? Right? The big evaluation piece that I was talking to you about was, well, we were noticing that we were trying to get them to hit 25 hours a week in direct billable kind of requirements. Right? Sure. And their caseload maintains because techs go and do their things. And if the sessions occurred, that graph itself does its thing. But the personal one, that's the one where it would, it wax and wane depending on what's kind of going on. Right. Neither they forget to schedule or whatever the small reason is. But what I was noticing is kind of did my own reflection where we would look at, I'd look at the team and all their individual scores. And what was fun is when we first, we were excited about this new process and this new idea, so we'd talk about it. And all the individual hours went, ooh, great. That kind of went up. And then we, for lack of, you know, not lack of a better term, but we kind of habituated to the idea. Sure. Right? And we just, yeah. That was fun. It was, it was cool. And we just forgot about it. And the scores went, boom. Ooh. How are you saying you had an A-B design? I do. Yeah. Actually, this is a B-A. Yeah. You remove the intervention. Yeah. So, and then I was like, well, that's not healthy. Right? We like to eat, go, go work please. Right. So we, so I threw the conversations back in. Huh. And they stabilized. Crossed everybody. So you did a B-A-B design. Mm-hmm. Yeah. So you evaluated your own, evaluated, evaluation tool. You evaluated your own intervention. Yeah. The intervention was that conversation. And now that we've, and became, made that like, I know you might have time, I know you might not have time to meet with me this week, but I at least need 10 minutes to go over these key pieces. Yep. From the leak before. Yep. So you're, ooh. Now you're even refining the intervention. Yeah. Only deliver the stuff that has control over behavior. Yep. So, at a minimum, you need to check in with me on the following. Ding, ding, ding. And now all the scores are going up to where they need to be and holding. I think that's invaluable. Constantly evaluating the performance of the, of the actions of the organization and the administrators in the organization are taking in order to, in fact, change within the organization is important. That's basically, in a sense, a type of treatment integrity. You're saying, did I do this at right number one? And then so did I deliver the information that was relevant? And then number two is that having an impact. If it's not having an impact, quit doing it. Yeah. If it is having an impact, keep it going. Like maybe tease some pieces out and test that and see what the effect those have. So when you're doing this OBM stuff, when you're in there, in that office, you have to be trying to maintain, evaluating your own behavior. It's like a form of self-management. I have a similar issue. There was a report that I generate and send out to people about the performance of the organization as a whole. It's got a whole bunch of little layers in it. And because of random things, I stopped doing it. And it just kind of died. I skip it as a weekly thing. Yeah. Skip it, start skipping it. Next thing, you know, everything tanks. All right. All sorts of problems started happening in terms of staffing, not staffing, but in terms of keeping clients in the door and all these fun stuff. So we put it back. All right. And the things kind of stabilized. Now, whether or not that particular intervention is what's driving that change is a different question. But in the context of the data we have, it sure looks like that intervention is what's keeping the behavior going in terms of certain behavior. I don't want to get too deep into it, because it's a really long example. So we had something, a bee, we took it away, we started losing some clients, and then not bringing in new clients as frequently. So we put it back in place. And again, there you go. Now we're keeping clients up and things like that. So there's probably a piece in that particular big package intervention that is what's doing it. Doing it. Don't know what it is yet. I'm going to evaluate that over the next coming years and figure it out. Yeah. Yeah. So evaluate yourself. Evaluate your own interventions. Well, just be mindful of those evaluations. Because just because it's wrong, if you hurry up and change things, you're going to cause a significant shit show. Did you just say be intentional in a fancy way? I did. I did say being intentional in a fancy way. And I know it's not very behavioral to say be intentional, but the point is make sure that whatever you're doing, at least it's targeted at a specific thing. That's what we mean by intentional in this case. Describe just more like time to schedule training and develop what you want next instead of... Train the response you're after. Yeah. Instead of just walking through and being like, Punishers and reinforcers and... Reinforcers for everyone. You're going to be like, No, here we go. We're going to hold a training. I know it sounds ridiculous because you already know a basic idea. This is a tiny change, but if we don't do it in this organized fashion intention, it's going to roll. And that gives you the opportunity for the person participating in the actual task to provide the feedback about how they can do their job better. Instead of a top-down approach, now you can say, Hey, we've got to improve this system a little bit. You got any ideas? What are you seeing? What are you seeing? Because I don't know. That's a whole other video on that. Yeah. Let's pause right there before we get into that topic because I can see we're cheering ourselves. That's the fourth one you were... That's the fourth in the trees video. Well, we just said this was the final three out of four. That's the fourth one. It is, but we can't do that in the trees. That's a whole different issue. It's true. I don't even know what it is. We'll go find out. Yeah, somewhere else. In a bar. Nice. Sounds good to me. Let's get out of this forest. Alright. Then go look at the trees. Ooh, pretty.