 OTAN, Outreach and Technical Assistance Network. So look and feel included. We're probably starting out with something kind of not so new here today. My guess is a lot of you have gone through this a lot. About half of you are definitely not new administrators. You've been through this many, many, many, many times. Some of you are new administrators. If so, a few of you, this might be new. But I'm thinking most of you have kind of been getting started. We've had a couple of meetings for new agencies. We've had a couple of the regular, you know, we owe it to accountabilities online. So I think the majority of you have gone through this. This is just kind of picking and choosing from the we owe it to accountability, you know, kind of picking and choosing some highlights, the agenda nonetheless, almost word for word, the same as what most of you have seen. And we owe it to accountability in other similar situations. We'll talk a little bit about NRS. I'm not sure if that's come across anybody's radar yet or yesterday, but NRS is National Reporting System. That refers to the feds. We'll talk a little bit about some ways that we're, you know, applying the federal funding in California. And then we'll move out of the agency and state level and more to the basic student level with basic data collection requirements. We won't get into all the gory details, but we'll talk fundamentally, what do you collect it intake? What do you collect it update? And then what are some things you need to do for follow-up? This is not a CASAS assessment session. We're not gonna dig into the CASAS assessment system either, but we will have a few slides that talks about how you apply that CASAS pre and post testing to meet state and federal accountability requirements. Everybody's heard about payment points, but pretty much everybody wants to hear about it again. So we will detail that. Some of the real in the weeds things we talk about in accountability will kind of stay out of that, but we will definitely identify what all the payment points are. And then a little bit on special programs and focus areas will probably stay out of that one. That one's a little bit more for accountability, but I think there are a couple of slides. There are a couple of things like selecting course codes, like designating things for Perkins, designating some other things for CTE that obviously have nothing to do with WIOA2 reporting, but are nonetheless state level requirements. So there's enough of those kind of things now that it's at least worth talking about. And then a couple of resources like the OTN website, CASAS website, et cetera. So a little bit of an overview. I left this here on purpose for most of you. This is something you've heard a lot, but if you really are new, this is important. That is national reporting system or NRS. Again, when we use that acronym, we're talking about the Feds. To be fair to them, they're not the Feds. They're contracted by the Feds to provide a means of regular evaluation for adult education nationwide. You know, it started in the 90s. It started as a voluntary program. 1998 is when I say it became not so voluntary. That's when the Workforce Investment Act or WIOA federal legislation was passed. WIOA was a huge piece of legislation covering a lot of different things. Title II was the component of it that pertained to us. We went through WIOA for a long time up until 2016. That's when the Innovation and Opportunity actor WIOA went into effect. WIOA covered a lot of the same things as WIOA. It was passed explicitly to replace WIOA. That third bullet on the slide shows how, yes, the train did leave the station on time. We started WIOA July 1, 2016. Since that time, WIOA has been the proverbial law of the adult education land. It covers lots of things too. I like to say it's the federal legislation that puts forth those valuable federal grant dollars in exchange for the detailed testing and detailed data collection we all know and love today. One more thing about the federal side, about the data, we also have the federal tables. That's a big job of the NRS. Those are some reports that a lot of you have used. NTE, you know, those are the ones that the feds put out. That is there is a very specific format that California and all states must follow when reporting all the data to the federal government. A little bit on WIOA, just kind of overview. WIOA-wide overview, this covers all the areas of WIOA. For WIOA and WIOA, we have four title numbers. Title two for both is the one that pertains to us. Title two under WIOA and WIOA, the section that covers adult debt and literacy. Title one are the workforce programs. Those are the ones that quite frankly have the lion's share of authority and lion's share of responsibility, you know, at the state and local levels. Why? Because the federal legislation says so. Title three is short-term workforce. Those are the one-stop centers. Title four, voc rehab that serves individuals with disabilities, individuals receiving workers' compensation. That again has been the same since 1998 for WIOA and WIOA. Some of the little graphics at the bottom not really new anymore, but what I like to say is where WIOA has been a little different from WIOA. That is all of that interaction going on. And WIOA, if we looked at a slide like this, most people would have no idea what we were talking about. And WIOA, there really wasn't any reason to know what these other guys were doing. But in WIOA, collaborations required at the state level. It's required at the local level. So interacting with your WIOA partners, 100% mandatory. I'm not gonna get into the table four or other federal tables weeds, but I do just wanna point out that the fed tables exist. The big ones in my mind are table four and table five. Table four is the one that reports measurable skill gains, mostly but not limited to those gains that students make between pre and post testing. You can get a measurable skill gain through pre-post for us in title two. That's by far the most common way students make gains. They can also get one by getting diploma or GED. There's also ways that they can get certain workforce outcomes if they're in integrated education and training, but keeping it simple for now. Table four calculates all that. It has all of those various levels, six levels for ABE, six levels for ESL, and it calculates at each of those federal levels what percentage of students achieved a measurable skill gain. Usually we're looking at it from a pre-post format. How many of our students at each level made a pre-post gain? The way the tables formatted is it looks at all of those measurable skill gains and just looks at the percentage at each level. I make a big deal of this because I do think, overall, this is kind of the name of the federal reporting game, what the feds really wanna know at each level, what percentage of students made enough gain on their testing and or achieved some other acknowledged outcome to move up to the next level to get that outcome. That's how they measure us in California. They do it the exact same way for all other states. They'll look at our performance, assign us goals based on this table, usually a couple of years in advance. And then we're expected to meet or exceed those goals at each level when they look and say, hey, California, you're doing a great job or not so much, this is how they're determining that information. The other big one, again, staying out of details, but just pointing out highlights is table five. This is the one for follow-up. By follow-up, what that means is its outcomes we're looking for for students that exit the program, not for students that are attending right now. That's just a fed say so issue as well. There's certain outcomes. You can see here employment, wages, entering college, that are very important outcomes, but for federal reporting, the feds are just more interested in those outcomes for the students that exit. They're much less interested in those kinds of outcomes if students get them while they're attending your program, they measure it more on what happens after they exit. Long story short is that makes the data collection process very, very, very, very, very, very, very, very difficult. We have to figure out ways to get data on students that have exited our program. So this crazy chart is showing table five. You can see we're looking at employment, wages, entering workforce training, entering college, a certain amount of time after exit. The way the feds have it designed is to use data match. On the slide, you can see there's the areas for college and post-secondary, where we have one data match with the chancellor's office, another one for employment and wages, where we have a data match with the EDD. Long story short is the one on the bottom uses a fuzzy match. It incorporates demographics. It's not perfect, but it gets enough matches to where it passes federal muster. As a lot of you have heard on the EDD side, they've always required social security number. As you know, that's a big issue. So that, you know, when we relied on that data match, that guaranteed exceedingly lousy results. Yes, the feds noticed. Yes, they did call us on it. Yes, they did demand that we do a lot better. That's what brought up that illustrious employment and earning survey since about the beginning of 2019. We've continued data matching, but we've also done that follow-up survey to try to bolster our data with more results. A lot of you have heard we now have goals. We have made a lot of progress, but we definitely still have a long way to go based on the goals the feds have set for us and so on. But again, this is the culprit of why we have to do that survey. This is the culprit of why we have those goals and why you listen to this stuff incessantly is we have this follow-up table for which has always been, you know, a lot of difficulty for us in California. So I just wanted to give the overview there lots more we could say. I don't see any Q and A, so I'll just keep on going here. If you have questions, feel free to ask. I'll just keep talking. You know, I think I have till 10, 15, so I better just keep talking, wind me up, doll, or I will never make it. Jay, we do have a question. Yes. Please define fuzzy match. Okay, specifically that means they use demographics, not just social security number. So I think specifically, but I'm less than 100% sure, but I think they use first name, last name, gender, date of birth. Again, not perfect, but that does give us way more matches than we get if we rely on SSN. So there's still a lot of people we probably leave on the table with that, but it gets enough matches that we can report good enough outcomes to where the feds leave us alone on that one, but the one that relies on SSN, obviously they don't. Okay, so this is what I like to call the transition slide. I also like to call this one the follow the money slide. So this shows how it kind of transitions from federal to state. I like to say the buck starts at the top here with the office and career technical adult and or octay. That is the component of the U.S. Department of Ed that's responsible for us. That is responsible for career tech Ed and responsible for adult Ed. So the buck starts there. Before I get into that, I'll point out there's the NRS that we talked about a couple of minutes ago, not the feds, but contracted by the feds from a data point of view and from a deliverable submission point of view, they're kind of running the show when we submit data which we just did less than a week ago. We reported to NRS, they have the website, kind of do a lot of the same stuff on a national level like we do at CASAS for California. So following the trail under WIA and WIOA, it's always been up to the state to decide who runs WIOA too. In California, it's been CDE all along. So CDE receives that money on behalf of the state of California disseminates it to use guys, the providers and then at the same time, does that federal monitoring to ensure it gets applied and used according to federal guidelines? And then I'll just point out there's CASAS at the bottom. We're doing a lot of the same quote unquote regular evaluation activities at the state level, collecting the data, aggregating it at a state level, submitting it, that is a lot of similar functions to what NRS does at the national level. I'll point out we're one of three leadership projects. I make less of a big deal about that ear because you're hearing from them plenty, but there's three leadership projects, CASAS, CalPro and OTAN each responsible for different areas. You're obviously hearing from CASAS this time. I think CalPro is yesterday and OTAN is tomorrow if I'm not mistaken. But sticking with CASAS here now and moving along, these are our deliverables again, kind of sticking, you know, a little bit at 40,000 feet above for this one, but we have quarterly and end of year data submission. I feel in my mind of minds anyway, it's a lot simpler the last couple of years than it used to be. That is the last two or three years we've had the quarterly data submission wizard and DE, which allows you to do WIOA2 data and CATE data in one fell swoop. It also includes both data submissions and both DIRs every quarter. So long story short, as you meet this requirement by using that quarterly data submission wizard and DE, most people anyway say it is easy, a lot easier now than when you are sending reports and PDFs and all that good kind of stuff. You basically repeat that process at end of the year. You run that same quarterly data submission wizard. You get all that done on or before July 15th. Once you do that, then we have that payment point certification letter based on your payment point summary report and DE that you submit. You know, long story short, as we have that certification process at end of year, we don't do that at the quarterly, but we do that at end of year, of course, because that's the data that affects your payment points results. So we have to be extra special, careful and make sure that the total you think you're due to earn is exactly equal to the total we think you're due to earn. You know, we have that process to go back and forth. Maybe it's fixing data. Maybe it's reconciling reporting. Maybe it's something else, but we'll keep doing that until we ensure that what you think you're slated to earn is the same as what we think. That's, you know, a big part of that letter that you sign showing that you agree that you're slated to, at least at the time of July 15th or August 1, that you're slated to earn X number of payment points. There's also that employment and earning survey that we already mentioned. Those due dates are the same as the data. It's a different wizard in TE. I think most people would say that wizard, you know, not that hard, but certainly a lot more complicated than the data submission wizard. A lot of decisions you make there, but long story short is there's a separate wizard that you use to do the follow-up survey. It basically uses all of those federal guidelines determining who the exitors are for a quarter, who's two quarters after exit, who's four quarters after exit, generating that list and then automatically sending that follow-up survey to those students, sometimes via email, sometimes via text. It allows the student to automatically respond. Some do, some don't. There's more process where you then can follow-up after the follow-up, so to speak, to try to get more results. Okay, this is just a quick table of the due dates. The due dates are one month after the last day of the quarter. So a simple example, we just had the last day of the fourth or the first quarter, September 30th, first quarter, July, August, September. Obviously it ends on September 30th. The due date is one month after the last day of the quarter. In this example, that due date would be August, October 31st. You can see the due dates are set up in that fashion. The one exception is the one everybody hates, July 15th end of year is a little earlier. I think most people figures that should be later than the others, not earlier, but because of that October 1 federal deadline, there's just no way we'd make it any other way. So that due date is a little earlier. You can see we have other deliverables in addition to the follow-up survey. And I think I put that all in one slide here for time. So we have these additional deliverables do each quarter. If you look at that beginning a year letter and the accountability training, we detail all these other deliverables. So we just point out that you need to do them here without getting into all the gory details, but first quarter there's that deliverable to do at least in one ELCIVX co-app. There's the CDECASIS training deadline on or before January 31st. There's the continuous improvement plan and IELCE report on or before April 30th. And then end of year, July 15th. There's some other, the WIOA survey is due March 31 and there'll be anyway more information on a couple of these, there's more to it, but this is a good summary, quarter by quarter, every quarter you've got the data, every quarter you've got the follow-up survey, most quarters there's usually some little goodie in addition to that, that you need to get done on or before that same streamline due date. I'll just say I still don't see everybody's question list wonders here. I don't know what's on. Oh, here it comes. This is Jim, you do have a few questions in the Q and A, do you wanna take those all the way to the end? Okay, sure, if I fire them up, I keep seeing nothing up there. Okay, we have one, this refers back to when you were talking about social security numbers. What happens when a student doesn't provide a social security number? If they don't provide it? Yes. Well, no grave consequences to the student or to you obviously, but there is incentive in the sense that that employment and earning survey that we described a couple of minutes ago only goes to those students without a social security number. If they have one, then they're good to go through data match. There's no reason to bother the student with that survey. So not a huge consequence, but there is a positive consequence if you provide that social security number. And then I'll just add, we've been talking about this a lot here the last year with federal goals and so on is how it helps our data collection efforts. We do have some goals. One of them is for providing social security number. There is a slide with those exact numbers I think coming up here soon. Great, and we have another question. Is the follow-up survey required when we have the student's social security number? Right, no, you still need to run the wizard either way, but the wizard will automatically look and see who has social security number and weed those students out of that survey wizard process. So the only students that get it are the ones that did not provide it. Okay, well, a couple more have rolled in. I see now. Yeah, all right. Now everybody's feeling a little safer there, huh? Okay, so well the IELCE report and the SIP be gone over in more detail in the data and account- The short answer there is, yeah, I see him now. The short answer is yes. At least for IELCE, I would expect to probably see more information on that after the new year. There's usually more training and things like that. Probably January, February as we're getting closer in the account regular accountability trainings, there's more information in terms of what to submit and when, as far as gory details on how to reconcile it. We don't really go over that, but for IELCE, certainly early, and I would imagine for the SIP, it'll be a similar sort of timeline. Okay, and then do staff proctoring CASIS, they might be meaning assessments need to be retained every year. What does the CDE training refer to? Okay, that's the, well, there's one CASIS. This is one that's really better for accountability, but yeah, one CASIS implementation and when we owe it to federal accountability per agency per year, yeah, that has nothing to do with proctoring. The CDE requirement is at the agency level, not individual level. For proctors, we don't really have a formal requirement to renew that, although if you are a proctor, we've always recommended every three years, you should renew that certification. Cheryl, I'll just say that if I answer it, the way you wrote it, that's an hour we don't have, need to be way more specific, not sure what you mean specifically there, that the wizard is just for the end of year, you're indicating a number of teachers, number of administrators, there's an area for certifications, there's a set of tables you fill out once a year, you don't need to do it every quarter, but you do need to do it annually, you just fill out those tables, when you do that end of year data submission. Okay, hopefully that covers it. Okay, sorry, I kept on seeing zero up here. I guess everybody's dutifully looking at it. So whenever I look, there's no little balloon there like the 99 plus for the regular chat. Anyway, go ahead and continue. Thank you, no, good, thanks. And yeah, the more questions the merrier, if we get zero questions, all I'm doing is scratching my head, if we have to give slides short shrift for questions, that's a good thing, not a bad thing as far as I'm concerned. But going back to the slides, hey, we've got the quarterly data submission wizard, I've already said, I think it's pretty darn easy, but hey, if you disagree, here's a link that just gives you the TE help for running that wizard inside the software. Similarly, and a lot more people would say, hey, there's reason for needing more help with the Employment and Earning Survey wizard, but here's a link to several help documents that help you go through those steps in that particular TE wizard. Long story short is we've got it organized by quarter because some of the steps do appear a little differently in TE depending on which quarter you're running that information. Okay, here are those goals. I knew what was coming up soon. I did know these percentages, but I still shrugged it because I knew we had the slide coming up, but I think this came out early November. So we've had these for almost a year now. That is, we talked about how we've always had a lot of trouble with this. Yes, the feds have noticed. Yes, the feds have called us on it several times. So you might say a year ago they threw down the gauntlet, hey, we're tired of talking in platitudes. So they gave some specific numbers a year ago. It's an either or that is, the goal is to get 45% social security number or 60% response rate on that survey. To be clear, they're not looking for 60% getting a job. They're just saying, hey, we need to get at least 60% of the students to respond to the survey. If we can get either one of those, they'll consider the gauntlet, which sounds good. Maybe a little more context makes it less good. And that it is a big poll vault, not a small one, again, unofficial results, but based on what we just submitted October 1, we're still at about 22, 23% social security number. We have bumped up to the high 30s now in response rate. I think we were kind of stuck at the low 30s for a good year. It does feel like we've made some good progress there. More on that maybe next week when we do that statewide meeting, we're still a long, long, long way from 60%. But there's at least some good news that it really does look like the 22, 23 year overall was a lot better in terms of survey responses than 21, 22. So there's at least palpable growth here. The other big update with this is ITIN or taxpayer ID. My guess is a lot of you have heard about this one, but this will be allowed in addition to social security number. Back to the, hey, we're always doing lousy and California narrative with this, the big reason, again, relying on social security for that data match. So EDD is leaving the door ajar, they're not opening it wide open, but they are at least letting it be ajar, say, where they're gonna allow taxpayer ID in addition to SSN. Not sure if that helps you or not. Some feel like it really will be a big game changer. Others not so much, but either way, if you have students that do not have SSN, but do have taxpayer ID, it's still helpful to provide that. There is a new field in TE to put it in and it will help with that data match and it will accept the student from the Employment and Earning Survey the same way it accepts students that provide social security number. The second bullet is just, it looks just like an SSN, but it begins with the number nine. Yeah, as long as the student responds, yes, as long as we get those survey results, counted by completed results that we can report to the feds. So yeah, if they tell you over the phone, that's the same as if they complete the TE wizard. And there was another one that got cut off here. You answered that one, Jay. It was regarding if the ITIN number can be counted. Oh, okay, got it. Thank you so much. Jay, when you are answering, can you read the question real quick because I'm not sure everybody has the Q&A open. Thanks. Okay, so I will just say, Bev, I think I answered your question. I'll leave you kind of hanging if you beg to differ. But anyway, I think we've got these goals in ITIN, so great. Here's one, I'm not quite sure why I flat this one down here, but this is just repeating what I just said that we realize that early deadline is unpopular, but we nonetheless need to continue it as is. It seems like it made a bad situation worse, but I think it has alleviated a little bit by attaching Cape to the same deadline. Most people hate earlier deadlines, but it does seem like having those two dates be the same as help more people than it's hurt. I'll just say a few years ago, we did change that. So it's all the same day that was actually was by a popular demand. So are the percentage based on enrollments or exiters? The percentage, you know, I think you just mean how do we do the denominator? It's gonna be the exiters for that particular quarter. So like we've got, I mean, so employment two quarters after exit, every agency has a big bucket of students that exit. And so it's looking at, you know, everybody who exited sometime in that, you know, quarterly timeframe. So that's the denominator. So yeah, if it's an enrollee that's still with you and hasn't exited yet, then they're not eligible for the survey. They're not eligible for data matching. So yeah, they're not counted in any of this data. I think that's what you're asking Laura. Okay, moving to great questions. I'll just say, but I better move along. So now we're kind of in a transition. Again, I did cut out some of that because we're consolidating two hours into an hour and 15 minutes. We're just sort of jumping right in, but I will note we're kind of moving out of agency level issues and state and federal level issues and more into student level issues here with this entry record slide. I'll also point out, I always like using those old school bubble forms to convey this. Nobody really uses those anymore. If it was 2006, you'd all be using them, but nobody's using them anymore, but it really helps to convey this to say on the entry record, on the update record, just saying. So on the entry record, meaning what you're required to record for the student at enrollment, you go ahead and complete that entry record that records that enrollment in a federal program. That is there an ABE, ASC, which is adult secondary or ESL, as you're recording that enrollment in the official federal program. As you're doing that, please do collect demographics, gender, race and ethnicity, date of birth all required. Please do collect education level. Please do collect labor force status. Please do collect barriers to employment. You can see some nifty little screenshots of what those fields mean on the slide. Here's a little bit more. This was kind of a marquee item for 2022 because we made goals required again last year after 10 years of it not being required. I think long story short is at least 98% of you were collecting goals over that time anyway, even though it wasn't required. Now that it's required, it really hasn't been that big of a deal. Another long story short is back in 2012, for federal reporting, it stopped mattering. Goals didn't matter any for federal reporting. That's why we stopped requiring it because the feds didn't need it for anything. But over time, we realized, feds are no feds, we really need that information. Okay, Jim, did you answer it, Jim, or did you want me to answer? The person was just thanking you for answering her previous question. Thank you, great. All right, so moving on to updates. I will sidestep here and say, since 1999, for those of you that are new, we've always had a little bit of a difference in the way we present entries versus the way we present updates. For entries or enrollment, we're really finicky. We say, do that as soon as the student enrolls. Day one, square one of your interaction with that student, you're required to document that student's enrollment and that student's participation in your program. That's pretty rock solid that you need to do enrollment or entry records immediately. For updates, it's always been a little looser. There's no exact timeline. We've just say complete update records after a substantial block of instruction and said, for updates, you can kind of tailor it to your own agency. Maybe it's monthly, maybe it's quarterly, maybe it's a semester trimester system, kind of how you like. So that's what we've said all along, that you need to do entries at this particular time. Updates more up to you. The last six or so years, though, that we've been in, the exception is for attendance, because we've got a short timeline. I won't be goofy quiz master and just say there is a federal reporting issue called periods of participation or POPs. It's just one of those things we don't have time for today. You can go to another session to get it. It's 10 or 15 minutes. We don't have right now, but for federal reporting, entry and exit is based on attendance. There's fields on the update that matter like not at all. In the old days, it was up to you to determine that. Now it's not. Now it's based exclusively on student attendance. Bottom line is that means you need to have attendance in TE for all your students because TE needs to calculate who's exited and who's not. It's based on 90 days of no attendance. So we're saying, yeah, at least once a month get your attendance in TE. Maybe you're doing it the old school way like we're showing on the screen where you're doing bubbling and scanning. More likely you're either entering it directly in TE or you're doing a third party import to get that information in TE. Either way, by hook or by crook, you've got to get that attendance in at least once a month. TE needs to determine for every student whether or not they're an exitor and furthermore how long it's been since exit for all that crazy follow-up reporting. I just got through explaining over the last 10 minutes or so if you do not do this, then you'll get false positives. There's nothing to undo. You just have to live with the fact that you bothered yourself and you bothered your student unnecessarily because you're not doing a good job getting your attendance in. Here's one more on update. This refers a little bit to in recent years, there has been angst people. We've had that information out about POCS for a while. We've had information out about how a lot of those update fields don't matter anymore for a long while, but some people still fill it out. So I've really been trying to discourage people from spending a lot of time, teachers beating their heads against the wall, trying to fill out fields that are 125% unnecessary. That's obviously a bad thing, not a good, but not to say, don't do the update. This is the one thing on the update that I say more important than ever. This is why you still need to do updates is we have all these outcomes. For CAPE, there's a lot of outcomes we need to mark. And on the WIO aside for things like IET, there's a lot of things now more important that we never used to care about, that now we do. So marking those outcomes for every student more important than ever. Okay, don't see any questions. So I'm just gonna switch gears here again, talking about assessment policy. Usually these are a few slides, but this year it's a bigger issue. With the long story short being, we've got those new CASAS tests available this year that is listening and reading steps for ESL, math goals too for ABE. So a lot of new tests come across everybody's bow here at the start of the year. For any year that CDE statewide assessment policy is the official document giving you guidance on placement, giving you guidance on pre and post testing, listing what CASAS assessments are allowed for state and federal reporting. It also gives you all those dos and don'ts around testing and it also provides those guidelines for local assessment. The local assessment policy moving to the next slide is what I consider to be probably the most critical piece from an agency point of view. That is the statewide policy. It requires you as local agencies to complete a local assessment policy and revise it at the start of the program year. So I'll say this is the last time I'll say this, tis the season to update it if you're doing it right now, quite frankly, you're a month or too late, but hey, better late than never. But at the start of each year, every agency should update the assessment policy. Who are your proctors? Who are your coordinators? What's your agency-wide testing schedule? What are your preferred tests for ABE? What are your preferred tests for ESL? How is your agency handling important issues such as test security? All of those things are things you should put in writing and list in your local agency assessment policy at a bare minimum if you get monitored 100% chance that will be something that gets looked at likely. It's the very first thing. So you need to make sure that's in good order. I always make a big deal about this, but this year more than usual because again, we've got steps, we've got math goals too. Not all of you are jumping head over heels on it, but most of you are at least kicking the tires on it and incorporating it into your agency's testing. If so, there's 100% chance you need to revise your agency's local assessment policy at a bare minimum to talk about how you're using steps, math goals too, et cetera. Even if you're not doing that, of course, there's still 100% chance you need to update that local agency assessment policy. But again, I think the other slides are kind of detailing what I've already been talking about. Okay, how often are outcomes zeroed out for reporting? I don't know what you mean, so I don't know what that question means at all. I'm confused about the two different types of outcomes. I guess I don't know what that question means at all. What do you mean slide 19 that teachers do? Sorry, I don't understand your question, Lars, and I still don't understand your question. Aaron, Mark, where does that policy live? It's on the CASIS California Accountability page where you get the CAPE data dictionary, the WIOA2 data dictionary. We've got attachments A through X. It's a little crazy, I admit. I believe the statewide policy is simply attachment A and that local assessment policy template is attachment B. Okay, hopefully that, Mark, I do think I answered your question. Sorry, Lars and Aaron, I don't understand either one of your questions. Yes, okay, is that? So, yeah, for WIOA2 reporting, it's annual, you know, that's something I usually don't get into because it's 10 minutes, we don't have, but I'll just say for federal reporting, I mentioned those POPs. So for federal reporting, it's based on periods of participation. Yearly is not what they use. It's all based on those periods of participation based on that federal 90 day rule, which is exclusively based on student attendance. That's one big reason why we make such a big stink and deal about that. Just say one, allow me to excuse me why I talk out of the other side of my mouth, however, that for payment points, it's the exact opposite. For years, for WIOA, it was always July 1 to June 30. For California payment points, it's continued that old school way where it's still July 1 to June 30th. Gotta say, it's way out of date now, but for pay for performance, you know, I think everybody feels like there needs to be something a little bit more finite POPs don't really allow that finite way to look at things. So it's easier to use a date range. We're getting there, Kristen, you know, a couple slides. I'm gonna wait until after we get through these. I don't think it will answer all of it, it will tackle some of it. That is, I wanna get into some of the things with assessments, I'll skip to this slide and then backtrack, Kristen, you know, this covers kind of what you're asking in accountability. This is three or four slides, but just to be clear, for A, B, E, A, S, E, you use reading or math, for ESL you use reading or listening to your question, it's either or you're not required to do both, you are required to do one or the other. You know, it's agency level choice on for ESL you can do all listening or all reading or you can do both same sort of choice available for A, B, E, but back to your question, you're required to do one or the other, reading works for both, listening ESL only, math, A, B, E, A, S, E only. I think that's what you're asking speaking, writing, you know, optional, I'll just say reading, listening and math are the ones that are used for state and federal reporting. You're welcome to use writing or speaking, but they don't necessarily have a role in this federal reporting like reading, listening and math do. I think that's your question, Kristen, I guess you can elbow me in the ribs if that doesn't answer it. Here's another one where I don't want to go into gory detail, but I did leave this year because I do think it's just kind of, hey, the topical issue, pink elephant, that is, hey, we've got a lot of new tests out this year. The most, the biggest new test year that I can remember in 23 years across us, that is we've got listening steps, reading steps, say on math goals too, all at the same time. So obviously lots of information out there on new tests way more than we have time to talk about now. So number one, if you have been trying to figure it all out, there's a lake, there's a bunch of pages on our website with everything you want to know about these new tests and more, I present these charts, there's three charts, one for listening, one for reading, one for math goals too. I'm not going to go over all the details, but I just want to point, well, here's some basic information like level and form number. I like to point out more for experienced users, I guess, but that's about half of you here. If you're a new user, I guess you can start kind of having it be this way anyway, so it won't be that confusing perhaps. But with the old tests, there was a lot more uniformity in my opinion across the different forms in the same series, across the different forms in the same modality and so on. When you look at these charts for math goals too, when you look at these charts for reading and listening steps, you can see things like scale score alignment, lots of variation, things like number of test items and test time, same variation. So it's not good, it's not bad, but it does require a little bit more attention as you're bringing in students to test. You might say, using the sensationalist example here for listening, where the test time's all over the map. Good if you wanna do listening, but just warning, hey, you just need to be kind of careful when you do listening and that is you need to set up different test sessions to make sure that the student doing the test has the appropriate test time. If this is the math goals too, kind of the best example, a little oversimplified, but you can see at the high levels for math goals too, you need 75 or 90 minutes. That's what you need to have it be statistically accurate. I think a lot of agencies are used to that nice round number of 60, maybe for some of you it's 65, for some of you it's 55, but your average agency I think has a pretty uniform test time. The old series overall were pretty conducive to that kind of consistency. The new forms are not at all conducive to that kind of consistency. Quite frankly, yeah, you do need to bite the bullet. You do need to do that extra scheduling. In this example, you do need to make sure if you're using math goals too that the lower level sessions are given 60 minutes and the higher level sessions are giving 75 or 90 minutes, which A means you've got to separate them out, offer multiple test sessions and B, yeah, that'll probably cramp your style. If you do this, you're gonna need to schedule 90 minute sessions. If you have somebody with an E level math goals too and you only give 60 minutes, yes, those test results will be skewed. Yes, in that example, it is 125% your fault because you're scheduling 60 minutes when you should have scheduled 90. Yes, that is absolutely something where you're artificially causing problems with your agency's testing. Sorry, there's just no way around it. You've got to do that extra scheduling. Just saying. The appraisal in the locator, the appraisal is kind of the full load, so to speak. That is at some roughly the same length as an ABC level test. The locator is the one that's designed to be able to identify that level more quickly. Prior to these new tests, the locator was an E test, but never had a paper copy. That is something a little different about the new test. There is a paper locator. There wasn't before, but the locator is shorter. It's designed to more quickly, put that student in a level. Sometimes people don't think the locator does it exactly right. The recourse is you can do the appraisal instead. That works just as well. It just means you and your students are committing a little bit more time, but that's always a good way to do it as an alternative if the appraisal isn't working. The other reason for appraisal is like Title I's or workforce programs, you may be doing it just to determine basic skills rather than a precursor for pre-post. If so, you might want to do that appraisal. The other thing is scale score ranges. That's 10 minutes or probably more like 25 minutes we don't have right now, but that's another thing that's been a big issue I'll just say to avoid sweeping the pink elephant under the carpet or whatever. That is scale score alignment. Scale score alignment is the short answer to why a lot of you spent years of your life over the last few years doing all that COSIS field testing and why we at COSIS spend years of our life on the same thing. That is all of that data is so we can align our scale scores to those federal levels. If we don't do that right, we have a 0% chance of getting federal approval. So we must align those COSIS scale scores to the A through E levels. We got to do it for every single series. We've got a large number of calisthenics we need to get through successfully to get federal approval. Long story short is that's a lot more difficult now than it was 20 years ago. We need to do a lot more analysis and a lot more alignment, which means those scale score ranges weren't the same as before, but it varies from level to level in modality to modality. There's no consistency with that either. We end up needing to adapt it by series and by modality. So a lot of you have benchmarks like 180 or 200 or 236 or 246 that a lot of you are used to using a lot to your advantage that really aren't doing you much good anymore. So you just need to look up those scale score ranges whenever it's time to figure things like benchmarks or test sessions out. Okay, it looks like no good. So I went a little bit overboard with that one probably. So with payment points, I'm going to just rattle off what earns payment points and then dig into a few of them. So we have some payment points that we call benchmarks not really a big deal, but a common terminology issue for the most part when we use the term benchmarks and payment points we're talking about the same thing but a lot of times Yahoo's like me always insist on using the term payment points never benchmarks. I think the technical distinction is in California we have that payment points, pay for performance system, all of those different outcomes that relate to federal reporting that gives you that pay for performance payment point is called a payment point. Benchmarks are those outcomes from federal table four that comprise a big portion of those payment points. That is all benchmarks or payment points but all payment points aren't necessarily benchmarks you might say. I don't know if it's on this slide deck but if you go to the website and look at those links there's all kinds of information on that. In the WIOA 2 accountability we've got math goals to listening steps and reading steps, those conversion charts just like you've known forever with the old series. I think I took some of that out of here though just because yeah, I needed to consolidate and I needed to trim somewhere. But again, back to here the benchmarks of those same outcomes we talked about with table four complete a level refers to those gains between free and post or you can also get it by getting high school equivalency or high school diploma, sorry about that. Okay, we also have student outcome data sets or SODS that relates to EL civics. We've got those three EL civics focus areas not so coincidentally we've got payment points that relate to each of those focus areas. For civic participation that's where we're using those additional assessments or co-apps that's another 15 minute rabbit hole we don't have time for. I think there is going to be an hour somewhere that focuses on EL civics. So you'll be able to get in those weeds a little bit better with that presentation but Civ part is the one where you're identifying locally which assessments to use and then getting CDE approval on the CASAS website. SIP prep is using CASAS assessments specifically citizenship interview or government in history and then there's that 243 not really new anymore but new over the last five or six years. That's that integrated EL civics where you're combining ESL and EL civics with workforce training from a payment points point of view that's using co-apps just like civic participation. The difference is for 243 there's a smaller number you can use. Not sure of exact numbers but there's roughly 60 co-app areas out there for which you can choose what to do with your students roughly 20 to 25 of those 60 are considered quote unquote workforce appropriate. Those ones that are workforce appropriate are the ones that you're allowed to use for 243 IELCE. Okay and then IET outcomes same thing as what we just talked about for the 243 sides but it also is referring to those that are enrolled in IET or integrated education and training. That's a checkbox you need to check at the class or student level. You're just gonna have to live with you're just gonna know that that has to happen but in any case in IELCE the goal is to have most of your students co-enrolled in some kind of workforce training maybe locally maybe with the employer maybe with title one maybe with somewhere else. In some cases though they may be doing the extra work on the ESL side but they may not be ready or may not be willing to enroll in workforce training. When we started this we agreed the ones that are enrolled are doing more work and usually you know merit a higher dollar amount. So oversimplified here but the data for this one is exactly the same but potentially the dollar amount might be something different. Okay this one kind of flaps in the wind here probably should have moved this up a little earlier but this is the attestation. We started this two years ago the last couple of years we've done this on January 31. We're doing the same thing this year but it's moving up to December 15th. So around mid-November this year because we're moving up this deadline we send everybody attestation trying to make that long story short. I think I guess I tried to get out of these. We do that HSE data match. The HSE data match is what really pays off students or pays off agencies I should say for getting GD or high set. We use that data match to verify the results you report in TE. What we found a few years ago is a lot of discrepancies we'd go back to agencies and verify. Yeah the discrepancies are 100% truth. So we needed to go back and redo the data match for several years in a row. So we brought up this attestation as a way to give everybody fair shake and review those totals after the data match and give you a chance to kind of wave your arms if you feel like the result was not what was right. So you can look and verify that after the data match. Sometimes the agency's right, sometimes it isn't but it gives you a chance to point out discrepancies. The other one relates to high school diploma. For high school diploma there is no data match. You're just marking in TE that they earned it but for auditing purposes that are looking at that certified list you keep at the district level in order to guarantee all those high school diploma payment points are on the up and up. At some districts you've got that list available when you're submitting your data. It's not a big deal but for some districts they might be a little bit slow getting that information completed. So if they don't have their act together at your district by August one you just kind of have to rely on what you put in TE but it also gives you a chance. Hey our district does this but it always runs late so you can use attestation to then go back and make sure that everybody you marked in TE is on that certified list. Okay so I guess that was just listing it. You know I'll just point out again for HSE it uses that data match for high school diploma you have to verify locally whether it's on that certified list there is no data match. Okay so moving to ELCIVICS. You know we've got a little bit of you know this is just showing how we determine who the quote unquote ELCIVICS students are. That is it's looking for the presence of an entry record in a class that you've designated for that ELCIVICS focus area. You know if you're looking at updates pre and post testing it's just following the same guidelines as for quote unquote regular ESL student and then for payment points purposes for WIOA2 reporting purposes ELCIVICS students are instructional program of ESL 100% of the time. Okay detailing civic participation again civic participation is using additional assessments or co-apps don't have time to explain everything about those but again those are those 60 different areas. There's a bunch of information about this on our CASUS website. This is where you select which areas to work with your students on locally and get CDE approval for each of those areas on our CASUS website. The maximum at the student level for this is three and order to get that payment point you need to achieve what we call a complete outcome data set that is you need to complete an entry update pre test and post test for that student in order to get that ELCIVICS payment point. For SIT prep we use CASUS assessments not co-apps specifically citizenship interview and government and history. If the student passes one or both of those they can get SIT prep payment points and SIT prep it requires that same complete outcome data set entry update pre test post test. For 243 we're using additional assessments just like we are for SIVPART. We're just operating from a smaller list. The match for ELCE is three just like civic participation. It has that same requirement for complete outcome data set. That's when I probably should have moved up a little bit so I there's a little bit out of order. I kind of rearranged this because we're setting it up for last time. I'm seeing this I should have moved this up a couple slides earlier but in the class instance record you're designating the program for that class or if it's multiple programs designate multiple this is where where TE is determining at that class level which ones relate to federal reporting for WIOA2 which ones don't that is specifically which classes relate to ABE, ASE and ESL that's important when running things like payment points reports or NRS reports at the same time at the class instance level if you're doing EL civics you have that focus area drop box you're selecting from one of three areas we've taught a few times already those three focus areas those are what appear on that drop box you select one and only one focus area if that ESL class applies to EL civics. Okay, a little bit getting in the weeds on a few things here, a little bit new this was about a year ago this changed where for a while we required you to select the focus area of course if you're doing IELCE and they're in workforce training you market either the class or individual level integrated education and training this shows that they're not just an ordinary EL civics student they're an EL civics student in 243 that's also doing that vocational related workforce training. Okay, moving on here sorry, cutting off we've got about four minutes I think we're in good shape time wise so a few resource slides here this is just a screenshot of the CASAS California accountability page somebody asked about this earlier so here you go proof in the pudding it's that section of our CASAS website for California agencies we're also I'll also highlight here that training and networking page I'll say the beginning of year stuff is on the CASAS California accountability page assessment policy data dictionary all those attachments et cetera the kind of up to the minute well not up to the minute but the more updated type documentation is on the training and networking page we have those regional meetings we have statewide network meetings as we generate documents from those kind of meetings we post them on the training and networking page so lots of wonderful information there too including that illustrious monthly PowerPoint that CDE and CASAS do every month here's just another plug for that same page with a little bit more information just reiterating what I just said also that order guide is on that same page that's what you can download and as a WIOA2 agency you can order CASAS materials materials free of charge including test units CDE might say hey wait a minute it's not free of charge it's courtesy of the California Department of Education regional and statewide meetings we've had those for a long time in COVID years we started doing stuff virtually for all the obvious reasons it was kind of an unintended bonus we did virtual meetings statewide you know kind of once a month hey what do you know lots of people showed up so we've continued to do that statewide WIOA2 statewide TE you know usually at the beginning of the month WIOA2 usually on Tuesday afternoons TE usually on Friday mornings we also have several regional meetings we've tried to move more regional meetings in person lots of different meetings available you can look on our website or that CA Adult Ed Training for the schedule finally we've got CASAS phone numbers CASAS email address and I think that's it there were some things I'm used to talking about that I skipped sorry for the hemming and hawing I'm used to this kind of being the full two hour thing you know so hopefully I didn't leave you astray with some of that hemming and hawing at the end where I had to cut some things out it's 1013 according to my phone so a couple more minutes so great we've got a high school diploma question no you don't need to do that that so sorry Jim you want me to so for high school diploma students you need to change their education level when they earn high school diploma no you don't I would discourage against it you know that education level actually needs to say the same quite frankly because it's documenting exactly what that student had achieved at the time of enrollment if we change it then we're misreporting because when they enrolled they didn't have it we want to accurately report exactly where that student was at the time of enrollment I gave you more than you asked for just to point out hey you know why you would never really want to change that okay are the requirements for testing student so it's not required the requirements good you're catching me on something I don't think I explained very well that is again the testing is required for those in we owe it to that is the ones in AB EASC or ESL so you're correct that's not CTE it's not required obviously you're listening to the class as person so sure great idea if you want to do that we certainly strongly endorse it we certainly would say there's lots of good uses for it but I will reiterate it's not required where CTE might use it to their advantage that's different than what you might do for AB or ESL is you might use those placement forms for CTE we would generally say you don't really need to do ABCD progress testing but it is a really good idea to give a reading or math appraisal or both upon enrollment as a way to just generally determine basic skills that is you don't need to measure progress but you may want to make sure they have the basic reading skills and or the basic math skills necessary to succeed in that CTE class before enrolling them that's where I think your average CTE program would say yeah testing might be helpful you do not need to get that you know the so this is a good both sides of my mouth questionnaire and is a is pre-post required for HSE HS diploma short answer is yes it's required the policy requires it however if you're looking at those payment point reports on CTE it will not take that payment point away for equivalency or diploma if they don't happen to get it sorry I have to talk out of both sides of my mouth hopefully you got the right answer there somewhere Aaron I'll say that was not your question Aaron that was zero percent your question and a problem completely because of my answer not your question but hopefully you still got the right answer there okay Kristen right yeah CTE is not covered under federal rule we owe it to so yes they are rightfully completely excluded the feds have no interest in including CTE and CTE and we owe it to so we rightfully exclude them a hundred percent yeah we can't we just can't do that or the feds will rightfully you know bop us over the head many many times okay great questions