 OTAN, Outreach and Technical Assistance Network. Good morning and welcome once again. As the slide indicates, I am David Stang. I'm just going to adjust this moment. Okay, so the bad news is that you just have me today, not Abby, unfortunately, to do this presentation. But the good news is that according to the agenda, we're scheduled for lunch I think around 12. But my presentation will not go nearly that long. So you may be excused for lunch quite a bit early. Again, I'm David Stang and I'm going to provide a brief overview today of Federal Program Monitoring. Let's see if I can go to the next slide. Use your arrow keys on your keyboard. They weren't working, so I'll just use my phone. Okay, so our agenda for the presentation is to describe or tell you a little bit about what Federal Program Monitoring is. I'll talk a little bit about purpose, why we do it, and I'll talk a little bit about the FPM office. As you may know, there are many different offices here at the Department of Education, the Adult Education Office, us for example, and the FPM office will talk a little bit about their role and how they coordinate Federal Program Monitoring for other program offices here at CPE. And I'll talk a little bit about Federal Program Monitoring program instruments, more specifically the adult ed instrument. And then we'll talk about instrument items or categories which are found in the instrument and evidence requests or documents which you will upload into a management information system here at CDE called CMT, the California Monitoring Tool, when you get scheduled for a review. And then lastly, during our presentation this morning, we'll talk a little bit about your role as AFLA administrators during the course of a Federal Program Monitoring review. Okay, so AFLA or the Adult Education and Family Literacy Act, which is why we are all here today, require CDE to conduct on-site and online monitoring of all adult education grant recipients. Okay, so anyone who receives federal funding for AFLA, at some point, you'll be scheduled for an FPM review. The purpose of the review is to ensure agencies are in compliance with federal and where applicable state law. Okay, and the current monitoring cycle is for this program year, 2023-24. Okay, the CDE is required to monitor the implementation or administration of federal and state funding. These are the types of agencies who receive AFLA funding, local education agencies or K-12 school districts, community colleges, coalitions, community-based organizations or CBOs, libraries, and other state agencies like the Department of Ed. Okay, I'm going to provide you with a couple tips during this presentation today. The first of which is please do not wait until your agency is scheduled for a review to familiarize yourself with the adult ed instrument. Okay, in a couple subsequent slides from this one, there will be links, several links for you to access one of which will have the adult education instrument, among other resources for you to review. Typically, we do our very best not to schedule program reviews for agencies with new administrators within the first or second year. You have somewhat of a grace period, but we do really encourage you to at least get a hold of this year's instrument, download it to your computer and look through it so you can begin to have an idea of what will be asked of you during a program review. And if you're up to it and have time, we also would encourage you to even begin prepping for it in small ways that make sense before your schedule for review. Okay, so I mentioned we talk about purpose, and as I alluded to before, part of the purpose of conducting federal program monitoring is to ensure agencies who receive federal funding meet compliance. Okay. It's also for agencies that receive program funding who are responsible for creating and maintaining programs to ensure they meet minimum fiscal and programmatic requirements. We refer to this as minimum compliance. Okay. During the course of a federal program review, we are not trying to take back money from you. We are just not in any way, shape or form a gotcha type of movement. We work together with you and help to make sure that you get to minimum compliance during your review. The purpose is also for the AEO to learn more about your programs and the services provided and to identify highly functioning agencies. There are many different characteristics of highly functioning agencies, and those are the types of characteristics that we look for so that we can share those with other agencies and so that we can learn more about the types of services being provided throughout the state. Okay, so what does the FPM office, their role? The FPM office here at CDE, they coordinate most of the reviews that are conducted through the department. I say most because the FPM office, the vast majority of reviews are conducted for K-12 school districts, and the FPM office coordinates those reviews. If you are a CBO, a library, or a community college or another state agency, our office, the AEO, coordinates those reviews. That's a little bit different. But in either case, agencies can get selected for review every two years, aside from the grace period that you get in your first and second year as new administrators. There are on-site reviews where we actually go on-site and do classroom observations. We often review staff and just get a better sense of the program and services that you offer. And then the other type of review is an online review, also referred to as a telemonitoring review. Conducted much the same way as an on-site review. We just don't physically go to where you're located. But both types of reviews involve a 30-day pre-review period where we review documents that you've uploaded into the management information system I mentioned earlier. We review those documents for 30 days and make sure they meet compliance. And if they do not, then we let you know so that we can go towards reaching them. The final actual week of the review is when we work towards finalizing the review and if there are any issues issuing findings and coming up with a resolution process. And then regional team leaders or RTLs work with the CDE program reviewers. And agency coordinators, typically at the district level or leadership at community colleges or libraries to conduct the on-site or online review. Here is a, and from my understanding you'll have access to these slides, but here is the link to the compliance monitoring web page. This is the FPM office resources that you have available to you. Where you can learn a little bit more about the process and what it encompasses. There are frequently asked questions. There are monitoring cycles. OK, 12 agencies are on a cycle, two-year cycle. Then there is also information about the criteria that is used to identify which agencies will be scheduled for review. And then there are also links to additional training. I don't think you need to access those at this point. Probably a good idea to wait until you're actually scheduled to access the additional training. But just as an FYI, these are the types of resources that are available to you at this link. All right. A repeat of the first tip for success, which should convey to you the portrait of familiarizing yourself with the adult instrument, right? Here is a link. Again, you'll have access. These shit slides will be shared with you later. You do have a minute to jot this down. But please do familiarize yourself with the adult education instrument. And if you have any questions, feel free to email me. I'll put that in the chat once my presentation is done. Or to ask your regional consultant. Again, you will know well in advance when you will be scheduled for a review. But as I say, it's just a really good heads up to go through the instrument beforehand to get an idea of the kinds of things that are expected to know. And also to get an idea of documents, which in some case, there can be many, many documents that you will need to upload into CMT. So if possible, please take a look at that and get an idea and even begin organizing a little bit if that's possible, some of the documents that are included in the instrument. So all program offices, like adult education, like English learners before and after school, these are the type of examples of other program offices here at CDE. Fiscal monitoring, all of these program offices use program instruments to guide their reviews. You don't need to worry about the other offices, just adult education. But all program instruments have items, instrument items. They're essentially categories that the reviewer refers to during a review. Our instrument and your instrument has 10 items. And again, once again, here is the link to the adult head instrument. This is just an overview today, folks. Trust me, the instrument is a detailed instrument. And it includes these categories and many of the documents that you'll need to become familiar with and to upload when you are scheduled for a review. Just quickly, and to give you an idea of the categories, there are 10 in our instrument, 10 items. And just to give you an idea of what items we have in our instrument, hopefully you're familiar enough with the adult head program by this point that these categories or items will resonate with you in some way. The first category or item is collaboration, alignment, and support services. So we will ask you to provide documentation in this area that demonstrates that you are collaborating with partners, that you have support services, that you're aligning some of your services and programs with other partners. AEO2 is financial accountability. Makes sense, right? AEO2 is a federal award. As a federal award, there will be some aspects of financial accountability. The data collection and program effectiveness, right? Hopefully the fact that those two are together gives you some idea of the significance of data collection, right, for program effectiveness, to have an effective program. Data collection is a key element. And then also staff qualifications are your teacher's properly credentialed duty statements. These are the types of things that we will ask you to provide to us when you're scheduled for review. Professional development. We have, you know, we work with a contractor, CalPro, who provides exceptional professional development, which is available to you. So we look to see that you are making professional development available to your staff, to teachers that's much needed and that will benefit your students. Number five is needs assessment. I think Diana alluded to her in the previous presentation. But we want to make sure that you are doing needs assessments on a regular basis of the students and to make sure that the program and services that you are offering is in alignment with the needs of your area, right? Okay, and then the next group of categories, the second half of our instrument is serving individuals with disabilities, 806. Intensity, duration, flexible scheduling, okay? We ask you to upload class schedules so that we can see that courses are being offered at appropriate times and not just at one time during the day. And that they're being offered at a rate and that the hours of instruction are commensurate with when you are pre- and post-testing. And then this is one that obviously would need to be here in the instrument, right? Evidence-based instructional practices and reading instruction, okay? Effective use of technology and distance learning. And then finally, IET, Integrated Educational Training. So these are the 10 items or categories you'll find in our instrument. And now we'll talk a little bit more about evidence requests. So within each item or category, the 10 that I mentioned, there are evidence requests. I think we have approximately 30, approximately 35 evidence requests for our instrument. And here on this slide are just a few examples of the types of evidence or documents that you'll have to upload. Under AE01, remember as collaboration, alignment and support services, we ask you to upload an umbrella MOU with the local workforce development board. So hopefully, well, I'll say this, if any of you are wondering what that is, find out soon that you are required to have a current umbrella MOU with the LWDB in your workforce development board area. And it outlines the services that are provided under the MOU and the cost sharing agreements that all of the partners agree to. And if you, this is why we say, please make sure you at least go through the instrument because this is an example of something that you should be familiar with now. Okay, AE02, something else that you should be aware of and make sure that you are doing is to make sure that you have a time and effort policy. Your agency, because you receive federal funding and many agencies use that federal funding to compensate staff, employees for their time and effort, you are required to complete or to have a policy which reflects what you are actually doing. That is, staff are completing time sheets. You're doing some kind of semi-annual certification. It depends on the agency and what your policies are, how you pay employees, but whatever your policy is, it needs to match what you're actually doing. And that is what we check for during an FPM review. Under AE03 data collection and program effectiveness, you need to upload a local assessment policy. And that policy you can access, if you're not familiar, you should be. You can access at the CASAS website. Your regional consultant can provide you a template. I imagine most of you are familiar with this or there's one somewhere in your previous administration or the person who held your role prior to you, I'm sure they were unlikely to add a local assessment policy that you can update or refer to. If you cannot find it, as I mentioned, there is one on the CASAS website or you can ask your regional consultant. Under AE04, which was staff credentialing and professional development, duty statements, we want to take a look at all of the duty statements for the staff that you work with. And then under AE06, individuals with disabilities, we ask for your ADA and IDEA policy. How is your program? How is your program? How is your agency? What is your process for making sure that students and staff have access? There's a process for when they need accommodations, all right? So just having a policy that says that you do not discriminate against individuals with disabilities is not sufficient. That would not meet minimum compliance. You need a process for when a student has an IEP, for example, what is your process for how you make accommodations for that student? How do you follow up with that student's IEP? If staff need an accommodation, what is the referral process? So again, you can begin to see the importance of looking at the instrument to give you an idea of the types of evidence that will be required to meet minimum compliance. And the sooner you do that, then make your life much easier and your staff if you get a little jump on that now. And again, these are just a few examples. There are, as I mentioned, approximately 35 requests. But the last one here under AE10, if you have an IET program, you are receiving 243 funding. Then evidence of co-enrollment. We want to see evidence that students are enrolled in both an ELL class, English language learners class, as well as some sort of training components. And that's what that evidence of co-enrollment is. Okay, final tip for success. Make sure your agency has the correct documentation and or an example for each of the requests. What a lot of agencies do is they look at the adult instrument, which you'll have links to in this presentation. It won't take you long to just go to the link and then bring up the instrument. And then what they do is create folders for each of the categories and evidence requests. So you can begin to just, when you become a little bit more familiar with the documents as new administrators that are reflected in the instrument, you can begin to just put those into the folders you've created. And that way, when you are scheduled for a review, you'll at least have an idea of what they are. And you can share those with your regional consultant to see if that is in fact what we're looking for. But by doing this, you get a little head start on the process because it is just a matter of time. If an agency has not been reviewed in three or four years, they will get scheduled for a review. Okay, so what's your role as AFLA administrators? As we've talked about and I've emphasized probably too many times this morning, becoming familiar with the instrument is obviously important. Ensure you and all additional users. So when you are scheduled for a review, you're going to get access to CMT, the CDE monitoring tool. So that information system here. Once you get access to that, whoever you delegate amongst your staff to help you with this process, and you will want to delegate, right? Because it's a big process, it takes time. As I mentioned, there's a 30-day pre-review period. But even leading up to the pre-review period, there are things that you can be doing. And even your reviewer can even look at documents prior to the 30-day period. So make sure everyone has access is the point here to CMT that you want to be helping with this process when you're scheduled for a review. You can assign additional users, so others can have access to CMT. So you are not the only person who is going into CMT and uploading what can be many, many documents. So you want some help with that. And then communicate with district personnel or their agency's leadership about who's being assigned to coordinate the review, right? If you're a K-12 school district as the adult ed director or principal, you really are the most appropriate person to lead the adult ed portion of the review. Sometimes school districts will assign someone from the district, which we do not. We cannot tell schools or any agency who to appoint as a coordinator. Oftentimes, when school districts do that, many of those people that assign at the district are not familiar with adult ed. It just slows up the process. So reach out to district staff to make sure you have that rapport or relationship so that you can communicate the importance of you having access to CMT. The system works. The process works much better when your reviewer, who are your original consultants typically, work with you and not somebody at the district to get all of the evidence uploaded into CMT. You are much more familiar with your program and how it runs and what forms of evidence would be appropriate to provide to the CDE. And then coordinate onsite scheduling and work with program reviewer during the course of the review. So if you're being scheduled for an onsite review, whether you're a K-12 school district or community college, library, and whatever type of agency you are, if you're being scheduled for an onsite review, make sure you work with your reviewer, your program reviewer to put together a schedule. So if you have an idea, your staff have an idea of what day one will look like, day two, day three, day four, if necessary. Okay. So it looks like my slides. I apologize. It looks like my slides are a little unordered. That's okay. We'll go through the review together. I had a little review for you here, but that's okay. Okay. Which AFLA agencies get selected for program reviews? So you all are representatives of AFLA agencies. You receive AFLA funding. So who will review, which of you will get selected for reviews? All of you at some point will be scheduled for review. All right. Which office at CDE coordinates most federal program reviews? Again, if you're a K-12 school district, the CDE, the FPM office at CDE will coordinate your review. If you are any other type of agency, a library, another state agency, community college or CDO, the adult ed office will coordinate that review. We will. And then how often may agencies be selected for review? You can be selected. It doesn't necessarily mean you will be, but you can be selected for review either online or on site every two years. Typically we do not review agencies every two years. Unless there are issues or concerns that we have. We usually like to review agencies every three, four years, not two. Okay. I see here the problem. Again, my apology for the slide being out of the quarter. And then what is the name of the document that we really want you to take a look at as soon as possible? That's the adult education instrument. And last question. What is another name for the documents agencies provide during a program review? Those are evidence requests. Okay. Again, I'll put my, as soon as we end here, I'll put my email in the chat. But that's the end of my overview of federal program monitoring today. Do access the instrument when you get a chance. And please email me or any of your. I am the one of the PM leads in our office along with Abby, Madina Lewis. So if you have specific questions, feel free to email us or your regional consultants. Caroline or Jim, I don't know if you have questions or if we wanted me to move on. Dave, you do have a couple of questions. Do you want me to call those up for you? Yes, please. Okay, we have one question. Where can we find the document containing characteristics of highly functioning agencies? We do not have a document that reflects anything or the characteristics of highly functioning agencies. That's an interesting question though. But the purpose, one of the purposes of FPM is for us to, as I mentioned, to identify those agencies. For myriad reasons, right, when we want to know more about how they provide certain types of services, how some agencies. The adult ed has a lot of moving parts, right? And much of it is there's a lot to it. The learning curve for people who are new to adult ed is can be pretty steep. And so we try to identify those agencies to do presentations when appropriate or to in certain areas to provide support to agencies that could use some additional support. And to let you know who those agencies are. So if you have questions, if you're not sure about something or you need support, reach out to reach out to your consultant. Again, that's an interesting concept that you raise. We do not have anything specifically that indicates the characteristics of highly functioning agencies. Okay, and we have another question to clarify. New administrators have a grace period of two years, and we'll have their review in the third year. Good question. Not necessarily. We try to, we do our best to make sure that we know that you have a lot on your plate. And for those of you who are new to adult ed, as I mentioned, the learning curve can be pretty steep. We try not to burden you with a review because it can be a lot. It's doable, especially if you're well organized and you, you know, start well in advance. And we have many tips. When you are scheduled for a review, we conduct training. You will be well prepared come come the time you have your review. But you have that grace period. It does not mean that you'll be scheduled in that very next year. But typically, we really try to review agencies. No more than every four years. Agencies that are highly functioning and typically don't have issues. We may give them a little bit more time. We may, you know, allow for an additional year, but we really do our best not to allow agency or not to go more than five years, four or five years for review. Okay, looks like our last question. When will we know if we are selected for a review in the previous school year or the summer before the review? Good question. Typically, we typically we begin the notification process begins in May and June. And we will let we will first initially let you know that the scheduling or identification process has begun. And I mentioned, I mentioned for K-12 school districts that there are monitoring cycles on the FPM or CDE monitoring website. If you take a look at those cycles, it will give you an indication of when you were last reviewed. But otherwise, we give notification to agencies typically in May and June, well in advance of when you're, you know, be scheduled for reviews are scheduled throughout the year. But usually by the middle of July, you should have a good idea of if you've been scheduled for the subsequent year. Okay, yeah, I think that's it for the questions. Okay. All right. Thank you everyone again. I'll put my email in the chat and if you have any additional questions or you need help accessing those links. Let me know. Thank you.