 OTAN, Outreach and Technical Assistance Network. We're from Corona Norco Adult School and today we're going to share our journey into developing a successful remote program at our school for testing in COSIS and civics. So I'm Christina Hyatt and I teach adult basic education math. And under the direction of our dean of students, I run the remote testing for the COSIS goals for our adult secondary education students. Hi everyone, I'm Marcy England and I am like the DJ today, so I just need to double check. Can you see I'm sharing my screen, right? I know it's like nobody likes hearing that am I sharing my screen, but I wish there was like some way we could just ask Zoom am I sharing it without asking everyone. Okay, and then I'm also going to manage the chat and I will watch the questions and kind of just make sure that this presentation continues to flow. So that's my role a little bit, but also I'm here to share the EL Civic site of remote assessments. I'm Marcy England. Let's see. I'm the ESL program coordinator at Corona Norco Adult School. I'm excited to talk to you about remote testing and what we've been doing. So based on that, I'm going to go to our next slide. Maybe there we go. All right. So it's always fun to start a presentation with some fun facts. So did you know that in 2020, puzzle sales increased by 300 to 370% over the same time in 2019 Ravensburger, which is like the gold standard of puzzles. It's the German puzzle company. They were selling puzzles at a rate of 20 puzzles per minute. Penny Gold, a pediatric neurologist said, when you are working on a puzzle, you can't worry about anything else. You can't let other things intrude other thoughts intrude. It's all about meditation for me. Another puzzle enthusiast said, it's my chill time after working all day. It's something I look forward to. So the puzzle fad of 2020 is something that I think we can all relate to in some way, shape, or form. And I know Christina, she completed well over 20 puzzles last year, you guys. And she's even sent me pictures of her completed puzzles. And I have to say that I have contributed to a few or several assists, although I cannot brag and say I completed any, but I am the one that walks by and says maybe this piece would go really well right there. So when Christina and I were talking about implementing remote testing, we really thought that this had a good connection to solving a puzzle. And we love an analogy in how looking at things from a different perspective helps you connect pieces. And so we're really going to hang on to this puzzle analogy and talk about how remote testing has been an enjoyable experience of focus and meditate, maybe not as far as meditation, but it's been an enjoyable process as we've solved problems and created a better picture. So today we're going to discuss the critical first steps we took to implementation. We're going to talk about administration of the assessment. We'll also discuss ways that we've worked to maximize learner outcomes and different ways that we measure learner outcomes and measure our successes. We're looking at things differently than just testing scores. We're looking at a lot of other positive things. We're also going to give you some troubleshooting ideas and we're going to share our pitfalls. We will tell you the things that you can do to avoid like, we've been through some bumps and so maybe if we tell you where they are, you will not hit them as well. So the format, Christina is going to guide us through remote process testing first and then we'll have Q&A with Christina and then I'll guide you through remote civics testing and we'll have Q&A for me or for in general. So that's what it's going to look like today. So just to give you a bit of background on our school, let me move this out of the way. Okay, so to give you a bit of a background on our school, just so you know where we're coming from in 2019-2020, we served over about 3,600 students, 28 had significant literacy gains based on cost of testing. So we were successful and we had a robust cost of testing program. Our programs include ABE, GED, high school diploma, ESL. We had 1,300 students, CTE concurrently enrolled and fee-based classes. Our technology accessibility to start off with in 2019-2020, we didn't have a loaning program, we just had laptops on school sites available for students to use but not really a program where students took laptops home. Thankfully due to grant funding and our consortium, we are now currently operating under a loaner program where we have enough laptops and Wi-Fi to provide students anyone that needs it with a laptop and Wi-Fi. So understand that that's where we're coming from. Our e-testing history in 2019-2020, only adult secondary education was doing e-testing. ESL and CTE did paper and pencil for civics and COSIS. All right, Christina. So when you're getting ready to do a puzzle, everybody has their first steps, right? The first things they do, like, for example, do you dump all the pieces and sort them first? Maybe you sort them by shape or by color. You maybe you're one of the people that turns them all face up. I personally like to start with the frame, right? Everybody's got a different way of getting started, but there's some critical first steps to getting going. And so likewise with our remote testing puzzle, we looked at what were these first critical steps? And we called them the corners. The most important parts of a puzzle is finding those corners and building around them. So the first corner we looked at was the who, right? Who is this going to affect when we test in a remote situation? Who needs to be trained? How do we who do we need to get on board with this program? For us, we needed to train our staff and increase their understanding of why we give the COSIS test even in the middle of a pandemic, right? So we worked to shift the mindset from pay points to student growth. Our COSIS test triggered digital badges. They triggered promotions and we use them for recognition. In our meetings and in our PLCs, we shared about the importance of COSIS test for showing this student growth. And once teachers began to understand this, they became our partners and making sure every student got tested. And I do remember when I first heard about remote testing, I think it was back in May. I thought, you want me to ask students to do what? And what are we offering them in return for this? I needed a mindset shift to. So by shifting my mindset to student growth, I don't even think about it now. Testing from home, that's just what we do. The second corner piece of our puzzle for testing was the training. We trained staff on the testing protocol. We got them the necessary certifications. Both EL Civics and COSIS have requirements to implement remote testing. So we worked closely with COSIS staff to make sure we were within their framework. COSIS has videos, literature, trainings and helps to get your remote testing program going. My favorite thing is they have a remote testing checklist. And who doesn't love a good checklist, right? We also train students. Some of our students have never used Zoom. Some of them needed to learn how to use a computer. We wanted to make sure that the students felt confident that they have the necessary technology skills to take the test in a remote environment. A third corner piece was that technology, right? Getting the right type of device has been very challenging and, in fact, one of our pitfalls at times. But luckily, as Marcy mentioned before, we were able to get Chromebooks and Wi-Fi hotspots into the hands of students who needed them. And then we also worked with the district IT department to make sure that those Chromebooks could work in kiosk mode so the students could take their tests online. The students need the right tech for technology. And they need to know how to use it. And they need to feel confident when they go into a testing situation that the technology is not going to be what trips them up. So we continue to work with the staff to get them trained on helping the students, on making the students feel comfortable with those pieces. And another important part of this corner of technology is making sure that the staff has the right technology and the right abilities, right? We all were new to Zoom a year ago. And so learning what Zoom can and can't do, learning to make Zoom work for you when you're a proctor was really important for me. And I'll talk about some of those things later. Another thing is when our campus went completely remote and we were only testing students from home, I needed enough computers and enough bandwidth to be able to do that from home. So these are all things that we had to consider as our critical first steps in developing our program. And of course, the fourth piece is the support system, right? So when I do puzzles with my family, I have a sister who she likes to start with the water. It kind of blows my mind, like why, right? But she always starts with the water and I have a brother who likes to look at the shape instead of the color. He's a little colorblind. So it makes more sense to look for the piece with that little arm or something. Whereas my brother-in-law, he cannot stop looking for that missing piece in an almost finished section, right? That bothers him to no end. And for me, I have to start with the structure, the house or the boat, whatever it be. So when working on your system, you want to look for a team that has differing strengths and different points of view, much like my family does when we approach the puzzle. Having that strong support system helped us work our way through a lot of the wrinkles that came our way. As questions came up, we had people we could go to for answers. Sometimes the pieces weren't fitting the way we thought they should, but getting a new perspective or having somebody else look at it or looking at it from a different angle were all ways that we were able to come up with solutions. So consider who's going to be on your team. Would it be your COSIS rep, your TOPS coordinator, the principal, your consortium, the experienced teachers on the campus, the techie people on campus, the district IT department. And for us, it was every single one of them. Another important thing to consider is the cost. So I kind of think of the cost as like the frame of the puzzle. It's going to determine the size of your testing program. And remote testing can be costly. I'm not going to sugarcoat that. But we looked at ways that we could minimize the cost and maximize our resources. For example, consider what do you have? And for us, we had lots of computers sitting in remote locations that we were able to bring back and implement in our testing program. You can also look at who do you have on campus. We had paraeducators and substitute teachers who were willing to step up and be part of the team. Look for the emergent leaders in your campus, those who can and will and are able to make that strong team to complete the puzzle. And last of all, look at how it could be better. It's easy to just look at a system and go, that's it. But we really need to constantly be thinking of improving the process and perfecting it as a way of reducing the cost and maximizing what you have available. And so for me, recently I started looking at, I realized I was spending a lot more time on communication than I normally do. And I had to step back and think about not only what it changed so I could reverse that change, but go even further and cut into what was being done before. How can I make it so I communicate less frequently and yet still get the message out to the people who need it? Marcy, you wanna talk about equity and you're on mute. Yeah, yep. What's a Zoom meeting without someone talking on mute? Okay, I'm glad I could do that for us all. So at our school, equity means everyone has access to learning. We work to ensure that the most vulnerable students are supported. This means bringing students on campus for testing when we can, but also offering remote testing for students that are not comfortable or cannot come to campus. It means creating systems that support all of our learners. For us, it means getting our Chromebooks to work and having staff to support technology problems. It means troubleshooting different technology types. Equity is listening to teachers as they advocate for individual students and preparing for student needs. Specifically with a remote assessments, equity means access, accommodation, and flexibility within the testing parameters. Oh, Christina, you're on mute. It's my turn. Thank you. Hashtag Zoom problems, yes. Right, well, I wanted to be like Marcy. She is my bestie. Okay, so I'm gonna start by going over the CASAS goals and life and work program and how that was developed and I'll stop for questions at the end of that and then Marcy will jump in and share what she did with Civics. So as we developed the plan, it really, I think about it as now getting all the pieces to fit together, right? You've got this 500,000 piece puzzle and now we've gotta figure out how the pieces are gonna fit. And so here's this key questions that we looked at. What needs to be done? When does it need to be done? Who needs to know about it? Who's going to do it? I honestly didn't think I was gonna be the one doing it but here I am because that decision was made somewhere along the line. And what training technology skills are required and what are the CASAS policies that need to be followed? And right here on this slide, for those of you who want later, I'm gonna be providing a template for developing your own plan. So that link is there when you want it later. And everything, we have our slides in a Google slide presentation. No, in a Google folder, in Google slides. So we'll be sharing the folder with you that has access. So any link you see in our presentation, any template, whatever, it's all inside that folder and I'll put it in the chat here in just a little bit. Perfect. All right, so on the next slide, I'm gonna go over what our school did or what we do for our testing program. I'm gonna not give you every single detail because we'd be here for four hours and that would be boring but I'm gonna give you some of the highlights. So for me, one week before I start emailing students to confirm that they have the required technology and they know how to use Zoom. In this email, I even include some Zoom tutorials because the number of times you've had a student come on and this is their first time using Zoom and they don't know how to unmute or they don't know how to mute were all things that we needed to deal with. So the trickiest part of the process is getting students to show up with the right kind of technology. And so one of the things I do to combat that is first, I send them an email and I tell them, here's the kind of technology you need but then I created a survey because it's not enough to send an email and expect them to read it. They're busy, it's more than two lines long and so Marcy, if you'll show the survey, I asked them instead of just reading, I asked them to do something to respond. This way I know that they read and they understand and it's a very short survey. It doesn't take them very long but I wanna know exactly what kind of device they're bringing to the testing session. And I wanna know if they understand that they have to be able to have a webcam and a microphone. I wanna make sure they understand the amount of time it's gonna take and the testing environment that they need to be in. So this survey has been fabulous because one, I now know that if they're showing up with an iPad and I can send them the tutorial so that they can download the app and get ready so the night they come, they're like, yep, I've got it, it's all on screen and we're not dealing with, oh, let me send you that, let me walk you through the process. And then the other thing, Marcy, if you'll go back, I got lost but give me just a minute. Okay, so when they, oh, and then the other thing is there's no surprises when they show up, right? Whatever technology they have, we're kind of expecting it and we're prepared for it because different kinds of technology require different conditions. So then the day before the test, I send that Zoom link. I don't send it in the first email. I'm waiting for them to respond to my email and there's a reason because I need them to think. I have to do something before I get this invitation. When I send the Zoom link, I assign them a time and I stagger those times. The first time I did it, I had all five students show up at the same time and it was not a good experience because I couldn't help five students on board at the same time, especially when each one had a different technology ability and a different technology platform. So now we give 10 to 15 minutes so we can work with the student one-on-one and bring them in individually. So then on the day of the test, the first thing I do is I open a breakout room and I have one room for each student that's gonna be testing. Immediately after I open those breakout rooms, and this is way before I expect the students to come online, I start bringing on computers in my office and I bring on one computer for each breakout room I intend on running so that I can see every breakout room all at once. I'm not having to scroll through and look at each breakout room individually. I can see all students and if one student is struggling with something or needs help, I can hear and see them immediately. One of the things though that we run into and I'm sure all of you can relate to this, whether you're running a program or not, is the internet, right? Did your internet ever go out? It's just me. So a couple of things that I have suggestion. These are a couple of tips I have. When you do the breakout rooms, there's a setting that says, automatically move participants into their breakout room. And there's another setting that says, manually add the students. So I manually add the students and the devices that are in my possession into the breakout room. But if the internet goes down, which it often does, right? It's lovely because as soon as it comes back up, Zoom knows to automatically move them back into the breakout room. And I'm not scrambling or having to move everyone back where they were. Zoom automatically just puts them back where they were and a lot of times students don't even realize it ever happened. And then my second tip that revolves around the internet is because I'm using sometimes six computers to do this, I need more bandwidth. And one of the things I now do is I turn on a hotspot for my main computer, the one that I'm the host on. And that one runs on one wifi connection and then the rest of them will run on a second one. So I just use my phone usually, I'm fortunate enough to have enough hotspot on my phone that I can do that. And it's made all the difference because if one drops the connection, I still have the other one up and running. So then the last couple of things or tips is after the test, sometimes students have a lot of questions. And maybe you're in a hurry or maybe you have a couple other students, but make sure you give them the time and answer their questions and provide them the resources. Direct them where they're going next. I made this mistake early on. I said, great, thank you so much for taking the test and we'll get back to you. And that was not right. There was a process in place that we needed to make sure students knew what to do next and how to do it. And usually in that end of the conversation after the test, I will tell them three times the next step. And later on, I'll show you another thing we've done to make that even smoother. Marcy, if you'll go to slide, two slides up, I think, sorry. So this is my workflow, right? Please don't look at this and think, oh my goodness, right? I'm kind of OCD when I write my steps, when I work on stuff and I write things like turn on the computer in here. It's pretty intensive, but I wrote this down because I'm training my staff or my team, I should say. My team to do this without me, right? I don't want this system to be my system. I want it to be the school system. And by writing it down, also I could look at the places where I can perfect it or make it better. So I'm not gonna give you a copy of this. Some of you are probably saying, but why, but why? And the reason, of course, is because this was our puzzle, right? This is what works for our school. So instead, I'm gonna help you. You guys have your own puzzle, your own way you're gonna need to look at things. And I'm gonna give you a document that's gonna help you develop those skills. You can see I've got all of the timing set up in here and I've added even more to it than this picture shows. I've given you some of the tips, some of the resources we're using. They may not work for your school. We have Zoom in Spanish because we have a lot of Spanish speakers, but maybe you need Zoom in another language. But I've provided some questions or some things for you to consider as you go through the process. Who needs to be involved? Where is it gonna happen? How's it gonna happen? How's the student gonna know about these things? How do we make sure we're staying within the parameters provided by CASA? So these are all, this tool is designed as questions to get you talking and get you thinking about how to get your program going. All right, and next slide, Marcy. Oh yeah, you're there. I'm not, thank you. Okay, all right, communication, right? So for me, communication is like assembling the vast blue sky of a puzzle. It's difficult and you usually spend a lot of time on it. And you're constantly trying different pieces that are all the same color, right? This is where that brother who's colorblind comes in really handy. So communication is something you have to constantly work at. And it's where I spend most of my time when it comes to the testing process. You just need to make sure that whatever your message is, that it's clear, it's concise, it's consistent, it's continuous and cohesive. We recently had a student that went through the testing process and I still chuckled to this day because Marcy calls her a good little communicator, which is exactly what she was. When she got your email, then you got three to four emails a day detailing exactly where she was and what she needed to know and how you were gonna be able to help her. And it's easy to think about, oh, that student. But I'm actually really grateful for students like that because they allow me to look at what's missing. What do we need to relook at, right? How can we deliver the message even better? Because while this student is communicating her needs, there were lots of students who probably weren't and had the same questions. I'll go back one more time. I was just gonna share my quick tip. So, oh, a couple more things. So the quick tip, right? One of the things we run into is we thought we were running a remote testing program and our first couple of go rounds, we had tons of students show up on campus and we realized we couldn't use the phrase go from home, stay at home, do not come to campus enough. So we went back and it turns out our message where students were signing up was conflicting with the message we were trying to give them. So we were able to look at fixing that message, making sure the message was consistent throughout anything that the student was seeing and making sure that every email I sent over and over again told them, you will connect online. You will not come to campus. We will see you on Zoom. And so my quick tip is make sure when you're communicating to students that they know exactly how this process is gonna go. Communication and miscommunication is one of the biggest opportunities for pitfalls. So when you're developing a good remote testing system, make sure you look carefully at what the message is, who needs to get the message and how it's going to be delivered. So one of the ways we did that is by creating a Google document or a Google site, excuse me. And we outlined the steps that students need for getting enrolled. It has the necessary steps, the forms, the tutorials, the contact information for each step of the process. And right there on step six is our COSIS assessment team. Just this little thing of telling them, hey, you're gonna get a message from Christina Hyatt, has really made a difference in the number of calls that go to the office. Because now instead of the student calling and saying, I never heard from the administrator, they know first to search for an email from Christina Hyatt and evidently it is there. All right, so the learner outcomes, right? So we talked about earlier that we weren't focused on pay points. We weren't focused on how this was gonna look for our school. We were focused on how we can show the student growth. But there was a happy side note that we saw the pay points. Even though we saw fewer enrollments this year, we saw higher percentage of students with paired scores. We saw higher percentages of students leveling up. And I also love that now, with a remote system in place, we can capture students who weren't available to come to campus. Even before the pandemic, there were always those students who, oh, I'm in Mexico. Well, guess what? We tested somebody in Mexico. We got that post test because of this remote system. And so it just allows the flexibility to test students where they're at and meet them if they don't feel like they can come to campus. We also are giving students digital literacy skills for many students these tests online or maybe the first test they've taken online, right? But now, high sets being administered online, GEDs being online all from home. And now they've got this opportunity to practice. As they enter college or into a job environment, they're gonna need to know how to navigate some of these online experiences. And by having this program, we're giving them the first opportunity to try it. And when they come around the second time, it's a lot easier. And so we can see that they are increasing in those skills. So the process was slow. We started right when CASAS allowed us to begin back in May. But we weren't that great at it back in May. And we're just getting better and better and better the more we practice. It's not that it's easy, that communicating with students or that moment when you're trying to onboard a student with technology, it's still nerve wracking. My team and I, we always talk about that being the most nerve wracking part of the process. But each time we put more pieces of the puzzle in, it gets easier and easier to put the puzzle together. So I am gonna pause right here and answer some questions. So Christina, we have a question from Peg. She says, how do you make sure the students read and understand the Google site or other information about your program slash testing? So the first thing a student does when they're getting enrolled is they have an orientation with our Dean of Students. And he takes some time to go over the steps and the process and he also gives them the website so that they have it. And everybody on campus knows if a student's lost in the process, here's the site to give them. And it became a one-stop shop of if a student calls and says, hey, I haven't heard from the testing coordinator, they could go, well, you know, it looks like you haven't done step five or you still need to do step four. And so it comes from him, but each step in the process, the office knows about it, the Dean of Students knows about it, the testing team knows about it. And when we finish testing with a student, we don't just tell them, we show them, okay, here's where you're at in the process. Have you completed these five steps? And there's just a couple more steps after they meet with us and we tell them at least three times in the phone, in the Zoom session at the end, and then we show them. Thank you. Okay. Oh, good question. Kathleen asks, is there any translation? Can you be more specific Kathleen or? Probably I'm thinking a translation on the Google site. Oh, I think that Google does translate, right? Am I wrong about that? I know for sure the forms, you have the translation option on the forms. And then Google translation really is just if the student is tech savvy enough to have that set up in their Chrome browser. But that's a good point is that we could have a little mini tutorial, like click here to translate this site. Yeah, that is great. See another piece of the puzzle in. Thank you. Diana says, how did you find the students who need tech equipment? And then she says maybe Chrome extension for the translation, I'm sure she's talking about. So how do you find the students who need tech equipment? So usually in the orientation, our dean of students does go over, if you're gonna be testing from home, you're gonna need some technology. But usually it's that first email I send a week out. Here's what you need. Here's the options you have for testing. And if they don't have it, there's another option on that quiz. And I did provide you guys a link to that quiz as well that you can duplicate it, make it your own if you wanted. And so when that other comes back, then we work with our team to either check out the device then or maybe find another device. And we were lucky enough that not just our Chromebooks, but every Chromebook issued by the district can now run the E-test. And it's just built right into the Chromebooks. So a lot of times that solution is, hey, well, do you have a K-12 student in your house with a Chromebook? And then we're able to do that. So that means, Christina, that you would use the K-12 device before they're issued a device through our school, right? Yes. And we've also enrolled students, issued a device, and then tested after as well. Okay, good. So do you COSIS test students with different devices in the same session? So I know that COSIS in their training, they recommended not doing that. And so I tried it the first time, but when you're only testing five students in a remote session, and this was early on, I just felt like it was too hard to find five students who all had a laptop or five students. And it's actually easier for me if I don't have five MacBooks at the same time, because that's a whole new dimension of working through the testing. So for me, I give them the options. My staff and I are well-trained on what the options are and what they can do and how to get the student onboarded with those options. And that's all part of our plan. I have all the devices written out and the specific steps that need to be performed for any particular device. Let me show that really quick. So that's here on the right, right Christina? Where you have like, if it's a lot of windows, a Mac, a Chromebook, an iPad and what to do there. Yeah. Okay, so we have with all, Kathleen says it's all the materials with remote testing steps. Can we use native language up to begin the test? I think that probably means like to explain the test up like up to that point. You know, I'm gonna refer you back to your cost us rep on that question because I don't wanna say what's wrong. But my understanding, I could be wrong was that yes, you can write up until the beginning of the test, but I could be wrong. So make sure you check with your cost us rep on that. And then here's a question from Peg. Do students ever lie about which device they have? If they're scheduled for a test or daytime? So two questions, do they ever lie about what device they have or like tell you the wrong device? And then if they're scheduled for a test and they don't show up, what happens? So we have had students that tell us the wrong thing and especially before I created the survey because they go, well, I have a Lenovo and then they showed up with a Lenovo Chromebook. And then I learned another puzzle piece place that a lot of people make Chromebooks, not just Google, right? Like, I don't know, mind blown or whatever. So lots of people make Chromebooks. Students don't always understand what they have but they've never intentionally lied about it. So sometimes students come thinking they're bringing one device and bringing another. We work with them and again that staggered time to give us the 15 minutes to work with them to see if they can figure out a device that works within our parameters. If not, luckily our district was allowing us to bring small cohorts of groups in the beginning. And then when we went completely remote we still would occasionally bring one or two students, usually one student in a room to do a test and through using that, if a student absolutely did not have the technology or the technology skills, that was our workaround. So there's actually a lot of questions popping up here which is great. We have time built in for this. Is there specific? Oh, hang on, I missed one. So the question, do you mean the school will provide a Chromebook to the students? So yes, our school does, we have the ability to provide Chromebooks to the students. Is there specific training for proctoring the COSSES test virtually? There absolutely is. And somewhere in here, I think back to slides maybe or one, Marcy. Well, I'm managing the chat. Let's see. Oh, good point. That's okay. Where do you want me to be? That one, that's the link. Right there is the link to the page where you're gonna find that checklist I talked about, going remote checklist. And it also goes through all the training. And I don't know how recently they redid their training thing but it's really lovely now because it says if this is what you're doing, then you should do these three trainings. If this is what you're doing, you should do these two trainings. And it's very specific and laid out. The training is not very difficult. It doesn't take too long to become an E-test proctor. I think it takes about 30 minutes. And then we went, we were part of a special group. And so we did a little bit more in-depth on the remote training process. So I can't exactly say how long that is because I haven't done it. I did the more extensive one but I can't imagine it would take too much more time than the rest of them. Great, thank you. Okay, so I'm rounding the end of this. How are we able to get a link to your testing document? I'll put that in the chat towards the end of our presentation and then Christina, where do they all live after we're done presenting? I don't think they... Okay, so I'll just make sure I get that in the chat sooner than later. And then I think there might be someone here from CASIS that if we want, that can answer the question about using native language up to the beginning of the test. This is Barbara Lehman and I'm a program specialist for CASIS. And yes, you can use native language until the test in terms of getting them started and getting them practicing on the sample, the two sample questions at first. But in an e-test, you can use native language to get them set up. Perfect, thank you for being here and for letting us call on you. You guys have done a really nice job and I know you were part of the pilot program and so I really appreciate your willingness to share all of this because it is daunting for many agencies. Yes, yes, it's a big puzzle. There's a lot of pieces in this box, but the reward of the picture at the end. And I don't know, Christina, I'm wondering if while I'm presenting you could put the link to our slides and everything in the chat. Have you put them in the Google? I don't know that you shared it with me. They're on the first slide and the last slide. Okay, perfect, yes, then I'll take care of it. Okay, so thank you for those questions and yes, Diana, we have really supportive admin and that is a big piece of the success is we have the opportunity to make mistakes too, to try things out, pilot. Okay, so now I'm gonna talk about implementation of Civics and I'll review the resources that were provided, the key steps. First, I'm gonna talk about implementation and key steps which include like reviewing all the resources provided by CASAS for California Remote Testing, determining your testing team and then identifying the co-app. So I'm gonna hop into that section and then we'll move on to my next section. So really the essential first step of implementation for CASAS Remote Testing is understanding the CASAS guidelines. So make sure that you've reviewed the parameters of these guidelines and how you see that working at your school with the systems that you're creating. You need to check that constantly to make sure that you're still staying in those parameters. I feel like I'm constantly just making sure, okay, am I still on track? Am I still on track? Because you wanna stay in the lanes that you're provided with and I appreciate the guidance but sometimes I start like getting a little farther and farther off track. So check that constantly. It was also helpful for me once I reviewed the instructions and went through it all to make like a cheat sheet for teachers and staff that don't need every single detail but they need to know the parameters as well and those that are administering the assessment for quick reference. And again, I'm talking about civics assessments. So the next step in implementation is really this step of taking inventory. This is about really looking at the school resources and identifying who can help. Is it teachers, classified staff, paraeducators? This is the step where you take the self-assessment of your school to identify what you have to work with. Also you need to look at what students have to work with at this step. What devices do they have at home? For civics, are they doing most of their zooms on their cell phone? Do they need to come by and pick up a laptop? What's going on? Are you using Zoom? Are you using Google Me? What platforms technology-wise are students familiar with and using? So this is about taking inventory and getting people on board. Some examples of questions you would ask at this point are like, do I have a technology team or do we have technology support? What will students be using for the test? Computer lab hours. Could we increase lab hours to open accessibility for students that have poor Wi-Fi or distractions at home that make testing difficult? Classified staff with flexible hours can be great at supporting this system. There's always that teacher that's eager, willing to innovate. And Christina said, look for emergent leaders. I love that term and I love the people that are just, oh, how can I help? The key is to look at the pieces you have and identify what you need. So just really kind of taking inventory. Then the next step is to identify the co-ops and teach an assess. Need to reflect on the needs assessment, the students' needs, the school resources and ask, is this doable? So one of the key things I'm gonna say here is don't get tricked into picking co-ops before you have done step one and step two. You'll end up with a co-op that you're using and you don't, that doesn't fit in with the guidelines or the parameters for remote testing or doesn't fit into the guidelines or the parameters of the resources that you're working with at your school. And that really varies from school to school. So make sure you do these steps one and two first. This will, if you don't, it'll lead you down a path where you'll end up committed to a co-op that you can't actually implement. So the third step of implementation is choosing the co-op. Check back frequently again to the COSIS website and look at the co-ops because they're constantly, right now they're reviewing them and they're updating them with that lens on remote testing and they're adding bits of language that include orally or electronically or and they're updating the co-ops so that they work fit better within the parameters of remote testing. The second, the first you need to review when you get to this step of identifying the co-op you need to look back at the needs assessment. What did your students say they were interested and need to learn about? And make sure that the co-ops you pick are also relevant to student needs. Then look at the school's resources to make sure that the curriculum that you have supports the co-op that you're teaching and that you have those resources available whether it's Burlington English which Burlington English made a wonderful crosswalk between the civics co-ops and their lessons that they have. So if you're using Burlington English ask your rep to get that for you. It's really useful, I use mine all the time. And then you need to ask yourself, is this doable? So it's important to look and visualize it. And the reason why I have a little snippet of this is co-op 47.2 internet safety is because I just wanted to point out looking at task three and task four what I do is I look at these tasks and I say I noticed that it says here on task three students can respond orally or in writing. This gives you some freedom if there's a problem with the online testing and a student quickly needs to move to oral and meet in a Zoom one-on-one with a teacher it gives you that freedom or that ability to be able to do this. And then in four it says, you'll see this task says optimally, optimally the writing task will be created on a computer and sent to the assessor electronically. So before when we were like really heavily paper and pencil we almost kind of scoffed at this because we were like oh, our students don't have computers this was maybe two years ago. Oh, they don't have computers or they don't have time to be online. And I think we were really underestimating our students and I think this forward thinking was there from costs us from the beginning. And now I'm so glad it's there because it's working better for us and better for your students. So for us, we did find that role playing was difficult because it's very time consuming and it requires breakout rooms with like one-on-one or appointments one-on-one with the assessor and the student. So that's something that we learned at our school was difficult. It can be done, but it's kind of time consuming. And then we also found that portfolio for us we've shied away from portfolio because it's difficult for us to verify that the work is student work and maybe and not like family work or group work. We want that portfolio to represent everything that the student created, which would happen in a classroom, but maybe not when students are working at home. So assessors have found that via Zoom they're able to monitor about 10 to 12 students effectively while they complete their tasks on Google Forms. And Google Forms has become our go-to tool for remote civics assessments. And most students complete the assessment using on a cell phone while zooming on their laptops so then the teacher can watch what they're doing. They can swipe the room and go through all those steps. And the link to the assessment is usually sent to students via email or remind during the class period. Okay, so assessment creation. So what's the first thing people do when they start a puzzle? You dump the puzzle out and you turn all the pieces face up. This is how I see the assessment creation. We dump everything out and we start turning things over so that we can prepare. When creating the assessment, I have found a few best practices and backwards mapping is now more important than ever. We use the platform and tools that students are familiar with and we must create and innovate when looking at the assessment and objective tasks. We've got to be creative and innovative. Test learning objectives, not digital literacy. That's a big key here and I'm gonna talk about that on one of our next slides. Okay, so backwards mapping. So have you ever completed a puzzle without looking at the picture on the box? It's virtually impossible, right? So we all take the puzzle box and we set it up so that it's sitting right there for us I think we all do. I don't know if that's how I've seen it done. So that's really how this backwards mapping works. We have to create the picture. We have to have the picture for the reference with the civics assessments. The first thing I do then is outline the objective for the team in a familiar format. So I go straight to the co-app and then I outline it and this is an example of what I do so that it's clear and written out in a way that everyone that's involved can understand. I use this format for every objective we do. I link the resources and assessment tasks to this format. This ensures that we're all on the same page. Next, I create the tasks. I use Google Forms as much as possible and create collaborative score sheets so that I can view work as it's completed. This is a new function in remote testing. Instead of paper and pencil testing where you can watch the scores and have the score so much faster. Often I will take tasks from previous assessments and recycle them to fit within the next objective's tasks so that the format is always similar. We're not learning a new format. We're testing the objectives or the learning gains here. It's important to be innovative and think of creative ways to adapt the tasks to remote formats. Consider all the ways you're testing tools. So all the different ways that you're testing tools. So for us, it's Google Forms can be used. So Google Forms has a video. It has images. It has charts and graphs. So you could use a video for respond when you see cues in the tasks that say respond. You could use pictures for it says identify or list. You can use tables for create a table. Be creative and when in doubt if you're not sure if you have any doubt always ask your COSIS representative. They're there to support you and they want to see what you're doing and they happily they get back to you really quick and say, yeah, this is great. This is great. This is within the parameters or they'll say, maybe do this or that. So I've had good experiences with that. Okay, so when creating the assessment, I cannot emphasize enough. You are testing the understanding of the objective, not digital literacy. And I know Christina talked about this too, but there's a lot of great technology out there. There's tons of great resources. But if your students are not familiar with the testing tool, it's going to get in the way of the assessment. So pick one or two platforms and stick with that and help students get familiar with the platforms and then stay there. Here's a hypothetical example. And I'm using hypothetical because it is hypothetical but it was almost a real example. So your principal or administrator comes back from a consortium meeting and says, hey, this school's been really successful with and I chose Flipgrid here. For the speaking portions of civics, we should try that out. So absolutely, you should consider and be innovative, but it should be utilized in low stakes, like low stakes in class assignments and activities long before it's ever used in an assessment tool. Keep things as simple as possible. This doesn't mean don't innovate. It means we're selective and you first teach the method before we use it as a testing platform. So there would be a lot of assignments and a lot of activities happening on Flipgrid before we utilize that or brought that in as a testing tool. Okay, when it comes to the actual administration of the test assessment, I have found it is important to be available and ready to support. Have a calendar with your schedule of availability. Have Zoom links ready so you can hop on and help when needed. Be the calm when assessors are nervous. This sometimes feels like, oh, it's so high stakes and what if this or what if that and everyone questions technology. You represent the calm in this. Be available just to say, it's gonna be okay. That's all right. I can email the printed version or we can do it another time or I can schedule something different so that everyone understands, yes, this is a test and it's important but at the same time, we have that flexibility and that accommodation. Plan for technology problems. I've had assessors call me in the middle of an assessment and say, it's not working. It won't let them post. And I just say, I have a backup, no problem. Or I can send you the slides so just kind of be as much as you can think through the questions. Christina has that great list of questions to ask. I don't have that for you, but I do it mentally. Like I try and think like what could happen and what can I do to avoid that to prepare if that happened? So the next thing I wanted to say is Google Forms has helped tremendously for us. With this is I have been able to immediately look at real time test results and share with assessors. And so I wanted to show you this is what it looks like. Okay, so it pops up. If you go to Google Forms and you click on responses, it pops up and looks like this. And then you click on the spreadsheet and this is what the spreadsheet looks like. I redacted of course all the student names and teacher first names, but before I started the assessments, I would go to sessions like this in conferences and just be like, where does all the information go? That's what I'm the most confused about. And I couldn't figure it out. Well, it actually is so easy to sort. And I just sort everything by teacher name. And then I look at the scores and I usually move this over to a written scoresheet because a lot of what the results are, I try and leave as open-ended as possible because we really need to measure the student's learning. And so multiple choice is something you're gonna want to avoid. So I need to look at this and grade all of this individually and then I can grab the scores. And it's just really beautiful. And I wanted you to just take a look at how easy and clear it is instead of stacks of papers and spreadsheets that you're creating, it just happens automatically. Okay, so maximizing learner outcomes. Some key ways you can maximize learning outcomes with remote civics testing here. So some things that you need to make sure you do is that everyone involved has the proper professional development. We have been so fortunate at our school to just have a lot of support from administration and a lot of extra hours devoted to professional development as we've moved into remote testing. This can be official PLCs or even shared YouTube videos with FAQs. I'm like the queen of YouTube videos and sometimes at the end of the video, I say, send me an email that says something so that I like some words so that I know you watched this and that I know you made it through. Give students plenty of opportunities to practice the digital skills necessary before the assessment, which I already talked about. Use resources that are already available and I'm gonna show you these, some really great resources that are provided through OTAN and CUSIS. And then focus on synchronous learning in civics and then find a partner to navigate with. It's really important that you have someone that you can say, I was thinking about doing this. What do you think? Or I was reading about task four and it says such and such, what do you think? How could I do this? And if you don't have someone, I can be that person for you. My email's at the end of the presentation and I love talking about and exploring the ideas, solving this problem and exploring ideas on how we can do that within the parameters. So here's some examples of maximizing learner outcomes. OTAN has created some great resources. On the left here is a picture of curriculum resources. In the middle, we have technology resources you can use for the assessments. And then on the right are ideas for how you can assess specific task instructions. So if it says create a list, OTAN has provided ideas on how you can do that. And the link to this is it's on the CUSIS website but the link is also here for you. CUSIS has digital resources on their website. On their website, start here. They've posted one of my favorite OTAN resources that helps you with task creation. Let's see if I can get it to pop up here. Okay, so this helps you with task creation. So maybe you need to create a sample identification card. OTAN has found a link like on the internet where you can do this. Maybe you need to create fake text messages for some of your curriculum or social media posts or newspaper articles. So they've created a list of places where you can just go here and get recipe cards or whatever. So all of these can and should be used when creating the assessment or building the curriculum because they're realistic and they're applicable in today's world. And they're already curated, they're already here for you in a nice list. So you don't have to reinvent the wheel. So then this is an example of how I wanted to show you getting students used to the assessment tool before the actual assessment. So in a PLC with teachers, I was trying to show them the same thing like please increase your use of the assessment tool and we use Google Forms mainly. I created a PLC exit ticket and it just was like which picture best describes your winter break and you just pick one. But instantly you're having an encounter with the same format of the assessment. If you're going to have students email, then you need to make sure that they're emailing, they know how to use email. Their first time emailing should not be emailing a high-stakes civics assessment or a piece to that. So just make sure you can use these as exit tickets, emails, quizzes, games, whatever. But you'll see this practice test here looks a lot like this PLC exit ticket. And so then when they get the assessment, it feels familiar and that creates increased learner outcomes. Okay, so here's an example of civics materials streamlined and stored on clear one-stop repositories. We use Padlet for curriculum and I created a Google site for all of the objective information. When we keep the process as simple as possible, it allows teachers and staff to focus on the actual assessment and curriculum to build student learning instead of searching through old emails, trying to find links to bits and pieces. This was a pitfall that we realized I was sending out like, here's a great resource, here's a great resource, here's this, here's that. And we realized that having everything in one location really helped have a clear process that ultimately has an impact on maximizing these learner outcomes. Okay, so for my last slide, I just wanted to talk about like, how do you find support and just jump in? So at Corona Norco Adult School, remote testing and online testing, it's not going away. We have a mindset of we're here and we're here to do this and we're here to create systems that are not temporary. We're gonna embrace this. When our students are back in the classroom, they will come to use, they will continue to use these methods to test. They reduce paper waste, we have increased test taking, response time, scores and feedback have increased. What started out as a quick fix to a problem actually has become a beautiful system. And it's important that we have a mindset to create smarter testing opportunities for students. And just really thinking about longevity and not band-aids that we're here to create good long-lasting systems. Okay, so I think that's about it for me. And... Marcy, I was gonna say, I must have the outdated slides. So can you grab that link and post it in the chat? And while you do that, I think Marcy answered the question about, you know, is this around to stay and maybe Barbara wants to chime in, but I don't see remote testing going away. It's equity, right? It used to be you have to come to campus, you have to do this, but what about the person that can't come to campus? What about the person who is homebound? We've provided an equitable situation for everyone who wants to be educated to receive the education where they're able. And then, let's see. And then the question about testing a student that's out of state, I'm not sure and I'm not gonna speak on whether or not you're allowed to test out of state, but I know I have a student who lives in California and her husband's work has taken them to Texas for a small time. And so we do that all through Zoom. She signs up for the appropriate time. I connect with her on Zoom. She downloads the platform onto her device using the instructions provided by OTAN. And then on our end, we're able to activate that test and let her take it while we proctor her and make sure that she's, you know, not cheating, not sharing the answers. And those are all part of the testing process that CASAS provides. The questions, the restrictions, asking to do a visual scan of the room, making sure they're in a room alone and, you know, watching them rip up their paper at the end because they're allowed to use paper for the math, only being able to use a calculator that's on screen when they're testing from home. These are all parts of the puzzle that CASAS training is gonna help you fit in and then where the student is doesn't matter from there. You can test them wherever they're at as long as they have a device that fits the parameter set by CASAS. All right, and Marcy, are you ready for some questions? Yes. Okay, so... There's one question that says, CASAS person, my understanding... I like that. That is that we can share practice tests, practices for tasks but not actually assessments. That is true. The sample tests, there are sample tests or assessments on the CASAS website. You can work with those. If you're talking about EL Civics, they are not as high-stake as the goals or life and work series because they're developed by each individual agency. So the limitations on those are much less stringent than the NRS-approved ones. So did that answer your question? In this environment, we're gonna assume she said yes. If not, there's Barbara. Yes, she was wondering about EL Civics and yes, yeah. And for the most part, every school's gonna be different. And so it's really better to get to know your students and teachers, I'm talking from the civics perspective and then make sure that I think that's the beauty of Civics is that it is tailored to your school, your environment, your needs assessment, what your students need. And so they really are unique to each school. And then another question was, can you automatically, this sounds like Melinda's influence, get the scores in the spreadsheet or do you set the form as a quiz? Yeah, thank you, Diana. That's such a good question. And that really for me was like just the gray areas. How do you do this? And so yes, the form, I set the form up as a quiz and then what you do, because I like the questions to be as open-ended as possible. What I do then is go through and you score each student's sheet individually and then that'll populate the form automatically with their total score. Now, sometimes there's a piece like task three needs to be emailed and there's a rubric. And so I still have the score sheets as a rubric. I keep those as their Google Docs and I have each one scored individually. It's still a lot less time consuming than the previous way with scoring everything manually with a paper pencil. But I hope that answers your question. Yeah, play around with it because it's fabulous how it works out. And then another question is there a clearinghouse of Yale Civics with practices, tasks and assessments that have already been administered? So there's not from us. There might be something that this might be a question again for our... This is Barbara and I put my email in the chat. But there is an Yale Civics exchange that we are doing in collaboration with OTAN and it's in the works. So we're kind of beta testing some things right now and I don't know the expected live date but when it's up and running, agencies will be able to download any of their materials that they're feeling comfortable sharing. They probably, and I'm not for sure about the tests specifically because you wanna do your own test for your own agency. You can, there are samples on the website. So you can look at those but each agency should develop their own assessments for a co-app and by looking at the rubric and working with their teachers. But that was one thing. Another thing I wanted to reiterate that you alluded to earlier was every meeting I've ever been that Dr. Zachary is in, she will say remote testing is a wave of the future. So it's not going away. And I believe that was your point and I was just going to share that. The other point is testing out-of-state students. I think if they are your students and they have indicated to you that they are going away with their husband but they will be back, I think that's fine. I do believe that you don't want to just accept somebody that is from searching the internet and saying, no, I wanna be part of your school. I believe and I can do some more research in it but it seems to me I was in a meeting and Dr. Zachary discouraged that. Yeah, I know for us at our school, we look for it to make sure they have a local address that they're using to verify that. So I see, Neda, thank you for that. Barbara, Neda, is your hand raised, Neda? It is, it's raised and it was just to confirm Barbara's comment on the Yale Civics Exchange. And the big reveal for that is at Summer Institute. So it's at Casa Summer Institute. We've been working on this for a little over a year. Our programmers are really working hard on it. So look for it, if you're planning on attending Casa Summer Institute, look for it there. If not, we will have a series of webinars after that that will reveal the Yale Civics Exchange. Very excited, and you're right, not the tests but the actual 30 hours of instruction and the material that's what you'll find on the Yale Civics Exchange. That's awesome, thank you. It really is because all these agencies have spent countless hours developing material and why reinvent the wheel? Right. Why redo the old puzzle? Right, I agree. Okay, so I think I've gotten to most of these. Did I get them all? For job application, do you use a Google Form too? I haven't thought about that one yet. So I don't know how I would do that. I don't know, I haven't thought about that. I think I probably would try and make it as realistic as possible. And probably a job application is a fillable PDF is what most employers would be using. I think it's really important to understand that when we're doing Yale Civics, we are teaching civics skills, civics life skills. And so we want things to be as realistic as possible. So I think I would probably use a PDF, a fillable PDF form. For that. Okay, Christina. Yeah, so what do you guys do with your puzzle when it's complete? Maybe you glue it and hang it on the wall like my aunt does with every puzzle they've ever done. I know for me, you heard, I take a picture of it, send it off to Marcy or whoever I want to see that I've done another puzzle. But the time's gonna come when we can greet our students back on campus. But as we said before, and Barbara reiterated and Carolyn Zachary is saying, remote testing is here to stay. And so this masterpiece we've created is gonna become a system that our school can use to meet our students where they're at. Thank you, Christina. I think at this point, I'm supposed to like really include like a really cool quote from the beginning, but you know, it's just really about having that open mindset and having expressed to your school and administrators like we need to try this, but we're gonna make mistakes. There's gonna be bumps, but let's support each other and just grow and figure this out.