 Well, just while we're getting set up, let me say thank you Sharon for your very kind introduction and for your invitation to participate in this webinar. I'm very pleased that and there may be interest on your side of the country in the work that we've been doing. So I am presenting this webinar as part of the Centre on Standards and Assessment Implementation, one of the federally funded content centres that was funded in the last competition. And this is, I suppose, the renamed Assessment and Accountability Comprehensive Centre. And it was the Assessment and Accountability Comprehensive Centre that we actually did the work under. And so I feel a little schizophrenic now. I'm presenting this with my new hat on with work that we did under the previous content centre. So my purpose today, John, if you can change the slide please. The purpose today is I wanted you to first of all get a sense of what we developed in terms of our online formative assessment programme, why we did it and some idea about the content and structure of the programme. We also have had a chance to implement this in a pilot and indeed prior to it becoming online we were able to implement some of the content face to face. And we have some lessons learned from our implementations and also I wanted to share with you some potential models of implementation that other states have tried. So that's what I'm going to try and focus on today in the session. Next slide. So in terms of the overview, here's the agenda. I want to give you some background to the programme just briefly so at least you know where it came from. An overview of what the programme consists of. And then I'll be showing you a sample of one of the modules, just a few slides of one of the modules so you'll get a sense of the content and its organisation. And then we'll talk about supporting implementation, some of the lessons learned and some of the models that have been tried in different states around the country. I put chat box in discussion because I would encourage you as I've left some spots for questions and comments. But if you have questions that come up throughout the presentation please feel free to put them in the chat box and post them and I want to make this as interactive as possible. Next slide. So as mentioned to you, this professional development was developed in a collaboration between the Assessment and Accountability Comprehensive Centre, which no longer exists, and the North Central Comprehensive Centre at MacKrell. And it was supported by the federal government and as such everything we produced is absolutely free to anybody who wants to use it. So there's no charge at all for using these materials. Next slide please. And so as I said any work that we do going forward will be done under the Centre on Standards and Assessment Implementation, the new content centre. Next slide. Some background. How did this program come about? Well, several years ago the AACC and the NCC worked with the state of Iowa to develop a statewide implementation for formative assessment. These folks were very keen. They'd been a member of the Formative Assessment Scass Rack from the beginning. Iowa was very keen to have a statewide implementation. We had a lot of support from the department. So a collaboration between the AACC, the North Central Comprehensive Centre, and the Iowa Department of Education. And so we developed the content and tried out the content of much of what you'll see today. When we actually came to put it online we were invited by the Department of Education. And so we developed the content and tried out the content of much of what you'll see today. And online we revised it quite a bit based on lessons we'd learned through our face-to-face implementation. So this lasted about two years, I would say. One of the goals of the project was to ensure that everything that we put forward in relation to formative assessment as a process or a set of practices was research based. We didn't want to sort of beat people over the head with research content, but we wanted to make sure that everything that we presented was grounded in the literature on formative assessment or learning theory and so on. And so once we developed our content collaboration we then piloted it with the Northwest Comprehensive Centre. So they actually did an online pilot with some of their folks at the State Department and some of their professional development leads to the State. And they gave us very helpful feedback. It was pretty brutal feedback in the beginning, but you know, it's all good feedback. You learn from your mistakes and we improved and certainly I want to acknowledge Claire Gates and the Comprehensive Centre for their sterling work and the State of Washington for helping us with refinements on this work. We were extremely grateful. So let me now, with that background, next slide please, John, go into telling you something about the program itself. I'm not quite sure why we can't see all the slide, but I'll just assume that it's a kind of closed procedure where you can fill in the letters for yourself. So in terms of the purpose of the program we have two core objectives in this. Next slide please, John. The first objective was to help increase teacher's knowledge about the process of formative assessment. What formative assessment actually entails in classroom practice? So we wanted to increase the knowledge base and then we also wanted to provide a support mechanism for teachers to implement this process effectively in their classrooms. And these were our primary goals, to increase knowledge and to increase skills. And the general view we have taken in this formative assessment module is one, as I said, that comes from the research, that formative assessment is not a specific test but rather a set of practices which have been shown when implemented effectively can improve student achievement. Next slide please, John. So the orientation of the program is that formative assessment is a process used by teachers and by students. We're very clear, as the literature is clear and many other countries in their implementation are very clear that it's not just something teachers do, students are equal stakeholders in the process. It's the means to provide ongoing feedback to both students and teachers while learning is occurring. It's not something that happens after the event and it's integrated into daily teaching and learning. So this is the view that we've taken in this program which corresponds very much to what formative assessment skills, views as effective formative assessment, what the literature suggests and the implementation of many countries around the world. So we do not see it as a particular test or instrument. You'll never hear anything in this module, this program about formative assessments. We don't see it as something to provide a score. We want to make sure that people are getting substantive insights into student thinking. It's not something that happens periodically as an interim assessment. It's something that is ongoing and embedded into the instructional process and certainly it's not something that teachers do. We regard the students as equal stakeholders in this. Next slide, John. Now we use this framework that I developed a few years ago as the glue, really, if you like, for the whole program. And this funds every single module and basically each module addresses one of the dimensions or two of the dimensions of this feedback loop we see as formative assessment in practice in the classroom. So, for example, we have modules on learning progressions, learning goals, criteria of success, on eliciting evidence, on providing feedback, on the classroom culture, on peer and self-assessment, which you can't quite see here, but that will be down on the bottom right. So this is the glue. And we come back to this at the beginning of each module so that participants can see, oh, yes, I see what element of the process we're on right now and where we're going next and where we came from. So we really try and keep this as the framework for our sessions. Next slide, please. Margaret, I'll add that when I played with the presentation format layout, I was able to change the whole slide so if people are not able to see the entire slide, maybe try the presentation view and again, that button on the bottom right hand side. Okay, thank you, John. So let me tell you about some features of the program now. Our goal in putting it online was to maximize the flexibility for its use. So we didn't see it as a one-size-fits implementation. We wanted to have consistent formative assessment content, but the implementation of these modules could be flexible. We didn't make it content specific, but we feel and certainly that's the feedback we've had that it's adaptable to specific content areas and across grade levels. So for example, in Washington they had a group of math people together and they were able to add content to it that was focused more on mathematics, still keeping the general idea of formative assessment, but some of the examples they built out in terms of the content. And we see it as adaptable across many different levels of the system, so that it could be used within the school, it could be a district, some people have uses on a regional basis, and I'm going to tell you an example of using it at a state level. So the whole idea was that we provide this content that how it got implemented was really up to those folks who were doing the implementation and wanting to learn. One of the things that I think got cut off here that we did make a significant part of the program was teacher learning communities. We have in each of the modules, in addition to content and activities, we've also provided work that can be done in the intervening period between one module and another for TLCs to engage in. Next slide, please. We thought this was important to do because the teacher learning communities, and I can't really see the quote here, but I will share it with you, that what a lot of certainly William and Thompson 2006, some of the work they've done and the work that's gone on subsequently with people like Caroline Wiley and Christine Lyon from ETS, they have been finding that teacher learning communities focused on formative assessment appear to show the greatest potential for improving teaching practice and student achievement. So the idea here is, and we think this is maximally beneficial, is if people are using content of these modules in whatever format that works for them and then engaging in teacher learning communities with the activities that we provide for the communities and of course anything else we want to add, that seems to be one successful avenue to implementation in the classroom. Next slide, please. And just quickly, to help us or remind us, this is from Christine Lyon and Caroline Wiley who've done quite a lot of work on teacher learning communities, and they talk about this model of learn, practice, reflect, revise. And so that's what we're seeing as our modules supporting the learning and then we suggest ways to practice formative assessment and then the learning communities give them an opportunity to reflect and revise. And we have found that teachers get to a deeper understanding of formative assessment and also teacher learning community provides support for different levels of understanding of education. So we've made that quite a significant component of our modules and certainly feel that that's integral to the work of implementation. So let me now move to some specifics in relation to the program overview. Next slide, please. Next slide, please, John. So these are the modules that we have. We have seven in total. We have an introductory module. And this is for people who are basically going to be implementing the program, be the leaders if you like of the program. It gives an overview and ideas about implementation, how to go about using the modules and the content of the program. And we have quite a lot about TLCs in there as well and the advantages of using TLCs and how to set up effective TLCs. And then the modules go through all the dimensions of the cycle, the feedback loop that I showed. All guests have been muted. You can unmute your line by pressing star six. So have I been muted? Can you still hear me? Oh, you're good, you're good. So we have a general overview and then the subsequent models are going to specific details about those elements of formative assessment I showed you in the feedback loop. The modules are quite extensive. And again, they are divided into specific lessons so that implementers or even if a group of teachers is doing it for themselves in a school, they can decide how long they want to spend on the content, take what they want from it, try implementation and then come back. So it's very flexible in that sense. In terms of supporting materials, next slide, John. Each module has three components. There's an extensive facilitator's guide. It is very extensive where we give some research background to the topic. We give notes for pre-planning, for planning during, and for implementation advice and then intercession advice, what you might be wanting to do in between lessons or modules. The PowerPoint presentation is provided as a self-managed Adobe Captivate with a voice script as well as a PowerPoint. So you can either use the PowerPoint presentation, just play it, it has the presentation, it has the voice-overs, or you can stop, you can discuss. It's very flexible as that, but there is a voice script. And then there are lots of handouts which comprise activities, information articles. We give articles for further reading if people want to engage in that during the modules. Next slide, please, John. And so you can use the voice-to-derby Captivate presentation or you can mute it if you want to and communicate the content as a facilitator. The script of the PowerPoint is available. And what we've included in terms of our learning structures are we have a lot of turn and talk where people are able to process or make sense of what they're learning about. Those are your turn where we ask people to engage in activities around the content. We have a Consolidate Your Learning, and that means towards the end of each section of the module that you're able to really do something to help you make sure that you've understood what it is that you're trying to learn about. And then the final thing is to try it out, and that's usually at the end of the module to take back into or into the schools to try it out in the context of the teacher learning community. So let me just stop here now. If you can go to the next slide, John. And if people have any questions or comments at this point, the next thing I'm going to do is show you some excerpts from one of the modules so you'll have a better idea of how all this content that I've talked about actually plays out. But I'll just pause for a second and give anybody a chance to answer any questions. Okay. And I've muted all the participant lines. So if you'd like to unmute your own line, you press star six. They may ask you to press star six again to mute it back. Or you can ask questions in the chat box. Well, I'm not saying any questions. So let's move on now. And what I'm going to do is to show you an excerpt from one module. And these aren't in particular sequence. I just excerpted them just to give you a flavor of how what I've talked about looks like in the actual module. Next slide, please, John. So this is module four where we focus on eliciting and interpreting evidence. And if you recall, that's one of the dimensions or elements of the feedback loop that you saw earlier. And I'm sorry you can't see this full slide, at least I can't, so hopefully most of you can. Next slide, please, John. Excuse me. So, oh, I can't see any of this at all. Okay. Well... Margaret, on the bottom, there's a little icon that looks like a little face, and this is pick a layout. If you pick presentation, you might be able to see the entire slide. Okay. There are three types. There's gallery view, speaker view, and presentation view. Yeah, I pressed and can't see. Okay. Well, never mind. Not to worry. Sorry about that. No, I just got completely lost. Okay. Well, I'll just keep on talking. And hopefully you can all see the slides. So, on this slide then, this is slide 20, John, that we should be on. Module has a number of lessons, and we open the module, and every module has the same format, where we identify what the content of the lessons are. And in the introduction, we give what the main messages of the lessons are, because learning goals and success criteria are integral to formative assessment. We also provide learning goals and success criteria for each of the modules. And as you can see here, there are five lessons in this particular module. So people have a sense, logging on exactly what they get to see. Next slide, please, John. In the next slide, you can see the main messages for each lesson. So, for example, lesson one, the main messages, the use of carefully planned tasks, questions, and observations can provide teachers with evidence of student learning during a learning sequence. So each of these main messages is posted. And in the facilitator's guide, we give a lot more information about the main messages and what facilitators need to bear in mind as they go through the module in helping shape the main messages as the content develops. On the next slide, you'll see, as I mentioned, that we have learning goals for each module. And here we want teachers to or participants to understand why it's important to elicit evidence and understand the criteria for quality evidence. We think this is important. Quality evidence is important in any assessment, and it's just as important in formative assessment. So I won't spend time going through all of these. John, if you go to the next slide, then we can see the success criteria. Now, these are for the participants to guide themselves through the module. And so at the end of it, they're supposed to be able to explain why, when, and how strategies can be used to elicit evidence of student learning. And at the end of each module, we want to model good formative assessment practice. So at the end of each module, we spend some time asking participants to reflect and to rate themselves with respect to the success criteria and then give them suggestions if they feel they haven't met the criteria of how they can develop further. Next slide, please, John. Here again you have the feedback loop, the next one, John, which I said it grounds all the content. And here we've highlighted the specific areas of the feedback loop that are addressed in this content in this particular module. Next slide, please, John. So this one, your turn. We normally try and give them an activity that's going to elicit some prior knowledge. And we think this is important for the presenter, for the facilitator, even for them to get a sense of what they already know about the topic. So in this one, we're asking them, what do you already know about the idea of eliciting evidence? And we do this in each module so that we start by listing some prior knowledge, which of course is what we want to model, effective pedagogy, and we know the importance of connecting new learning to prior learning. So we want to make sure that that's included. Next slide. And then once we've gone through these various introductory aspects, then we begin on the content. And as I said, we focus on research-based content, although we don't necessarily call that out in the slides, but it's certainly there in the facilitator's guide. And we've also added a reading list in the facilitator's guide of all the work that we have used to come up with our content. So if people are interested, they can go and have a look at that and read more, read the actual sources that we take our work from. So next slide, please, John. So in this slide, we're really talking about a notion that Fred Erickson in the beautiful paper, I wish I'd written the paper in about 2007, talked about taking stock, that formative assessment is about taking stock of learning as it is developing. And so we spend a bit of time talking about this idea of taking stock, that you don't let learning progress a long time before you take stock of where students are with respect to where you want them to be. Next slide, John. And then we also focus on this notion of getting inside students' heads. And I think this is particularly saying in the context of the Common Core and Next Generation Science Standards, we know that the Common Core is supporting or encouraging teachers to develop higher levels of thinking among students and deeper learning among their students. And so if we want students to be engaged in deep learning and being able to think critically and deeply about their learning, as they are learning, we need to get inside their heads. And so we talk about the importance of substantive insight. Laurie Shevitt has a wonderful paper where she talks about, you know, we don't need scores, we need substantive insights into students thinking. And this whole module is focused on how teachers can externalize students' thinking in a variety of ways through evidence collection so that they are able to get some insights into how students are thinking about this deep content to support them towards success. So that's very much a focus of this module. Next slide. We take a lot from knowing what students know and I think this is an important quote. This is one of the foundational works that we draw from from the content for this program. And what the committee said, that good formative assessment requires radical changes in the way students are encouraged to express their ideas. So a lot of this module is focused on how teachers can help children express ideas in ways in which that gives them insights into thinking. Next slide. And finally, in terms of some of the research that we address, this one from black and colleagues, Paul Black and colleagues, I don't quite know what's happened to the image. There is an image of the book, Assessment for Learning, which I'm sure you're all familiar with. And what they talk about is more effort has to be spent in framing questions that are worth asking. And so, as you well know, a lot of formative assessment is premised on teachers' effective questioning and probing of student thinking. So we spend quite a bit of time in this module talking about questions and helping teachers think about how to develop good questions that are going to get at student thinking and then to give them the insights they need so they can keep learning moving forward and develop that deep understanding among all students. So next slide, John. And then, you know, after we've done some content and we have a turn and talk and here in this one, we are asking them to think about questions that are worth asking, meaning ones that require some thinking and not necessarily a yes or no answer or what I call the guess what's in the teacher's head answer. You know, what am I thinking and therefore everybody has to try and guess. And then we, so once you give them an opportunity for activities, then we get into the idea of there's no single way to collect evidence. Again, I'm not quite sure what's happened to the image here. But we spend a lot of time talking about how the evidence they gather must be aligned with the learning goals and success criteria. There's no single way to get evidence. There's lots and lots of ways, but as with any assessment, it has to be aligned with the goal. So if you're assessing state standards, they've got to be aligned with the standard. Similarly, informative assessment at the micro fine-grain size of the classroom, then evidence has got to be aligned with the actual learning goal that's the focus of the lesson and the success criteria. The indicators of progress, what are those demonstrations of understanding. And it may not come as a surprise to you folks, but I have to tell you that over the years that I've been doing this work, it is astonishing to me how hard it is for teachers to come up with the learning goals and the success criteria. They typically want to go straight to what the students are going to do in the lesson. And it's very hard for them to think about well, why are they doing this? What are they actually going to learn as a result of it? See, John said it's very difficult. And then when you ask them, well, okay, so if students are learning this, what are the indicators? What are those performances that you're going to be looking for in what they're... I really like what Patrick Griffin always says in terms of what students say, do, make, or write. What are you going to be looking for that's going to tell you and equally importantly tell them that learning is progressing or it isn't. And there's something that needs to be done about it. So one of the modules is purely focused on learning goals and success criteria. And this one now is talking about aligning evidence. So if one of your success criteria is that students are going to be able to explain a particular concept, then they better have an explanation task. So we really spend a lot of time on this evidence quality here. And I'll just go to the next two slides quickly because time's moving on. So when we talk about evidence gathering, we talk about the importance of planning, planning ahead. It's not something that's just serendipitous in the classroom. It's actually planned along with instruction, the kinds of questions and discussions, the observational specific tasks that students are going to engage in so that you will have the evidence as a teacher that you need. Next slide, John. And of course that's not to say that teachers can't get evidence on the fly or things don't happen spontaneously. But it can only be used effectively as evidence if teachers know what the criteria are that they're looking for and suddenly they see it in a way that they haven't anticipated it. So we really stress the importance of planning. So just go quickly through these next slides, John. I'm going to race through them. So the importance of formative assessment, making a decision, whether it's to carry on with the planned lesson or to make adjustments instructionally or to provide feedback. Next slide. And then we give them an opportunity to generate possible strategies for eliciting evidence. And we develop this idea in terms of your term. This is one of the activities that they would actually do in the session. And then they have a next slide, a consolidate your learning activity. One of the handouts and they're really building their knowledge here and they continue to meet with their TLC after the module to think about ways in which they can collect and use evidence. So and then finally, the try it out is when they take it away with them, they select one of the strategies, how they're going to use it and share your experience. And one of the things that we did learn from our pilot with Washington State and the NWCC is that teachers want to try things out immediately. So we've built a lot of time into the modules for trying things out. I'm a little leery about just trying out a whole load of different things, which is why I think trying it out in the context of the TLC is really important because they can keep coming back to it, revising it, and so eventually it does become part of their independent practice. And so that's just a quick ride through the content of one of the modules and excerpts of one of the content, some of the content in lesson one. So let me pause there again and see if anybody has questions or comments. Things aren't clear at the moment. I know I'm packing a lot of information into a fairly short space of time, but that's the nature of the beast here. So I will stop talking for a second and see if there's any questions or comments. And again, you can use the chat window, or if you'd like to press star six, you can unmute your own line and ask questions. Let me move on now then because I want to end really with some ideas for supporting implementation. So if you're thinking, well, we've got these materials, they're pretty flexible. I could use them in a number of different configurations. Then how might I do that? So the final section of this webinar is intended to help you think about that. So next slide, John. It's pretty clear from everybody who's been involved in formative assessment, effective formative assessment. Unfortunately, some people still think that formative assessment is about giving more frequent mini-tests. But anyway, effective formative assessment really does require a change in practice. For some teachers, it's going to be a big, big change. For those of the standard deliver mode, it's a big, big change. For others, it may not be quite so much. It may be a bit more planful than perhaps they have been. But I'm pretty certain that for all teachers, there will be some change. And for some, it's a big paradigm shift. So one of the things that we think is important in considering this module is to think about ways in which this can become a cell for people. Next slide, John. And so we know that it does support effective teaching and learning. I mean, there's a literature in that, and there's a lot of anecdotal experiential evidence from those of us who've been working in this field for a long time and have seen how it can change teacher practice for the better. It also is focused on the individual students, unlike other forms of data which get aggregated. This is very much at the individual student level meeting individual students where they are. I certainly think, and that's what we found through talking to states and thinking about these modules, it definitely supports other key initiatives. For example, response to intervention is very much part of the response to intervention where we pay close attention to all children's learning, some children then needing more attention than others, and so it goes on. And certainly we see the formative assessment as being a critical aspect to the implementation of the CCSS and the Next Generation Science. And that's certainly something that the states that I've involved with through the formative assessment state collaborative are focusing on this year, formative assessment through the Common Core. And the educational effectiveness, in fact some states have included formative assessment in one way or another in their teacher evaluation because they regard it as essential practice to being an effective educator. Next slide please. So one of the things that we have found, or a number of things rather that we have found, is the lessons learned. And these are several aspects of our implementation starting with Iowa several years ago that we've seen are important. First of all, sustained leadership. This isn't going to happen unless there's leadership to support the modules, to support professional development and the leaders at the local level who are committed to supporting this. It's just not going to happen. Similarly, this idea of commitment to change. If people think, and I do underscore this, because if anybody who's on this webinar right now is interested in this, then I think the interest has got to be not for quick fix, but for a longer-term commitment. Because as I mentioned earlier, we're talking about changing teacher practice and that can take time. And it requires ongoing professional development, which is why we put together these modules with the TLC component to it so that there will be support for ongoing professional development. It's going to need time. People need to commit time to this, but we wanted to make the program as flexible as possible so that we could be responsive to people's time constraints. They do need resources, so we've provided these resources, which we think are useful. In other implementations, we've realized that access to experts is important, to have the content knowledge, and so we've put a lot of expert knowledge into this content. And peer coaching is an important aspect of implementing new practices, and so the idea of the TLCs has become very prominent, as I mentioned earlier. So let me just conclude now with a few examples of some implementation at different levels. So this was a state implementation. It was the state decided that the schools and school improvement were going to engage with formative assessment for two years, and so these modules were used as part of that process, the content of these modules. The idea was that school teams would come together three times a year for two days, and this was when this module content, it wasn't used in an online format, but it was used, the content was used as it is. There would be two days, three times a year of this content, and then in the intervening periods, there would be school level coaching. There were coaches assigned to these school teams and webinars with experts, because we wanted to make sure that it was access to experts and also the kind of school level coaching that was important. We see the TLC now that was included in this particular program as a proxy for the kind of school level coaching if external school coaches aren't available. And this was pretty successful. Some people saw it as a great opportunity to change practice. Other people, as you would get in any mandated professional development, were less inclined, but in general, I think it was very successful and the content really made sense to the people who were in the face-to-face meetings. In the next one, this was another state in combination with a comprehensive center, and this was a multi-year project. This state actually started with their own folks building capacity at the department level. They didn't want anybody who was out in the field from the department talking about formative assessment in a way that wasn't consistent across the board. And so they spent some time building their own capacity. Then they were able to embed it into state initiatives. For example, in this state, there was a very long-standing initiative in math, and the math people spent a lot of time incorporating what they'd been learning into the math professional development initiatives. A further development in this state was that the regional centers, I'm sure you will have them, they're called different things in different states, they were involved in building capacity of the certain personnel in the regional centers so that they could provide the ongoing support of this content into the schools in their regions. So it was really starting with the state, then getting embedded into state-wide initiatives, which personally I think is a terrific way to do it, then building the capacity of regional centers so they could use this content at the local level, at the regional level to support districts and schools who really wanted to make a commitment to doing this work. Next slide, John. And then the next one was the state, again, the same content, some of it was online at this point, and they had a one-year implementation that was collaboratively engaged in planning and implementing this with the math PD providers across the state. And this was mainly led by the comprehensive center but the state did participate and they used the content of these modules, some of it was online by that point. So different ways of using it, you know, that's perfectly possible with this and the leadership of the school decided this is going to be a two-year long or a year-long initiative and we're going to use these materials. So it can really be used from the school level right to the state level as we saw with one state supporting the department then into state initiatives down to the regional centers. So as I said in the beginning it's intended to be flexible, we feel that the content is solid with respect to the research that it's grounded in and we sincerely hope that you think it's useful and might take the opportunity to use it at some point in your regions. So with that I'm going to stop again and maybe there are some comments, questions, ideas of anybody's got any ideas bubbling up, we'd love to hear them. And I'm going to use all the lines hopefully correctly. All guests have been unmuted. So if you'd like to ask a question either you can use chat but of course if you'd like to speak on the phone just tell us who you are so Margaret knows when she's talking. John, can you hear me? Yes. Okay, this is Kathleen from Virginia. I know we're working on formative assistance but is there a way that we can look at all of the modules? Yeah, John, I know we're going to be meeting to talk about this but this seems like compared to Rick Stegen's work and we've looked at that as well. We tried that a few years ago with PLCs and it didn't work well. Of course I think that was prior to more web-based and online kind of programming and I'm thinking how we lay this out will depend on and I think you went back a slide if you could a couple of slides. We'll go back one more. I think it was this one right here. And so the webinars with experts if we... I'm thinking through this but see the experts as being very important because I don't know how else to get this to a school level. I'm not sure if they have the capacity to get there on their own. So in this particular state I know this was a two-year project how did those webinars play into to the overall... what did they do four times a year? Well, I can tell you because I ran them four times a year. We basically... I basically use my formative assessment to decide what the content of the webinar should be based on the face-to-face institutes. You know, I could see what they were struggling with or I could see they needed more help in certain areas. So that was one way the content was developed. The second way was that I asked them to bring ideas and things they'd been doing and showcase them. So we'd have maybe three or four people talking about what they'd been doing some of the challenges and then it was used as a... almost like a PLC where we're all talking about what will be a good way forward here. I think the access to experts is important and as I say part of why we wanted to build strong content is that we wanted to get that expert knowledge into the content and I think it is advantageous if you can have some expertise that's supporting the ongoing implementation as well. I'll say, Katherine, I'm glad you asked that question because as you know, we are going to come to Richmond to share these modules. People in other content centers as well as SEAs are welcome to come see that introductory session. The idea was to give people more information on the material because as you see, so much information in these modules and then in the different region states could work with their comprehensive centers as to a model they'd like to develop so let's say Virginia would like to develop this and we had actually planned on maybe developing that online community with you and then you could decide who the experts are, how many times they would need different kinds of activities go ahead, Katherine. Go ahead, I like this model, that's good. I was going to say simply because funding is in everybody's mind right now but because funding is limited, getting Margaret all the way up to the East Coast and have a special occasion so we want to make sure that other people who might be kind of close to the region would be able to attend. And John, if they can't attend I might be able to arrange to have it streaming video. Okay, great. Do you know what the date is? We've already scheduled this date. Right, Friday, November 22nd. Okay, what I will do if we schedule that date I will check to see if I can and if it's Friday it's probably possible. I'll check, okay? Okay, and for our fellow comprehensive center staff that are online, please feel free to contact me or Margaret or our coordinator Kim Cook for attending that day if you'd like to do that as well. We'd love to have you. We'd love to have you. Yeah, beautiful place. And then Margaret and Nancy Grigdon ask how do we access resources if they can't attend that day? Is there a process they go through to contact you? Yes, if people want to contact me I can send them a link in the password in the description. Great, and I'm going to before I show Margaret's email addressing to come back to that I do want to do a very quick commercial for a webinar going on tomorrow on a formal assessment as well from one of the other content centers if you have not registered you can go to their website and register there. But if you do need Margaret's contact information there's her email address and she's very quick to respond. I want to know if there are any other questions either on the chat or on the phone. John, I will check and I think you probably made that point with Shelly in Virginia, Shelly Levin Rider. Yes. Okay, I will check with Shelly and see if we can make sure that it can be we might be able to live stream it which would be good because then if they want to join they can, okay? Great, awesome, thank you. And again this webinar and this idea of focusing on formal assessment the idea that several states in the Appalachian region that is served by the ARC were interested in this concept of formal assessment so we're trying to leverage those funds, leverage Margaret's limited time just trying to reach as many states as possible and we're willing to share that work with our other companies and centers and of course other states who they work with. So at this time I have three o'clock you do have Margaret's email address you want to follow up if you would like in that Friday, November 22nd state please feel free to contact me or you can simply reply to Kim Cook who sent out all the information for today's webinar I'd like to thank Margaret once again I know there's so much information you do such a good job getting through it and if you are like Margaret and you were not able to manipulate the slides or see the slides in a way that was beneficial to you we are testing the system you will receive an evaluation link and it would be helpful if you would tell me about the operating system and what browser you are using and I can give that feedback for our technology team. So thank you everybody for joining us today and I will be in touch with the follow-up email and questionnaire and thanks again Margaret. Thank you everybody thanks a lot John, bye bye. Bye.