 I'm going to talk about an observation instrument that we developed and that we use in our school. So I'm going to talk about how did we make it, how we linked that to our development program, some of the problems that we have in our center, and then maybe some of the more systemic problems that exist with observations. So last November there was a QQI event up in Crow Park and there was a lady from Turkey, and I hope I'm getting this right, it's Tijian Askit, and she was talking about part of your quality assurance that you've got to try and assess teaching and learning. So you've got to know is it working, is it not. And so when she was talking about observations, she recommended using criterion reference observations. And in her center, it was a university setting, they were using ballet, which was kind of the standard for them. So that's great and that's really nice, but that's not really going to work for me working in a mainly general use center in Dublin. So nice idea, but maybe not going to work for me. So we decided, okay, we like this idea criterion reference, what will we use? So we decided, well, why don't we just use the delta ones because that's a pretty good standard, you know, and it's externally validated, we don't have to think about it too much. But that was a little bit high for where our teachers were practicing because we had one delta qualified teacher as she left us high in. And now we have another one who has just passed through the module two. So it's a little bit kind of beyond that level of practice. And so what we kind of figured out was that we have a certesal in our center. That's good, isn't it? And Peter had kind of done all the work. Peter, they said, there's our certesal coordinator and he done all the work and made some lovely trinity certifiers. So they sent over to trinity, they said, these are great. So why don't we use them for a start? So that's where we started with it. Okay, we train teachers in our center and we train them with the philosophy of the school. So we think this is how we would like people to teach. So let's use those descriptors to kind of assess what's happening with our teachers that we have at the moment. So that's how we made it. Yeah, that's how we made it. So here's what you need if you want to make your own descriptors. You're going to need some sort of external standard. I think that's really important as a baseline. I did sit in the room maybe about three years ago with three other academic managers in my center where we made a list of what I think good teaching is. I don't know, I don't think that really worked. It wasn't validated from anywhere else. It was just my personal feelings, your personal feelings. I kind of think you need to benchmark it against something. And then it can evolve. I think you need to have a link between those descriptors and what your practice is or what you would like your practice to be. So again, thinking about one of the most powerful questions you can ask is if I do something, what effect is that going to have on the classrooms? What's that classroom going to look like? What does that teaching look like when it's finished? We did toy with the idea of, well, let's scale it, let's have newly qualified teachers on the CERC TESOL loan, let's maybe change that and go up to maybe a Delta standard later on. But again, we're kind of operating there on the principle that, oh, you've got three years experience, you must have developed all the time. But as we know, experience doesn't always have equal development. And so again, not really having a Delta standard teaching force. I don't think that's really realistic to use those. And so the next thing we did was we trial it. So we made it, said, great, that's lovely. But we trial it. We decided, okay, we'll take some of our experienced teachers, we'll go in and use it with them. We'll get some feedback from them. How do they feel about it? Is it working for them? Is it helping them to identify things that they can use, or problems that they're having in their classes? We have two standards. So we've gone here to Dublin, one in London. So we tried it in both contexts. And we had, I went to London, I used it there. Somebody from London came to Dublin, used it with our teachers. So that was interesting to try and see, can it work in both places? Yes? And so here's how we use it. This is what it looks like. So this is a little sheet. You've got kind of different aspects of teaching. They're further subdivided. There's kind of three sets of criteria for each of them. Kind of like at the amazing, wonderful, aren't you? And a brilliant teacher at the, you're doing all right. And then at the, maybe let's do a little work on this. So you kind of got three things. So we use it in a kind of a seven step process. I only realized it was seven when I counted them when writing the slides. So the first thing is the time is easier that you're going to observe them. You know, let them know. And so what we do is we'll say, hey, we want to observe you in the next couple of weeks. What day is it? I think that's fair enough. You know, a teacher needs to have a bit of control over when you're going to come in and disturb their classes. And the teacher submits the plan. We just used the regular lesson plan format. I know we did previously have, you know, this special lesson plan for the observation, which, you know, I don't really see the points of that. It's just more work. And so that's their normal plan. So you say, this is what I'm going to do. A pre-observation meeting. And again, we use like a little interview schedule with that kind of some questions to ask about, you know, what are you going to do in the lesson? Why did you choose these materials? Why is this good for these learners? What have they done before? What are they going to do next? Then it is the observation itself. And so during the observation, we try and fill this in as we go, but we also try to keep a running commentary. So when I'm observing, I like to use my computer. So you kind of hear the tapping. The tapping is constant, which I find is good, because it doesn't mean, you know, the teacher says something and then it's... If you keep it kind of nice, even stream of things going on, there's just less kind of panic. You can hide behind the screen, which I think is good. Afterwards, we'll write the report. So we have this. We have to kind of blow by blow what happened. And then we have a little report that kind of says, this is the summary of the lesson. This is what happens. This is what it looked like. And then maybe one or two recommendations. There's a post-observation meeting, but it's not straight away, tell me what you're going to do next. It's, here's all the documents. Here's what I wrote. Here's the timeline. Here's the little report. Take that home. Read it. Think about it. Decide just something that you want to add. There's a little negotiation there. You can say, oh no, I did that because. And so now that can be included in the report. And then finally, there's a little conversation about, okay, what are you going to do? So this is where it gets interesting, because I might have an idea of, I know what you need to do. And the teacher might come back with, this is what I want to do. And so it's that negotiation between the two of us. So this is what comes next. So here's my lovely picture. So we're going to do this, but that just gives us one kind of idea. So I've seen that one lesson. But that doesn't really tell me what's really going on with that teacher. So I need a bigger picture. So we use things like process-based feedback. So like asking students, what's it like in this class when the teacher does grammar? What's it like in this class when the teacher does listening? And students are pretty good. So we have 15 students in the class. So every class is observed by 15 people. And those students are normally reasonable to experience. They've been in lots of classes. They've seen lots of teachers. So they've got some pretty good ideas on how things work. They might not be able to tell you, well, they used an inductive approach. It was quite, you know, test-based test at the start. They're not going to be able to say that, but they'll definitely be able to tell you, well, first we tried the exercise after the teacher explained. And that's enough for you to be able to say, oh, yeah, I know what the teacher was trying to do there. And maybe look at the lesson plans, maybe from conversations you have with the teacher or conversations you've overheard as you've slowly passed by the teacher's brain. Or just your instincts about what kind of teacher is that person. I kind of feel like we've got 38 teachers. I think I have a pretty decent idea of what most of them are like as teachers. You get a kind of feel for that with the people you're working with. So this kind of full picture when you have it, helps you to say, OK, I've got a really clear idea of what this person needs to develop. So the observation is just one part. It can't be the whole thing. So now I've got an idea. This is what they need to do. I need to think, is that realistic for this teacher at this moment in time? Because they might come back and they might go, yes, I want to develop an action research project on how to motivate C1 students. And you're kind of like, yeah, but I don't know if you're quite there. So it's time to maybe bring that down a bit or push it a bit higher. And then also you need to think about how you can structure that. So you shouldn't necessarily be telling them what to do but you need to maybe put a shape on it. Deadlines are important. So enough preaching. And here's some of the things. So the problems we have, we have maybe four of us in our centre that do the observations. And that can be variable. We did standardize. So like two people in one class is observing. We did a couple with videos as well. But it's still a bit variable and it's still a bit what I like versus what he likes. And that leads to inconsistent messaging and that's not really that good. And the instrument works really, really well for people who came through our training course and work with us now. Not so much for people who train in other centres because maybe they've learnt a different style. And so do we want them to change to be more like our style or do we accept that there's lots of different ways that work and that's a really interesting question that's probably a longer top. In an observation it's just one little slice and it's a really interesting question as well. Can you tell if learning has happened or is happening in that observation and isn't the whole point to know whether that teacher's effect has been making really happen. And then the classic of course for us is that the teacher's development's desires not quite matching maybe their development needs. So those are some of the problems we had if anyone has any solutions I would be delighted to hear. And then there's some problems that everybody has. So most girls have an observation system and it's all different ways to do it and we're really, really bad at observing. So the Sutton Trust did some research and a report on their MET project which was done by Bill and Melinda Gates Foundation and they said it's about 50%. He may as well flip the coin to decide if it's good teaching or bad teaching. Which is really, really serious. So like there's a blog by a guy called Robert Coe and he was kind of saying it's all about our instincts so we bring all our biases, our preconceptions into the classroom with us and if I went in and observed the class and then you went and observed the same class later if that were possible, so now maybe by video we would probably disagree. So we're really, really awful at it which kind of throws into question a lot of things because insert T-celled delta, dip T-celled delta one of the main ways we measure good teaching is through observation. So the question I have is why aren't we talking about that? So is this going to be another thing in 20 years where we're kind of going okay, it's like learning styles all over again where we know it's bad practice but everybody still includes it. Look how long what Cambridge took learning styles out of this syllabus maybe a year ago, two years ago. And it's, you know, and yeah, yeah, okay, so really, really recently so if this research is coming in now that we're not very good at observing surely we should be talking about the implications of that on all our major qualifications which is kind of a big, a bigger kingdom we can fit into this one that we tend to talk. And so, here's an alternative. I really believe in process-based gene feedback I mentioned that before asking them what do you do in this class and let's make our judgments based on that. And I do think that the likes of Ruth Weynard and Thomas Farrell had great ideas when they said we should use evidence-based so we can count things. We can say how many times a teacher does this how many times a teacher does that how many times students do this or that and then we can decide is that a good thing or a bad thing because we can link it to the outcome of the lesson we can link it to how effective it is and then things like guiding reflection tasks based on that so we can ask questions and there is the danger of course of reflecting for what you want to hear which we heard about.