 All right, Karen, when you're ready, I think we can start. OK, welcome, everyone. We are here as a part of week two for DL MOOC. I'm Karen Fassenpower, and I am part of the DL MOOC team. And we are so glad to have you here with us, whether you're watching live or watching on the archive. Last week, we had a few technical problems that hopefully we've sorted out for this time. And we're all learning, and we really appreciate everyone's patience as we're going through this process. Last week's session, as well as this week's session and all future live sessions, will be archived at DL MOOC.net, and you can watch them there. We're really excited about all the things that happened in week one of DL MOOC. We've had a great turnout and tremendous community participation. We have now over 3,600 people registered for the course, over 1,500 in our G-plus community, and others participating on blogs, on Twitter, and elsewhere around the web. We've designed DL MOOC to be a connectivist MOOC, where the main content isn't really the readings or even these panel discussions, but instead the learning we do together as a community. And these panels are really meant to be a jumping off point and a spark for your own explorations to take back and apply in your own learning context. And we hope that as you do that, you'll share with us on G-plus community through our Tweet of the Week, the Put Into Practice activities, and as well as all the discussions that are going on in the different smaller groups that have started up as a part of DL MOOC. These connections and these experiences are deeper learning for us, and they're really designed to be the main part of this course. Finally, before I turn it over to our panel, we'd like to have you invite your friends and colleagues to sign up for DL MOOC. It's not too late to join, and in fact, we've designed the course so that people can come in and come and go as they like. So please email, post on Facebook, and Tweet for people to join us if they'd like at DL MOOC.net. And now I'd like to turn it over to Rob Rudin to introduce our panel. Thank you, Karen. And for those of you who are just joining us, this week's session, two sessions, are about looking at student work. Overall, the MOOC, of course, is about deeper learning, and we want today to connect the what, why, and how of looking at student work to deeper learning. And just as a reminder, the elements of deeper learning as articulated by the Hewlett Foundation are content mastery, critical thinking, problem solving, collaboration, effective communication, self-directed learning, and academic mindsets. And those last two, self-directed learning and academic mindsets, are a particular concern of the Rakes Foundation, and they lump those two together as agency. So as you can see, we're talking about deeper learning as kind of a mix of 21% of these skills, and content mastery, and student agency. Without further ado, I want to just go to our panel for introductions. I'll introduce myself first, and then we'll go around and get right into our discussion. My name is Rob Reardon. I'm the president of the Heidekai Graduate School of Education, which is an organization that actually devotes a lot of thought and time to looking at the work both of graduate students and K-12 students. I was a classroom teacher for 25 years in Cambridge, Massachusetts, where I came to really value deeply the processes of looking at student work with colleagues. And I've been deeply influenced in that work by several of tonight's panelists. So moving on, Steve Seidel, please introduce yourself. Steve. Your mic is off if you're talking. Steve, let's pick up with Steve a little later. And Ron, how about you? Edric, can you hear us? Yes. Are you guys able to hear me? Edric. OK. How about if you give us a brief introduction? OK, I'll go ahead and get started. Hello, everybody. My name is Edric McAllagham. I'm a seventh grade humanities teacher at High Tech Middle Chula Vista. Chula Vista is located by the US-Mexican border in California. Some things that connect me to looking at student work is just one of those daily practices that we have here at our school that helps us to, one, really get into an understanding of how much our students are learning the content and what we're getting into in our projects. And then also, it helps me fine tune my projects and just the work that I'm working with the students. And Edric is here as a practicing day-to-day teacher and one of his students is here with us, also, Issa. Hi, my name is Issa, and I am a seventh grade student at High Tech Middle Chula Vista. And I will be giving you guys a student view on the topics we are going to be talking about. Great. Who would be next? Joe, how about you? It sounds like we're having a little trouble connecting with Joe. Carissa, are you there? And Ron? He's out, too. Hi, Rob. It looks just like Edric and Issa. We'll be the only ones joining us for right now. We'll work on getting any other panelists back right now. But maybe you could talk to just them. OK, so why don't we just let's begin the conversation, then, amongst the three of us. Edric, how about if you talk to us a little bit about the ways about why look at student work? I mean, what do you gain from looking at student work? And what are some of the ways that you go about it in your classroom? In your classroom and with colleagues? OK, first I'll talk about looking at student work in the classroom. We find that not only just as educators or teachers within some of my colleagues at looking at student work, we find that valuable. But we use that process in the classroom as well with students. So students looking at student work using a critique model that we learned through Ron Berger's work. So we find that extremely valuable for several reasons. One, the student receiving the critique or receiving the information from that critique from the group of students. They're learning how to fine tune their work to make it better. And we also learn that the group of students who are helping out and giving the critique and looking at that particular student's work, whether it's an essay or a particular section of a project, they learn not only how to help and give constructive feedback to one of their peers, but they also learn a little bit something about their own work as well. So when they leave that critique session or looking at that student's work, they can go out of it and kind of look at their work with a new set of lens so that they can fine tune their work. Issa can speak a little bit about that. Yeah, I think I learn a lot from critique myself because it's a great way to see other perspectives about people's views on all sorts of things. It, of course, depends on what you're critiquing. And I think that I also learn a lot about how it's OK for people to kind of help you out on work because I remember in the beginning, I really didn't like critique very much because I kind of found it offensive because people were pointing out my flaws, things like that. But I think I've really learned to be OK with how they're trying to help me and trying to improve. OK, I think that we're back online with some of our other panelists now and we'll ask them to introduce themselves. And then I want to pursue a question related to what you guys have just said, Edric and Issa. Carissa, are you there? Yes, I am. Great. Yeah, so I'll introduce myself. Hi, everyone. I'm Carissa Romero. I'm the associate director of PERTS. And PERTS is a center at Stanford University that develops and evaluates programs that teach adaptive academic mindsets, which we'll be talking about in depth later in the course and that improve student achievement. And today, I'll be bringing my experiences both as a researcher and as an educator to the panel discussion. Particularly, I'd like to credit Joe Boller, who's an expert on math education. We've been working together to design an online professional development for math teachers. And a lot of that experience has greatly shaped my most recent thinking on deeper learning and deeper learning in looking at student work. Wait, thanks, Carissa. And Ron Berger, Ron? Thanks, Rob. I'm sorry I was bumped off for a moment there with technology. I am Ron Berger. I'm chief academic officer at Expeditionary Learning and was a classroom teacher for 25 years. And it's been a, I would say, a passion, almost an obsession of mine for my entire teaching career and career in education to be thinking about collecting beautiful student work and using it to improve teaching and learning. Great, Ron. I'm just, let's see, if Joe, are you with us? I don't think we have Joe back online yet or Steve. So we're just going to continue with our conversation and welcome them in as they appear. Ron, Edric and Issa have just been talking about looking at student work and particularly how useful it can be in terms of understanding better what's one's own work and in terms of looking at student work in order to improve the work. We know, I'm sure, that there are other possible reasons for looking at student work. And I know that you've spent a lot of time thinking about the production of beautiful work, how looking at the work contributes to that process. Could you say a little bit about that question of why you look at student work? What are the different benefits of it? Sure, thanks, Rob. I think there's a lot of reasons for looking at student work. In terms of lessons, for planning lessons, for tuning lessons, for assessing the efficacy of lessons, looking at student work is the best way I know. For understanding student thinking and building a sense of student mindsets and student capacities, looking at student work is the best method I know. But my personal passion around using student work is around creating a vision of what we're aiming for. So I think as teachers, we spend a lot of time hoping students will create great work. But if we haven't sat down with our colleagues and looked and talked about student work together, we don't know if we have a common vision of what we're aiming for. And it's through those discussions that we get a sense of what we hope we'll achieve with students. And for students, I think students spend a great deal of time listening to teachers, describe what they want in words, and sometimes they get a written rubric of what teachers want. But those students have no vision, really, of what a great essay is, what a great geometric proof is, what a great creation is, until they actually look at a beautiful model of it, an example of it, done by a student, and then it creates a vision in your mind and heart that's way beyond the words that can describe it. So the main thing for me for student work is that it creates a model and a discussion point for what we're aiming for. Right, thank you, Ron. And Joe, I think Joe is online with us now. Joe, would you introduce yourself briefly? Sure, my name is Joe McDonald, and I'm a professor from NYU. And I got involved quite a while ago when thinking about ways in which we can read complex texts of various kinds. And one of the complex texts I most like to read is student work. Sometimes it's individual student work that's like Ron, the kind that Ron has been collecting and curating brilliantly display, and sometimes it's other kinds of student work, including whole school district collection. And Joe, you've done quite a bit of work around how to go about looking at student work. Could you say a little bit about that work and what would you say to people who are saying, well, it sounds like a great idea. I want to dive in and do it about how to get going with that and what are some good structures for looking at student works? That's a lot of what I've been doing is developing what I call protocols, other people call them protocols as well. Using the word in a kind of narrow sense can be used for a lot of other things too. But it means finding ways to talk, let's say, about student work, or about problem solving or other things that enable us to segment certain ways of looking and talking. For example, in looking at some of the work that Ron has collected, it's often helpful to separate a description and judgment instead of saying, oh, this is a great piece of artwork. To say first, in a very constrained way, here's what I see there, and what do you see? And other people say, I see this, I see red, I see straight lines, I see, and to forestall people seeing, saying, oh, I see a terrific piece of work that's right on standard or whatever the house they might be inclined to say. So these protocols are really strategies for constraining the way we talk about what we read. I'm reminded, Joe, of what, in King Lear, what Kent says to Lear, see better Lear. And perhaps protocols offer us a way, and we need to be thinking always of ways to see better. And sometimes a quick judgment can get in the way of our really seeing well. Steve, are you with us? We're still waiting for Steve Seidel of Project Zero on Harvard to come on. Carissa, I'd like to go over to you for a second and ask you a little bit about the connection between looking at student work and the academic mindsets that we are trying to foster in students. Yeah, so first, just to give people a brief summary of what we mean by academic mindsets, just the way that students think about school and learning, and they're kind of four main ones. So in students' words, those beliefs are, I feel like I belong here, I think I can do this, this work has value for me, and I believe my intelligence can grow with effort. And I think one thing that's really valuable about looking at student work in the way that we're describing is that it ships the focus of what school is for. And so what Issa said earlier kind of really brought this out for me, really highlighted this sleep. So she said at first it was kind of uncomfortable, it felt like people were just being critical. And I think that a lot of students view school that way. They see school as a place where go to be judged to learn whether you're smart or not. And looking at student work in the way that we're describing shifts that focus to learning, especially the way that Joe is just talking about it, you really wanna describe what students are doing. You don't wanna just say whether this is good or bad. You wanna engage in the process of the work with them. And so I think what's really valuable about student, looking at student work in this way is that it really does bring this focus back to learning goals. This is a question for anyone. I think I'm getting some idea from you, Carissa, about this. But when we look at student work, what are we looking for? Right, so I've been thinking a lot about this based on what Ron sent. And what I noticed about all of those tasks is they were all really relevant. So they all had something to do with going out there in the community, doing something that people are actually gonna use. And that's, again, very connected to academic mindsets. This work has value, it has purpose. And also, they're really open tasks. Students can engage and be more creative when they have these kind of broader tasks. And that's just kind of what I noticed from the perspective of mindsets in it. So thanks for sharing those, Ron. Anybody else on that question? What are we looking for when we're looking at student work? You know, it depends a lot. There's a protocol called what comes up for you. And you take a piece of student work, like some of the work that Ron's brought in, and around the table, you ask people in a kind of game-like way, they can't repeat what somebody has just said. To answer the simple question, what comes up for you when you look at this photograph, when you look at this drawing, when you look at this poem? And then there's much more complex protocols where, for example, a town, a little rural town in Minnesota called Oatana, which is sort of a former mill town that was engaged in a lot of identity shifting, mostly because of immigration and other forces, wanted to know an answer to the question, what is the purpose of an education in Oatana? That what they meant by that is what can we infer from the work that's done by our kids and the work that's asked by their teachers, asked of them, what can we infer about purpose? And so they collected this massive amount of student work, all the work of five third grades and five seventh grades and five 12th grades on a given day, this is several years ago, and then they had a whole bunch of people from the community and lots of teachers sit down and for a couple of hours, they had a seminar on what they might infer from this gigantic collection of work that they'd all spent a couple of hours browsing. Well, so it was like a slice through the day, just from one day, and then what can we say, looking at this work as evidence, what can we say about our values? That's the name of the protocol, it's called the slice protocol. Uh-huh, okay. I wonder, is Steve with it? No, Steve is not back with this. I do want to get back to, I want to continue on with the discussion about the purpose of looking at student work and I'm thinking about an experience I had with it myself. It can be one thing to look at the work of other students and then another thing to bring to a group the work of your own students. And I brought the work of one of my graduate students to a tuning, a looking at student work session at one point and I was kind of, maybe we had the wrong protocol, Joe, but I was interested in what can we learn about this student? But I also brought along with the work, the rubric that we used, I used to try to communicate some of the expectations to students and the whole conversation turned out to be whether the work was aligned with the rubric. And I learned a whole lot in that session about rubric construction and how I might better communicate expectations to that student, but I learned very little about others' perceptions of what that student might be working on or what they saw in the work. In other words, the rubric offered a filter for the work that worked against my purposes and yet worked to my benefit. Yeah, that's interesting. As the method for discussing what we read when we read student work changes, as you say, the filter changes, the purpose implicitly changes and the benefit shifts too, but there's a great story because it demonstrates that there are other benefits besides sometimes the benefit that you set out to gain. There's really no better way of thinking about your teaching than having people talk about your students' work. I wonder, Edric and Issa, getting back to you and to your classrooms, can you tell us about a time when you kind of really learn something by looking at student work? I mean, a concrete time when you learn something. Looking at student work either with peers or with colleagues. Issa, you want to take that first? Sorry, Edric? Issa, go ahead. So, I don't really have a very specific time, but I know that every time I look at someone else's work, I kind of learn about their way of thinking and it's throughout the many times that I've done this, it has really opened my eyes to looking at things with a different perspective and looking at things in a completely different way. And it's just like, I think it's really cool how that works. Cool, and it sounds to me like you might be what you're talking about also, the same thing from different perspectives and so forth that we're talking a little bit about building a culture of conversation about the work and the way that Ron was referring to earlier, that in having these conversations, we begin to discover what our shared values and aims are. Definitely. Edric, anything on that? Well, I think that the looking at student work, I think that it really reflects on others because it helps people understand each other more and it could simply be a way where like, I remember, I forgot if it was, I think it was last year somebody wrote a piece that was very, a writing piece that was very personal to them and after they shared it with the class, it really helped us connect with him more and we understood each other more and it helped us build kind of a stronger relationship with each other and it was really great. Right. Edric, you wanna chime in on that question? Sure, and I'll take it using the idea of like looking at student work through multiple perspectives. One of the things that, anytime I'm looking at student work, in my past practice, especially when I was working with a large population of ELs or English learners, I learned a lot when I, about a student's work, about my own practices and about how well or how well I'm scaffolding a student or maybe I might be lacking in that. So what I try to do is I try to, when we take some time to take a look at student work, I try to focus it not only just looking at student work at the end product, but also looking at student work throughout the process, because it allows me to take a look at the student work where they at, are they understanding the material that we're covering? Are they processing that material? Are they turning it into their own learning? And then it helps me guide whether or not I should take a different direction with that particular assignment or project, and helps me, one, understand what they know and what they're able to articulate, and then two, whether or not I'm doing a good job in the project design or lesson design, and it allows me to make the adjustments that's needed throughout that process. Great, so that really informs your own practice as you're doing that. Definitely. Carissa, in math, what does student, looking at student work, look like in math? Yeah, so the kind of thinking that we wanna see when we're looking at student work in math is that students seem to feel like the task is really open, that they're able to kind of make it their own and look at the problem from different angles. So you would imagine that if you give an open task, you wouldn't expect everyone to come to the answer in the exact same way. So it's a really different approach than kind of the typical drill and kill learning. So the kinds of tasks that we're suggesting and that we're looking at student work to make sure we're achieving comes from the background of complex instruction. So really opening up a task to allow students to be creative and we wanna look to see that students took these different approaches and we wanna see the approaches that they took. So really getting insight into the way they solved the problem. So when we're looking at student work in math, we're looking for thought processes, we're looking for where the thinking is going and evidence of the thinking. And then also that rebounds back to instructional design. It tells us a little bit about how our design is or is not fostering the virgin solutions to problems in math. Does that catch it a little bit? Right, if you have a typical task, you're gonna see everyone solve it in the same way because they're just kind of learning the rules. And if everyone's solving the problem in the same way, they might learn the rules but they're not really learning the concepts. And so that's not really useful information about whether the students understand what you're teaching. Ron, I think that from actually the article, the interview that you did for us and unboxed a few years ago, I became aware in a way that I hadn't been before about how your background in art and architecture had really influenced your thinking about student work and about how to go about looking at it and working at it. Could you say a little bit about that, about the roots of looking at student work for you? Sure, first Rob, I'd wanna say that I just wanna rename some things I thought were very important. I just got said and that's both you and Joe pointed out that it's first important to determine what the purpose is for looking at student work and then to choose the right protocol of if the purpose is to tune your lesson or to assess the quality of that, then that's gonna be one protocol and if you're using student work to inspire students, that's an entirely different protocol. And I think Carissa was pointing out, pointing a really great example of if you really wanna know if the problems you're giving kids are the right problems, you really need to look at the result of those problems in their work. My particular passion is around looking at student work to inspire the quality of what students do, which is a different thing. It's using student work with students, although I think all those other uses are equally important. And my issue is that I think that the quality of the work that students do in school really matters. And I know that seems like a silly thing to say because of course it should, but I don't think students get that message. For example, I was in public school for 13 years and there was not a single piece of work I did that any other student looked at or commented on or cared about. And in contrast, Issa was talking about the fact that in her classroom at High Tech High, it's a regular practice where students are analyzing and critiquing and caring about the quality of what they do. That sends a very different message to students that the quality of what they do matters. And so for me, my background in both carpentry and architecture and arts comes from this ethic of craftsmanship, that what you do, you should do well. And I think that having a culture in your classroom and school where everyone is discussing your work and critiquing your work and pushing the quality of your work sends a very different message about the value of doing things well. I think that, wait, are you still talking? Oh, okay. I think that it's really important to do like exactly what you're saying before is kind of compliment people on their work because what Mr. Vak does sometimes is he'll take the work that he thought was really good and he'll put it up on the board and have the whole class kind of read it and look at the work that we've been doing. And it really inspires people to do their hardest work so then they can get recognized by the class and they can kind of feel special like that. And I just think that that's a really great way to help people become inspired to do their hardest job. Great. Thanks, Sia. We have an interesting question from Trevor in the audience and it has to do with teachers bringing in work to share and it goes like this. Teachers are sometimes concerned about showing work or exemplars too early for fear of squashing creativity. What are your thoughts about the timing of sharing and reviewing work examples? And I think he's particularly talking about exemplars. Ron, could you go at that for a second? I know you've thought a lot about the when, where and how of models, of sharing models. Sure. I'll try to be quick. I was a little long last time. I'll say briefly that there is not one time to introduce exemplars. It can be at the beginning of the lesson, it can be partway through after kids have had a stab at it themselves, it could be close to the end when you're trying to push quality. But I am actually an outlier on this question. I think it's crazy to deny kids the inspiration of looking at great work in the worry about squashing their creativity. To me, that's like saying, we want kids to create beautiful music but we're never gonna let them listen to beautiful music first because we're worried that might influence them. If they've never listened to beautiful music in that genre, how will they know what they're aiming for? So I'm not so worried about squashing creativity, I'm much more worried about inspiring quality. Well, very interesting. Thanks, John. Mia from the audience asks how critique can be a lever for moving school and classroom culture in a more positive and productive direction. Karissa, I wonder if you had some response to that. Yeah, so I think it's interesting that there were two questions about critiques here because I think there's kind of in some classrooms that don't practice deeper learning regularly. There's a culture of thinking mistakes as negative. So students tend to view mistakes as a sign that they don't have the skills and teachers tend to view mistakes kind of with sympathy or even worse with a lot of criticism. And really what you want teachers and students to be doing is thinking that mistakes are good. Critiques are valuable. They show us what we have learned and how much we have to learn. They help us know how we're progressing toward our learning goals. And so I think looking at student work, especially during the process, while you're still working on something, can help move a classroom culture towards the direction that school is for learning. We want to be making mistakes. We want to be critiquing each other and that's why we're looking at work together all the time. I think it's important also to, in looking at student work in that way, and I'm just referring back to the notion of a positive direction. I think when we're looking at student work in that critique mode, to point out what's working. And on the presumption that students can learn, we can learn more from finding out what it is that we're doing that's working and that's effective than we can from what we're not doing. In other words, if someone tells me do this, what you're doing, I learn from that. If someone tells me don't do this, then I still don't know what's working in that. Ron, did I sense that you wanted to add something? Oh, no, I actually just loved Carissa's response to that because even though I am obsessed with beautiful, high-quality student work, I also believe that we should collect the early drafts of work so that we can talk about the fact that all great work starts out with struggles and problems. And Edric pointed that out that he loves looking at that kind of work with students because students can see that people, we all make mistakes in process. Can I say, Rob, can I say that... Go ahead, Joe. Yeah, I think the kind of mini culture that the person who's brought the work and the community of people who are looking at the work, the mini culture that they introduce is really crucial here because I think both are important. Critique and also what Issa said before, what she named as complement are equally important. We want to know what's working and we also want to know and need to know what's not working so that we can improve that, particularly when work is still in progress. And there are a couple of protocols that have evolved over the years that deal with this in a different way. One, it's probably the first time that we use the word protocol for this kind of effort is something called the tuning protocol. People have used the word already in the conversation about tuning. And that was invented because a group of people were coming together to look at work that was still underway. And the corporation that had funded that work was going to be present. And I happened to be the facilitator and was really worried that it was gonna be too heavy on the critique on the part of the funder and that it was gonna be too heavy on the defense on the part of the people who were bringing the work. And so we just invented this set of constraints that you had to give what we called warm feedback, which was appreciative feedback. And you had to give cool feedback, which was critical feedback, but you could not mix the two. You had to give each its due and that the group was collectively responsible for keeping both in balance. So that's one way that people have classically done. I know as a writer, I love sitting down with other people and asking them to give me cool and to give me warm feedback and not to mix the two so I can take each into account and not think that one's sugar coating for the other. Another protocol I just mentioned quickly is the charrette protocol, which comes out of architecture. Again, work in progress and people are brought together to look at the work in progress and hear the people who've put it together, describe it and so on. And the culture of the meeting is that we all own this work. It is, this is our work. And sometimes people who do this protocol actually ask people to use that first person pronouns and talking about it. Well, I think what we have to do here is so there's critique, but it's implicitly self-critique, artificially self-critique in order to create this culture of equality with respect to critique and compliment. So just quickly related to highlighting the good and the bad, something that I learned from Joe Bowler that teachers sometimes do in their classroom is they use peer assessments and what the peers do when they look at each other's work is they give each other one or two stars and one or two wishes. And so the stars are meant to represent what students have already learned and the wishes are meant to represent what the students learning goals should be. Cool. On that point, I just want to mention one thing about, just kind of connecting what we're talking about here in the academic growth mindset. One thing that's very important that you'll see in some of our classrooms here is the idea of using exemplars, but using exemplars from struggling students. So maybe let's say it's of a narrative piece and they have a great opening line or a hook that we also maybe put that work up, celebrate that work with the whole class when we're doing a whole class critique so that that student who's struggling can feel like, oh, I did something great. I learned how to add that one component in my writing. And then at the same time, after we separated and celebrated that student, we go out and take the time to say, okay, now let's look at the next few sentences. What's one thing that you would change if this was your writing? And then we help give that student critique. I think using that practice, especially with our struggling students, helps to build their confidence level. And so they feel like, okay, look, they celebrated my work and then they gave me help as a community. And so everybody understands that we're a community of learners and we build each other up and then building that just that motivation factor with some of our students that sometimes come into our classrooms that don't have that from the get-go. Yeah, Adryk, I love that point. I think one thing that I didn't say that's really awesome about mistakes is they don't just help you know what you need to learn, but people actually make really great leaps in their knowledge when they make important conceptual mistakes because it highlights a major misunderstanding. And so in kind of American classrooms, we often tend to ignore those important conceptual mistakes. But I heard this really interesting piece on NPR where a teacher in an East Asian country actually had a boy go to the board because he got the problem wrong and all of the other students in the class were trying to help him understand this conceptual gap. And by the end of the class, he actually took him the whole class. But by the end, everyone was clapping and he felt really good. And you would imagine that in an American classroom, you'd feel really uncomfortable. They're making this boy that got the answer from the front of the classroom. But it would be great kind of, I would like to see more American classrooms like that where we're celebrating mistakes and recognizing the value of them. I love that phrase of celebrating mistakes and celebrating questions, Carissa, fabulous. There is a very interesting question coming in from our audience about technology and looking at student work and how, what are some of the ways that technology can kind of foster and facilitate our looking at student work together? Would anyone like to tackle that question? Actually, I have one thing to add to what Ms. Romero was saying because I think that it's really great that at HTMCV, we have kind of a culture like that where like, I remember in sixth grade, our teacher would say fail to prevail because everybody has some point where they're not doing so well or some point where they don't understand everything. But in order to really get a full, in order to learn, you have to have a point to start learning from. Issa, how about Google Docs? Do you guys use Google Docs to look at each other's work at all? Yes, definitely. That's a big part, another big part, about high tech middle is that we actually just started this thing called reading response journals or started them up again. And one thing we're gonna be doing because they're a weekly journal that we write about whatever book that we're reading at the moment. And we are going to be discussing with other people that are shared on the document, discussing about what we thought about different things about the book that we read or the responses that we put. Right. Rob, that used to be harder to look at student work, used to take more work to get the work on the table, so to speak. Now everybody's got a video camera in their cell phone practically and we can take pictures of dynamic things and we can share them instantly around the world and have groups in remote locations offering us feedback and going deeper in examining our collective work. So it's just, the technology is a great, just affords much benefit for this kind of stuff we're talking about. Now that you mentioned it, yeah. I mean, we're in one of our projects involving some educators around the world. We've had synchronous conversations going on with people in India, Australia, San Diego and Boston all at the same time. I mean, pretty late in the evening, one place and early in the morning, the other, but looking at student work together and adult work, project designs. It's unbelievable, unimaginable, just a few years ago. I remember, Joe, we were with, was back in the 70s. We were both working a little alternative schools in Cambridge and Watertown, Massachusetts. I mean, who would have thought that we would have come this way in the world? Anyway, so I don't think, go ahead. Who's next? I'll jump in for a second on the technology. We've seen some great examples from the community already with the deal, student work hashtag, just tremendous student work that they're sharing through websites and digital portfolios. And I think the technology provides such a phenomenal opportunity for students to share work with an authentic audience. And one of the projects I work with is a national writing project site called Youth Voices. And it is all about students sharing work and it's particularly focused on student writing that's passion-driven. So the students are coming up with their own topics and they're giving peer feedback. And I think that's a really nice place to look at how technology is facilitating some really phenomenal sharing of student work. Wait, by the way, here I am monitoring, but anybody jump in at any time that you want to. And I want to share something. Just another piece. Oh, I want to share two things. One is about this notion of separating warm and cool feedback. And bringing them together can be very counterproductive. And the worst way to bring them together is to separate the two with the word but. I really like that part you wrote about your grandmother's boot. But what is missing, the only thing that's heard is what comes after the but on that. So the one of the things we're trying to do is if we're talking about our work together is get the but out, so you speak, so you speak. Then the other thing I wanted to say quickly, if I can say it quickly, is that looking at student work for us can also help us to look at bigger goals. It can help us run, as you were saying, to what our shared aims are for the student work. But it can also help us look at our institutional aims. At High Tech High, for example, we're integrated in our classrooms of diverse and untracked. So one big question for us in looking at a piece of student work is where is the evidence of both access and challenge in this work? And I remember one particularly powerful session where a teacher brought in three pieces of student work, of student writing, one from an English learner, one from a student who was kind of moderately proficient as a writer, and another from an advanced writer in the classroom. The assignment was to write a story that was a mystery and that used code. This was an integrated math and humanities project that used code in the solution of the mystery. What emerged from it was that the English learner wrote a piece that was gifted at sharing dialogue. It could have been turned into a play right away. The advanced learner wrote stuff that was impeccable in its mechanics and so forth, but didn't have the same kind of life in it. And yet, it was everything that we would have wanted from a well-tuned kind of piece. But the point being that we can value in looking at our work, particularly around questions like access and challenge, we can come to value the different entry points that students have to the work and the different ways that they can shine. And I love exercises that force a group of teachers to take a bunch of work and particularly work that seems at first inferior in some sense, didn't meet the standard, and say, OK, now we're going to, as a group, look for the strengths in this work. Let's go around the table. Everybody's got to find a strength. And the next person has to find another strength and using this sort of game-like way of doing it and forcing people to go deeper and deeper. And because, of course, those strengths are not false compliments. They're strengths in which somebody who may have challenges in, let's say, writing or drawing or whatever it is, might build on. So it's an interesting point to include both compliments with the critiques. And obviously, there's a lot of value in that. But I think another really interesting way to have students respond to critiques less defensively in a way that helps them more is by communicating higher standards. So there's a really interesting study where researchers asked students to write an essay. And this method that they use is particularly effective for African-American students who are often more sensitive to critiques on their work, because they know that there might be bias in society. And so they're kind of looking out for that sometimes. And so what the researchers did was they put a post-it note on the essays. So they gave the same kinds of critiques to all of the students. But for some of the students, they saw this little post-it that said, I'm giving you this feedback. Or I'm giving you these comments because I want you to have feedback on your work. So very uninformative. And the other post-it note said, I'm giving you this feedback because I have very high standards for you. And I know that you can meet those standards. So explaining that there are high standards and assuring students that they can meet those standards. And those students were actually more likely to revise their essays. And their essays were better than the students who were just told that they were getting comments for the sake of having feedback. That's interesting. And that has to do with fostering mindset through critique. And it's the message of the teacher that says, this is important work. You can do it. I know you can do it. And my feedback is in the interest of us getting there. That's right. So it's very interesting, Carissa. OK, we're running short of time here. And I want to both honor the time and give all of our panelists a chance to have one final word or one final comment about the conversation, something that you've been wanting to say, or something that you want to say by wrapping up. We're just going to go around quickly. Let's say a 30-second to one minute comment to wrap up. And you may pass if you wish to. Who will start? I'll start. I just have something really brief to say. One thing that I really like about looking at student work and the way that we're describing with a deeper learning lens is that it's really valuable for me. So as an educator, I feel like I learn a lot more when I'm looking at student work with a deeper learning frame. So it's not just useful for the students, but it's useful, I think, for us as educators. I talked a bit about protocols, obviously. And I'd like to recommend some sources where people could find out more about that stuff. One is my own book. I'll do an Oprah thing for a second. It's called The Power of Protocols. But beyond that, there's a terrific collection of protocols also on the website of the School Reform Initiative. So schoolreforminitiative.org. And you'll find lots of them, and in particular, ones for looking at student work. All the ones that I named tonight. Great. I'm just going to jump in really quick about the importance of using protocols. There's a negative saying that goes, when all is said and done, more is said than done. What I love about using protocols is that in the time frame that you use it, whether it's a 30-minute protocol or an hour, there's a lot of work that gets done during that time. Whether you're using the protocols with students or you're using the protocols with colleagues in trying to push that particular assignment. So it's just very important that to find a protocol that's going to work for you, and then use it with your students or with your colleagues. I just want to say that I'm so happy that I got to be here. It was really nice of you guys to invite me to do this. And it was a really great experience for me. Rahman, you guys were. Yeah, Issa, that was wonderful. Thank you. I'd like to say that I hope that people listening and watching this don't view this as a choice. But I hope that they listen to all the different uses of student work that people were giving and the different protocols that Joe mentioned and that Carissa mentioned and use all of them. Because I think student work is important in all of those settings. And I think every time we use student work with educators and with students, it sends the message that the quality of what kids do in school matters. And that's a message that's transformational for kids, to elevate their work as something that's worthy of being looked at closely. I will also offer a resource. EL has an open source center for student work. It's available right at the EL website. And what it is is the content of the suitcases I've been dragging around the country for the past 20 years scanned and put online so that people can download for free examples of exemplary student work from K through 12 students. And so if that's ever of use, I would welcome people to go on and find things and download whatever is helpful for you. And I call it the museum of student work. And the URL for that and the reference to Joe's book, Power of Protocols and other references is up on the URL for this course. I would say by way of a final word that one of the most powerful experiences in looking at student work for me was as a public school teacher in Cambridge, Mass, I took a portfolio of a senior of mine to share with a group of people over at Harvard. A thing run by Steve Seidel. And a kindergarten teacher from the district had brought a portfolio from one of her kids. And I looked at her work. And she looked at the work of my student. And we met in the center of the room. And I said, I see my student in the work of your student. And she said, I see my student in the work of your student. And there were these through lines in the work between a kindergartner and a senior. We saw the same students kind of 12 years apart. It was just a powerful experience for me to see that there was something universal about what we were doing. And I want to make the point that this happened in a public school setting. And this kind of work that we're talking about can happen anywhere. All of us have come from different contexts and so forth. But this work can go on anywhere. And that's it. I'm going to turn it over to you, Karen, to wrap up. After I say thank you to all the panelists, Issa, Ron, Joe, Edric, Carissa, who I missed. Edric, I got everybody, I think. Thank you all for being a part of this conversation. Karen. Thank you so much, Rob. That was such a lovely story. And thank you to all our panelists and everyone in the DL MOOC community. Lots of tweets and conversations going on out there. Keep the conversations going. Before we sign off, I want to just quickly do a couple shout-outs to some things that have been going on in the community this week. We have over 25 small groups that have formed. These include a group of Australians who are doing all kinds of exciting things. We have groups looking at high school science and engineering, professional development for teachers, project-based learning, equity, a new group for looking at global arts education. And there's a new group that will be going up in the next day or two around assessment and deeper learning. We also have a lot of people blogging about their DL MOOC experiences and deeper learning in general. After our last session, Kale Burke, who's a secondary principal in Canada, was prompted to write about his thoughts about what we ask students to produce. And I thought I'd just, since that relates so well to what we're talking about, I'll just read a short excerpt from that. He says, relative to the attributes that we have determined our students need to be successful at and beyond our school, are the artifacts we are tasking our students to produce, allowing and requiring our students to demonstrate these attributes? Are they allowing and requiring our students to demonstrate deeper learning? So there's some food for thought. This is posted on the G-plus community as well as many other blog posts. Go engage in conversations around that. We also have a deeper learning story bank on the DL MOOC.NET website and have had several new contributions to that, including a story about the linkages between curiosity and grit. And that came from Monica Reddy at Napa High School. And we invite you all to add your own stories of deeper learning to that story bank so we have an even richer resource. There was a lot of discussion about protocols on this session. We are going to have a lens into the classroom live session on Thursday at the same time, 4 p.m. Los Angeles time. And this will feature a teacher, Brandon Cohen, who is going to present a classroom dilemma that a group is going to use a tuning protocol to examine in more depth. And we hope that this will be a useful process for our community to observe and comment on, and also that some of you might go on to do protocols like this yourself, either face to face or in online environment. And we'll be doing these lens into the classroom sessions each Thursday night for the remainder of DL MOOC. We're also always looking for teachers as well as students from our community to join in those discussions. So if you might be interested in joining the Hangout for one of our lens into the classroom sessions, shoot us an email or send us a tweet or write us on G+. And then just lastly, there is so much good stuff going on in DL MOOC. It really can feel overwhelming. And I think even those of us on the team who are living this, it's hard to keep up with everything. And I just want to remind everybody to pick and choose what makes sense to you. Do the activities that work for you. Feel free to drop out and come back. We've really designed this MOOC to be flexible. And we want you to take part in the way that works best for you. And that's all I have for tonight. Thank you all so much. And we will see you online.