 Felly ydych chi'n gwybod i adeilad yma o'r ddweud o Doug Fisher. A yn ymweld y profesor fwy Lleinu Qwyddo, rwy'n dweud y Lleinu Qwyddo i ddeud o'r ddeudio'n ymddangos, ac mae'r rhaid i'r cyflwyno'r rhaid o rhaid o'r cyfle ni oherwydd, ac mae'n sgwrn iddyn nhw. Fe ydych chi'n gweld i'r ddweud i'r ddeudio'r ddweud i ddweud, ac rydyn ni'n ddweud i ddweud i ddweud i ddweud i'r ddweud. Why I'm gonna be talking to you today about much of my personal experience waxing a mouc in my on-campus courses. Not sure how I can control the slide deck there, who would I be interacting with to do that. I can just forward it, so if you can forward from the title slide to, are you seeing the slides up there? Rhydd i'n gwerthu. Rhyw arwain amser? Yn gwerthu? Rhyw gwaith? Rhyw i'n gwerthu. Rhyw gwaith o'r ddau i ddau i sgrinio gyda'r iawn i ddau siaradwyr? Esperau sy'n rwy'n rwy'n gwaith. Metodd roedd yr awd fel y bydd honno'n gweithio, mae erioed i'r cwrn nhw. Rwy'n gweithio'r cwrn nhw? Rydw i'n credu i chi felly rydyn ni'n ddatblyg â chyfeiddiol? Rydw i. Rydw i. Fawr, roi, rydyn ni'n credu'r cwrn nhw? Rydyn ni'n credu'r chyfeiddiol. Rydw i'n credu? O�로. Sio maen nhw'n dod o'r ffordd oedd y maen nhw'n ffordd a hynny oedd unrhywethaf hwnnw ar lawer y cyfaint a wynau ychydig o'r ffordd yma. Rwy'n sy'n dweud hynny ym mwyn yn gwrando. Mae'r hyn sy'n ziw i'r A gydag y gallwn y fflaenwyr ymlaenwyr ymlaenwyr yn cyflaenwyr ymlaenwyr. Mae'r robot yma ar y fflaenwyr ymlaenwyr. Ymlaenwyr, yw'r fflaenwyr ymlaenwyr? Mae'r ddweud yw hwn yn ei ddweud? Yn ymlaenwyr? Yn ymlaenwyr? Yn ymlaenwyr. Yn ymlaenwyr? Yn ymlaenwyr. Ystafecor amsiddoriad Ysgyrff er gwaith Mae service travelling Mae'r SIM fan cofyd Jenf or cuestión Mae'r SIM fan cofo'r SIM fan cofyd Rep vowels Rep vowels Rep vowels Rep letters Rep letters Rep letters Rep letters Rep letters Rep letters Rep statements Rep letters Rep letters Rep statements Rep letters ac cyfwyriau sydd wedi sicr ar rahan o blwyd o'r cyfacol yn y cofort yn fawr hynny, a'r dymurau yn cael ei ddarparu oedon nhw a chyfnodd aelod ar gyfer yr Argyn — ond rydyn ni'n grywod yw'r dwylo gyffredeol o'r math angen ond mae'n gyfwyrdd yn y cwmp, ac yn dwylo'r math angen ond, sydd o'n ef yw'r gwmpau, dwi'n fyddwch i'n goflau'u am Cyfwyrdd Cymru, when I taught my undergraduate database course and my graduate machine learning course. So, you are pretty, I'm sure, familiar with the idea of flipping the class. So, here is a week-two schedule for an example of what I would do. I would have students watch the videos from Jennifer Wydham. this is in database. She was the Stanford faculty member teaching the MOOC. This is after the MOOC was done. This was actually the first time I did this. This was before Coursera was founded. And so there were no terms of service to worry about. The videos were just up and online. I had students watch those. I adopted Jennifer Wydams' textbook in database to make for a smoother experience for the students because they were both reading and watching her videos. You can see in the middle that I would also quiz students on the material. So they would watch the videos the week before. They would walk into class. It was a Tuesday, Thursday class. They would take a quiz on the videos and the reading that they were supposed to do. And then we would do some kind of in-class exercise. Here's an example of Jennifer Wydams' screenshot of Jennifer Wydams' video in database. This is a good nuts and bolts lecture. I think I will always want to do the inspirational lectures in person. But for nuts and bolts, I'm a big believer in video. So you can see Jennifer Wydams' there in the bottom right. She's interacting with the screen. She's playing PowerPoint. Students would watch these kinds of videos for the most part of the nuts and bolts variety. And then they would walk into class following a quiz and they would do some other exercise. In this case, this other exercise was to watch a video by Hans Rosling on visualizing data. You should be seeing Hans here. The year is 1948 and he's talking about a visualization of economics and health in 200 countries over 200 years. Just do a sanity check. Are we all there? So they would watch a video like this, a very short video in class, and then they would spend the class reverse engineering a database that would be sufficient to support Hans Rosling's visualization here. And it was a much more interactive experience than just listening to me lecture, giving the same kinds of nuts and bolts that Jennifer Wydam had already done. Now you might imagine if you're a student and you're watching Hans Rosling lecture and then you're engaged with your colleagues and with the professor in small groups working on a problem engaged in active learning that things might improve. And things did improve. When I first started doing this, I was worried quite frankly what other people would think of my using other professors lectures. I was worried about what Vanderbilt would think. I was worried about what my students would think, what my colleagues would think. But it's been a great success and everyone seems to have liked what I did, which is one reason I'm director of the Vanderbilt Institute for Digital Learning right now. But these are class ratings and you'll notice that I guess the surprise for me was having my students watch other professors lectures. My instructor rating has increased across virtually all classes. In my field of special interest, artificial intelligence, it's held steady. So these are ratings that are taken at the end of the semester and they're on five point scales and on a five point scale. And I've done this now in three classes. I've highlighted in green, the green box, the database progression at the bottom there of that green box. You can see in my pre-moof use of materials, my ratings were three is average, four is very good, five is excellent. They were decent ratings, but they were nothing to write home about. But those ratings have increased. The means have increased as I've adopted MOOC use and as I have adjusted to it. So that now at the top in spring 2013, this semester ago, things are quite good. I think if you look at the bottom observation, just that summary observation in the red box, again, the experience was that the instructor rating was typically going up or at least holding steady. And as importantly, I think the standard deviations are going down. Now there are some confounds to all this. I was really excited about the course. I don't think quite frankly that the fact that the students were watching another professor's lectures, even though they were very good, was primarily responsible for this increase. Although I think it was responsible for some of it because they could go back, they could watch the lectures over and over again. They were very good lectures. But I think in large part this is due to the fact that they're engaged in active learning inside the classroom. The confounds here are probably several. I was a lot more excited to go in and work with a class in smaller groups than I was to lecture from PowerPoint. Quite frankly, lecturing from PowerPoint has become very old. So that's an example of a confound. I was just a lot more enthusiastic about some of these classes than I had been before. But you know, as a good academic, I think if you're going to be using other people's content, you start producing your own. And I started producing my own for my AI courses first, then for my database courses. And I would simply do voiceover PowerPoint. What I noticed after putting up, producing some of this content and putting it up online, because I had been impressed by the very good nuts and bolts lectures of some of my colleagues. If you don't know what a nuts and bolts lecture is, it's describing an algorithm or a mathematical proof in some detail. It's not a kind of inspirational lecture or a big picture lecture. It's sort of getting into the trench and working on something technical. And again, I think these are much better online. And I started doing my own nuts and bolts lecture. And I was happy to find that I could do outstanding nuts and bolts lectures too, so long as I was willing to put the time into them. But what happened after I did this is other students started finding my lectures online and started using it. And this was quite a revelation. When this first started happening, you can see the student comments here. They were coming to my YouTube channel to look at my AI videos. These were students in a UC Berkeley MOOC in artificial intelligence. So MOOC students coming to my YouTube station to remediate their understanding of some of the content that had been described in that MOOC. So this was good. I went back to the discussion board and verified this. But this is sort of an example of updated statistics on my YouTube channel. And at the top left there, you can see, you can get a glimpse of the pattern of performance over the number of views. 37,000 views is small by MOOC numbers, but it's a lot more views than I've ever gotten before. And, you know, if you look elsewhere, you look on the far left or far right rather, see the subscribers 121. This is the first time I've ever had followers of any kind in my life. So this is kind of a kick as well. But, you know, I think, and this is going to be something that I come back to is you look at a pattern like this and it impressed upon me to think that much of our educational data mining work so far as MOOCs are concerned is looking at patterns of student behavior within the MOOC itself. But really I think where we will go, I hope where we will go, is understanding that a MOOC is, I liken it to the sun and the solar system. And there's a much larger ad hoc community that grows up around a MOOC. And really what we want to be looking at eventually is that larger community and mining the data from that larger community. So my hypothesis about my YouTube channel performance is that these high points are synced with MOOC use. And to some extent I verified that. Now, something that I've done since this experience of students using my MOOC is at X has just opened away for closed, what they call closed instances of courses. And I'm now using that 188 MOOC, a material from that 188 MOOC from UC Berkeley as part of my current artificial intelligence course. And I don't show you a screenshot here, but I can go in and they see this closed instance with the UC Berkeley lecture material and quiz material and test material. And I can go in and I have gone in and swapped out some of their material and substituted some of mine. I've augmented the material, the lecture material from UC Berkeley with some of my own. I've augmented it with lectures from University of British Columbia. So I've gone in and I've customized my course using UC Berkeley content, Vanderbilt content, University of British Columbia content and probably down the road other content as well. This is something I'll highlight in a second that something that I think is really exciting to me is that this MOOC activity has made me feel for the first time as a teacher that I am a member of a community. I mean as a scholar I've been a member of a research community for a very long time, but I have never felt like I was a member of a community as a teacher. When I walked into the class it was my class and quite frankly as you get older the idea that you are now a member of a teaching community is a lot more exciting than the whole lone wolf idea that you walk into your class and it's your class. That's just an old concept for me now, something that I don't enjoy nearly as much as being a part of a larger scholarly like community of teachers. Just to show you that we can do all the kinds of content, it's not all video. This is a wiki book that I started that's community driven. The intent here is to use, well we're running short on time so I won't explain it, but community driven content in the form of wiki books, open source materials, the AI course uses a text which is freely available online as well that I can adopt and use in my lectures because of the licensing. The next time that I teach a machine learning course online I really expect that I'm going to be customizing it even to a greater extent than I did before. These are the two main themes I want to stress right now is one of community which I've already talked about but also customization. I overheard the last speaker talking about University of Edinburgh's AI planning course but if I ever teach a machine learning course again I think I will customize it by drawing on, if I can, material from different sources. Not just the machine learning course from Stanford on the Coursera platform that I've used before but another machine learning MOOC that may be offered from Washington and some of the learning content that's in the AI planning courses, natural language courses as well as some of my own content. So I'm now getting pretty comfortable with the idea that I can customize courses, I can draw from different sources given my edX experience and students aren't freaked out about this, they like it. They like looking at different professors for different content. That's not something I had thought that they might like but they seem to. So educational data mining, just some questions suggested by this experience. I think we want to look beyond individual MOOCs and mine data from what I call the MOOC solar system. How do MOOCs students interact with YouTube, Wikipedia, other sources? There's going to be a challenge there, how do we get that data? Who's going to give us the data? Is Google going to hand us the data for this kind of thing? How do we link users in Coursera course or edX course or Udacity course with users in some of these other open source environments? So some real challenges there about how do you get that data, how do you link it up and of course how do you mine it. I think, you know, one thing that I've been tempted to do is start creating content intentionally. So not just putting, when I put up material, when I produce material now, I tend to be busy right now but when I do I intend to be pretty strategic about it. And I'm going to put up content that fills gaps in MOOCs. I as an expert in a MOOC know where students are going to have a problem with content in a MOOC and I can fill that content in. I can bridge those points of difficulty in advance with content of my own. And I think we'll see other examples of that. I want to have my graduate students do this. I've had graduate students in my advanced courses produce and upload educational content as a requirement of the graduate course and I think I will do that more and I will have them look to MOOCs to figure out exactly what content will be most useful given the current resources on the web. If we go below to benefits of the on MOOCs, you know, I think there are some real, so flipping the classrooms an obvious one, I think there's real course design issues. The previous speaker mentioned that online learning education has been around for a long time and that's true. But how do you take as an on-campus instructor, I think this is a new question, and incorporate a MOOC in full or in part into an existing on-campus course in a way that's ideal, in some optimal way, in a way that maintains the faculty engagement which is the second bullet there and which is not a no-brainer at all. I imagine faculty members incorporating MOOCs and then sort of becoming very disengaged, unengaged with their courses. These are things we're going to want to better understand. And then with these local learning communities and just the learning material, how do students interact as a local learning community in their own class with the global learning community that's part of the MOOC if in fact the MOOC in the on-campus course is synced? I'm going to ask Fiona for a time check. How am I looking? It's okay. We can spare you a few more minutes. How much longer do you have to go? We will ask you some questions. I can go a very long time. So I will hit a couple more points for you and then I will close. Brilliant. Okay. So this is, you know, even in October 2012 you could complete, essentially a computer science degree online and for free, given the MOOCs that were up from different places. And I think that you will with all these resources, not just MOOCs, resources on YouTube, Khan Academy, elsewhere. You can, well we know students are already customizing curricula if you will. They're looking at different ways of working their way through a sequence of courses. So they might take the intro course from edX in programming. They might take a second course in data structures from Udacity and then move on to some Coursera courses from different universities. And so they create their own sequence and we can certainly do this. But once students start creating their own sequences and start embedding these within larger social networks, we now open the possibility of crowdsourcing and that some of these trajectories through existing online courses and other remedial material are endorsed or not endorsed by the community and then become consensus favorites. And so, you know, I think crowdsourcing curricula is going to be something that's going to emerge in the near future. Vanderbilt and the University of Maryland have just joined forces to offer not the first MOOC sequence, not the first MOOC but the first MOOC sequence. So these are back-to-back MOOCs with a kind of loose intended to have a loose coupling and so that students will get something added by doing the sequence above and beyond simply taking two independent MOOCs. Again, I think you'll see university partnerships that create these curricular constructs that then are vetted by the community. I'm going to go to the very, this is not the very end but this is my last slide, design strategies for MOOCs. We want to start designing MOOCs with local learning communities in mind. Not just so much of the MOOC landscape now is opportunistic. We use MOOCs for local learning communities but how can we design? What are the design criteria when we actually start deliberately designing MOOCs with local learning communities as well as global learning communities in mind? Design MOOCs for remixing. Again, I think customization is going to be a big thing and if you can create your MOOCs so that I can swap in certain things and take out other things, I think it will be a benefit. And finally, design MOOCs with research opportunities in mind. So we can use MOOCs for research questions. Right now, much of the data mining is again opportunistic. Let's look at the data and see what happens. But let's say that midway through a class, I ask half the students the following question. If you finish with distinction, would you be willing to become a community TA for the next offering of this class? I ask that question of half the students. I don't ask it of the other half. And I look to see how such a question as simple as it may seem affects retention rates, affects completion rates, affects the rates at which students obtain distinction. Another example that we might ask is we're interested in looking at MOOCs and other kinds of social media. We want to have students tweet some online lectures and then take quizzes. Half the students tweet, half the students don't in real time, and we look to see how this improves or doesn't comprehension in quizzes and whatnot. And so moving from opportunistic research to deliberative planned hypothesis testing is something that I think certainly a Vanderbilt you'll see more of, but I can't help but to think you'll see it everywhere if it's not happening already. I guess I'll just stop right there, Fiona, because as I said, I could go on a long time. It's lovely. Thank you so much. Thank you very much. It's really weird talking to a screen, so it's really odd that I can't see you, but I'm just going to ask if there are any questions for you. Does anyone have any questions with us? Oh, yes, we do. I'll just get the microphone so you can hear. This is Helen Gersby, I'm from Unia in Norwich, and I was just wondering if we could then transport you back a few years to the beginnings of dabbling, mix, and can you repeat that? Fiona, can you repeat that? We're just going to come up to the front. This is Helen Gersby from UEA, and I just wanted to ask you, if we could transport you back in time a few years to the beginnings of your dabblings in the world of MOOCs, is there anything that you would do differently? Because that's kind of where we're at at the moment, and I think we'd kind of like to learn from your experience, and if there are some tracks out there, we'd quite like not to fall into them. So is there anything that you would change or do differently, or is the fact, I really like the way you just describe it as serendipitous? Is the serendipitous thing just part of what we should embrace? Thank you. Well, I think to some extent that serendipitous is something we should embrace. We should try certainly new things. I am sort of naturally conservative in doing these kinds of things, and I think that that conservatism was an advantage. One thing I was told by the higher level administration here is, you are in your current position as director because you did some interesting things, and you did them by the rules. So, for example, if you have MOOCs in your courses, this is something to think about. If you use MOOC materials in your classes, at least it was the case that some of the terms of service for some of these groups, for Sarah, for example, said that you needed explicit written permission if you were going to use this material as part of a tuition bearing course. Now, the problem is if an instructor requires that students turn in assignments from a MOOC as part of their on-campus course, the instructor is not in violation of those terms of service. The students are in violation of those terms of service for turning it in, and you don't want to, as an instructor, put your students in violation of the terms of service. So, those kinds of legalistic traps are things to think about as you move forward. Student, you might start off slowly. I sort of naturally started off slowly. I did not rely on any of the material I did in the lectures. I quizzed the students myself, had them do the different assignments than the MOOC assignments, so they were watching lectures, but I was still remaining very engaged in the course. Again, I think a possible trap, and it's to some extent something I have felt this year for the first time, but I think it's probably just due to my new job, rather than the way I structured the course per se, is I think we want to be aware of getting too faculty becoming disengaged from their students because they are relying heavily, or they start relying too heavily on the MOOC material. There is also, I think, you want to think about how is the online material going to relate to the on-campus, for the in-class experience. Creating in-class activity takes time, and if you don't take the time to create that in-class activity, I think you will fall down as an instructor. The students won't like it, so there has to be some relationship between them. Thank you. We've got time for one more short question. There's no more questions left. Thank you so much. That was excellent. Thank you very much. Thank you. It was a pleasure. Thank you. Thank you very much.