 Okay, welcome everybody to the third blow up the lecture series of our series this year at ANU. As we always do at ANU, I begin by acknowledging that this is the land of the Ngunnawal people and we recognise their unbroken connection to the past and the education that the Ngunnawal have engaged in both in past, present and future, and we pay our respects to their elders, also past, present and future. Welcome. Just a technical note before we get going, this session is being live streamed. If you are shy about appearing on camera, to the left of me is the area that will not be filmed. Okay, and you'll notice that there's only one person sitting there at the moment, Alistair. So if you would like not to be filmed, could I ask you, rather than walk right in front of the camera to get to that space, you may wish to go out the back door and then come back in again. I hope you've enjoyed meeting one another and talking but I am really just here to welcome you all. If you haven't been to one of these events before, we have live streamed before and you can still download and listen to the previous sessions where we had the CEO of edX kick off, Anant Agawal, then we had Sanjay Samar from MIT, the head of eLearning, come and talk to us and now we have even more excitement and the youngest panelist yet. We'll be appearing this evening and I can see he's brought one of his friends which is terrific. So without further ado, I'm going to hand over to Philip Clark who is really our emcee for this evening. So thank you very much. All right, thank you very much. And welcome, it's a new experience for me too. So let's hold each other's hands and go gently into the early evening. This is sacred ground for me, this lecture theatre because it was in the front row here that I had my first ever university lecture. So I'm not sure about the flood of memories that are coming back, but mostly good. But these were the days of course when universities lectured to students who had to attend the lectures and if they didn't, they suffered accordingly and that was the only way that information was imparted. Seems a strange time really, but people would sit here and record things with a pen and paper and if they missed it, there was very little chance to get it back. Nothing like today's university experience and what we're going to be talking about tonight is something of course that's to use comic book terms and more extraordinarily, unbelievably amazing development ever and that is the explosion of online learning to massive groups of people. The massive open online courses, the MOOCs as they're, I think it's a pretty dreadful acronym actually and I sort of wish someone had invented something slightly more fancy for it, but there we are, we're stuck with it. MOOCs until some bright young thing comes up with a snappier name. But that explosion of learning goes far beyond students these days simply listening to things or learning online in a conventional way. This represents a massive explosion and reach of the universities and all the learning that they have out to audiences, is that the correct term to use, out to people who have never had the opportunity to get that sort of and get access to and to engage with that sort of learning at that level. It's an amazing opportunity when you think about it and how it's done of course is one of the things we're going to be talking about tonight. It's all very well to think of the possibilities that the digital age gives us which are effectively limitless in terms of the information that we provide, but as I know our own institution, the Australian Broadcasting Corporation, we still exist. Although bloody but unbound, I think is what is how we're looking at ourselves, have found one of the great contradictions of the digital age of course is that the more successful you are, contrary to normal business rules, your costs should come down, but of course in the digital world the more successful you are because of service and transmission and digital costs indeed the more expensive it is. So that's an issue to be looked at I guess as well. But how should MOOCs operate? How should they reach people and what are the current hurdles that stand in the way of this form of learning becoming more and more ubiquitous? I think there's some of the questions that I have tonight. We've got a gathered a panel which I want you to interact with. I hope they'll interact with each other and it's a chance really in the next hour and a half or so to come to grips with some of these things and to ask questions because that's what a lot of people are doing in this space right now. So let me introduce our panel tonight. We've got from the University of California at Berkeley, Professor Amando Fox. And I'll detail Amando's experience with MOOCs shortly. Professor Gabriel Bama from the ANU from the School of Population Health and Research here. And Gabriella I think, she told me earlier is a solver of big problems. No, she didn't say that. She said I like to work in that area. I don't think she would claim to be the solver of big problems. But perhaps working in that area. Ben Niles is the President of the Postgraduate and Research Students Association. Welcome Ben. And the youngest panelist as you correctly observed is joining us tonight to Sam Parkinson. Sam is a year eight Tilopia Park School student. And he's joined at least by one other classmate I think up there as well. So yes, we've identified you now so you have to ask a question later. So welcome to you all. And there are different levels of engagement and experience here with with massive open online courses. And I suppose if we could point to it, it's a crudely point to it. We have Amanda who's got a massive, a massive, I mean that word's going to be used a lot tonight. He's got a very large involvement in MOOCs. We have Ben who's tasted some of us. We have Gabrielle who's involved in it here at the ANU in terms of getting an ANU project going. Universities everywhere are clamoring to get into this space and asking questions about how they do it and at what level they should do it and what are they trying to achieve as well. And poor old Sam here has been lumbered with being the future. That's the thing when you're being young though. When you're young that's the trouble. No one ever asks you questions. They just give you titles. So Sam, whether you want to be the future or not, you're it. At least for the next couple of hours. So yes, exactly. I always remember one teacher saying to me once you're the generation that's going to solve all the problems that we created, I thought thanks very much. I should fix them yourselves. Anyway, let's get started. So I thought we would start by getting each panelist to detail their involvement at the moment, what they've been doing. And we should start perhaps by explaining to others. Is everybody familiar with what a MOOC is? Broadly familiar? Okay. But I know some people, no one wants to put up their hand because there will be some who think, I think I know what it means. Yeah, I guess so. So Amanda, maybe we could start with because your experience that you see at Berkeley. What you're doing? What are you doing first in terms of this particular area? So I actually have a number of perspectives and I'm never really sure which hat I ought to be wearing. I'm a computer science professor there. And that's really until about a year or two ago is all I had ever been. I was not a MOOC expert. I was not advising universities on how or whether or why they should do MOOCs. That all happened after I sort of did a MOOC by accident. And it was really just because the founders of Coursera, who I assume many of you have heard of, one of the first for-profit companies to really sort of take the MOOC movement public, as it were. Our faculty colleagues at Stanford, who are very good friends of ours, and my colleague and I at Berkeley had developed a very successful software course. Stanford didn't really have a correspondent course and we were invited really as a professional courtesy and experiment to put the course up. And all we really knew is that this would involve hundreds of thousands of people around the world potentially looking at our materials and doing the homework assignments. And the only reason we said yes is because at Berkeley we had been trying to think about how could we offer this course to more Berkeley students. It had become very flattering for us, a popular course. And we had thought about doing automatic grading and creating special tools that students could use to check their programming assignments without having an instructor intervene. And we thought that if we accepted their invitation, it would really force the issue and we would have to get our story straight about whether it was possible to do. And sort of a few months after that Berkeley realized MOOCs were going to be a thing. They were exploding in the news and because I had done one and not zero, I was an expert. And they asked me to advise and be the liaison to edX with which Berkeley had just formed a relationship. And since then I think I have learned a lot more about education, about the politics of education. But most importantly for me, my view of teaching has been almost completely transformed. I would say that even if I never cared about doing another MOOC or having hundreds of thousands of people take my MOOC, it has permanently changed the way I teach in the classroom. And I think if I have an ax to grind today, that's the ax. This is not a flash in the pan kind of a thing where you can watch YouTube videos and learn stuff. This is potentially a way that can be transformative in what we do in residential education as well. And that's my bread and butter. The Berkeley students pay my salary and first and foremost I have to do the best job I can for them. So that's what I do. I thought that what we were doing is creating MOOCs. But what we're really doing is exploring this new technology avenue that I think could have a profound effect if it's properly handled in how universities teach their students. And ultimately high schools and middle schools as well. Okay. All right. Gabrielle, now you're embarking on a project here, I understand, at the ANU. Let's talk about your involvement and what you're trying to achieve and what you understand to be the nature of the task. So I might just start by saying the ANU has done two MOOCs successfully, one on astrophysics and one on engaging India. And my colleague Mike Smithson and I are part of a group who've been selected to do a MOOC in 2015. And there will be four MOOCs in that lot. Mike Smithson and I are doing a MOOC on ignorance exclamation mark. And the exciting thing about that for us is that we've never had the opportunity to teach that before because it doesn't fit neatly into a curriculum. So even though it could fit into all sorts of places. So Mike, most of this is Mike's brain child, if you like. He's been working on ignorance for more than a quarter of a century as a PhD. He asked, as a PhD, and we get, he's got this great collection of jokes. I wish I had some slides, I can show you. He's got terrific cartoons. But when he was a postgraduate student, he asked the question, well if knowledge is socially constructed, well maybe ignorance is socially constructed as well. And has pursued that. So he's got a background in mathematics and in sociology. He's done, he's very akin to, knows a lot about statistics but also knows quite a lot about social construction of ignorance and knowledge. And has really developed a whole range of thinking about ignorance that there hasn't really been much of an outlet for. And so he's, and mostly we think about ignorance as a bad thing. And one of the things that he's really contributed is the benefits of ignorance. So if we didn't have ignorance, we wouldn't have any surprises, for example. We wouldn't have any freedom, for example. So there are a whole lot of ways of thinking about it. So Mike and I teamed up about a decade ago and my interest is in complex problems and how universities can help people who deal with complex problems do it more effectively. Again, that's not something we're terrifically good at. So we're really good at universities at thinking reductionistically. We're really good at thinking about things that have got perfect answers, that have got clear context, that have got set values. And I'm interested in things you have to think about systemically, where there's values conflict, where the context really important, where there is no perfect answer, there's just the best, worst or least a best possible or least worst solution. And so what I've been interested in is how ignorance fits in with that, because that actually triggers a lot of that thinking. And so he and I have put a MOOC together, which allows him to talk about all the stuff that he's been working about. And then I do a piece at the end which says so if we look at the lectures that Mike's given this week, how do they help us deal with complex problems more effectively? So it's been very effective for us in terms of bringing out an area A that there are very few people who are expert in, and B where there's actually nowhere else that people can go to get what we're presenting. So that's an exciting part of it. In terms of the work that I'm doing on complex problems, there are lots of people who work on complex problems, but we don't have a unified community. We have people all over the world who do pieces of it. And so I'm excited about, in the future, looking at how we could use the best knowledge that we've got, given that there aren't very many people, but bring those people together and make that available to everybody who's interested in this. So when a university doesn't have a strength in an area, but the strength is distributed, how can we use a MOOC to get at that distributed knowledge and make it accessible to people? So for me, those are the really exciting things about what we're trying to do. Okay. Well, thank you, Gabriel. Ben, now Ben, I understand you've put your toe in the water or tasted a MOOC, you've not completed one. And this is a topic we might talk about later, actually, the issue of completing MOOCs, as opposed to having a look at them. Ben, tell us about your experience so far. So I'm a postgraduate student here at the ANU. My undergraduate was a comments degree down in Melbourne at Monash Uni. And I'm doing a coursework masters here. I guess my interest in MOOCs has been just an experimentation rather than for the sake of seeking to complete a course, really from, as Phil was saying, I guess, the perspective of the future of education. For me, it's really a completely new format that I think everyone has the opportunity to experiment with and trial, particularly outside, for me, of your necessarily field of expertise. I think MOOCs offer a whole new range of possibilities to students. And I use the word students because I think as soon as you enroll in a MOOC, anyone whether you're currently studying, whether it's in high school at a tertiary education, or have completed and are no longer doing the other forms of study, become a student once again. Even if it's just a dabble, it provides a whole new world of opportunities. So for me, there's quite an interest in what are the possibilities and what's the potential of MOOCs going forward. So that's a little bit about my background. OK. Well, thanks, Ben. Sam, have you had a go? Yeah, I've had a really small dabble in MOOCs, but like I am interested in computer programming. And like in the world of computer programming, one of the big ways that teaching happens is through online courses. So they're not, they don't have the massive element of MOOCs where you have people interacting, but they still have the element of online learning. And I have, you know, seriously completed some of them and, you know, coming to MOOC, looking at MOOCs, I feel that they're really different to these online courses in a way. Yeah, OK. But so you would say as a young coder that, in essence, your engagement with learning has been almost totally online. Yeah. Well, in the field of programming, I mean, definitely. Yeah. And yeah, I've sort of seen good things about, you know, the whole potential of online learning. How has that, how has a learning experience felt? I mean, I don't know whether you can compare it, but comparing it to another form of learning, a direct classroom learning, you're in year eight, because so you're getting a good dose of that as well. So how would you, how would you look at both those learning experiences for you? I mean, how does it feel? Yeah, well, I actually find them really hard to compare, because I mean, when you're doing, you know, direct classroom learning, like for example, if you're doing maybe a science, you know, science, you'll be, you know, you're in the science lab, you might be doing practical things. And, you know, it's kind of, it's very different, like online learning programming in a way to online learning for many other things, because if you're doing computer-related things, you are doing a practical thing as soon as you've started off a computer, right? So it's, it is a really, I don't really know how to compare them at all, I think they're different. Because they're so really completely different, they're completely different. And, and exactly, and that's the essence of it too. All right, look, we've got a raving microphone. Where's our raving, yes, and you're monitoring our Twitter feed as well. Excellent, excellent. So we have a raving microphone here, yes, the hand in the ear. So please, anybody who wants to join this discussion, there's an opportunity for you to talk about it, if you want to talk about your own experiences, or ask a question of the panel, or raise some questions that we should be asking. We've got one at the back here, yes. Actually, it's, we're out the door. That's right. We're really at a hospital pass up there, don't we? That's possible. This is where I had my first lecture, 40 years ago. Oh, dear. And speaking of lectures in here 40 years ago, one of the advantages of you needing to be here was the horror of the Socratic method, I'm sure you remember it well. I do, yes. But the purpose of the Socratic method, apart from terrifying students, was to establish that they had actually done the reading, and that they did actually understand the subject matter of the course. They couldn't steal it from someone else. Well, we know that online this is the major problem. In fact, it's a problem with universities generally, because we've gone away from reliance on exams. So now people hand in assignments and you've no idea whether they've written the assignment or not. You don't have it. It's not in handwriting any longer. It's all done in some type script. How do you know it's theirs? And in fact, we know that there are a thousand for-profit companies out there that are making billions out of writing papers for students. There's just one that's been caught recently in Australia, which goes to show that some of them are occasionally stupid, but most of them aren't. What are you going to do with MOOCs about knowing that the work that people are doing is their own? How are you going to give a qualification to someone who's completed their course or even a unit in their course by doing MOOCs? Well, that's a good question, Amandam. Can I go to you on this because it really gets back to a question that I was going to wrap up with, but you said, now let's get to it first, because that's really the key question is, how do you know that you've had any success in MOOCs? What is the measure of success? How do you measure? Yeah, there's a great many threads to pull on. So thank you for asking the question. I'll pay you afterward, according to our agreement where you were. No, I'm kidding. Perceptively asked. Let me let me try two ways to answer it. And then I'm sure there's going to be many other threads the other panelists want to pull on. I'm going to also in answering it, I'm going to partially use the Socratic Method. Observe, Sam, have you cheated in any of your online programming courses? No. Why not? Because, well, then I wouldn't actually learn programming. And thank you. So hold on. This is this is a he said to me very profoundly important answer. This is the most important thing you're going to hear all night. He actually wanted to learn programming. There's a deep philosophical question that I think a lot of universities, certainly in the US, possibly here as well, have been struggling with for quite some time. And I think one of the effects of MOOCs is that by surfacing some practical issues such as cheating, which I agree is rampant. You know my answer to your question, how do we know that students work as their own and a MOOC is that we don't. And we tell them ahead of time, we are not going to we're investing zero effort in monitoring for cheating because Berkeley's MOOCs are not offered for credit. There's no tuition being charged. This is for your benefit. If you want to cheat on a course that isn't graded, for which you're not paying, you know, more power to you. But the deeper problem that I think it reveals is that I think universities have a little bit of an identity crisis around things like what grades actually mean and what it is you're supposed to be measuring when you sign off on a student having earned a certain degree. In fact, I just finished reading a few months ago a very provocatively written book, Excellence Without a Soul, written by a long time Dean of Students at Harvard that tackles this along with many other issues. So the pragmatic answer to your question, in my view, is it's an open challenge that may or may not ever be resolved to detect with definitive accuracy whether someone is cheating in a MOOC. But I would argue that that maybe is chasing the wrong question. If the purpose of the MOOC materials is to empower people like Sam who want to learn and the role of the university or of the mentor becomes, in a certain sense, to certify that they've learned, then those processes are separable. Exams are one way to calibrate them. You know, we have exams in a controlled setting and it's proctored and you're observed. But there are other ways as well. For example, a lot of courses in engineering require a design project that is open-ended. You can cheat on a homework, but your project is different from the other teams project and you can't cheat on that. And it becomes very quickly relevant, very quickly clear which students have actually prepared for the design project by doing the background work. In my field, which also is software like Sam's field, a lot of the companies that recruit our best students and we have awesome students at Berkeley, I have to say. It's a great program. It's a privilege to be there. And the companies tell us, yeah, we don't really look at their grades. We look at their code portfolio on GitHub, which is a very popular site where programmers who work on public projects that they don't wish to keep proprietary, they will post them for everyone to see. We ask them to come in for an interview. We ask them to solve some coding problems. We want to observe how they think through the problem and you can't cheat through an interview. So I'm not suggesting that universities ought to outsource the certification process to employers and graduate schools, but I'm suggesting that we can separate the transmission of material and providing materials that students can use in a self-guided way, maybe with some instructor assistance from the process of someone all saying, OK, now that you've learned the material, let's sit down and find a way to assess what it is you really mastered or not. And for my part, I'm going to continue to spend this much effort, that's a zero for people in audio only, on actually monitoring for cheating in the MOOC itself. I'm going to continue to rely on the other mechanisms in my campus course and in my interactions with employers and with graduate admissions committees as the sort of final cross check. That's one opinion that I'd like to hear of. Anybody else? Gabrielle, what do you think? I mean, what is the measure of success? Well, let me just address that question, too. I think that you've hit them now on the head that certainly the way that I've been thinking about it, too, is you've got to separate the provision of the information from the assessment of how well people are doing. So let me just share with you something that I've been thinking would be a fabulous thing to do. So just imagine you get the best people in the world to put together a master's course, in my case, to be on dealing with complex problems. And so these are people from around the world. You've got the very best people. Anybody can listen to those lectures. But each university sets up its own way of assessing that course. So the assessment for it is run through a specific university with a much smaller group of people. And what they do is they use whatever they like of the online material as part of the teaching portfolio. They can add to it if they want to. They can use parts of it or not all that. But they design their own course drawing from what they like of that MOOC and perhaps other MOOCs and perhaps some online teaching. So what it does is, and I think that's something you raise too, it really changes the way you think about teaching a class and the resources that you've got to draw on in teaching a class and the exciting things you can do around assessment. So you're not spending the class time with students talking at them. What you're doing is you're spending the class time with students working on a problem or they're presenting or whatever. And you can give much more intense tuition if you like so that you can go back to really the kind of psychratic method you can actually, and if you're a tough teacher, you can, you know, students can't get away with bullshit if you like. They really have to think. They really have to have shown that they've done the reading. They really have to show that they've listened to the lectures and you've got the potential to really do something exciting. It won't work if we try to make education cheap. It'll only work if people have a lot of time, if the teacher has a lot of time to spend in the classroom and to work with students intensely. If we're trying to, if we want to make MOOCs a cheap form of education where people do self-assessment and that's the end of it and they get a degree, that's never going to work. That's, then we're going to go back to the problems you talk about. But if we take this to really think about education deeply, it gives us a wonderful opportunity to do some really exciting things. Anybody else? Yeah, question here. Yeah, so I'll just add, I also sat here in my very first lecture a couple of years ago. What was it? I think it was foundations of Australian law or torts. Yeah. Yeah, I'm also in Sam's position. I did a like an online coding course on Code Academy, which was like teaching me how to use HTML and do a bit of basic programming, something I don't do here. And I found that sitting at my desk, learning how to type code miss some intangible benefits that sitting in a lecture has. So there's no interaction with students. There's no face-to-face interaction with lecturers and things like that. Do you think that MOOC should address these sort of intangible benefits or whether they're relevant at all to MOOCs? A minute. OK, sure. I've always got something to say. So you should pick other people, but. So let me be controversial and say that, no, I don't think MOOC should address those differences, although I think they're extremely important differences. To me, that's a little bit like saying, this is a great hammer, but it doesn't screw screws in very well. Should the hammer address that? The MOOC is a different kind of tool, in my view. And I think you're absolutely right that, you know, Gabrielle earlier referred to the fact that according to one of the very widely held theories of learning, knowledge is in that socially constructed. That the idea that you're interacting with colleagues is an integral part of learning the material. Learning is not something you can do in a vacuum on your own. Now, if you look at the way people are using MOOCs, and I know some of this anecdotally, because we survey the students in our MOOC, and, you know, there's respondent bias and all that stuff. But modulo that, we find that students, for example, who live in the same city will, on their own, organize little study groups where they'll meet at a coffee house, for example, or students who are at a particular university. We have students who are using our material, even though they're enrolled in a degree program at another university, the coursework that we're offering is not available to them for whatever reason. But they are already colleagues, and they're in a lot of the same courses. So they'll find a dorm room or a university classroom that's unused, and they will re-inject this critical ingredient of interacting with your peers. In fact, I and others are trying to do research on how to enable some kinds of interaction, even when the students are physically separated. Can they interact over chat or over video? Or what can we do to sort of bring part of that back into the MOOC? But it's not because the MOOC needs to solve all the world's problems. It's because we recognize that part of what learning is is socially constructing new knowledge with your peer group. And to the extent that we can make it easy to connect things to the MOOC that enable that, that's hugely important. And by the way, let me just, I love code. Code's what I do. I eat, live, breathe, drink code. It's great. Well, I don't drink code. I drink other things. But an unusual thing about code specifically, and I think one of the reasons that it was sort of first out of the gate for MOOCs and these other online coding courses, which predate MOOCs, you know, if you're taking a MOOC on, let's say, physics or chemistry or medicine, you may have the best simulations of a rocket taking off or simulations of surgery, but that's not the same as cutting open a cadaver, right? When you're coding online, you're doing the real thing. There's no difference between the activity you're doing in a coding course online and actually coding. It's the same activity. So in some sense, we should almost, and I say this as someone who loves coding, we should back away from it a little bit because it is such a special case. It's also a special case because the people who do that, the people in that field, like me, are also the ones who have the technical wherewithal to create the tools that are being used for MOOCs. So I would propose that we temporarily take it out of the equation. It's suey generous enough that I don't know how much we'll be able to generalize from what has worked well for coding schools. That's what does raise a question. Is it suitable for all forms of the imparting of knowledge? Gabrielle, is it or are there some things that really oughtn't be attempted via a MOOC? Well, it's a challenge for our MOOC because we're ours isn't technical. We're not imparting technical knowledge as a lot of other MOOCs are, right? We're trying to give people some analytical skills and that's a bit higher up the food chain of learning. And so we're really very strongly exercised about how do we do that? So what we can... So one of the things that is happening on edX is that the MIT has developed hangouts which allow people to get together. So we're looking at how we can bring that in. We're looking at how we can have meetups as part of what we do so that students can interact and so that we can facilitate learning. But I think for the sort of work that we're doing, a combination of MOOCs and I've forgotten what the little ones are called. They're called SUKS. SPOX, that's right. It's just to give you another acronym which is a specific course. So if you combine the massive with the specific, you can kind of get the best of both worlds and that's the interaction that I'm interested in the long term. Is the point to pick up something you raised at first up, I mean this was the issue of credentialing and credentials, etc. Is to talk about credentials really to miss the point of the discussion about MOOCs anyway? Ben? Yes, I could jump in. I guess a lot of how I see MOOCs is and perhaps this addresses more of philosophical question. When you develop a MOOC and perhaps Armando and Gabrielle can touch and Sam even, I find that there's this concept of providing something free and open to everyone. And perhaps in that sense, accreditation or assessments sometimes isn't necessarily the goal of... Which is traditionally we thought of that we thought of an assessment and credentialing and how well did you do and what qualification did you get and from which institution? So we can rank all of that and have a menu of scores against which we can assess whether you're any good or not. Something for those of you who were in attendance or maybe watched online. The previous talk with Sanjay was something that I found quite interesting that he raised was I guess the philosophical question of this is something that can be provided to people all over the world from different backgrounds perhaps from rural backgrounds or low socioeconomic backgrounds or in countries or places where people don't necessarily have the opportunity to access university education or a higher form of education like we have privilege to here in Australia. And I'm not sure the kits that you guys have had on your MOOCs yet and how you've been able to track that but it was something really interesting that Sanjay raised in something I guess going completely away from this whole element of using MOOCs as a tool to replace necessarily lectures and courses that are being offered within a university but also offering them free to the public online as a way of I guess advancing education throughout the world and providing something that I guess didn't exist 10 or 15 years ago where now I guess you have the flash in the pan learning something off YouTube or Googling something and checking it up on Wikipedia however much you want to rely on that as a source but now to this you know something really specifically developed by some of the leading academics from around the world that can be accessed by anyone from any background in any age back with you Sam you've got to go through the dreadful gatekeeping that happens at school of course and that's you know that is a form of credentialing because you can't avoid it and you know we've designed a system to force you through it so that'll come out in year 12 with the dreaded HSC and the and the and the ATAR system which will determine initially at least if you were to go to an institution that will determine which institution you get to go to but you said earlier that you didn't really see any any comparison whatsoever between that that form of credentialing and what you were doing online that is because online so talk some more about that about about what's what's more important to you well I suppose like online is it's a really awesome opportunity because I mean you can go on and you can learn anything whereas you know with the traditional forms you don't have that you've got to you know you've got to enroll you've got to make a big commitment and yeah I mean to some extent the issue of of authentication and credibility is is tied up with that whole issue of the authentication isn't it because people want the thing whether it's the degree from a prestigious institution or the admission to a particular course so therefore whatever it takes whatever it takes which is the antithesis really of what we're talking about with MOOC education well you know that there is somewhere that there's I'm coming to your question there's a shake up waiting to happen here right because I think that again let's try to sort of pop up a couple of levels from the question I think is really being asked here the idea that there's a finite and relatively small number of elite institutions that an accreditation from that institution is considered to have economic value it has a return on investment you pay some tuition but it will likely put you in a selection pool for better economic opportunities the entire model of filtering and ranking and scoring and ATAR and all the other crap that goes along with it is based on the fundamental assumption of extreme scarcity that education is a product that is in very short supply relative to the demand now I don't think that MOOCs are going to fundamentally change that equation I think that there is such a thing as a scarce resource of great instructors having face time with great students and you know as was pointed out earlier that the interaction with smart peers and with a mentor who could sort of be a tour guide through a body of knowledge that's not going to go away but having said all that there is a tremendous amount of stuff that we can now offer to people very very cheaply because of the blessings of technology and Moore's law and what the missing piece that I think we haven't seen yet is what are the intermediate credentials if you will to which value is going to be assigned that are not a university degree but that recognize some combination of you've absorbed the knowledge and you've proven it to me somehow you've come into my office and you've written a program on demand I can certify that not only did Sam claim to have taken the course he can actually write code to solve these different problems at some point someone will be the arbiter of a system in which that metal has value and that will be a profound change for universities because we've been the only ones holding the bag of metals until recently it may be that I plan to be retired it may be as many HSC students have found too that the credentialing has a very short shelf life and usually lasts as long as you're halfway through your first university course when people work out whether you can actually do the work or not regardless of what your past performance was yes to a question at the back there my question is can MOOCs coexist and cooperate with more traditional forms of online distance education so I've been teaching a course online at the Australian National University for five years it's got a course code the students pay money they have assessment the only thing is they do it online so it uses money at the same techniques as MOOCs but there's a small number of students they get charged a lot of money do these I mean do these two forms of online education coexist or will MOOCs replace other such forms Gabrielle? well I you know I think what we're going to end up with is well so one thing we could end up with is that the delivery becomes free to anybody and you could do that in your course now if you wanted to and it's the assessment the accreditation is what you're actually paying for and you can pay for it at an elite university or at a community college depending on what your ability is and what you want to and you know what you care about and what you can afford to pay I guess so it seems to me that that's going to be part of it but I think the other part of it is that there is there is some advantage in the big but there's also some advantage in the small and I think it's figuring out what those two are and what we would lose if we lost all online education I think we've been thinking a lot about face-to-face classroom teaching and what MOOCs help us do about that and flip classrooms and all that sort of stuff we haven't thought as much as far as I'm as in the kind of reading that I've done about what it does to classical online education and I think what's been interesting to me is that I don't think MOOCs have learnt a lot of the lessons from classical online education either I think there's you know there's quite a lot that we can learn from the experience that's been gained Yeah, we'll go to yes a question at the back then do you use the voice? Do we need do we need the microphone? I think Yeah we do then we'll go I think we've got some tweets we'll go to yes we'll go there I've got you sir down the front yes perhaps you need to look at a hybrid model you know why should it just be the one thing perhaps the core material that would be digested in a lecture could be observed on on a mood say you know an hour before the lecture with the students operating privately and then you come to the lecture theatre having logged points in the in the lecture and the the lecture having reviewed some of those and plucked some that he'd like to talk about and also to have the students there interacting with the material maybe for two hours rather than just the one core one hour on the lecture which may or may not be more relaxing for for the lecturer and perhaps it's more engaging but giving the people there more more opportunity to meta think over the material that people worked in teams you know you know and in law if you've got to do a little hack work you don't get to do anything clever because you're there and if it's only you by yourself you're you're just grinding through the material and research you have to grind through it or economics like Danny Rodrick beautiful writer you know he's got a head professor that does he's an I'm a crunching from this sort of thing you know if that's all you're doing the the hack work you can't do the more powerful thinking because you're probably exhausted and such like so I'm just thinking of those sort of techniques where you use a hybrid and more enabling people to engage with the material rather than just sitting the other it's a lecture like he's talking at 300 words a minute and writing with two hands you know which is not not really a very lovely experience and so it could be more powerful but just moving it around I think is is what you need to do and try various things in various forms anybody want to comment on that yeah a lot of us are doing that the TLDR a lot of us are doing that some people call it flipped classroom some people call it you know section-based or project-based or a problem or a laboratory-based learning but there's many variations of there's some stuff you can do online to sort of gather the background and then there's some stuff where you go and work with peers and perhaps under the supervision of an instructor or somebody on the teaching staff and I think because now I'm going to try to address your question and also the previous one about whether MOOCs can or would replace existing online models it I think what MOOCs have forced us to do for better or worse and here I'm referring to both university residential courses as well as online courses is to really I hate to use this word deconstruct the value proposition inherent to each of those courses what is the thing you're paying for when you pay for an online course that you're not paying for when you see the exact same assets packaged in a MOOC one possibility is you're paying for accreditation by a trusted source somebody who is an authority and a teacher and an expert signs off that you know you're worthy to pursue a career in this field another thing you might be paying for is a very experienced tour guide we call that a teacher and it's you know most of the knowledge that I teach in my software engineering course I didn't invent in fact you can find most of it on the internet there's too much of it my role is to actually tell the students what's important and what order I believe they should do it and that's my opinion and then go to 50 other top universities talk to 50 other top professors and get 50 other opinions all equally valid it's really a matter of what the student is after and what kind of career they want to pursue so again I think it comes back to what is the scarce resource that we're charging you for because that's sort of basic economics right when we charge you for something the premise is that there's some scarcity behind the resources a finite amount of it and we need to charge as a way of doing admission control it could be teacher time it could be accreditation it could be the sophistication of the accreditation process and I think universities have sort of bundled all of this stuff together and say you get the degree all of that stuff has been rolled in somewhere and I think MOOCs are forcing a lot of these instruments to sort of be broken out piece by piece and say okay well where is my tuition going how much of my tuition is teacher time how much of it is lab facilities how much of it is access to other intelligent students who are my peers and from whom I'm going to learn even more than I learned from my instructor and I think and it's a salutary exercise for universities to have to go through that and a very painful one but I think that's really the root of both of the questions about my piece of bolt okay all right no we've got some Gabrielle I'll go to you just a few weeks away one point and that's to pick up on what both Ben and Sam said which is that MOOCs also give you the opportunity to taste stuff so that you don't have to you don't have to sign up for a whole course but you can if there's stuff that you think you might be interested in and I think they're really important for people starting out on their learning their lifelong learning is to that they can expose themselves to a whole lot of things that that that they think might be interesting but they don't know before they kind of enroll in a course by the time you get to university and even now at school you're having to sign up you know you're having to strip your options back and MOOCs allow you to keep your options open and to to learn about a whole lot of stuff that you wouldn't necessarily learn about otherwise and I think that's a a great advantage Heaven's above being at university and then going to things outside your course yeah quite how extraordinary a concept I know I agree we've got some some tweets have we yeah we've we've got a question from Glenn who's watching the live stream hi Glenn he's got a question for Armando and would like to ask you about the feedback you got for your MOOC and what worked and what didn't it depends on which student you ask and I think this comes back to the idea that you know we got so first of all generally speaking we got a lot of very positive feedback that was very flattering for us as instructors the fact that there are students many of whom are actually working professionals with full-time jobs who are taking time out with really no remuneration to them to take the time to work on our material is sort of the highest compliment an instructor can receive so a lot of the feedback we got was thank you for putting it out there we've literally we've had people say because of the knowledge from your course we've been able to sort of start a small company and break into a better employment situation so those things are all very flattering at the same time there were a number of students who really didn't agree with the way that we taught the material it's not that they weren't interested in the material but we went too fast for them or we assumed a level of knowledge or a way of thinking about programming that maybe was not what they were accustomed to we or maybe there was something about the pacing of the course that they didn't like some students had suggestions as to how they would teach the material differently and with respect we said great suggestions we're going to keep doing it the way we're doing it because it works for our students and it works for us so we've learned a lot from doing it there's a lot of people who liked it there's a number of people who really didn't care for it and that's good feedback too it doesn't change what we're doing but I think it reinforces the fact that different learners will have different ways that they want to be presented the material and to the extent that what we're doing for our students at Berkeley and let me make this perfectly clear Berkeley did not get into this to make money because there isn't anything to be made Berkeley did not get into this to put great courses out there for free in the rest of the world although as a public university we love it when that happens Berkeley got into this because we think we need to stay on top of our competitive edge for teaching this is something we're doing because our faculty and students are going to benefit that's our first job so everything in my MOOC is there because it serves Berkeley students in some way and to the extent that other students around the world can benefit I feel great about that things that I wish had worked better I've discovered as a matter of humility and as an instructor that I had certain ideas about the level of proficiency of certain students in terms of their ability to write high quality code code that's clean and elegant and concise and whose design intent is transparent I have found out through the MOOC that the level of proficiency is not as uniform as perhaps I had thought and then I thought well surely it's because these students are out there in the rest of the world if I look at just the Berkeley cohort surely as a group they'll be much stronger and doing the MOOC forced me to look and they're actually not right there's almost as much variation across our students as there is in the rest of the world so it actually has helped me gain new insights about the students I teach and it's causing me to make changes in my material that again they're for the benefit of my students but it was all these other students responding to our material that allowed us to draw those inferences so as an instructor it's probably been the biggest learning experience I've had since I started doing this job 10 or 12 years ago does that answer your question? Not yes Yeah well yes it goes some way down I'm sure I've got a question down the front here yes just one moment sir and we'll get the microphone to you Hi I looked at the EDX thing on the computer and I was interested to find when I looked at their they had these samples of course they're just one lesson from each course over a range of physics, chemistry and various other topics but what interested me was to find that the most effective sample was a straight filmed lecture all the other samples had fancy computer graphics and all sorts of other things were not as effective as that straight lecture now this probably was because that lecturer was a very good lecturer I think for those it was a classic physics lecture of the simple pendulum in which he stands at one side of the lecture theater and he has a pendulum ball this big holds it to his forehead lets it go whizzing over the other side of the lecture theater and then it comes back and it sort of stops just short of his forehead so it's a very spectacular demonstration but the lesson also contained the other things where he did the conservation of energy and the simple algebra of the pendulum and so on and it was very effective the interesting point to me is it was much more effective than any of their other samples which had obviously involved a great deal of computer work and so on but they still weren't as effective as a simple lecture has this been the experience so far that we that we find that's the new teaching tools that emerge of of graphic interface and so on are more effective than than simply one expert human talking to others I know you want to answer this so first of all let me just say that I took freshman physics from that professor that's Walter Lewin at MIT and he is an extraordinary, extraordinary presenter he was legendary even when I was an undergraduate which I think was in the Jurassic period or something so I appreciate that a thing that I think a lot of successful MOOCs do have in common and the students react to this when we survey them and we ask them the ability of their presenter to be clear and engaging and interesting and lucid and transparent has a tremendous amount to do with the students perceived effectiveness of the teaching having said that, effectiveness is a word that is almost by definition subjective so I think what you probably meant to say was if you don't mind my putting words in your mouth that what you found most effective was Walter Lewin's presentation of conservation of energy using the pendulum but would Sam find that as effective as an alternate presentation of the same material in a different format using different tools? Open question, open question it is my claim we could ask Sam yeah absolutely, it's a follow up yeah can we get the mic back? So we're just monitoring our feed here we're not checking Facebook but that's another question well of course I'm a theoretical physicist so I realised I could be biased so I tried this out on a small group of my friends none of whom were physicists and they all found the most effective lesson was the same one that I had found now admitted it was only a very small sample so probably not statistically significant go real the point that Amanda is making is that what MOOCs allow you to do is to develop courses for different kinds of learners so what we've got now is we've got a one size model that everybody's got to take basically and what this allows you to do is to look at different ways so just as you said you found some students really liked your style of teaching and some students didn't and what we can do then is to develop a course that caters to those students and I think that's really one of the powerful things about MOOCs is that everybody's and different learning styles can be accommodated I think that's really important but I also have to say I reckon if any of my physics lecturers had gotten up and put a wrecking ball and swung it out their faces I would have been pretty engaged as well I mean there's an issue here it's the point you make about teaching styles because universities once of course never ever recorded their lectures and said you had to attend then they started recording them which did lead to a lot of students not attending lectures not because of laziness not because of laziness but actually some students and this has been my direct experience in talking to them some students actually prefer not to go to the lecture but prefer to listen to the lecture in their own time because they find it a better learning experience because they can stop they can interrogate online a point they don't quite get or whether the students have said and they can work through the lecture that way now that's not my personal preference I'm with you I would prefer somebody to speak to me but it's been an interesting development isn't it so it's not just absenteeism for its own sake it's actually because some students prefer this what about you Ben? I was actually going to jump in at this point too I think perhaps what Gabriel was mentioning is also something that's really quite new in the world of education generally this opportunity for and this is where I see it I guess university is almost showcasing some of the best teaching styles that they have on offer and students being able to then kind of search through and sample other styles of teaching I guess what Phil did touch on regarding online lectures is you can also then speed up your lecture to 1.5 or two times the regular speed when they're speaking a little bit too slowly but not if it's a Monday he said you come at the end I guess maybe it's something Sam may have experienced too but even earlier this semester in studying for exams one of the concepts in my law course that I struggle with I then decided to just search online and find other videos and documents produced by other universities on a very specific concept that I thought might have explained it a little bit better and I guess that was a way that I was able to sample this kind of almost free and online sharing that universities are starting to do by putting up lectures online and things like that I'm not sure whether Sam you've ever I totally agree with you like when I'm trying to revise for my tests or do something like that I am, it's just amazing to have all of these resources available by the internet and I mean this isn't really a MOOC specific thing but if MOOCs are going to allow you're going to encourage people to put more of these resources out there because I mean there are people doing all of these lectures every day and I mean if a MOOC means that they put these lectures online I mean that means that there's so much more available and yeah I think that MOOCs can actually let you better utilize what's already being done because you have access to a certain person's explanation even if they might live halfway around the world and this person's explanation is good well why aren't you using it? Sure How important are the kids? How important is talking to other kids? Oh I think it's really important because you're not doing that online though I mean you can, you can I know but it's not, as we all know it's not the same Yeah well I've had a look at a lot of MOOCs and I think some of them try and really integrate discussion but I mean I totally agree with you it's not the same having an online discussion as it is So you'd still rate that as a very important format as a human, as a learning experience so the idea that you're actually talking with other kids about what you're doing Yeah totally So there's another dimension of MOOCs that we haven't talked about yet which is that they actually also give you research capacity So one of the things we're doing in our MOOC is that the lectures that I'm giving are being given in two different modes so that students can choose the one that they prefer and that gets to some of what you're talking about but you can do other kinds of research as well so what Mike's doing is he's doing surveys of people you can do randomized controlled trials on MOOCs you can do a whole sorts of really interesting and exciting additional research that involves a student and you can give the student feedback on that so getting students involved in projects is also a really good learning experience Is there a fundamental... We should have started with this as a fundamental thing What's the target audience? Do we think about that when we're designing a MOOC or we're just sending a ship out on the ocean? Is there a target audience? The target audience is third and fourth year UC Berkeley Computer Science majors That's my audience Now, the good news is apparently there are tens of thousands of other people in the world who are in some sense part of that audience that are not in my classroom and I can tell because they say they like what we do and they say they've learned a lot from it but that's my audience Okay, can I realize what you're thinking in that targeted way? So we've got different perspectives on this so Mike's course, Mike's part of the course is suitable for basically any discipline because ignorance is relevant to every discipline so it's much more widespread, if you like and it's applicability but he's thinking about an undergraduate level I'm thinking about for the complex problem stuff I'm thinking about people who've got experience with complex problems but I'm actually starting to realize that that might not work because the people who are going to enroll I think part of it is if you have the bar too high it alienates some people and you've really got to think about what are those people going to do when they get to this piece of the course so they're going to get annoyed because they can't contribute or work so that's a bit that we're curious to see what the impact of it is but ours is kind of a much bigger audience and what we really want to do is to get people excited about ignorance so I'm hoping you're all going to see that Yeah, a question over here I've got a question that's more specifically for Ben and Sam but everyone can contribute if they want I'm actually designing a MOOC at the moment on an introduction to actuarial studies and I've been pondering the role of assessment within MOOCs because obviously, given that we don't have the accreditation issues with MOOCs you then wonder what is the purpose of assessment and in my little dream world I ditched assessment altogether it doesn't exist you're doing it because you want to do it but then when I thought about it we live in the PlayStation and Xbox age of trophies and achievements and all that sort of stuff and I thought, well maybe that becomes the new role of assessment as a form of motivation so I'm curious to know from your perspectives what is the role of assessment and you can define that however you want within an online learning experience Sam, do you want to go first? Is getting a good mark important or is understanding it more important? Yeah, well I suppose having any form of assessment and stuff is always a real motivator but I also think personally from my experience that if you're just listening to a lecture then that's great, you're absorbing facts but I mean a lot of the time assessment forces you to think analytically about these things and that's obviously specific to whatever subject you're doing at MOOC on but I also think that as well as that if you're teaching in schools I think that assessment is a really important thing because I don't know what this is like in university, it's probably there but assessment is a real point where you've done a piece of work and you've been forced to do this piece of work so now someone can give you feedback on this piece of work and I think that that's probably that's where I think assessment is going and that's where I think it's gonna be definitely the most Can I just clarify something? I'm certainly not talking about formative assessment I'm not talking about summative assessment in the sense that these MOOCs still look like university courses in a lot of ways you get a grade at the end so I guess the question is more on the lines of is that important? Is getting a grade important? There will obviously need to be formative assessment which is just practicing essentially what you've learned but what's the role of getting a grade? How important is that? Is that important? Yeah well I suppose it's always a nice thing and it's a motivator I just yeah well it is a motivator but it's not gonna be necessary for anything yeah I mean especially if it's not 100% you know if you ditch the accreditation side of things I mean they're kind of you're looking at what is the point of a grade I mean if you're not looking at feedback you know what is the point of a grade? And what do you think? I guess one of the handy parts of being past the president is that you hear along the grapevine that there's someone creating a MOOC on actuarial studies somewhere in the university I must say when I first heard that I thought to myself who would want to do a MOOC on actuarial studies? But this is also coming from the guy who will now check out the physics pendulum swinging video so I'm sure I'll dabble in your actuarial MOOC one day regarding actually assessment and coming out with a grade at the end of it perhaps I differ on Sam's thoughts a little bit I guess if it comes down to a little bit of almost yes what's the purpose of a MOOC are you there for just to explore and you know broaden your horizons in terms of learning but I think the opportunity that MOOCs also provide is a first-hand experience I would say the advantage of producing a grade at the end would be that someone could then be like oh actually I shouldn't have become a lawyer as I may one day find out and then think to myself actually I'll go and pursue actuarial studies and maybe then find accreditation you know through a different course the one huge disadvantage I guess I would see from it is someone maybe being discouraged that they've put in all this work and completed the course but not come out with the results that they've expected and I'm sure for all of us including Sam who's ever taken any test we've all experienced that oh what is this I thought I deserved an A plus or an HD so I guess it's a bit of a dichotomy that exists one would be encouraging people to then take the next step and maybe plunge it a little bit deeper and pursue that course but the one big downside that I would see would be yeah discouraging also yeah before we move on from the question while you're passing the mic it'll be quick one of my colleagues who has another Berkeley MOOC on artificial intelligence they tried an experiment after the first successful offering of their MOOC they had set a pass threshold of some number and on edX you don't get a letter grade you get sort of pass or no pass the second time around they as an experiment they did pass with distinction if your final number of points was higher than a higher threshold you would get sort of a little virtual gold star and you would get a nice signed letter from the instructor saying good for you and that's it I mean it's still no credit and if you look at the histograms for the two when they did the fur distinction there was this big cluster of grades right at the top where the threshold was so that's Sam as a programmer do you know the site Stack Overflow oh I love that site for those of you who don't write code for those of you who are not fortunate enough to participate in humanity's greatest creative achievement of the 21st century you guys go right in there's actually I have a point this is a forum where programmers ask questions and depending on how many questions you answer and if you see an answer to a question that you think is correct you can upvote it there's this very complex system of reputation points and as you gain more reputation points you actually achieve editorial powers on the site so the more you contribute the more authority you have to delete an appropriate post or to close off a line of discussion that isn't going anywhere it's amazing about the site it's a traditional game and in fact a colleague of mine did a sort of human studies experiment carefully looking at the logs and there's a certain amount of social engineering you have to do to tweak these reward systems but the bottom line is if you put a counter in front of somebody they want to make that counter go higher and their great goal was they tied the counter to editorial power which gives you recognition as you are an editor but it means that the people who use the site also curate it it's genius so I think there's actually many possible roles for assessment but I think the bottom line is if people can get more points there's this thymotic urge for recognition that will make them want to get more points if at all possible and if you can find a way to use that to advantage if you can get those people to say great news we're inviting you to be a volunteer teaching assistant for the next offering of the MOOC we actually have done this and it's the reason that our MOOC is sustainable the best performing students are honored to be recognized as TAs and give back and actually help future cohorts so there's something happening here and I think assessment has a role to play if only because of the gold star counter okay yes, question you've talked quite a lot about the benefits of MOOCs in terms of teaching feedback on your teaching research gathering data and so on I'm interested in particularly Gabrielle's view and Amanda's view on the marketing potential of MOOCs so have you or some of your colleagues done a MOOC as a sort of standalone course or piece or bite size chunk and then the demand has been so big for that that it's led to the development of the program or things that have happened in the classroom as a result so what's the benefits in terms of maybe recruitment or other benefits if you like of the marketing benefits Gabrielle so we haven't done our MOOC yet we don't go live until April next year but certainly one of the things we're trying to do is to get people interested in the topic so marketing is a huge piece for us you're just making me realize that I haven't thought about what we're going to do with all this increased demand if we are successful in our marketing so I'll put that on my to-do list but certainly we want to we think we're on to an area that's underdeveloped and we really want to raise awareness and interest in it how about it there's many kinds of marketing but I think for an instructor one kind of marketing is tied in with our egotism that our way of presenting a certain body of material is a really good way so although this wasn't the intention doing a MOOC of part of our Berkeley course put our approach in front of a lot of people many of whom, not all of whom, responded positively and some of the ones who responded positively for example were other instructors at other universities who then asked could we use some of your materials in our classroom setting which is where the whole SPOC concept came from some of the ones who noticed it were professionals, employers who are going to recruit our students who said wow, this is the way you guys are teaching this material we think is really great we're predisposed to want to recruit students who have gone through a program that uses your ideas so I don't know how you characterize the ROI on that marketing but the psychic income for the instructors has been really great I know that there's a lot of sort of institutional level this increases brand awareness and I'm sure those things are all true I have no idea how to measure that and that, I think most instructors don't have that view of the world so I would like to defer to an administrator who has that view of the world to address that part of it but I think it's rare for teachers to have a voice that could get that loud and get a little bit bluntly and for the ones who are good teachers like Walter Lewin and his physics lectures it's great for them and it's great for the world so if that's marketing that's our view of it I get a sense that universities do see this as an opportunity to here's the dread for word because it's to increase their brand power I am for the ANU actually and I speak to my colleagues and other universities involved in edX and this and I think it's still an experimental space in that sense and it's a question of whether or not it's a direct marketing link to recruitment or whether or not it's about raising the brand profile and awareness of your institution or whether it's about giving some reflection of your on campus experience or whether or not it's about things like alumni and broader community engagement so different universities are using it in different ways so some universities are using it as a marketing activity I don't know how directly you would be able to link it to recruitment though I think that's a while off but one thing that I think is a really interesting development is in alumni and philanthropy and Harvard in particular has gone very being quite creative and big in that space offering alumni only courses asking their alumni to choose what sort of courses they were like taught and using them as an engagement tool with their alumni which in Harvard's case in particular always ends up contributing to their extensive and jealousy inspiring endowment that allows the university to do the work that it does so I think that they have lots of different lots of different potentials and I don't think anyone can tell you straight up this will definitely assist recruitment or this will definitely assist our profile this will definitely assist our alumni engagement as it is in the teaching space it's still experimental space and we're learning but it's a case where universities institutions feel that this opportunity is too valuable not to be in even though we're not quite sure down the track what will happen was there a question over here yeah sorry so I had you so I'm interested so far we've had a discussion about taking existing course and moocifying them and putting them out there but I'd be interested from Amanda and Gabriella about how institutions like Berkeley or ANU might use MOOCs which we don't own but is material which is out there by others and build that into programs that we might accredit and for Ben and Sam what would be what would you like to see if you come to ANU and our programs you're constrained to our programs would you like to see compulsory as part of your degree programs that you should and you should be asking you to go out there and look at other material and accredit a resource which is made available from elsewhere Gabriella so certainly that's where my thinking is I haven't done the administrative details about whether or not it's feasible and all that sort of stuff but it seems to me it's nuts not to do it particularly where the resource is scarce which is in the field that I'm interested where there aren't very many people who do it and who are good at it that you wouldn't draw on the best people in the world as part of your course just strikes me as nuts so I think that's certainly something that I'd like to see developed further Alan do you see that as part of the university I mean if I understand the rest of your question is that do universities now evolve via MOOCs to say well look the education is not just what you're being offered here at Berkeley and you can read widely but it's also you can go directly somewhere else and that's part of the Berkeley experience too I mean in the U.S. it's already fairly common for there to be some level of cross-articulation agreement for certain courses and again I think as with so many other things MOOCs are kind of forcing this issue and they're accelerating the conversation about well suppose I claim to have taken a MOOC on some topic that Berkeley offers a required course on can't I just take some sort of exam or other an interview with a faculty but whatever it takes to prove that I know this material why should I have to waste my time repeating the course and by the way sitting in a seat that could be occupied doesn't have that course so I think as a matter of reality this is going to happen I think it'll be painful for universities because we're large bureaucracies that don't move terribly quickly you know cross-articulation even within the University of California system is already sort of an ad hoc and messy process but I think we're going to be forced into doing a better job of it I think instructors are going to borrow materials from each other and sort of start blending other instructors' ideas into their courses but I think there's also a role for saying if you can learn software engineering better at MIT than you can for me go ahead and learn it and you know Berkeley has expectations of what we're going to assess to make sure that you know it but if you're willing to meet that assessment criterion you know it makes no sense for us to force our way of teaching upon you if there's a better way available also if it works better for you so I think as a matter of reality this is going to happen and you know before MOOCs the idea that you know a student could go online and choose five different versions of introductory physics that just wasn't in our conceptual vocabulary right we live in the age of miracles and wonder if somebody had told you ten years ago ten years ago right I've been teaching only a little bit longer than that if somebody had said within ten years you'll be able to type in a key phrase for a course or a topic instantaneously get links to a dozen courses all for free with videos and interactive assessments taught by the best professors at the world's best universities with no charge by providing an email address and by the way you can also talk to people online and it'll all be free so what's happened at the university Gabriel they do more good stuff you go there to do the real no but it's a you're describing you're describing an institution well you're not describing an institution actually at all we just you're describing the end of an institution you're describing a learning environment which has really leapfrogged the institution the evolution of an institution that doesn't really exist anymore Gabriel do you have any comment on that well I guess as I said earlier I think that the real benefit for universities is that we can put our effort back into real teaching and real learning so it's not the transmission of information that can be done effectively online but what we can do is you know tutorials and practicals and things that become dead because we can't afford them but they actually then become the stuff of learning again and that's and you know what we all of us enjoy pontificating but what we also really enjoy is interacting with smart people and getting their ideas and watching them grow and and you know you go back to having the ability to do that it's incredibly exciting okay question over here general in the red top so just to follow on from this point sort of ten years ago there were five different textbooks available on a particular topic one of which was the recommended textbook for a particular course but in theory students would have had the ability to go and access any other textbook around obviously the internet gives us a much larger set of things that you can look up but the concept of having multiple different resources to look at is not really that new one thing that most of the online resources seem to have in common is short bite-sized chunks of information followed by essentially some exercises to check whether you've understood what's just been presented and I think that mode of learning can have a lot of very good benefits for a lot of students but one of the biggest things that I got out of my university education here at ANU was the ability to teach myself things and I think the ability to pick up a book read through a significant chunk of it and understand what the book's telling me that I value very much value from my university education is there a problem with the fact that most of these mooks are condensed into bite-sized sort of chunks that we're preventing students from learning more lengthy you know from more lengthy broader documents anybody like to take this on could I this was again something that I think John touched on in the last series one of the physics professors here at ANU and I guess it touches on this new concept of flip classrooms and how that also perhaps ties into mooks really providing students with bite-sized chunks of a lecture 15 or 20 minutes that are easily digestible as opposed to one or two hour long blocks and then following that with some form of assessment essentially providing students with a manageable amount of work that they can do in their own time and then working off that I think the difficulty in that sphere perhaps Armando and Gabriel can touch on this is that the amount of work that goes into producing a high quality 15 or 20 minute production is actually quite large so that initial expenditure of time and money into the development of a MOOC or an online course that's specific in that sense of providing bite-sized chunks and relative assessments is quite difficult and then afterwards I guess the updating and upgrading of it in terms of resourcing and multiple resourcing not being a new concept, I think like you mentioned with the internet things have changed drastically and even now I guess if I can draw on a personal experience even with law readings I'm never going to go out and read the multiple and Albert will probably agree that you would never read the other available option simply because I guess it's it's not appealing I think the opportunity that MOOCs provide is it's not only a different resource but it's kind of the best possible alternative to what you're already receiving and it's easily digestible. I think perhaps you touched on it it's really a matter of providing students or those engaging with this new online form of learning with an option that's better than what they're already receiving otherwise I don't think people would look to it as an alternate source of or alternate resource. Can we just put you on the mic to follow that question? Sorry I may not have made my point completely clear. I agree with you it's an incredibly attractive and in the most case very effective and better way of learning things but I'm worried that we're cutting out something by focusing attention on these more easily digestible things. I don't know that I agree with that. I think you're raising some important points but I think you may also be conflating some points that ought to be kept separate. So regarding the observation for example about you learn how to teach yourself you learn how to learn how to go off and read a fairly lengthy tome about something and absorb it. I have read probably 50 programming books but that's not how you learn to program. You learn to program by reading a chapter and then struggling with something to get it to work in your code and there's nothing that can substitute for that experience. So teaching myself how to learn is a combination of digesting someone's explanation of a concept and then trying to turn around and apply that concept in a novel situation that isn't in the textbook. I think to the extent that these bite size boundaries say okay we're going to build up a series of complex concepts that you can do a difficult task. As the instructor I have an opinion about some boundaries I can draw around the concept such that at each boundary we can make sure you're still on board and then when we go to put all the concepts together that's when you struggle with the material and apply it to your own code. I think really a lot of people sort of learn that way now. I think I certainly do and what the MOOC experience made me do is actually sit in the student's seat and say if I were the student where would I be drawing these boundaries? What were the concepts that I had to sort of internally master one at a time before I could put them all together? So I don't think that we're sort of taking away the ability to absorb long chunks of stuff because I don't think that ability was ever really there. I think a lot of knowledge you may think you're learning that way but until you have to actually apply it to a new situation I don't know if I'm convinced that someone has really learned it. I think they read a little, apply, read, apply and they get stuck and that's the point where I think the breakdown actually is useful. I completely agree with you that that's the way you have to learn from the lengthy time but I'm saying are we actually preventing students from learning that specific skill? I think I agree with you because I think we don't know whether by giving people bite-sized chunks and that's all we're giving at the moment whether we're doing something or what is capacity to follow a complex argument through to conclusion. If you think about philosophy or other areas where you can't get the idea across in a 30 second grab or in a 10 minute lecture you've got to kind of layer it and build it up and we don't know yet whether... I think this is one of those cases where computer programming is different. We don't know what it does to people's ability to think for a long time I guess that's the... I think that's an open question. OK. Back to you sir, you wanted to... Ben said one of my degrees is in law, my other degree is in science and in my science degree I do chemistry which requires a lot of like hands-on technical lab skills. So my question is is there like a box of subjects that can be put in, it's like the MOOCs category that can be covered by MOOCs and is there like other subjects that can be excluded by MOOCs and do you think that MOOCs should be developed in some way to cover these sort of subjects as in like particular lab skills and chemistry or I don't know I'm sure there's other subjects that... OK, we can discuss Jane Austen online but can we actually examine the chemical properties of a substance online? I think the... not to cop out but I'd say let's ask that question again in five or six years. I think where MOOCs are now does anybody remember the web in 1994? OK try to... those of you who were born OK or try to remember the web in 1994 and try see if you can predict Facebook right? Technologically almost nothing changed about the web except the computers got faster and more people got broadband but none of the technologies that were used to build Facebook were fundamentally absent in 1995 people just hadn't fought far enough ahead to figure out all the new ways that could recombine them so I think it's kind of too early to say you know the MOOCs of today in five years are going to look really simplistic and that's as it should be so I would say let's get together again in five years and see where the results are going because I think it's too early to call I've got two questions at the back here just while we take the microphone is computing power or Moore's law albeit withstanding is computer power the single biggest hurdle at the moment to the expansion of MOOCs? No but I think it was until a few years ago I mean the idea that you could put video online and have it be distributed everywhere unlimited free that makes no economic sense there's no reason to believe that should ever work but there is YouTube right the idea that computing power in the United States you can rent computing power from Amazon for about five cents an hour for a non-tribially powerful computer so for you know a few dollars an hour we rent enough computers to do the fairly intensive work of automatically grading a lot of student assignments so I think a necessary condition was computing power to get cheap enough and accessible enough to a lot of end users that the fabric was there now that the fabric is there people I think will unleash their creativity on what you can do with it so I think we're kind of just at the beginning of this I'm an optimist just a brief comment that I think some of the issues you raise certainly have been tackled by traditional online and distance learning sort of at universities so I'm aware that at UNE they actually have tested students controlling their experimental sorry their chemistry instrumentation remotely so people can control their GCs and HPLCs and things like that and run analytical chemistry experiments from anywhere in the world so that's quite effective yeah okay we've got time for one more I think do we have one more there one more question if not I think we might wrap up we've had a good grapple with the issue I'd like to thank you for coming and also to thank our guest tonight Professor Amanda Fox from the University of California Berkeley Sam Parkinson from Year 8 it's a bit early to come to university but I think one of the things we've learned from the discussions tonight is that that question itself may be a rather a rather superfluous one too and Gabrielle good to see you here in your contributions and Ben too thank you for your contributions as well and thank you for coming and we'll wrap it up this lecture series for tonight on the subject of MOOCs and where they're going thanks everybody for coming and thank you to the panel