 Everyone, we are back again with our eating up webinar series. Today, we are very much pleased to have with us Mary-Nik Jolavni Hill. She's going to introduce, present as a very interesting topic, something, an area for research and deepening where we are all involved some way, since making strategies for MOOC research. We really much thank you very much for your participation. I know you are at a conference or away from your office and anyway, you are available today to give this webinar with us. I will just steal a few moments to introduce the event, which is offered within the Eden NAP, Network of Academics and Professionals, to its membership. As you know, as Network of Academics and Professionals, we try to involve the community of our members, of the members of Eden Network of Academics and Professionals. Of course, we support networking of individual members also through these kind of events. We try to support effective meeting and communication among our members through different channels. The Eden Chat will follow this webinar at six, so don't miss it. And it is coordinated, of course, by a steering committee that I'm representing, but I'm glad that today also Alfredo Soeiro, who is the Vice President of the steering committee, is there with us supporting Mary Redd today. Here you have pictures and names of the people involved in the steering committee so that you can match your face to the names. We are all involved in promoting our activities. You have also the possibility as NAP members to participate in our platform, so don't hesitate and go to the NAP members area where you can find information about all the people connected there. Being a member includes a series of services and benefits that are listed here, different ones. I just mentioned the fact that you can delegate up to 30 individuals in the NAP and that you can attend conferences at reduced fees. And the incoming conference in Bruges will be a very interesting one, so I really support your participation in that. Our challenge is our involvement is aimed at professional development, first of all, so through webinars, through EdenChats, through communication, through networking events at conferences where we organize this road to an event where people can meet in a speed-up meeting, introducing their own research interests and creating new research groups. And this is very important, I think, when we meet at conferences. And so, I won't join for a still more time, but with this slide I really recommend you to register for our 28th Eden Annual Conference in Bruges, from June to 19th. And, if there is any discrimination in quality or if you hear some random people going by, we've tried to get a place in the hotel which is quiet, etc to do this, but as you know anything could happen, so my apologies that happened. So, the Emux conference was very, very interesting. We did this presentation, Kroger did the presentation earlier on today, and it's part of a wider research agenda that we have in the National Institute for Digital Learning in Dublin City University. We were very much focused on MOOCs and on research in general, but particularly because of this research agenda into digital that we've developed with Professor Mark Brown. I'm head of the IDEAS Lab, which is an innovative unit founded to particularly look at how we can ingrain digital innovations within to the wider university curriculum. So we get to do a lot of research, a lot of testing, and a lot of evaluation and working with faculty across the university. So we have a huge remit in my institution, but one of the things that we thought would be interesting to discuss today would be what we have learned through our last few years of MOOC research, and just to talk about how these things are very, very important. These have linked well actually into some of the main issues that have come true in the Emux conference, which was attended by most of the major platforms, both within Europe and globally. We had Coursera, we had edX, we had fun, we had future learn, we've had a wide range of inputs, and it was really interesting to see that some of the issues that they are now grappling with as they are moving into the next phase of development, and we know that development is linked around issues around recognition, recognition of short courses, then moving into the micro-gardensial space, space, and then moving into full stackable online degrees. I think from our point of view that was very, very interesting because I know our institution is very much going in that way and looking at that, and we will have a major announcement with respect to this in the coming weeks, but to give you context, it was very interesting to hear of the notions of, we know that Class Central in their last research over, they have identified nearly approximately 452 ways that badges and alternative digital things has been done by various stakeholders who are providing digital learning via short courses and by longer courses. So there's a shakeout happening in the field, which is really, really very, very interesting from that perspective of it. And I suppose our research agenda in DCU and in the NIDA is particularly to have, is particularly to look at issues where trying to shape the research agenda. So from that, this is why this research is particularly important. Now, I've spoken a lot thus far, but what we all hope to do is that the three of us will have a conversation. I'm sorry, I've just muted or I've still my video because I'm very cognizant that I'm on a Wi-Fi from the hotel. So just in case I know the audio will go through. So just in case anything happens. So what we'll do is Kruger is going to lead out on the PowerPoint that we have here beside us. And then Elaine and myself will join in with individual research that we have been doing. So Kruger is engaged in research, in his own research, and I might just let him say a few things about his own research agenda, and initially to start off and then allow Elaine then to come in and to introduce herself as well in her research. So just Kruger. Hi, everyone. And so, yeah, I'm a second year doctoral student and I research on the online courses that we have through the FutureLearn platform. My own research is geared towards the psychology and motivational aspects of language learning. So it's quite a specific field in one way, but it also has much broader resonance and is involved kind of usage of a lot of different methods and instruments. And it's been really interesting to actually grapple with those issues and design appropriate research. So a lot of what I've been looking at over the last couple of days has been to do with seeing how other people are researching, what the broader research agendas look like with other institutions. Part of our presentation today was talking about how we're in a kind of certainly unusual, if not unique position in the kind of MOOC that we run, the language MOOC, it's quite interesting and unusual in that regard. And so that's my kind of research, a blend of sort of psychology and then obviously the technological aspects as well. And I work closely with Elaine on that as well. So to bear witness because this is to Elaine and I'm going to freeze. Hello, so my name is Elaine and I am actually a final year PhD student in Dublin City University. I'm very similar to Croher. I'm looking at the psychology of learning, but specifically at emotions and the different emotions that people feel as they learn in a MOOC. It is obviously a very prominent area at the moment. A lot of people are becoming very interested in this in the wider educational area. And it's beginning. There's a good few studies now happening in the MOOC area as well. But kind of different, I suppose, to previous studies is I'm looking at self reports of learner emotions. So really getting it straight from the learner's mouth. What is, how are they feeling as they progress through a course? And also a little bit different from most studies so far is it's really embedded in the theoretical framework that has come from the field of education and that is the control value theory that Pekron designed in 2006. So it's very much relevant to a wider field, but I'm looking specifically at MOOCs and language MOOCs in particular, and that being our Irish 101 series. And I think Croher will tell us a little bit more about that as we go along. And I hope to input with some examples from from my approach as well as we talk through the presentation. So I look forward to that. So I hope that format works and we hope that the video link and that the audio link for today remain constant throughout. From my part of it, I've a particular interest into the macro and the micro implementation of innovation and change within higher education on a wider scale. So I think we have a breadth of experience and research that we engage in. So we're, we'll move on now to the presentation. But at any time, if you would like to ask us particular questions or you would like us to engage with some of your own responses, please use the chat and we'd be very grateful for, we're very grateful for questions on that. And again, if at any stage we fall off the Wi-Fi connection, our apologies. So we'll start the presentation now. Great. So we'll probably essentially follow the way we did present this at EMUX, it was an interesting discussion and we had actually quite a few comments as well. So I think it kind of did its purpose or did its job in that sense. But to introduce the topic itself, so you're seeing a slide here of the first of our courses and it's in a series of courses into the Irish language and Irish culture. They're delivered through the Future Learn platform and there's eight of them at present. Now, we launched them last year in January 2018 and since then they've been really incredibly successful. But also when you come to a conference like this, what's interesting is they're quite unusual. Elaine and Mourade have done an excellent study that actually looked at the number of language MOOCs that exist. And I think, guys, it was about 150 or something a few years ago. But the vast majority of those MOOCs that did exist were English, French, they were global languages. So by my count, I think we're one of only about three minority language MOOCs. There's a Friesian one and a Maori one in New Zealand as well. But I don't think there's another one at present. So we're in a... Yeah, and we're also, when we use the term minority language, we're talking about lesser use language. We understand that there are some global languages spoken by millions of people, which are termed that. So we understand that. So just with regard to the research that we did on those language MOOCs, one of the interesting things was that English, Spanish and Chinese were the main global languages, but it was English for particular purposes in very specific domains. So that was quite interesting. And also with respect to Spanish, I think we felt that from the research and the findings that Chinese was coming through, but it was coming through in more of a holistic way. I suppose with this series of MOOCs that we have relating to the Irish language, they're co-funded by the Department of Culture, Heritage and the Gweiltocht, the 20 year strategy for the Irish language. So we had a huge input, not only from funding-wise, but from our department, our government department. And the reason that they were put out was because of this global connection. So the government saw that MOOCs were means by which we connect to a diaspora as you are maybe aware. We have a huge diaspora of Irish people globally, most particularly sent to and sent in North America and in where you'd expect from England, Australia, New Zealand, etc. But I think one of the things we'll talk about is that a lot of data is given about MOOCs and it's all about how many registrations and that. And in the initial research on MOOCs, it was all about numbers, how many people are registered and the huge figures and that was the thing that was important. But I think what we found in this is that, yes, we've had those big numbers, but when we're going into a lot of the clickstream data, etc. What we've been mostly focused on and what we've been trying to do our research on is the interaction patterns and tying that up with other data formats that have come through the MOOCs, such as discussion where we've had textual analysis, etc. But we'll speak more to that through the presentation. But it's just interesting in that. And as well for reporting, you can imagine that our government is very keen on how many people have registered to do this, how many of them have finished it, etc. And what we've been able to report is on levels of interactivity, but also give them some research into what the global diaspora and people who are interested and who have taken on this, the sort of trends that they have with respect to their participation and their motivations. And I think that's going to have to do with your own research in that space. So we might move on to the next slide. Yeah. And actually, that sets it up really well for kind of what we were talking about earlier, which was when we're approaching from a specific perspective, I mean, we're researching on technology and MOOCs, but we're very well embedded in wider literatures, so linguistic literatures and the like. And one of the most interesting things about that is you can kind of put on a critical hat. And there has been actually a good bit of literature in the last couple of years that we've drawn upon pointing out this clickstream data and this in a sense, and it's something that you even see at the conference, massive emphasis on particular forms of data completion rates often very important despite the fact that there, you know, there's a wider debate about how useful that metric is, but even the likes of using data, these billion data points to actually say, look at what a typical learner is doing. But that it does link through to and it's always scary to use the kind of philosophical words, but a kind of neo-positivistic view where you're sort of measuring engagement in a very specific way, a static way, and that even though we have these massive reams of information, it's much easier to conduct a study in a MOOC in the sense that it generates so much data just through people participating in the MOOC, and both the actual interpretation of the data often when you read a report that's based on a MOOC or MOOC research is often a very narrow framework or it's essentially just looking at X and Y, seeing what the relationship is between the two, but that some of the more interesting things, as Morade says, actually relate to the social dynamics and the interactions and what kind of meanings people draw from things as well. I think that's very important. So we did put in this fabulous picture downloaded from Shutterstock, but just trying to talk about these theoretical bridges. So if you have a massive stream of information on the one hand, and it is in and of itself not necessarily useful if you don't have a particularly salient or good way of connecting it to some broader insight that you're trying to say. And it seems often with a lot of MOOC research also that there's not as big a focus on tying it to the wider literatures that already exist. So we did make reference to Martin Weller study, a very interesting one which was to do with the various literatures relating to open educational resources and the like and showed that if you actually cluster it out by references, citations, the MOOC literature is quite peripheral. It doesn't link through to where there's already been findings, maybe 30, 40 years worth of findings that are probably very relevant in MOOC contexts. So there's a danger as well that if you're not aware of that literature, that you're kind of reinventing the wheel, or maybe you're not drawing from these really rich and existing funds, things that people already know. So I think from the theoretical values are really, really important. And, you know, we do, we have and we've done quite a number of literature reviews, both systematic and scope. And there is, we have a plethora of empirical, of empirical studies, etc, which, you know, we have no issue with. But I think at this stage, as cover said, it's linking that that's the theory is re or two theories is very, very important, whether it's both from an inductive or a deductive process or both. Sorry, we can't hear very right anymore. Something might have happened. As I told you, she's and that's actually she told us she's in Naples attending a conference. And are you there? Something might have happened to them. The connection. Let's wait a couple of minutes and see what happens in the meantime, sort of survey here, which is your experience with MOOCs and actually more than attending MOOCs, which is your experience with design of MOOCs and you use of MOOCs within teaching and learning activities. I hope we're back. Are we back? Can anyone answer? So sorry, anyone answer this question? OK, so apologies for that. It was, as I said, I think we leave the video off because just in case we're running it too tight. So again, apologies for that. But as I said, we are in a hotel lobby, so we're doing our best with what we have. So we'll just take it up from there. I think we could probably go to the next slide. To entertain the audience. It's better that you go. Excellent, OK. So what we had started looking as well was, of course, at these different forms of research. So I think Moraid was mentioning about context and the importance of this and field specific findings and one that helped us as we actually started looking at this has been this distinction between variants and process based research. So obviously, when you see this at first, it kind of sounds like it's very similar to a split between quantitative and qualitative, but it's actually quite different in a sense. And variants, as a course, if we have, you know, could be 100 persons or could be 10 persons, but we're comparing them across case and we're looking for kind of an average and an aggregate and most MOOC research definitely seems to fall into this category. We look at interaction. We look at kind of social dynamics, how people are working with course materials using a variant based logic. But the other one that you can use that's equally valid and definitely there seems to be far fewer studies is this process based approach where you're looking at it actually across time unfolding. Weirdly enough, MOOC runs themselves are fantastic case studies because you could literally have the even as learners start to get used to each other, they start to interact with each other. But it's rare enough that a study actually takes that sort of lens and it seems that and even the last few days we've probably noticed a lot of this is to do with generalizability that a lot of learner or a lot of researchers I should say really want the idea of coming up with generalizable findings that can work across many different types of MOOCs and but there's actually ample space there to try and understand these process based contextual case studies using individual runs and the like. So it follows the course as well that I mean all of our research probably falls into the the domain of mixed methods research. We make use of a lot of different sources of data as we do conduct it and we've got kind of three examples that we'll use and of our research and kind of giving applied examples because it's always better and more useful I think for someone who's listening if we can kind of show what we mean in context so we will talk about that but we'll also talk about the importance of this concept of field specific knowledge so there's not just MOOC research it has to surely tether to something deeper and bigger also pragmatically for researchers I think it's probably better it gives you more kind of ways to pitch your research when you're talking to people as well and to interact with other researchers and maybe find collaborators as well. Okay. So these are our three examples as I was saying at the top this morning we can't claim who have invented any of these terms they're quite well established terms but also as I say talking about them in context might be useful for any other researchers out there also so we'll take each one in three. Yeah so I think I'll speak about the data triangulation because it draws very much on what I'm doing and it gives us I can show some examples from my work on how we've used this to really enhance our data collection until there's so much more than we would have got from just one method alone. So as I said before I'm looking at emotion in a MOOC and getting self reports to collect that kind of data. So if anyone I'm not sure if everyone's familiar with the method of experience sampling but this is when you collect feedback from individuals be that about their opinions their feelings their thoughts at regular time over time but within the moment that it's felt so in an educational context that would be wilder engaged in a learning process or in a task or activity and that's what I did I measured learners emotions at multiple points throughout the course and this was usually immediately following a task in that course. So there was our course itself was three weeks long and so I had 18 data collection points during that course so there was about six per week and these were these were kind of mapped against the different tasks types that we had in the course they were discussion quizzes articles which were kind of usually very text based and what was our other one videos. So they were kind of four different tasks types and I looked at people's emotions during these different tasks types and these were true very very short concise surveys that they answered following those tasks and from that we got a very big response thankfully so for over the 10,000 people who enrolled in the course over 3,000 responded to at least one of those surveys. So a nice large sample from which to interpret the data and they for one example we're referring back to the data triangulation was for a Pacific task learners were it was a video and learners were introduced for the first time to a conversation using the Irish language. They'd been kind of introduced to phrases prior to that but this was the first time that they heard two people speaking the language together. They responded to the surveys the sentiment or the emotions reported during in that survey were extremely positive high reports of curiosity enjoyment pride and these shown through really strongly and that was one that was from our survey results but when we went then and looked at the task or the task itself and the comments posted below that task they were extremely negative and this was not represented in the as clearly in the survey data but it gave us a new perspective from on which to look at this task and really see that it gave us something to think about with regards to survey results but also how people post in a course and maybe it is a truly representative of what people feel are the majority of people just happy to move on and they don't want to post and how much of those kind of high representative are the comments to the overall sentiment with the course. So you can see from those two we had very quantitative survey results for which people just were asked to report from a list of emotions to us kind of doing a thematic sentiment analysis of comments and we got two very different findings from that which both have have value depending on which way you want to look at them and kind of if you repeat that over time it's that's how you get the value of it as well. And I think then you also had a data triangulation designed into your research design. Do you want to have a little chat about how you did that? Yeah, of course. So I'd read actually a lot of the literature about mixed methods and kind of came upon a nice sort of phrase a little bit don't sound at first with this idea of using them dialectically. So that means that when I was designing my survey I knew straight away that it's really important in any sort of survey designed to really account for the fallibility of an instrument. And when I say fallibility I mean it can be statistically valid. It can come out with some great results. But where it gets very interesting is if you give people the opportunities and responses to actually comment on the survey itself. So for example at the bottom of the survey I specifically asked for feedback were there any questions that you thought were a little bit off or anything that you'd phrase differently and ended up with really a wealth of information from even little things that are actually very telling about the research. So for example I had a question at the start about what one's nationality was and I had a couple of responses from people saying that they were kind of offended that they could only pick one nationality because you had a lot of people who said well I'm Irish but you know I've lived in England all my life but I'm Irish so I should be able to use both. And that's very interesting straight away about the relationship between for example the learner's national identity and who they are and obviously in a motivational study that's very telling sorts of evidence and this is this kind of supporting evidence argument as well. And but also from the start I used kind of like a battery of responses to kind of measure people's motivations but I also left open boxes at the end aware that any scale I could create would not encompass every motivation there'd always be ones and so when I thematically analysed all those open responses came up with some really new questions that were then filtered into the survey design for the second one specifically because I was kind of saying from the start this is sort of exploratory it hasn't really been done before we kind of know a lot of reasons that people have been doing it but there's bound to be novel ones things you would not expect like people doing it for health purposes where they're talking about you know Sudoku I like learning a language that I find kind of difficult and I'm doing it just for fun I'm doing it because my grandkids can speak Irish they go to an Irish language school and I want to be the cool grandparent that was an actual response and so everything under the sun you can imagine it was there and it's important to account for that so that's another form of triangulation where we kind of exploit the weaknesses of the methods because for example with a survey what you ask is what you ask you don't get another you know chance to expand or elaborate on the data you have and that might be why data triangulation is actually really really important that it gives you that opportunity to delve into you know developing our teams etc that have horizon from one method which is very very important I think for my own research point of view and some of the research I've conducted related to this and policy implementation is that independent of whether it's textual data that you're mining or whether you're looking at interview data is that that triangulation is a really really important feature so I think for us in the MOOC context we decided that this was the an approach that we should use to validate not only the types of questions but also to allow us to increase the types of questions that we are answering because we wanted to ask a little bit more complex than what pure survey data would allow us and you can even from the types of things that Elaine is doing and what Corcoru is doing is that there are areas that yes you can have a very you can use a survey instrument and you will get that as you said today that table at the end absolutely and it's underwhelming and it's underwhelming so that you're able to sort of drill down more into that that was very very important but that also gave you the ability then to ask not only the question which the table represented but then to have deeper questions and I think when we were looking at MOOC research and doing our literature reviews and both of you have done quite enough in depth literature reviews and you're in the two areas we're looking at it's that there is an absence of that and that some question types are not being asked so I think data triangulation from that perspective is allowing us to ask is allowing us to engage with deeper questions and I think broader questions from a research point of view absolutely so we'll just move on to longitudinal research do you want to take this for sure so obviously the word is used a lot and I mean it literally means looking at something over time and but essentially the way we were looking at it I think from the start is this concept of change researching change and similar to kind of what we were talking about with the survey instruments anything you take if you take a single survey instrument is a snapshot at that very particular moment so that's that notion of as opposed to me change it's changing as it's an active form exactly and that's what's really interesting too is that and even when it's tracked over the length of a course or over multiple courses and that's where you definitely do see a lot of kind of longitudinal surveys studies which are very interesting but the way we were kind of looking at it too is we've noticed even on our own MOOCs when you see the same faces the same names over and over again and another way to look at it longitudinally is to flip it around and instead of saying you know a thousand learners this is server response at XYZ point individual learners and track them over time so for example I've only interviewed one person so far but I'm engaging in interviews and part of the plan it's about their motivations to earn the language but part of the plan I actually have is to get back in touch with them in maybe six months time because I'm really curious we have a series of MOOCs so they get kind of progressively more difficult and I'm curious to if these MOOC learners and the one interview I did do for example she's very positive she was ready to move on to the next steps will she actually be doing it in six months time she's atypical in that as we were saying about completion rates and stuff most learners don't in the kind of technical sense complete a MOOC there's a lot of literature as to why it's problematic to even look at it that way but most don't but the ones that do are actually kind of probably very interesting and unique just in terms of they're not being that typical and so sometimes we're kind of we always want to generalize our results and that is very important but the other side theoretically is actually the outliers or the less typical persons are actually also really interesting and we had a great discussion and actually a question came from the end about this concept of broadening the focus from say the platform activity the clickstream stuff what they engage in to actually looking at beyond the MOOC does that influence in my case say their motivations you know talking to people around them and things yeah and I think that's the point I think that we're trying to get we're not criticizing the studies that have taken place to take because in every field and in every burgeoning field there is a sort of a you know what seems to be a pathway that people follow as researchers follow as it becomes more come as the research becomes more refined and I think we have you know we don't and I think we should be cognizant as researchers in MOOCs that we're not a historical and we're not a contextual and we need to understand and that's why the broader focus and if we know that learning happens not just in the formal learning environment we can't just disaggregate that from the lives of the learners and also from the wider social context in which they are in and also from where they have come so I think that's it and just coming back to one of the points that Kuhn had made earlier I think we have in our Eden paper that's in Bruges we're looking at the serial MOOCs and the serial so you'll get to hear and plug in our own paper if you want to come self-promote and ask the Bruges conference and we'd be interested to hear your thoughts on the paper that we've developed looking at and trying to track learning over a MOOC series as opposed to just an individual MOOC we know that it's not just an individual MOOC because we are quite cognizant and we'll be I suppose theoretically we'll be aligned with steak for where you know where you case study and instrumental and intrinsic case studies and within that we would come from that sort of I suppose methodological standpoint and tradition so looking at the next slide and I saw the iterative for just redesign I think that's key isn't it really? Oh absolutely I mean and it's kind of it's something that I know that I can speak of as well but it's been so interesting seeing the talking about something frankly and organically as well so you'll notice that there's a little Irish there and that's a Shanukkel which means a proverb Neil Siegun lucht there's no wise man or woman without fault and that's an important one to for any researcher to probably bear in mind is not necessarily about a humbleness but it's about the fact that the way we actually move knowledge forward is by acknowledging where we didn't do as well as we thought we would or where you know we ran into like road blocks or speed bumps or something and we also like this concept of the logic in use that Maxwell developed so he pointed out and actually it's a lovely paper if anyone wants to look at the reference at the end he pointed out that whenever you're reading a study what you essentially get on the method logical side is a reconstructed logic it's a story of how it was done and it's very clinical it says everything about this was you know XYZ everything's very neat but that's almost always not exactly how it went down because the logic in use was that you made decisions at certain points you might have to chop a question out if you might have to looked at a different method of data collection that this logic in use is actually in a way the more useful for researchers because it shows that problem side also but it does tend to get cut out I mean often studies don't have enough space to kind of talk about these issues but where there's real kind of value in these kind of papers particularly in a budding field like MOOC research is that a lot of the kind of problems we face are probably quite similar you'll notice probably survey fatigue and respondents just won't maybe answer a long instrument there's actually probably a point an inflection point past which they're just not going to you know their legs that they need to serve and yeah so we've all kinds of like protects your own but that's a really cool thing though and I suppose with all the different survey instruments that we have that we can we now know where enough is enough hmm which question almost yeah and what questions they're falling down on so it gives you that iterative redesign where you can sort of say okay well let's really think about my survey instrument do I need to have those sort of two introduction questions or should I just cut to the chase and get to the really these are the really good stuff so I think that's been important and I know on our third point here I think Elaine has particular to just like talk to platform limitations because when we were going initially on to the I think we were one of the first universities to engage with this type of research and that was challenging for our platform providers yeah definitely it was something because when I when you come up with a research idea you have this idealistic view of how you want it to work and how you see it happening in the course and when I was coming up with my research design the course itself was only in the development phase two so it was a learning process all around but what I wanted to originally do was to have the surveys as a pop-up that would appear in mid-learning tasks so they'd be embedded in the course itself and they'd be it wouldn't rely on someone to have to do anything extra to answer the survey they wouldn't have to click somewhere else they wouldn't be moved off the platform to answer a survey and I think that's key when you want to get in the moment reports like I was I didn't want to interrupt the learning process I wanted this to be part of what they were doing and to be nearly a reflective step in what they were doing but because of the way the platform was designed this just wasn't possible and it was also the GDP or thing yeah so we had to have so many we had to have a very lengthy disclaimer in advance telling people that they were leaving the platform that they were now answering a survey that belonged to Dublin City University it was no longer part of the platform site and while this is all understandable it did inhibit the way that the survey came across on the platform it looked a lot more substantial than what was originally intended for it to be they probably spent more time reading the disclaimer than they did answering the survey oh they did yeah absolutely and so that was and it just it kind of it interrupted what I intended to be a very seamless process and I think it's something that we all kind of have to and platforms need to address if they really want research and more research that doesn't just associate itself with the background clickstream data if we want to hear from the learners themselves on the learner for you to sit and what we've seen in so many of our comments is that learners want to be heard they want to give us feedback and we very much phrased this the surveys that we conducted as can you help us can you give us something that you want us to know so we can help you and that was kind of and we really find that there were so many people more than willing to tell us what worked in the course what didn't work and many times unprompted in the discussion steps or just the common forums under each task people would converse and they would have discussions among themselves encouraging each other and not so after reporting what they felt but what I think is really interesting is that going back as well to what Cara Hur and Merade were speaking about there as people don't talk about often enough what doesn't work and it usually stops other people from making the same mistakes we were all tending to report on how great something worked out instead of saying well I actually didn't intend for it to be that way and that would it opens up conversations to a wider view and we know that there is a publishing bias against that type of framework I think there was a very good article recently that actually mapped that out where the positive findings were significantly more likely to be published than findings were indicated well there is no significant difference or they didn't do their hypothesis failed or whatever so I think that's a really, really important just to I suppose link into the Emux conference here it was quite interesting to hear and I don't know whether of course this could just be the rhetoric of the major platforms you know talking about they were going to look into more of learning behaviour and learning responses and they were certainly from what and it appeared that they were echoing each other so it seems to be something that they must have got back in the service that they are conducting and the engagements that they are having with learners because it was definitely a clear message that they were trying to communicate to the MOOC research community I think in the last day they don't know what you're talking about that was Yeah no it definitely seemed to be coming through that we can't just go off what we think we know we really have to open it up now and see what learners want from these things in any kind of field if you're looking to provide a service you have to hear back from the people that are using that service and that definitely seems to be the way that the MOOC platforms are now moving we have the basics now we know how to make the courses we know the kind of those basic fundamentals we now need to go beyond that and look at the learner but in that way it hasn't been that this has been happening in other areas either like things like emotion research are only coming to the fore these haven't been addressed elsewhere either but I think it's really important that we do jump on that trend as it comes up because this will help in so many ways to support the learner because it's like a 4G experience it's not just interacting with content it's enhancing the whole experience for the learner and I suppose it behoves us in the MOOC research community to keep pushing the boundaries with the platforms I know that when we first had the initial conversations about the type of research a lane particularly one to conduct it was challenging and there were tensions there because we were asking them when we need to have this particular design in place and we went through a very stringent ethical process on our review board in our own university but I was delighted that we did that because we were able to prove that what we were trying to do had value and was ethically sound and of course that is a very important key part of any research design and I think a lot of by facing challenges it doesn't mean you should stop because following what we did the platform has changed a lot of their disclaimer policies because they realise that more of this needs to happen and until you approach them and explain why this needs to happen a lot they just aren't, if enough people say it then it'll become more of a conversation that needs to be addressed but I think it was very, it's interesting though because platforms or any commercial platforms they're looking at the bottom line they're looking at selling the certificates people upgrading, etc, etc so whereas if you, it's simple it's very, very simple if you're the person who is engaged and it's the learner, it's satisfied they're more likely to stay on the platform then they're more likely to engage in other courses so they see, I suppose, the monetary yeah, it's the interest of the line they're much interested in the line and it just behoves us as researchers to keep that in mind and not, well not keep it in mind that they have a stake in the game as well but also that we have, from a research perspective we have a stake in that we are pushing forward the most robust and rigorous research designs where for particular questions and I think that is just something as research community relating to MOOCs that we have to keep doing and that's nearly sometimes independent of the platform but though it doesn't matter which platform it is but we just need to keep doing that and I think that was clear from some of the presentations that were done as well that there is a little bit of a movement towards that absolutely okay, are we yeah, I think we've done one slide yeah, so we've done one slide for implications and conclusions and I know we've chatted quite a bit so if anybody wants to throw up a question or anything, I know we're coming towards the end we'd be very happy to do that so I heard you want to yeah, so just to conclude kind of, I guess it's kind of a lot of this is what we have been saying but I mean, it comes from our experience or 15 months worth of kind of interesting journey and research is the first point is definitely about this role of theory that it is very important and I do think, I don't know what my colleagues think but I do think that my opinion anyway is that when you do read a lot of MOOC research there just isn't a strong theoretical framework or it's ignored or it's ignored exactly where it's not it's considered secondary or that it's just if some case is taken for granted that people just know that it happens I find that a lot with a bit of motion research was that people would make statements such as emotions play a significant role in the learning process without building on that or saying how and why do you think that yeah, or even why it's desirable like because you kind of got to think at that level having it as and tease it out properly and you know because then you actually function as well like yeah, you know I know it's so important and like this and this is where we've had an advantage probably is because we do read a lot on other subjects so it actually bleeds in quite well to the MOOC stuff because you see these kind of debates they're already happening in other environments there are a lot of these questions the things we were saying there about process of variance like these are foundational kind of social science questions across the board like nobody agrees necessarily on these so it's kind of interesting to see that process but it does appear that if you were looking at the literature and that Valetiano's paper is a very good review there does appear to be a heavy weight in the one direction which is that kind of variance-based thing so I guess the second point follows kind of naturally which is the importance of methods and also to look at some of those alternative designs that probably don't appear very frequently case studies they don't appear very often in MOOC research thematic analysis of comments is something that I would have thought it's like very easy to do and quite kind of foundational because you have that information but it's rare enough that you see that either or a real CAO exactly yeah and even like frameworks that you may not even necessarily agree with but like discourse analysis you could use that with you know comments and conversations so these things about the questions that aren't being asked as well is probably an interesting direction yeah and I think you know I think today there was a you were at the presentation or was it yesterday about the you know that there could be AB yes on edX on edX and that was very interesting because on some of the platforms again it's the tension at the platform infrastructure and limitations but they've come across and they're now moving into where you can do that type of control testing and I think that's going to be even from a variance perspective even if it's that it's going to be actually fascinating to see what comes out of that and the people and the universities that are using that for research how what they'll come out of it and will they change their questions so like will the questions get any more in depth I don't know will they just improve the quality of their findings this is a big question isn't it and I mean it does and it seems to come back as well to the functionality of a lot too which is you know the great thing about having a theory is you're feeding back into something course you know the study's not just about well I've shown the link between X and Y that's often very important but what does that link actually mean and are there ways to make does it follow that that makes something better or you know what are the kind of criteria this is something that actually came up in a research seminar that we had in the NIDL the National Institute for Digital Learning and Professor Mark Brown actually drew on this point where once you're finished a study that shouldn't be the end of the questions you should only be asking more questions then or what questions does this study now raise that I know this what else should and that's really it draws on all the different points that we've made on getting different types of data drawing out a process much longer as long as you do not research and also doing the iterative redesign of new questions where are we going now and that's kind of what that is not what questions are we not asking and why aren't we asking those questions and are they important are they not and I think one of the issues that came about one of the last sessions was you know the notion of scalability and I think that's where the MOOC platforms are in their head with regards to a lot of these issues so they're looking at it from a point of view of how can they scale even more complex learning experiences say in discipline where you have both the knowledge and the skills which is very much in language learning and how that can scale up to in those disciplines so I think that for us will be another interesting way when they start coming to grips with those type of issues because then of course then the learning design will change and then the learner experience will change and potentially then you know the areas that we're interested in the motivation etc that that will also change so I think you know we try to give you a flavour really of the research that we're conducting and also try to include some of the highlights and you know where the platforms are going because the platforms are leading us as well a little bit in this way because how would they develop and how they we know it does have but it does bring to bear and you know an influence on some of our research and how we do it so but I think interestingly enough I think it's always very clear you're on the right way when a platform tries to copy what you're doing or start getting very interested in sentiment and emotion because they they know that this is an area that they're going to have to think about because at the end of the day particularly the commercial ones they do all have a monetization agenda whether it's they are businesses you know Harvard and MIT told I get X immediately from the gecko that you have to come develop that sustainable business model and you know now with future learn with their venture which you know that they've got funding from and from seek we know now that they're also very much into looking at how their business model is going to be and of course even though all of them all of them and I say that they have a very good and that they're one of their rays on death is that of access to education and providing education at the end of the day they are businesses and that so we have to be very cognizant of what they're trying to do as well so just a very quick girmila mahogany which is thank you very much and if anybody has any questions we're happy to take that and take them and will I hand back to Antonella or Christine or are you okay so I don't see any questions coming oh see Monica in the chat you've had a few and come up I'm sorry and just apologies again for the break in the connection but hopefully you have a back and everything has been going fine so I think Monica has I think we need e-learning for our students and teaching process because we respect their time you know so thank you so much thank you so much thank you thank you for your availability in you know in difficult in a difficult situation for connection and for being away from your office but notwithstanding that you are you have been wonderful you and all your colleagues PhD students and so on and so forth but now I really would like to remind you the Eden chart at six so please if you have other you know thoughts or in these two hours you have the time less than two hours actually you have the opportunity to reflect on what we discuss today with Maria and her colleagues don't miss this opportunity on on the Eden chart where Meredith would be available again thank you so much for this hard day with us and I also would like to remind you that next webinar will be held and hosted by Alfredo Sorreiro and again a very interesting opportunity for all of us it will be devoted to portfolio and how to build a portfolio you know technologically supported anyway thank you so much to all of you and have a nice day don't forget also to register to our conference in Bruges Bruges is fantastic don't miss it here we have we have the title for the next webinar so I want to remind you the webinar hosted by Alfredo Sorreiro on June the 5th same time 330 Central Europe summer time possible use of a portfolio as European Development thank you so much bye bye