 Well, welcome again everyone. I think we will get underway now. It's now five minutes past the hour I think we've given most people a chance to come online We are of course recording today's session. So if you have to leave at any point, don't worry the Video will be available. This is I think the third in the education in a pandemic series of Eden webinars Eden the European Distance and e-learning network One of the peak bodies in Europe for the promotion and support for new approaches to teaching and learning using technologies. I Think today's topic couldn't be more timely One of the urgent questions facing us as educators today is how do I? Probably design and manage assessments in online learning environments Or perhaps that should be how do I manage alternative assessments in online learning environments because we're well aware that many of you Have had to generate new forms of assessments in life of the challenges We all face and in particular the difficulty of running traditional examinations We want to explore a number of questions But mostly we want to listen to and hear your questions so we can respond to them But certainly issues of academic integrity will no doubt come up How can we ensure that our students are not cheating when we move to these alternative assessments? What measures can we put in place to ensure that learning is happening and to assess it effectively? I think rather than take an approach that's a little negative We would want to focus on a much more positive educational opportunity that new alternative assessments provide So I think without further ado what I will do is Give the opportunity for the panel to introduce themselves, but if I haven't already done so my name is Mark Brown I'm the director of the National Center Institute for Digital Learning based at Dublin City University If you're wondering about my accent I'm originally from New Zealand so that helps to add a southern hemisphere perspective Although I'm sure we have some colleagues I see from South Africa here And hopefully from Australia or New Zealand as well The one thing just before I introduce my colleagues again It's just to reiterate that in your questions rather than use the chat box Can you please use the Q&A tool as that will help us collect them and make sure that we respond to your questions as much as we can So without further ado, it's my privilege to introduce you to Frado Who's going to just introduce himself for a few minutes rather than me talk about him I think it's probably here better to hear first hand. So Mike over to you Yes Good afternoon or Good morning because there are some people from South America My name is Alfred Sweron from the University of Porto. I'm a member of the executive board or executive council of Eden and I've been working on the learning On university life and learning for a long time And I think that I have to congratulate first Eden and the president and the Secretariat and my colleagues in the board to have chosen this series of webinars especially this topic I think it's really important. So I will start talking about Perhaps before you start could we introduce the other two panelists sure we have and then we'll hand back to you Okay, perfect. Oh, so Lisa, would you like to just give us a little introduction? Hi, Mark. Thank you. Yeah, my name is Lisa Marie Blaschke I'm a Program director at the University of Oldenburg at the Center for lifelong learning Where I'm responsible for the management of technology enhanced learning program Also a program in which I teach. I was a former member of the executive committee for the Within Eden and I'm currently a chair of the board of senior fellow of fellows for Eden So that's all there is for me. I'll pass it on to art or no Thanks, Lisa. Yeah, so I'm Orna Farrell. I'm a colleague of Mark's from DCU I work in online programs there. So our DCU connected. I Suppose platform and I'm program chair of three online programs in the area of humanities I also do research in the area of online student engagement e-portfolio and teaching online So back to you mark Right. Well, thank you for the brief introduction. They're just helpful to contextualize each of our presenters will just talk for about 10 minutes and any longer they'll get the evil zoom I And then the plan is that we'll try to devote at least 20 to 25 minutes for Answering your questions with the size of the group I would just ask you again post your questions in the Q&A tool lots of questions are stacking up there, which is great So without any further ado, I'm going to hand back to you Alfredo and you can start off. Thank you Okay, so I apologize for breaking what we had agreed but it comes a little bit from Let's say The intent of speaking about something that I'm very interested in this question of management I think it's one of the most important issues in learning and especially on online learning Currently in the days that we are leaving that it's really important to Correctly assess What students have learned so I have just two topics and They are different one of them is how to help the assessment choice And I will talk about Web tool that it was produced on the European project After a thesis produced by dr. Hit the Falcone She's she was the one who made the model But you'll see how the model was implemented and how you can use it to find your correct assessment And the second part of my presentation is about an experience I've been conducting for some years this last year with a certain variation which is the use of e-portfolio for formative assessment online and It was not done online. It was a blended experience, but it can be reproduced easily online. So that's why I bring it here and So like I said the project That I wanted to talk to you about was distant and five years ago now. It's probably very useful for some of you and What was the purpose of this European project? We wanted to know What the students have learned and for that we needed to align In some way the correct assessment with the learning outcomes as you know the learning outcomes or Competencies as you want to call it. They are very different mostly we assess knowledge But we probably need now to assess skills because we are online We also need to assess attitudes and we have to find proper assessment for these three types of Learning outcomes or competences and that's what the project did And you think it's simple? Probably not It's something that it's really difficult We are choosing our assessment methods based on mostly On our experiences student and the teachers, but we need a rational approach to this To this choice or this alignment between the different type of learning outcomes This is the web tool that I want to talk you about it's public. It's free and Anyone can use it and let me be clear. It's just an advice. It's not black and white It's not mandatory, but it's an advice and it's an advice based on You can see the model that was behind this project, but as you can see on the last Oops, sorry on the last Row you can see that it's based on the revised balloon taxonomy that several types of perspectives of cognition and You what you do in this tool is just write your description of learning outcome and When you write you choose the dimension of the revised balloon taxonomy There is more appropriate to that learning outcome. That's your decision. Of course as a teacher That's your decision and then you'll find on the last box You check the assessment methods that are proposed Let me tell you that at the time it was very difficult to find the proper Procedure to do that currently with artificial intelligence probably to be easier, but at the time we used Matrix that was multi-dimensional and We use the theory of Anderson at all to align the learning outcomes with revised balloon taxonomy and This is an example. For instance, this is a decision-making from a project Where there is a framework for the Competencies of civil engineers in Kalohi probably some of you know about it But for instance for decision-making in that project, you have a descriptor that has three dimensions knowledge skills and attitudes and how can we evaluate as you know most of them of the Exams or assessments that are made are written. So you have Multiple choice or you have a set of questions that you provide to the learner, but do they are adequate to evaluate attitudes or skills so This this last row are the three methods that the model proposes as Probably the best ones to evaluate The type of learning that we are intended to do with the tool you have the address if you want to know more There are several examples. I mean the partnership worked very hard It was a very interesting project one of the best in my life and I enjoy it very much That's why I'm talking about it because I think it's useful and proper in these days the other Example that I wanted to to bring to you is the Deplication of e-portfolio. This is something that I've been done been doing for years these are on the last year of a master that we have for professionals engineers and This time We made a Very big change. Let's say that the evaluation of the e-portfolios was not done at the end of the semester But start I started doing it every week. So The students had to put in the e-portfolio There what they think Where the competences they acquired in that week they had it to justify with documents and other Thoughts but mostly they had to do something that it's not very usual in these e-portfolios They had put the percentage of them of the competence that they had acquired Some of them were of course very optimistic a hundred percent in all of them But then I had a weekly meeting with the students to discuss what had they done and It was very useful for me to understand the benchmarking of the plan of the discipline on the other hand to follow up The progress of each one of them and with some of them I had to tell them you have to look into this in that book or in that article or that link in the web But you have a problem with this Competence so this is what we did. There are some results that I think that are interesting from this application I was just reading a paper from Orna that says that the e-portfolios were a failure in Ireland Maybe this is something that I haven't seen so maybe she will change your mind. We'll see but anyway What happened in this e-portfolio is the teaching with remediation because yeah, I really did it The e-portfolio is compatible with aeropast the new aeropast that will come in fact very soon The conscience about the choir competences I was in an engineering school where the competences of the program were written even in the rooms in the classrooms All the competence they were supposed so they know what they are there They are not there because we want to torture them. They are there because they are supposed to acquire some Competences and that's very interesting with this proposal. Of course Reflection about what they have learned and it's transparent when these discussions every week It's transparent what I'm supposed to do what they've learned and how I can assess what they have done because this Is very good to assess what the students have learned And just some final remarks. It's in a minute mark. I'll finish in a minute And One of the things I've learned through all my years. I'm not young. I'm very relatively old is that assessment is very teacher dependent and Teachers are very possessive about the assessment. They are very emotional when this discuss assessment It's not rational the discussion about assessment. I don't know why I'm not a psychologist as a sociologist But it's true another one is that We don't have many assessment methods based on rationality. You need something that it's rational that at least Aligns the learning outcomes with the assessments we choose another one Like I mentioned already is that we have different type of competences or learning outcomes This is very used we use it in professional terms for Finani some of you may know, but it's the Federation engineers doesn't in Europe and we use it because it's Very useful also for competences besides the learning outcomes another one is that We probably are not doing assessment according to the learning styles and the portfolios give us more Knowledge of what is the student and how he learns? So I think it's very useful from that perspective and the last Is that this question of assessment and accreditation and qualification and validation of competences of people? I think it will be the main function of universities, but at least the crucial function of universities Let me tell you a story if I can it's only 30 seconds Okay, 30 seconds. It's I belong to a council on engineering continuing engineering education and once the The director of Stanford the CPD of Stanford arrived in the breakfast and he said 10 years ago or 12 years ago All our courses are online And everybody at the table said what now Stanford courses are online all of them Yeah, and how what is the business model? Well, the business model is that if you want anyone wants to certificate from Stanford that they had the Competences of that course they will go to Stanford and pay So think about it and that's about it. Thanks mark for the patience Well, thank you very much. I've just been keeping a watch on the questions coming through And there's a question there, which I think we could probably just answer right now Around whether the first example you showed the tool is still freely available And if that's you could post the URL with more information Okay, it's in the It's in the the slides which I think will be available afterwards, but I already done for all panellists. That's no problem That's great. I think rather than taking any other questions now We'll just continue on and at this point Lisa. I'm gonna hand over to you Great. Thank you, Mark. Let me put my slides up here quickly everyone I'm gonna talk to you about What we've done here at the University of Oldenburg and and prior to that as part of the Part of the Masters of Distance Education e-learning program at University of Maryland and that is use of Portfolios and portfolios for assessing learners and applying a backward design in doing so Now the master of technology enhanced learning mTel program the focus of which is developing skills and competencies to support the implementation of Flexible and media supported educational projects and programs within organizations and the course where we applied the backward design was the theory principles and history course Which is really the basic course for for the entire program and from their students move on to learning more Basics their courses in research Learner support the design of courses budgeting globalization and then to management And so in focusing on this first course, which was the foundational course We really had to think about what kind of learning outcomes did we want to have just not for not just for this course But for the program in its entirety So what is backward design that the approach that we used is the backward design backward design approach And this was really starting with the end in mind. What do your students want to learn? What are the key ideas and concepts? Really focusing on on what do I want to achieve at the end? And then how will I have students achieve these desired learning outcomes and how will I assess these? What kinds of rubrics will I use and then how will students achieve the desired results and the knowledge and the skills and the way that we did this? We first looked at what kind of learning outcomes did we want to have and these were things like you know That they have a knowledge and understanding of technology enhanced learning that they that they can apply it within their work environments that they could use use technology for learning and then that they and then that they complete specific learning activities To be able to achieve that and that so moving from the desired learning outcomes We first defined what it is We wanted them to have at the end of the course and even at the end of the program And then moved into what kinds of assessments would we use to assess whether or not they've they'd achieve that and then Specific learning activities so the specific learning activities and the assessments It was important for us that our students because they're within a technology enhanced learning program that they also learn a lot about different technologies Now I'm not going to go through this entire slide because we really don't have a lot of time to do that You can go back and look at it later We identified a number of different learning activities that we could use within within the within the course Things like a reflective learning journal Different things like connecting with the research community using an online mind map for the personal learning environment working on a group project That that sort of thing So looking at the entire course at the end we organized it according to waves of distance education and e-learning and how it's developed over time and we looked at it Not in terms of the eras but really as waves of development over time and during that time During the as we move through the course the students developed a group grid where they were looking at all the different factors that were that that emerged in each of these different waves things like societal political technology economical factors And how those influence the development of distance education e-learning over time and then they also had to do follow a researcher within Distance education and e-learning over twitter or some other form of social media and report back to the class So they would start developing their network over time then during that time. They'd also Keep a reflective learning journal which documented many of the things that they learned how they learned And then those would be reviewed by the By the instructor now these were all past fail formative assessments. It wasn't required for them to get a grade on these particular activities And and it wasn't until they submitted their final portfolio that they received a grade So past fail was the grade that we gave for the many learning activities They did during the course and then the final portfolio, which would be their final reflections their personal goals would be in the final In the final grade and then participation would be a self-assessment activity on their part Now this is an example of a portfolio that a student did a paper portfolio She's put together an executive summary talked about course learning outcomes talked about key competencies that she's achieved And then she includes all of the different learning activities the results of them in her portfolio This is an example of the e-portfolio where student the students actually moved this portfolio into an e-learning environment This reflects what helen barrett talks about the two different phases of portfolios one where it's more of a reflective activity and the other where it's more showcasing your abilities So why did we use this approach? And I think this will be helpful for you if you're thinking about assessments and how you want to redesign them for the future And in your online learning environment It it really focused on what students need to learn instead of focusing on okay I've got to complete all these tasks. How do I do it? What are the key topics? What are the most important themes that they need to need to learn? It emphasizes formative feedback and development Giving a pass fail on the different learning activities It also draws on their intrinsic motivation because what we did was we defined unique and contextual learning activities for the students In addition, it encourages reflective practice where they have to really think about What they've learned how they've learned it and as an instructor it gives you a holistic view of the student You see how the student is interacted within the classroom You get a really good idea of of who that student is You get a picture of who that student is and that the reflective learning journals is another way of really getting a Another way of communicating with students It also helps them to develop their digital skills By using digital media and it gives them a platform to showcase the competencies and skills they've acquired Some practical guidance on how to use e-portfolios and the backward design and this kind of learning learning activities Get to know your students find out what it is that what where where they're going to need scaffolding where they're going to need help Create an environment where your students can explore create collaborate connect shared reflect And it really should be a place where they feel safe and where they can try out new things and where they are really involved in In in designing their different activities Or their different assessments Use the portfolio as a place for students to explore and reflect Give them an opportunity to think about not just what they've learned, but how they could apply it within their context Uh, it makes sure that you design learning activities that can build on skills and competencies And then the really the really important role as an instructor is to guide them with your feedback and with example with your example keep the learner at the center and make the portfolio the centerpiece of of the course And then here are some resources that I pulled together that can possibly help you in your Future assessment development when you're thinking about what kinds of things What kinds of things you want to do with portfolios and also there's a number of Examples here that you can go to links that will that will guide you in getting your online assessments Getting your assessments into the online learning environment Getting your assessments into the online learning environment. So did I make the 10 minutes mark? I think that it was fine. You'll you'll still get your badge And um again with one eye on the time that we want to have as much time for q&a I'm going to just go straight on to uh, orna because I know she's going to have our Element of portfolios and then we launched straight into question time after that So orna, sure. I'm just going to share my screen I actually don't have as much portfolios as usual mark So hi guys, uh, I'm dr. Rowna Farrell from dcu for those of you who've joined a bit late My handle is there on twitter if you want to get in touch or my email um, I was thinking about oh, uh, I was thinking about this This interesting pivot or whatever people are calling it right now It's very complicated, but in some ways it might be an opportunity So I picked that Churchill quote to kind of illustrate that so out of intense complexities intense simplicity emerges So maybe this is an opportunity to kill some sacred cows Get rid of some dead wood uh And and the sacred cow. I think we're all thinking about is this um And the question I was thinking to myself about this was do we really want to replicate this online? I mean, is it that good that we want to keep it? um What do you think? um You know, how do you feel about the current exam structure? One thing I find about exams? There's a lot of ritual and process around exams bureaucracy checking papers security And is it worth all that effort? So I'll be curious Do we want to go for this proctored version of the same thing or should we take advantage of this uncertainty? And to get away get rid of the dead wood. So I'll be interested to hear what you think Um, one thing that was interesting There was a great event online that maha presented at and mark as well last week called gas to goes global But Sheila McNeil made a very interesting comment and it stuck in my mind And she said do we really want to create an atmosphere of trust or surveillance? So if we go down the online proctoring road, is that just further surveillance? Or do we want to go down the more developmental trust road? So I think that's an interesting quest question to think about Here is a lot of gifts um So again kind of high level concepts and then I'll get into the kind of more detailed assessment bit But uh, this is a very nice framework by cahoo about student engagement And and I adapted a little bit to think about some of the Some of the factors of covet 19 and how that might be affecting student engagement um, so she is kind of four different categories of of um Thing Uh over on the left there you have structural things. So there are things like life load and I really like this concept So this is the sum of all the caring responsibilities a person has it could be children It could be older parents if you name it it could be all of those things And I think everyone uh, everyone's life load is considerably different right now And I think we need to think about that, you know, uh, they could be frontline workers Their their spouse could be a frontline worker. So there's a lot of issues around How much time does that person have? Um, the other thing is institutional supports a lot of these are done in bricks and mortar still Even if your library is good about buying ebooks and good about subscribing to online journals There's still going to be a gap for the students. There's still going to be things they can't find And certainly I know from my own students. There are stuff they can't find particularly the history students They're looking for archival materials. Some of it is digitized. Some of it is not and the archive is closed So what do you do? um And the other structural thing which is important is course design Now I don't want to get too fussy about course design because I think the kind of Change to online teaching recently has been remote Emergency teaching, you know, do your best to try and help your students support them. And I think that's fine But if this keeps going on people are going to have to think about more orchestrated ways of designing their courses The psychosocial stuff is interesting teaching support is an obvious one Bit of work I did recently about online teaching Part of a project called open teach We did focus groups and questionaries with students and we asked them what made a good teacher essentially an online teacher And the things that came up weren't content knowledge professionalism the things they wanted and they thought made a good teacher was empathy support flexibility responsive and good communicator So I think those those attributes are worth thinking about and it kind of echoes some of what maha was talking about about caring A caring approach the other thing which is really happening. And again, I'm hearing this from my students Is technical problems My students are online students. So they typically would have a good laptop good broad band They would have the tools they need, but there's a lot of students who've been chucked online The shops have sold out of laptops The Dell supply chain to China has been interrupted And so they may not have the tools that they need to learn Also, they may not have the broadband. I also hear about people sharing laptops. So when we're designing assessments If you want a short timed exam, is that going to be possible? Is the student going to have the bandwidth to do that? Other things belonging again for campus based students They may not be used to this kind of online community thing. So there might be a little bit of work needed there And the last end behavior is time and it relates to life load Um, and again, how how people's time You know, how much time will they have to engage with assignments? And will there be flexibility of time built in so that if they want to learn at one o'clock in the morning As many of my students do that that's actually, you know, facilitated So I hope you enjoyed those gifts dcu um Developed some principles for crisis assessments, which which are quite nice and and quite simple So there's four principles there. So validity. So try and maintain the connection with the original course learning outcomes Equivalence. So is it roughly the same as the the the exam? It was originally intended to cover uh proportionality and academic integrity and you know different ways to ensure academic integrity So those are the kind of overarching principles that have been guiding our Crisis assessments. So in some in most cases that's changing exams into different types of assessments Again, it very much depends on the module and the course and how many exams they had um in my own course courses We have a program focused approach to assessment and feedback and to be honest, we don't actually have that much exams We have a modular course designed for online adult students and With a lot of choice involved as well that that has advantages and disadvantages But we did still have a few exams. So out of the 32 modules I think we had 10 exams still so we still needed to kind of rethink how we would do those But I thought I would give you a flavor first of our approach to assessment and feedback every year around march we create a google sheet like this And we align the assessment plan with the module learning outcomes and the program learning outcomes Um, and then we then bring that to our program board and we discuss and you know And one of the aims of creating this matrix was to increase the variety in particular of assessment types Uh, also just to make sure that all learning outcomes were covered in particular areas We struggled to cover in the past were Oral communication because you know, that's a bit more challenging online Uh, the other area was collaborative learning opportunities. So those are two areas we we sought to improve Um, so now we have this matrix going for about six years The other thing is when we started the project if you looked at that matrix Nearly everything was either an essay or an exam So so I think we we've gradually improved the variety of assignment type there And all these are online assignment types as well So it might give you a flavor of what's possible and that was just for the history section. We have four other discipline areas So what did we do about the ones the the exams we had? So as I said, we had about 10 modules with exams Luckily the kind of review process had already happened. So the extern had reviewed them The intern had reviewed them So we had exam papers essentially that were through our quality assurance method So what we decided to do was to turn them into open book take home Assessments and or exams, but I'm not allowed. I should call them assessments because that's what they now are Um, and the reason is so we give students a week to do them We use our vle loop, which is Moodle We have turned it in turned on. Oh, sorry. Actually, it's Urkund So it'll go through that that mechanism. So and we've also told the students by giving them a model Example of what they're going to get that we do expect referencing and that plagiarism will be Followed up on and we always very clearly on each module page Have the plagiarism policy front and center and as students submit they tick every time And sign that page and and if you look at some of the kind of best practice guides that's in that zone The other thing I think was important in our decision was that This approach is low bandwidth Yeah, so asynchronous mainly Uh flexible as well. So people could you know work away for a week at any time That that they could and that's important to us because You know for our students They have a lot going on in their lives even outside of coronavirus life load wise And so that's our plan Here's some other ideas which I drew from a resource a dcu resource And And the link is at the bottom I'm going to share the google slides link. So it's down the bottom for the whole set and you can Keep them. I'll also tweet it out later with the hashtag Um, so the portfolio lisa and alfredo both talk about that And I'm a huge fan of e-portfolio for many of the reasons they they describe. They're authentic. They're personal You really get to know the students. They're hugely hard to plagiarize although. I have heard of people buying reflections online And the other thing I really like about them is they're incremental So you build them up over time And the best ones, you know are like every week or every month a little bit And add a little bit more and look back on the previous one and what have I learned And it's very very iterative So I think it's a lovely approach to learning But I don't see in a kind of a crisis I must get an assessment ready in a couple of weeks that it will work But you never know it depends on people's timelines The other one I really like is a collaborative wiki. So I use the the tool within moodle And actually what I started doing was getting the students to pick a wikipedia page critique it And then use that as food for thought for their own wiki Yeah, so instead of saying oh wikipedia is awful. Don't use that And actually get them to critique it and apply their critical thinking skills The next step I'm hoping to do is get them to live at wikipedia. I haven't quite got there yet Another really nice one is an asynchronous online debate. So we use this quite often So you pose a you know debate style question for against You tell you know give the student guidelines about how much they can write what type of information you're expecting Source quality And then you also build in loops to make them comment on each other. So, you know You know comment on leases or comment on two other students And then at the end you get let them choose their best five Other stuff there. There's some interesting ones like the vlog a blog podcast online presentation is really good But some of those I worry about because of the bandwidth and the technology requirements So I'd kind of urge a bit of you know careful consideration there The last one I like as well the peer assessment We use that a bit as well. So for example Create a short video explaining a psychological concept And then we used the workshop tool in Moodle to pair up groups of students to give feedback on that You know that example, so we've used that in lots of different ways There's also some other cool stuff one of my colleagues avon costolo and mature if he's here Uses a tool called pure wise Yeah, I see I'm getting a warning there And that gets students to create their own quiz questions and then comment on each other's questions So I think there's some good ideas. Have a look at that that link if you like One other thing that's quite a nice one about academic integrity. This is from a project A u project that dcu is involved in so they're kind of principles for academic Integrity but for assessment design And so just to pick out one or two because I reckon mark is going to start giving me dirty looks But some of them I've already covered covered ensure they're authentic current and relevant Integrate them across the program Include elements for students to think Demonstrate their own work and what I would say is make them as context-bound Personal as possible because that that makes it much harder to plagiarize Collaboration so this and from this Australian resource I in my head collaboration would disincentivize Plagiarism but this Australian resource That mark was Referring to in the chat as well would actually suggest collaboration increases the chance of plagiarism Which is interesting So very very interesting resource have a look contract cheating and assessment design We've mentioned a lot of these but things that are very hard to to buy Vivas and actually that would be I saw a question in the in the q&a there early. How do I ensure? Um, you could how do you ensure that the person actually created that so the viva tool is very good? And you could do that online Uh, or I mean I have used that in the past for students that I suspected of plagiarism But but couldn't necessarily prove A short viva. Well, no, I think I'm gonna uh start to stop you on that note. I'll just show your webpage though Um, well that can sit there in the background. There you go. I thought you'd like that Um, we have uh, just a massive questions coming. Okay. Well, I stop sharing or or not sharing Um, and I think we'll go into a q&a mode What I'd like to ask the panel members to do as well is to go in into the Q&A tool when they are able to or others are speaking Answer some of the questions there. We'll do that as much as we can But just to get things started Um, the questions are falling into a number of um, sort of class clusters if you like There are some bigger questions around and on it to some degree you challenged us in the start of your presentation Um around the sort of paradigm shift that was also one of our opening questions and the challenges that assessment bring about Then there are some questions much more pragmatically about online proctoring what systems how we stop cheating Um, then we have some questions around Uh, the actual pedagogy of assessment in relation to workload Principles and so forth. So I think what we might do is just start at that bigger question Um, and I'm going to ask um, lisa you to come in first if you don't mind and then we'll go to alfredo Around the bigger question around how you get people to engage in alternative assessments that might clash philosophically or pedagogically with their normal approach Oh, wow. Well, that's not an easy question to answer. Um, I think when you're It isn't just the instructor that needs to reassess or rethink the way that they're going to Do assessment. It's also the student that needs to needs to To really accept that that new kind of of of assessment It's really I think you need to to To rethink how you're going to assess students It isn't going to be a matter of giving them a test anymore at the end You really need to give them a more formative A more formative experience where you get to know the student where you get an opportunity to To see their work over time to provide feedback so that they can improve on their Improve on their work as they progress through the course How do you convince instructors to do this? I don't I'm not really sure because we don't really have much of a choice right now If you're going to move into an online learning environment and you want to survive and you want to have a good Solid pedagogically based Teaching and learning environment. You're going to need to rethink how you do your assessments and how it is that you present your your online Your online content. So I really I don't think that it's a question of I don't think that we can easily convince instructors I think right now in order for you to really Survive in this new environment. It's going to be essential for you to rethink how it is You're going to approach teaching and learning Which is going to mean thinking about assessments and coming at it from a from a new perspective Alfredo, do you have any thoughts? well, yes, like I said My experience after all these years is that Um teachers are very possessive concerning their Evaluation of students. I don't know why but It's very difficult to discuss Assessment with any teacher believe me. It's my experience my personal. I don't know about yours so What I think is that has to be a decision of the instructor. It has to study I I'm I'm going to ask you frankly How much time did you spend researching? the correct mode of assessment And then you'll see that most of the people that are teaching and assessing they didn't They just used assessments. They learned when they were students and their colleagues do etc So my point in Eden in another conference is that we need more research more training On this question of assessment. It's the less studied part of this teaching and learning paradigm In order, I'll give you a chance to come back in on that bigger question around the Sort of beliefs that people have about assessment And how you address that to some degree you touched on this when you talked about working in a programmatic way So you might wish to expand on that Yeah, um We experienced a lot of challenges Early on When we were trying to change that and I suppose what we tried to use was a very evidence based approach So we would those who who didn't agree with the change we would present them With the evidence, you know on that assessment type and show them that listen the program currently is not assessing communication Oral communication. We need to get a learning outcome here And also we used feedback from the students as well who who were looking for different type Types of assignments and not just, you know writing 12 essays every year But it was very gradual. I think that was the thing, you know, we'd find someone who was kind of Interested and open and we'd work with them one year And then we'd use them as an example to the other colleagues And then a few more would come on and over time Uh You know that the I suppose the super tanker started turning But it was very gradual and I do think there's something about pedagogical or philosophical beliefs some Some people just really believe that essay is the right way to assess An elitist student for example, uh, and they just refuse to believe there's another way Um disciplinary resistance. I call it Okay, um We're mulky processing here folks trying to answer as many of your questions in the chat as well I think before we move into perhaps The area that a lot of you have an interest which is around academic integrity and cheating Let's take an educative approach and ask the panel about the designs that they might have and ways in which they would address in Assessment that's alternative assessment to an exam But maybe scheduled over a longer period perhaps 24 hours. I know a few people are suggesting So what sorts of assessments would you choose for a Defined time period, but it's not really meeting the criteria of continuous assessment I'm going to give you Alfredo the first chance to come in this time just to share the mic Okay, thank you. I mean I I use a lot turn it in so there is In my university there is this tool that we can use so I use turn it in for For homeworks and others, but in terms of for instance of problem solving um it's very easy to to verify If I'm an engineer, so I teach engineering. It's very easy to verify if there is Pleasure or multiple solvers in the same program at the same time I think that for instance if you want to do a multiple choice and you have a large bank of questions You can adapt For each student an exam that it's different from the next one. So it's it's not easy to to to have a pleasure is when you even when you do a multiple choice or Other type of computer based assessment And Lisa do you want to come in and in particular talking about some of the design considerations In the way that you need to manage An assessment over a time period What we did one example that I can give you And and this is where we you really have to be creative in the design of your assessments You have to make it contextual. You have to make it relevant to the student For example, one of the assessments we've used is to have students define a strategic plan And in in that case it has to be a strategic plan within their institution And then they have to take that institution and then do a comparison How would I implement something for example within my institution? How would I implement e-learning within my institution? What kind of challenges are there? What kind of strengths does our institution have all kinds of weaknesses? These are things that that wouldn't be really easy to Copy or to find or to plagiarize in that sense because what you would have to do Is in this particular assignment is to take that contextual environment and and to apply it to the assignment It also intrinsically motivates the student because it's something within their environment that they're actually addressing within the assessment Another thing that I found really helpful in terms of the portfolios in any case Is when we have the reflective learning journals, we can see certain writing patterns with our students We we know how they're thinking we know how they're addressing certain topics We see the same sort of thing within the discussion forums So you're able to get a really holistic view of who that student is the writing abilities where their shortcomings might be What things that they need to explore further And so if they submit an essay later that doesn't fit to those two other forms of assessment where you kind of go Well, this doesn't sound like the same student that did this reflective A reflection within their learning journal or has been posting in class Um, I don't use turn it in but I'll often go into google and just enter some of the text in an essay and say okay Is this you know, this doesn't sound like the student that I've come to know within my class And so uh, so it's a good opportunity. I think to to identify where there might be plagiarism Thank you Lisa. Um, orna. I'm just going to come in and move the conversation on and give you the first opportunity So you're the one thrown with the question at the deep end, but um, something that's coming through In the various questions. I think is this issue of so if even if you are rescheduling the assessment So it's over a 24 hour period. How do you know that it is the Person who is meant to be responding? I the identity of the person Preparing the assignment. This is an interesting question in many ways because for written assignments outside of examination That question comes up regardless At dcu. We utilize a system called studiocity that provides Formative feedback for written assignments and gives feedback within 24 hours before the assignment is submitted Some people see that as cheating But what we know is that from the research that lots of students already asked their parents their guardian another student Give feedback on their assignment before it's written So um, it raises an interesting question about how important that really is and are we looking for love in the wrong place? Um, I'll that note. I'm handing over to you orna Yeah, I mean we've I've had discussions about this before about this whole idea that we have to verify the identity and But but what I ask is do we have to verify the identity for coursework? Um, you know, why is it suddenly more important for exams? But of course we have identity verification I mean if you're using your your your vle, you know, the student's id is there It runs through a plagiarism checker Like irkund and we have internal quality methods. So we have what we call a moderator checks a sample of The the student work And then the markers themselves if they notice something untoward in in the assignment Uh, they will pursue that as well. I mean the obvious identity checker is to do a viva style one mark But I mean that's just not doable on scale. So what we do is if it's suspicious we follow up with an interview Um And but and if the quality assurance Triggers, you know, this one looks dodgy will look into it but otherwise, um You know relying on both the the markers I which I think is far superior than Uh plagiarism checkers like turn it in or irkund. They sense if something is wrong Um, and then the we have the further quality assurance Check and then actually at the end of the year the external examiner would often view a large sample of the assessment So those those measures are in place And just From the the moderators role before I asked Lisa to come in and give her a moment to think as well one of the um items bullet points, I know from the many guides that you can access at the moment Is that having written assignments that ask people to draw on their own experience? Whether for example, if they're a nurse or a teacher or even an engineer where they're being expected to draw on the context in which they've had to Do much of their work over the course of the semester. That's just another strategy to enhance the academic integrity of what they're doing Lisa, do you have any other ideas or thoughts? As I mentioned before mark, I think a lot of it has to do with finding out the relevance of the of the of the topic for the student I mean even my own kids my own kids will say to me, you know, mom, they give me assignments. They're not interesting They're not things that I'm interested in and I said, well, what is it? What is it that you're interested in? So sometimes it's it's making it the assessment open enough for them to identify Identify the relevance. What are the things that I can take from this assessment and apply to my context? How can I How how can I um make it my own assessment? How can I make it my own? Uh my own way of assessing uh what I've learned and and asking them to define it I think I mentioned this in one of the answers that I wrote Uh in the chat box is that sometimes it's also looking at, you know, how would I assess my own learning? Coming up with your own um with your own learning goals. I mean, that's that's I mean, I know that's a bit further than a lot of uh, a lot of our face-to-face Instructors are ready to do at this point But it's another pathway that you can take in order to make uh To make assignments really relevant to students is that they are able to say This is the kind of a Assignment that I would like to do um and then getting the approval from the instructor to move forward and I think that's one way of also addressing that And making it so that they're more intrinsically motivated and less chance of plagiarism because if they're motivated They're not going to plagiarism. Most problems that we have with plagiarism are because students One aren't interested in the topic and two don't have time to get uh to get their assignments turned in Thank you, Lisa. I think some very uh, sagely advice there that um, certainly drawing on the the research evidence That the issue of plagiarism in most cases is actually not out of being deliberately Dishonest but much not really understanding what's required. That's certainly a study in Australia that I'm citing More often at the moment and also the concern about cheating But we're getting down to that the dirty side of the conversation using that word cheating So, Alfredo, do you have anything practically that you would advise or what's your own institution doing to address the concerns around cheating? Um Like I said, we we have the tool turn it in available But for instance, it allows Like Lisa was mentioning it allows complimentary oral discussions and exams so we can that's what been rescheduled for this emergency in these months that are missing until the end of the year so you you have to find probably a set of Of assessment modes that fit not only the purpose of verifying the learning outcomes But also the type of students and disciplines that we have that is a diversity For instance, I'm having an exam next Friday. What what the students will do is that they will share Their screen and their sound with me. So it's a way, you know, but there are probably other ways because Like it was mentioned, I prefer the e-portfolio. The e-portfolio is being used even for professional Use like doctors and engineers and and nurses and they are being used to verify their competencies Their cpd. So engineers island. We are talking about two artists here Engineers island uses the e-portfolios to verify the competencies. So if we do it for professionals, why don't we do it with the students? So that would be my my advice Find the right ways to do it because there is a large variety of options So one of the questions that has come through in relation to the use of e-portfolios is that Often the advantage of an e-portfolio is around the critical reflection that the learner is able to draw on And the question is how long do you need to be able to engage in that critical reflection as a portfolio? Something for another day when we have the luxury of more time What can a portfolio still be used as a form of authentic assessment to address some of the issues? I don't know who wants to pick that up. So i'm going to let you decide on this occasion I'll take that mark and Yeah, when I was thinking about this that that would be my instinct the portfolios I've done what students have typically been over a long period like a year an academic year And I think if you really want that developmental experience Um, that's what you've got to do because if you give it to them over a few weeks, they'll write it all in one day Um, but the whole thing about making it incremental and having incremental deadlines is the structure of the of the assessment The frequency Prevents students from doing that You know, so if you give that to them over two or three weeks, they're going to write it in one day That's just my two cent But do you think lisa or alfredo? He say a little bit this year where we did the weekly Either the results were completely different from doing at the end of the semester because the dialogue and the observation of what they've written And the discussions about what they were supposed to learn and what they supposed because they had to prove It's not only saying they have to prove that they acquired that so It's another way. Of course. These are 22 year 21 year olds, but I I think it can be done in higher education without any problem Lisa do you want to come in on that note before I come in and change? Yeah, the only additional thing that I would add is is to reinforce what what orna and alfredo have already said Is that if you do this over over a certain period of time over weeks And then have it gradually feed into their portfolio and whether this portfolio is used for final assessment or whether it's used for Formative assessment or whether it's used over, you know, a couple of months or a year or throughout the whole graduate or undergraduate experience Let the let the learner define that, you know, together with the instructor But if you do do it do it over a weekly or bi-weekly so that they're continually submitting So you don't have a situation like learn or orna was saying that that this all gets done done within a matter of a couple of weeks And I realized this these are these are examples that that are long-term examples that many of you are faced with With the problematic of you know, I have to find a solution right now. I have to do things right now and and it's My best advice for you is to do the best that you can To provide the students the support that they need to be flexible But to gradually work toward a design that is going to be to be useful within a online learning environment This isn't something that's going to happen overnight. We didn't learn how to become online instructors Within a matter of of of days or weeks This is something experiences that we've gained over years and years of Teaching in an online environment working with students So I guess my last piece of advice would be, you know, don't push yourself too hard This is to give you guidance to give you direction to give you some ideas on how you might move forward But you know, you won't be able to do this Right away, but we are really looking at problems that that are going to become long-term problems We're not going to be able to go back to the old normal There's going to be a new normal And so what we're trying to do is to provide you with some guidance on how you can adapt to that new normal Thanks Lisa and I think you you cued us all to also flipping this a little bit to think about the learners here that When we're choosing alternative assessments and I'm conscious a number of questions have come up about workload And also the appropriate level of challenge. How can you ensure it's at the appropriate level? I think we do have to think very carefully about The assessments the alternative assessments we choose because not only may they create an additional work for us They may create more additional work for our students And I'm just going to give you a quick example from my own university Where some staff who have a very contemporary view of assessment a very authentic oriented approach We're going to propose that their students produce small videos as distinct from a written essay And of course very hard to plagiarise a small video of yourself doing a quick pitch In relation to something But the difficulty there is we would have been introducing to students a new technology that many of them would have had to learn In addition, bandwidth issues right now are actually quite a challenge for some of our students depending on where they are located in Ireland But also I don't know about you but assessing a video is quite tricky. You have to actually view the video Not to suggest that you don't Read an assignment as such But there's a lot of time that comes with that So those are just some of the fish hooks that my own university has identified around the workload And then one last thing is what we've discovered is that in moving to an alternative assessment giving people 24 hours, maybe even sometimes 72 hours in a gap to respond to something Um, we haven't perhaps taken sufficient account that the students of their full time might equally have a number of other Assessment tasks coming in in that same window. So I think if you're not yet at that point I'd be encouraging you to expand the window for the assessment period or the alternative assessment period Again, are any of the panel members want to come in around this issue of workload? The challenge That's that the appropriate challenge or anything else or no Well, I suppose that's what I was trying to capture with that student engagement framework there from cahoo that I mean, I'm just thinking about my own experiences with small children and cats and Also, even my children's schools emailing completely Really big Lists of homework and having too high expectations And I think when you you're on the receiving end of that if if our students feel like that I think We're doing something wrong So I think we have to be very mindful of the life load stuff the bandwidth is as you said mark A lot of people talking about their broadband is is really not very reliable And cool things like short videos are great exactly as you said But as long as the students know how to do them, have they ever done that before do they even you know Do they even have the right technology to achieve that? So I think why while You know, you might be gravitating and I personally would be guilty of this. Oh, that sounds like a cool assessment really different Maybe this isn't the time to do it. You know, this is the time for the simple Not the brilliant solution as Lisa was saying Um, so that's why that's my final word on that Folks, if you're wondering, um, we're way over the hour now. We got underway a little late and and we had to keep As much as we can the time the expectation is that we might continue with the q&a for another five minutes or so We're going to struggle I think to answer all of the questions that have come through But hopefully we're touching a lot of the points in this conversation now One just to change tack and I'm going to get again into the sort of challenging aspects There have been a number of things the challenging and they added nuts and bolts. He sort of level A number of questions about online proctoring systems. What are the best systems? Would they Use a system now? Are there open source systems and the so forth? I know I have particular views on that I've had to share those in my own institution But I'll let the panel start off on the first note as to whether you think An online proctoring system is the way to go right now Oh, I'll start. I think it's expensive and difficult to implement. So I I would skip it first one. I would prefer something that it's individualized like it was mentioned by the panelist already and and Define assessment for the individual But don't try to verify the to replicate what it was shown by or not to replicate what happens in the exam I think That's a problem with e-learning also something that we have to Deconstruct what the idea we have of higher education and education the space to face and we think you have to rebuild a new scenario for e-learning we use a lot of the face-to-face in the We try to replicate and that's very difficult. That's my My contribution Lisa any thoughts on that has Oldenburg explored online proctoring at all No, we haven't not at the moment and so I really can't say much to that topic I was right in the middle of answering a question in the online chat. So My mind is somewhere else Let me I said I had views in order since we're at the same institution I'll just jump in because it has come up on a number of occasions So I do want us to have addressed it In my own institution one reason that we're not exploring online proctoring is even the bandwidth Durations and we have it from very good sources that our major providers internet providers prioritizing Basic infrastructure and particularly health care support So you can understand that support for education does not quite stand at the same level as support for our health systems And we know we use Moodle as our BLE or our learning management system that there have been small outages no more than a minute or two minutes from time to time Completely outside of our control And even though we've quadrupled the support that we have in terms of the size of our pipe and the like So I personally would not be recommending any kind of live proctoring At dcu. Horner actually was leading the charge here with a pilot in this space But I would not be wanting to live pilot But ultimately and I'm on a soapbox here if that expression makes sense and I'll stop in a minute I think this whole conversation about alternative assessments and we started on this at the start first question The first question that came through really challenges us to why we assess learners in the first place There's lots of things that people learn that we never assess In fact, you could argue some of the most important knowledge is not assessed And I'm going to quote from a famous book from david perkins many years ago We talked about the need to move from away from training memories to educating minds and I think To some degree each of the speakers have touched on this in the sense and challenging some sacred cows about why we're assessing and how So that's me standing off my soapbox But I certainly would not be advising people to be involved in a live pilot of a proctoring system But very happy to take any questions directly via email because we can share our experiences as to why we've made that decision Panel members, uh, that was a lengthy sermon for me for a minute I think let's just look an eye on the time give you all one last chance To come back with a final comment and why you're just preparing yourself with the final comment Comment so I don't completely drop you in it as you multiprocess answering to the q&a I think again People are asking whether online assessment can be Academically have academic integrity. Can it avoid cheating? Can it raise the Challenge to the same level all of these questions for me Actually are asking the question of why we're assessing in the first place And some principles which each of the speakers really touched on order in the last talk was quite explicit around those principles One of the things I think you need to be thinking about an alternative assessment is Who are your learners and what do you want them to learn and therefore? What is the best assessment technique task approach for those purposes? And that's my little I guess last word at a conceptual level before I ran round up thinking everyone So reverse order this time We're going to start with orna and end with alfredo if your last word on alternative assessments and the challenges we face in managing assessment Well, I'm going to go with my own acronym and the kiss method Keep it simple stupid. Uh, no, but I I really think uh, you know Considering the times You know simple low bandwidth asynchronous flexible assessments Are the way to go don't over complicate things Really good communication with the students, you know, we've been having lots of things like town halls q&a's uh model assessments and So and we've been we've been trying to be very transparent with the students about, you know, our plans And so far that's worked well for us Yeah, I can only reinforce what what orna has said as well I mean, it's important that you're there to provide feedback that you communicate with the students that you support them that you're flexible This is a very challenging time not just for us as instructors, but also for our students If I were to incorporate any kind of assessment That I've mentioned today It would be the reflective learning journal because this is an opportunity for for you as an instructor To see where the problem areas are for your students. Where are they struggling? How can you provide additional support? What kinds of things are they having? You know really having struggles with It also gives you insight into how they're learning and what they're learning. Are they are they really? Learning the things that that you want them to learn And how are they applying it within their context? So I think the for me the the reflective learning journal would be an opportunity for them To reflect not just on their learning but on the current experience that they're going through right now writing can be a very um A very helpful process in working through a lot of the things that that we're being confronted with a lot of new things That we've never been confronted with before And so that that would be the assessment that I would recommend Okay, my turn. It's it's very simple. I I presented in the Eden conference in w some years ago Research research and research. We you have to study more this question of assessment online especially but in general but online It's more needed at these days and the other thing is Is that we have great networks? We have great um People that are willing to share their experiences and what they have done I think Eden is a great space. We're going to have a virtual conference in next June so take advantage of it and join these networks where we share these things that each one of us Is trying to develop and improve Well, thank you Elfrater for that um little prod for me to give Eden's conference a a plug Which I had to share the slide so I can now tick that box as well Want me to share the slide mark? Yes, thank you I think that was my job. Sorry the last slide. Uh, is it So, uh, you've got an opportunity to uh Attend and participate and contribute virtually to the uh Eden conference in June No doubt we will be having lots of talks about The experience we're going to have over the next few weeks in my own university Most of our face-to-face teaching that would have been has been now completed virtually last week There might be a little bit still going this week and then we move into this alternative virtual examination slash assessment mode And I assume that most of you will also be doing that over the next month We haven't been able to address all the questions I hope we've managed to give you something to go with there have been lots of very useful resources Posted and we'll try to follow up when the video is available with links to some of those as well But on that note, I really want to just thank Orna Afredo and Lisa for their contribution today I should also acknowledge that Tim One of our executive members has been behind the scenes trying to keep a track of our Questions coming in and helping us prioritize those and then Diana Who's hosting the zoom room for us from her institution? Thanks to her and all the other administrative team from Eden who helped to make these sessions useful We're not waiting another week for the next session. Um, the next one I believe is in a few days if I have got my correct Schedule right, but the next one is very timely too because it's on how you evaluate In an online environment. So having gone through all of this and done the assessment I think evaluation will be very timely. You might just want to check the website for the precise date I don't have that at my fingertips here. Watch your emails otherwise, but on that note Thanks all for attending participating contributing and until next time. Have a good week