 Hi everyone, my name is Sonja Burns. I'm from Perth, Western Australia and I'm here today to speak to Geraldine O'Neill about her national forum for the enhancement of teaching and learning research fellowships, which I'm honoured to share the findings with everybody that Geraldine's got to tell us about. I've got over 14 years experience in higher education in Australia and I'm an active researcher in teaching and learning around quality with a special focus on work integrated learning and industry engagement, which is about authentic learning experiences for our students. The ultimate aim of the fellowships is to improve the quality of higher education and inform policy and practice in Ireland's higher education sector. So with that Geraldine, welcome. Thank you very much, Sonja. Nice to be here. So Geraldine, just to start, Geraldine, would you like to give us a broad overview of what your research is about? Sure, yeah, I'm delighted to. So it's about assessing work integrated learning. Work integrated learning is a term I think that many people struggle with because maybe not a term that's used very much in the Irish sector, but it includes all the sort of work based placement type stuff that happens off campus, but also looks at activities that happen on campus and in the curriculum. So just on that term, so work, assessing work integrated learning, that spectrum of assessing those contexts, particularly looking at assessing assessment and feedback because feedback being a part of the assessment. So I really wanted to explore this quite complex area of assessing work integrated learning. Initially, what I did was through a series of webinars with the sector, we asked people what were the challenges in this area. And interestingly, what came up was authenticity, which is like meaningful assessments. But also what came through was, well, it's okay to be authentic and it's great, it's authentic, but it also needs to be consistent, consistent cost placements, consistent across different assessors. So this was coming through. So therefore, I decided to look at developing, exploring this consistency and authenticity in work integrated learning. So what I did was that I can go through the methodology a little bit more, but very quickly, I did, you know, look at surveys, I did some workshops and I interviewed some researchers to develop an understanding of this. And one of the key aims was that I would come up at the end of this research, with some solutions to some of these challenges. So really, I want to interrogate, yeah, some solutions to these challenges. So just, if you, when you think of the term authenticity in terms of assessment, can you just give us some of the key themes that have come out around the challenges for people to both design and implement authentic assessment in a work integrated learning context? Yeah, it's interesting because when people talk about authentic, one of the things that comes through a lot is this idea of real life, real world, you know, very controversial terms when people don't like it, but students use it a lot. But it's the idea that it's meaningful and linked with maybe the students identity, who they think they are, but also who they, where they think they will be in the future. So whether it's, you know, a researcher or whether it's a particular discipline. So that idea, real world, and that is a term that's often used with it. But actually, it has a broader kind of connotation as well, because when you look at the sort of literature around it, it talks about things like, but also around student impairment, and then being able to self monitor and do other things. So that's another aspect of it. So it's quite a broad term. And I'll get into the findings a little bit later, because it's not always me doesn't always mean the same to everybody. But they're the kind of key things when you're talking about authenticity, this idea of meaningful, valuable, you often to the students, but it can be to others as well. And interestingly, as I'm talking about it, sometimes this can be in tension with the idea of consistency. Because it actually, because it's meaningful to the person, and it's meaningful and valuable to them, and it empowers them, sometimes them being consistent across context on things can be a bit challenging. So that's my sort of working definition of authentic, a term sort of, yeah, that is to say meaningful in real life is probably the most common understanding. But when you delve deeper, it has other other kind of understandings as well. Interesting, look forward to a more full some conversation at some date time. Tell us why you chose this topic. What interests you about this topic in particular and with a focus on assessment and authenticity? Well, I was involved in a national project around assessment and feedback for a year or two, I actually worked with the National Forum for a year or two, and I was involved in a national project around developing assessment and the understanding of assessment, what it meant. And but as part of that project, and my ongoing work in University College Dublin, and assessment and feedback have been that suppose my, my sort of, I don't know, the theme or that my key, I suppose, topic area. So I'm always interested in assessment, it's so complex, it's so challenging. But it's really, really interesting. And I do think that if you can crack assessment or enhance it in some ways, you can go a long way to enhancing student learning and to getting the policies right. So I was always interested, assessment has always interested me from, from lots, lots of different angles. And I also think it's a really important thing in relation to, you know, students employability. And that's why I suppose I'm very interested in this idea of employability. It's kind of can be seen like a dirty word in higher education sometimes that it's, you know, they're really firm employability. But students are really, really valuable. An interesting research study done by the National Forum again on student success and what students felt success was in fact employability and being an employable graduate came up for students as a really the top, the top one. And secondly, interesting was, was kind of grading and assessment. So if you look at the student voice and this is what they're looking for, even if we feel maybe higher education has more kind of aspirational and sort of, you know, other kind of purposes. But you know, the student voice are saying, we need to do this and we need to do it well. Very good point. And I hear you loud and clear about the difficulty of assessment in our current higher education environment. Given that you just mentioned employability, and I get the word itself can be contentious, but that you need to assume there's other stakeholders involved then in the process when you you bring in the notion of employability. So can you tell us firstly what you see as the role or what's come out in your research as the role of the stakeholder in terms of employers or community agencies? Firstly, and secondly, given you just mentioned student agency, perhaps the role of the student in this process ensuring authenticity. Yeah, it's interesting. So what I found I suppose both in the literature and it was coming out in my own findings is the literature and authenticity, and I suppose even your own works on this, which is around the authentic framework, really talks a lot about the audience for authenticity. And so the industry partners are really, really key. And it also comes out in Rola at Jawe's work as well in relation to that the industry partners are really, really key for authenticity. And I suppose I know even from my own practices that my previous life was an occupational therapist. So I was actually involved a lot in working with sort of practitioners. I'm using the word practitioners, by the way, to kind of cover the idea of like industry partners, healthcare workers, that wider term. But they're really key in relation to because if you think that authenticity includes the stakeholders, they're really key in assessment. But one of the challenges is that we don't often support them enough, they don't often see that they're they themselves are employed by different systems. So the so we're expecting a lot of them. The students end up with them and working with them and learning a huge amount from them. But we need to support them and to be clear, I suppose, in are they involved in assessment, you know, are they involved in feedback is and you know, as I get through to some of the findings as well. But but certainly we need to certainly involve them as partners in this in the assessment in this in this wider assessment process, because they're key in relation to to ensuring that the students come out with the competencies, learn like different things from being on their placement, but they're very much part of the assessment and in particular the feedback process. I suppose the other thing in relation to them as stakeholders is that if they are involved, I mean, they do also, you know, also get some opportunity to influence the type of graduate who's coming out. And so in some ways shape how they might contribute to the industry and in the future. So it's a two way process and they're a they're a key partner. Yeah, key partner. And the student. Sorry, the student. Oh yeah, the student. Well, the students, I would be always an advocate for the students as partners in this in empowering students. And I think, interestingly, we have a working definition in Ireland of assessment, which includes assessment does learning. And the student is very much a partner in that piece. And I think they have a role to empower be more empowered, particularly in the work integrated learning aspect, because they go into it with such unique identities and skills themselves, and they go with their own needs. So I think a really empowering students in the process is important for valid assessment. But I also think it's important for upskilling them in in empowerment and in self monitoring, and being involved in judging how well they are doing in this, it's a really, really key skill. If we can give students the skill to self monitor and to self evaluate, we're really setting them up to be judges of their own skills in the workplace. So I think that's one of the most powerful things to give students in this process is that they can self evaluate and know how well they're doing themselves. So it's really, really key, I think. Yeah, totally agree. I think there's lots of barriers to making that happen at the moment, but couldn't agree more. So why do you think this topic is important for and potentially impacts on the higher education sector, including those of us who teach, those of us who learn, and those of us who lead in the higher education institutional environment? Yeah, I might start with those who lead actually, because I think sometimes work integrated learning is seen as quite a binary in the curriculum in higher education. I think sometimes it's seen as that, you know, students do stuff on campus and then they go off on campus and it's like this other world and the two, you know, don't come together. So I think even the language of work integrated learning, which really pushes the idea that this is a spectrum of learning I think is really key for policymakers. So I think from this idea that the topic and how we describe it is really important. It's seen very binary at the moment, so that there's those out there and those in here. And I think that for leaders and policymakers is really in key. I think it's really important because I think certainly in Ireland, and I know internationally this is the same, that there is a real push, you know, for graduate skills and sort of employability. And I think, you know, so I think it's important from the kind of national level and the sector level, the policies that, you know, that we look at this in the curriculum. From the teachers in this space, I do think that there are those involved in placement and those types of things and they are very committed to it. But I think other, let's call them teachers, lecturers, whatever in the curriculum really needs to look at their work in relation to where they fit in with this. And I think your own, you know, Bosco Inference, that lovely framework that you're involved in, Sonia, you know, really kind of had a place, really strong place in this where you can look at curriculum and see what role you have in this in relation to your modules on campus, you know, and I think it shouldn't be seen as this binary of, you know, there's the workplace and then we do this theory practice divide as well. So I think it's really key in that way. And I think I've mentioned the students, I think, again, they have a key role in being involved in empowerment and shaping how this might look for them, I think. So they'd be the key things why I thought this topic area was important. Excellent. And did you have much of a base of previous research to build on? I know that their assessment is a particularly challenging topic for many people to work with. But did you find a lot from which you can build on for your own research? Yeah, no, there was, it's quite a well written topic. Actually, there was a lot of literature on it. There's quite a bit around authentic assessment. But that's really popular at the moment, I think, because I think that is actually coming from a base where people are looking at traditional assessments and thinking that, you know, the traditional exam is not working for us. How do we make it more authentic, more engagement? So a lot of this strong engagement and authenticity, you know, sort of, as I say, some of your own work. But also, there was a really nice study done by Villarro et al. And they looked at sort of all the literature on this. And they actually came up with the idea that when they looked at all the literature, they said, well, this idea of realism is coming through this real world is coming through in a lot of the literature. And this cognitive challenge is coming through in the literature. And also the idea of this evaluation of judgment, this idea that students, you know, judge, and again, Rola, Giawi's work in that space is three principles of authentic assessments. So authentic assessments are a lot been written about it. And not without its critiques, actually, because a lot of people like the word authentic. I mean, it's interesting, I myself gave a workshop on it a few years ago. And, you know, we had a great discussion around, you know, well, does that mean mine is not authentic, if you know, so there's a challenge for that term. But in essence, the idea of engagement and that a lot been written about work, integrated learning, a lot done on, you know, the idea of internships. So that, you know, so there's a lot of literature, Smith, Smith et al's work, you know, around kind of, you know, the critical issues in work integrated learning really bring I think one of the things that came to me a lot was the idea of the tension between in this space with assessment that's valid, you know, does it measure what you wanted to measure? And is it reliable? And from that, I thought, well, actually, taking that a little step further, you know, is that well, authentic is, you know, is it, is it kind of genuine and real and valuable, a little bit of extension of validity in some ways, with a little, little, little more, but also consistency. So I think that there's been some work in this space, but lots of work on assessment is very topical. Lots of work and feedback. Even the National Forum's work that I was involved in a few years ago around, you know, what is assessment, and a lot of stuff around, you know, internships and placements and medical education that, you know, Goverton and Van Der De Fluten, I think, did a lot of work around, you know, what is validity in medical education. I mean, so a lot of different disciplines are looking at this. You know, those that are, you know, the professional bodies and those involved, medicine and nursing and occupational therapy and social work, a lot of, lot of work in relation to work-based assessment. The core of different things, clinical placements, work-based assessment, internships, you know, field works, a lot of different language on this, but a lot of literature on it. So yeah, so lots to build on and coming out from different angles and things, but yeah, yeah. Yeah. So and you raised some interesting things though, because when you talk about validity and reliability, et cetera, of course important, but I still think we're driven by traditional perceptions of what assessment is and sometimes this stuff doesn't fit into all of that. It's very, as you say, complex. So given that really detailed background, which has been very helpful, thank you. Can you tell us a little bit about your methodology and how you went about gathering your data and then interpreting that data? Yeah, sure. I suppose the three main methodological methods, I suppose, but I suppose the overarching approach I wanted to do was more participatory action research and it was an approach that I had done myself a few years ago and in essence, just to suppose to explain it a little bit, it's that you work with, you work with people in general to help them make changes to their own practice. So it's really trying to suppose, again, it goes with my philosophy of empowerment of students and empowerment, it really tries to empower. So a centerpiece, I suppose, in my work was working with disciplinary groups and I've been nine disciplinary groups who had a student, a practitioner and an educator in all the groups in the workshops, they were three hour workshops, working with them to develop solutions to their own practice. So the methodology actually, I'm quite proud of the methodology because it really actually worked with them to come up with solutions to their practice. So that was the key kind of center part. But before I did that, I actually looked at, I was lucky to have access to the National Student Survey data, which is a survey, a little bit like the, I think the Nessie in Australia, which is a student engagement survey. And it actually is a survey that asks students around their engagement. So I had actually access to the five years of that data. So I was able to look at the question, what could institutions do to improve, which again is around solutions, asking the student voice that's kind of fit him at the philosophy. And so I had 95,000 comments on that I was able to access. So I was able to look at that and then do my workshops, which came up with some solutions. And also I was able to, I had the, was able to talk to seven researchers before I did the workshops around their understanding of work, integrated learning assessment. And that was really very valuable to get kind of expertise. So what a nice mixture of student voice from these surveys, seven researchers who gave me great insight into what they were doing and the literature and what they thought. And then these, you know, three, sorry, then these nine workshops, which was around developing solutions to their discipline. And then I'm going to finish at the end of the year coming back to the seven researchers, just to sort of present it back to them and see how it could actually impact on practice. So given the range of participants and the different backgrounds, which I think is intriguing, I've got two questions, sorry, two questions in one again. The first is, were there any themes that resonated across all stakeholders, that is themes they shared? And were there anything that was particularly something that came from, you know, particular stakeholder and not necessarily another? Yeah, yes. Across them all, I suppose, they all, I suppose it was really particularly looking at the consistency and authenticity and things that came with it. Certainly what came across them all was it's really important to have clear expectations, you know, that this came through a lot, you know, we need to be clear, you know, transparency needs to be really clear. And in some ways that falls in the middle between consistency and authenticity, it is not clear, you know, like it won't be valid and it won't be authentic and it won't be so. So that came through a lot with everybody. The idea we should be talking about this and this interstitial dialogue came through a lot between across them all, you know, that it's, you know, we need to talk about it, we need to, it's not something that's solved easily and we need to make sure that we all get our voice heard and it came through. In relation to authenticity, the students, interestingly, the student voice in the large-scale data, the survey data, nearly a third of the comments of all the open comments were related to work and authenticity and real, particularly around the realism side. So the students are really pushing, you know, this idea, it needs to be practical, real, you know. And interestingly, the researchers also really felt strongly about that, you know, it's really important, it's authentic, in fact, you know, it's much more important. Consistency is great, we need to be able to stand over but you know something, if it's not real and if it's not kind of empowering and if it's not, you know, that's really important. But there was maybe a little bit more of a mix with the consistency piece in the workshops because the consistency in the workshops came through a stronger than authenticity, which was interesting. So there was a couple of common things, the interstitial dialogue expectations were certainly common, but when you start delving down into the weighting of authenticity and consistency, it was a little bit different among different stakeholders. And I want, I suppose that was your second question actually, wasn't it, the difference? Yeah, yeah. And in relate to delve into that a little bit more. So wondering, let's think about our university staff, our teaching staff are in the main hardworking, passionate group of people. And I think they believe in this, but what did they cite as key challenges for them in actually achieving ultimately an authentic framework for their assessment? Yes, they interestingly, they struggled particularly with authenticity when it came to work integrative learning. And when I say that more so than the practitioners and the students, they struggled with authenticity when it came to things like writing a report back in the institution, having done a placement, that they found they really struggled with the educators and the students a little bit also in that space. So there was a particular challenge with, interestingly, there was less challenge with authenticity and stuff where people were doing things like problem based learning on campus and project work. Everybody was delighted even that was on campus with authenticity where there was a little more of a challenge for educators and students a little bit too is the type of placement where they go off and they have an experience practitioners are not involved and they come back and they write a report or they write a reflection like posts, post placement. And that's when there was a particular challenge for authenticity, actually, that type of context. So when you're on campus, and you knew you were on campus, and it was better than the exams, it was like, this is great, because they're doing that stuff. But the report, even though they had been on a placement and came back, that was when the most interesting when educators were most challenged by the idea of authenticity. More so sometimes as I say, than the on campus one, that they weren't out of placement. So interestingly, and you might be particularly to Ms. Sanya, because the this idea that authenticity is a continuum from placement, so from from campus to placement isn't necessarily a very linear direct line. You know, you know, and actually, and in fact, it was interesting when I talked, I know you were asking about the educators particularly, but talking to one of the researchers, they were involved in quite really elaborate authentic projects on campus. And they said they had actually really great success with them. And they struggle sometimes with placements, where the practitioner wasn't involved. And they students came back and as I say, did a presentation or did a report or wrote a drop afterwards. And they felt that actually, it didn't feel that authentic. So I think, I'm not sure if quite answering your question, but that was really that was the most challenge with authenticity from the educators. Was that kind of context? Does that make sense? Yes, it does. Yes, absolutely. So were there any other I think you've highlighted some of the findings? Is there any other findings that you have initial findings that have come out that you haven't that you would like to tell us about today? Yeah, I mentioned that authenticity was a challenge in the that type of placement with, you know, where they weren't assessed in placement and they came back. Interesting, although I think one of the interesting findings is that although the student voice with that national survey came up as realism and authenticity been really important, when it came to sort of placement types, and this was from a lot of the workshops, the student's biggest challenge was consistency when they were out in placement. And I thought I was surprised that actually this and in fact consistency came up in the workshops as bigger challenges than authenticity. Can I just ask consistency in what? So consistency in how the students were assessed. So things like, you know, they and this linked a little bit some of the findings with how they were graded. Consistency, for example, when they might go to one particular placement and if they were graded ABCD or percentages, but often ABCD, that if they go to such and such a placement, you know, they give all As, whereas if they go to another type of placement, you know, they don't do As, they only give Cs and Ds. So this type of inconsistency in context. And then within the sort of placement, and these are probably more in placement, some of these examples, that within the placement context that, you know, Mary would mark easily, and, you know, if they had Peter as a kind of a kind of a therapist or as a kind of a co-worker, you know, that they marked, you know, harder. And so this kind of inconsistency and that was often linked, I found in the findings, and this came up quite a lot in the idea of the grading scales. So if they use like a more pass fail or competent, not yet competent, they actually, there was less challenge with consistency, they were allowed to kind of just relax a little bit. So that's why consistency was coming up, I think, in a lot of placements. And that did lead to one of the some of the findings in relation to competencies. This came through a lot again in the workshops and in the interviews, that in writing competencies, you know, for a placement, so this is like, say, for example, the form that they use when they go out with placements, if this is a long list of competencies, there was a huge challenge with consistency, because people, sometimes people didn't even an opportunity to see things on the list, that came up, you know, there was a very interesting comment from one of the therapists actually in one of the disciplinary groups, and they said, we were making up scenarios, so we could actually tick a box, you know. So this came through as a challenge. So one of the kind of findings, one of the key findings is that there's manageable sets of competencies, and that they're not this kind of like, you know, 95 competencies, that there's some sort of a manageable set of competencies. And linked with that, I think another thing that came through is, if you have this manageable set of competencies, and then you could empower students to self-regulate and work with partners in industry, then you can actually take those broader competencies, but bring them uniquely to the context. And I think, again, this has come up in the literature before, but this was coming through as a lot of solutions. So yeah. So when this to me brings us back to our traditional notion and model of assessment, and it actually doesn't work. So, Jordan, it's intriguing. And as you know, one day, let's hope we work together because I can see that we're very think very much alike, and have done similar sorts of research. But to finish off, what do you think your findings have? What impact potentially have they got for higher education policy and practice locally within Ireland and perhaps even globally? Yeah, absolutely. I think the first thing that I would say is that we have to recognize in our policies and practices that employability things are very important to students. So I think we do need to make sure that our policies support the development of these skills. Another thing I think that matters, and I think this maybe is in an Irish context. And I think in some ways Australia and Canada and some international are ahead of us in this way, but I think the language matters. And what I mean by that is I think we need to start using terms like work integrated learning, which I know have been used a lot in Australia and Canada, but not probably everywhere. And I think this, it's really important that we start using this in our policies or the reason I think that is that there's, again, it takes away from this binary in policies of, well, we've got placement officers over here. We've got the education staff here with the students here and we've industry here and they don't mix or they don't meet. So I think in the language it's important that work integrated learning suggests a more integrated curriculum and staff and people and that dialogue. And equally, I think assessment language matters when it comes to the word assessment. When we talk assessment, we need to also understand, as in the Irish definition of assessment, that this includes the graded piece, the feedback piece and the self-monitoring piece, the self-evaluation. That when we talk assessment, we're talking that all foreign as learning and I think that matters. The other thing I think that is really important in policy is we need to build an interest stakeholder dialogue into our practice and policies so that we have opportunities to meet. We need to build in forums that get these people together in the room and looking at that we need to look at how we actually progress that kind of dialogue. We need to empower students in the work integrated learning processes. I know students as partners is coming into a lot of institutional policies but we need to look at make sure it's coming into the work integrated learning pieces. Competencies need to be manageable. I think this has implications for professional bodies and across the world. When they're really looking at their accreditation piece, when we're looking at quality sharing processes, we don't need a long, long list of competencies. I think we need to look at how we get the right amount of policies and the right pitch for our competencies. I think institutional grading schemes need to be seriously looked at in relation to how we grade in this space and how we actually, many of them talked about when we're using the grading scales because it's in our practices, our institutional policies and therefore, but we really would prefer to use, you know, like pass, fail or competent like incompetence. So with the institutional grading schemes, we seriously need to be looked at and how we value different types of evidence and transcripts with different types of evidence. I think this would be a really transformative change and I think that many internationally would probably feel the same in that space. And I think that my last point of this is practitioners, if they're involved in assessments, which we do want for them to be authentic to that practitioners are involved, but they need support in this space. They need to be resourced. They need to professional development to this space. We can't just land them into this and expect them to do this consistently and be authentic, be involved authentically. So I think resourcing and professional development for this broader group of practitioners is really, really key. Otherwise, we can't have consistency and authenticity. I think this is where it would fall down. So there'd be my key suggestions for a kind of policy and practice. Well, Jordan, it sounds like a very interesting and certainly important part of research in higher education and it's definitely of global interest. So we wish you luck in bringing this to fruition and we look forward to seeing how it all comes out in the end and reading the final report. So thank you for your time today. Thank you, Zanya. Thank you. Thank you for your time as well. It's great talking to you. Thank you. You too. Bye. Bye.