 Welcome Peter. Thank you. Thank you very much. Hi Vicky. Hi Vicky. So welcome to my session on how to design a programmatic assessment for lifelong learner skills. I will start by asking how many of you are familiar with programmatic assessment or the concept of programmatic assessment already. One cool and come in, come in. We're also doing a recording so and what kind of. How do you know programmatic assessments. Sorry, medical education. Yeah. Yeah, that's the traditional place in which programmatic assessment happens a lot. So I'd like to first start with an introduction about me who am I, who are we dive into learning motivation and the aims of assessment we have then into the topical programmatic assessment, how to employ e portfolios for programmatic assessment and lastly I'd like to introduce Portflow as a solution. So, my name is Peter with an extra I just like dream, you just say Peter and dream. And I work as a product manager for Portflow at dream for a little over two years now. So as an education list, I studied educational science and technology at the University of 20 in the Netherlands, and previously I worked as a product strategist for a little over six years at Genzi aimed at creating attack at primary for primary education. So I'm here to continue to pursue my passion for promoting intrinsic learning motivation using at tech and ownership of students here at dream as a product manager, aimed at higher and further education. So looking at the challenges we are facing in our education at this moment, in which change is known and embraced as a constant so we are preparing students for rapidly changing future with increasing rapid changes due to changes in climate technology and also globalization in a lot of unpredictable unpredictability unpredictability. Yes. So we really don't know what the future occupations of our students would look like. However, what we do know is that change as a constant requires a lot of flexibility, and also responsibility at the end of our learners as well. We've put an increased importance on to lifelong learning and a key focus on the development of the wide set of skills and competencies, rather than just grades and courses. So if you ask yourself why are are our students learning. The ideal answer would of course be because they want to expand their skill sets, because they want to expand our knowledge, because they want to have a big well paid job. But practice also tells us especially in higher and further education students are learning because they are expected to research has shown that the more students mature in education so after finishing primary education second education going into further and higher education. And more and more focus is put on stimulants of extrinsic motivation and their intrinsic learning mode with motivation. As a consequence, suffers suffers suffers from that. A lot of things you might hear students say is I'm not going to this next class. If it's not mandatory, because I'm sure it will be a boring one so they really perceive their education as consumers, rather than directors. And many current practices in place in our current education system that inhibits the intrinsic motivation of our students, such as grading assess assessments. Big data of course especially since the possibilities of digital education, digital supervision supporting famously or infamously grown since the pandemic. On the other hand, we try to counter these effects by more offering more personalized flexible and self directed learning based around competencies and skills. But how can we achieve that. Before we dive into that answer. We're talking about assessment, a little intermezzo on the purposes of assessment we can have. So, in essence you can discern for main purposes of assessment. The first one is to use assessments to have a tool to sort and select students so classifying students by their level of ability. And whether they can transition into the next year of their program, etc. The second one is stimulating focus so pointing out content in your courses that is especially relevant or especially important. So directing the attention of your students to the things that you think are important. Next to that is of course, monitoring the process of progress of the students track development on their competencies, the learning outcomes, have they understand what they should understand by now. And, lastly, but not less important is improving the learning process itself so reflecting on your practices as an instructor. Understanding the learning process through the weaknesses and strengths of everything that has happened in the tools you've used the activities you did etc. Unfortunately, the three first aims of assessment are rather focused on extrinsic motivation. So, focus on the outcomes rather than the process itself, and only together with the fourth one you can focus more on the intrinsic motivational so allowing monitoring of process for example progress for example to focus more on the learning process also improving the learning process for the students. So, introducing programmatic assessment as a concept. It's definitely not a recipe so it's not that doesn't tell you how to organize your education but it's a concept aimed at improving the learning process, both for the students, as well as for the teacher. It's about continuous measurements rather than a final high stakes assessment at the end of the term or the end of the course. It's about longitudinal assessment and a holistic view of the students both in their academic competencies but also on their social growth. So focus largely on qualitative feedback, rather than grades and marks, and really aimed at growth encompasses three types of stakes, which we call low stakes intermediate takes and high stakes valuations assessments so called data points. A big characteristics of these especially low and intermediate stakers assessments are that there are no distinctions made only feedback and ideally only qualitative feedback. And then at the end of a certain periods or a certain moment you can have a high stakes assessment and is largely integrated within with learning outcomes, competency skills, etc. We see a lot of instances programs that employ e portfolios for the purpose of programmatic assessment because especially because they are especially suitable for this purpose in which students can set goals or the student can be held by the program by setting goals, which can then again be competencies outcomes skills during the learning process they can collect evidence of their activities what they have learned the feedback. And these this evidence is not only aimed at the result of the learning process but definitely also about the process itself. And they can link these evidence is to goals in their portfolio, manage access who they want to share their portfolio with externally internally within and outside the education institution involved stakeholders take our best to identify a request feedback to anyone evaluate reflect on the process and then after a couple of cycles do a final assessment based on their evidence the intermediate stakes assessments and use their portfolio or private portfolio for showcasing purposes. And this diagram gives a schematic overview of the different stakes of assessment so in the low stakes, you have a lot of them, mainly consisting of assignments activities, experiences, consisting of artifacts and feedback ideally, and then at certain points you have these intermediate evaluations based on the gathered low stakes aimed at one or multiple goals objectives outcomes, and then at the end of the term, for example, or at the end of the course, you have a summative assessment based on the results of the intermediate stakes assessments, usually. And our advice here is to use a student owns portfolio, rather than a institution owned portfolio for this purpose. For both the assessment and development of the students, as it allows for authentic assessments like I just mentioned students are able to clearly identify any stakeholders that have anything valuable to say on the learning process. You also embrace the flexibility of students, you offer students a way to reflect and prove their efforts on their personal learning journey, you offer personal control. Like I said, you can have students set their own goals, augmented with the goals that you set as an institution or as a program. You can allow students to manage access initiate feedback and really elaborate on their reflective learning at the moment they seem relevant. Of course, you can also schedule this process. You can also design the learning activities and by allowing them to build their own portfolio. So, lastly, and workflow of course is a solution that fosters this process allows you to build a personal students owned portfolio within the community currently operable with canvas and bridespace and moodle will be one of our next integrations. And the advantage of a student owned portfolio is, in our opinion, one of the best strategies to teach responsibility like I mentioned in the beginning responsibility for this is a key factor of success in this uncertain process of unpredictability by fostering ownership of the learning process. And appeal to the intrinsic motivation of students harness their flexibility to deal with the imperfect, unpredictability, and also by addressing diversity of students are really fostering inclusivity of all kinds of students and their own stories into their own personal portfolio, promoting strengths rather than weaknesses, promoting their focus on the competencies they want to achieve, they own and also their learning preferences, enabling students to pick to paint a full picture of themselves as a learner, and also to allow them to reflect on that holistically, including also possible extracurricular activities. The portfolio is fully integrized integrated integrated and synergized with the VLE LMS, meaning that portfolio builds upon the sound foundation the LMS already offers VLE already offers by minimizing the points in which the system would compete so for the end user, the VLE in this portfolio is being experienced as a seamless integration into the VLE, meaning that they don't leave the VLE there are no new tabs opened. They can import assignments from the VLE into their portfolio and link them to the goals objectives skills in their portfolio, and they can also create snapshots of parts of their portfolio and submit them in their VLE to be graded in the VLE. And, importantly, it's very important to offer a wide range of flexibility, not only in the sense of how and when you want students to learn, but also in the dimension of where you want control to be at what moment in time, or in what context. So, as we do recognize students as the most flexible partners in embracing imperfect ability. We think it's important to really support the wide spectrum between teacher driven and student driven education. Meaning you could have multiple scenarios at any given moment, depending on the abilities of the students, but also depending on the context of a course the aims of objectives of a program. And you can decide to build more student driven education later on in their program and scaffold their pro progress towards it. Together with the VLE portfolio offers teachers and instructors and program managers toolbox, rich toolbox. To offer that scaffolding to their students by giving them instructions, deadlines assignments check-ins but also appraisals grading valuations and using templates and feedback along the way. And you can use any of these tools, depending on the purpose you have and the spectrum and where you want to be on the spectrum of control between teacher and student driven education. So, when we look at portfolio itself. There's a rough learning trajectory we usually use to explain the features of the portfolio. Starting with the planning phase in which students can set the goals, define the structure of their portfolio, and in which they can be scaffolded and held by templates, by instructors in the programs. In the learning phase they can collect evidence of the learning experiences, access, determine access, gather feedback, discuss feedback. In the tracking phase they can initiate so called progress reviews for the intermediate state assessments, gather insights on the learning progress towards their goals. And in the end in the proving phase they can create snapshots of their portfolio for assessments grading and showcasing. Going through these phases one by one of course in real life these phases are far more intertwined than we do presume here. And this is an example of of workflow and abstraction. In this case integrated in canvas. So, usually it will be added to the global navigation the main navigation of the VLE as it's not tied to a specific course it's really a course overarching portfolio every user will have their own portfolio. And structure their portfolio with these folders you see down here which we call collections and these folders usually refer to learning contexts. So that may be projects, courses, internships, modules anything the students has been learning within, and the student can create these collections themselves or they can be imported through a template. And then within these collections the student can add goals and goals can be anything the student is working towards so it can be a competency skill outcome learning objective. And the student can then link these goals to their collections and link evidence within the collections to these goals. In order for these goals to provide a course overarching overview of what has been happening has happened to that goal and like collections goals can also be imported with a template to a student's portfolio. In the learning phase students can add evidence which can be elements assignments. Which they can import into the portfolio, follow uploads from their own computer Google Drive one drive links to anywhere on the web, pre text evidence for quick created evidence or video evidence, and they can also browse existing evidence and also reuse existing evidence to link to their goals and collections. And one of the advantages of linking or importing actually elements assignments is that you would also able to import the feedback that you received in an element assignments to really show that authentic feedback within your portfolio, and to continue the discussion and reflection of feedback. Next to that it's possible to have multiple force versions for each evidence so you can really start by uploading a draft version of a piece of evidence, sharing that with a couple of peer students. And then iterate with newer versions, in which you indicate what you have changed, enabling anyone who has extra portfolio to browse through all your evidence versions, while still seeing all the feedback that you received free to these versions. Everything is conveniently labeled, really showing the interest of learning process of the students and making that visible. It's possible to share your portfolio with anyone within the organization. Through the integration we have but it's also possible to invite people outside of the organization to your portfolio or just to one specific collection, for example if you have a collection that contains all your evidence from an internship. You can also invite your internship assessor outside of your education institution to access your portfolio provide feedback, which is then be, which can then be granted through a two factor application. You can even send directed feedback requests and all the feedback is covered in threaded discussion so visually identifiable what comment is about what piece of feedback. And you can also enrich feedback with file attachments with formatting and also create audio recordings for more nuanced feedback. And then in a tracking phase, when a student wants to know where they're standing how their development is going towards their goals and goals achievements. You can initiate a so called progress review, which would gather, not only qualitative feedback but also quantitative feedback on their progress towards their goals. So in that case, they can invite people to their portfolio and ask them for a quantitative review of their portfolio based on the evidence that's currently there. The goals they have selected so they can select a set of goals. And then every person you have asked for feedback for a valuation score gets the opportunity to raise your performance on each of the goals that you have selected. And that's an aggregated into these kinds of insights. So you can see the insights aggregated per role so yourself evaluation, the evaluation of your coach evaluation of an expert maybe also evaluation of peers, teachers. You can also see your progress, again, aggregated per role that has evaluated you. And then lastly in the final step. The student can create a snapshot of their portfolio or part of their portfolio in which they are able to add a plea description and submit that portfolio within the VLE or an external grading application for summative assessment. That same snapshot can also be used for showcasing purposes, meaning that they can create a showcase link of that snapshot, which would then give anyone with that link access to your portfolio but then without the feedback. So just to finalize versions of your evidence in the way you have organized and curated your portfolio. And of course also possible to create a export of your portfolio or specific snapshot. We're also integrating with badges my credentials. Very important to also offer more flexible education and recognize the earlier smaller steps and achievements of the students. And we have a case study of one of our successful partners long time partners that has transformed their education using workflow and embracing the unpredictability by allowing students more ownership of their learning. And of course we have a lot of our institutions that can vouch for that. Any questions. I think that's it then. You have a question. Yeah. So, how do you have any evidence or improving the experience of the neurodiverse students using this approach. I don't think we have any scientific evidence on that we do have we did have some conversations with our customers, especially with our longer going customer. And that indicated that also neurodiverse students that they feel they have more by the fact that they have more control of their portfolio they feel to be more in a safer space, which they control. So they decide when who how to ask for feedback. And they decide have control over how to deal with the feedback they received how to organize their portfolio but also how to prove that they are, they have achieved a certain competency. Yeah. Yeah. Yeah, that's very important. Welcome.