 Welcome to this afternoon's session. Sorry about the slight delay just trying to get the technology as usual I'm trying to get the technology working but we've got to go now. We have two presentations in the next hour and the first presentation is Paulo Santos who's going to be talking about digital competence and we are ready to teach in three European countries to say that so you make sure you're at the right place you're on the right one. Okay without further ado or to you. Thank you very much thank you for joining. Good afternoon my name is Paulo Santos I'm a research assistant in the Southern University in Germany and today I'm going to talk about this research some findings of this research that we conducted in connection with OpenTeach project and then we're talking about digital competence and OEA readiness and of teachers in three European countries namely Germany, Greece and Portugal. So I'm having this presentation we're having this project along with Professor Weinberger who's also here head of the Department of Editech is also going to talk about some interesting also research connected to OpenTeach in terms of quality assurance and so on after this presentation. So for this one now first of all I think we're all aware of the affordances and obstacles of OER and OEP and then the background from where we started this project and this research was looking at this connection of OERs in the movement of digitalization of of education right so we know that OERs can be a catalyst for digital transformation in education so OERs can promote this movement but it's also we believe it's two-way like this movement of digital digitizing education can also help establishment of open practices and so on we know that through OER we can also expand the access to high quality educational content we can promote different learning experiences with teachers that are technology enhanced and so on by this we can encourage collaboration in different ways that technology can afford and we can support different ways of dealing with different learning scenarios and support personalized and adaptive learning and ultimately promoting more equality and access to education right so we all know this I feel like I'm preaching to the choir but just to establish the context where we started this but like when we are when we're thinking about paradigm change and if OER and open education bring this kind of paradigm change we know that this kind of movements would likely take time right and this doesn't happen from day to night and we all know that there are some non-barriers and studies that show these barriers to be concerns about quality lack of relevant content in subject areas also difficulty in finding OERs time constraints from teachers and educators also lack of awareness about OERs and the benefits of OERs also copyright concerns in each country and culture is different there's these different laws and things that sometimes teachers are not also don't have the time to go after that and sometimes intricate and so on and also important here lack of necessary digital competence which is one aspect that we focused on this first part on this on this research this is all taking place as I said in the context of this project called Open Teach this is an Erasmus project that's we are focusing mainly on these three countries Germany Greece and Portugal and that the main aim is to our target group is primary and secondary school teachers and our the main goal is to to empower teachers to use OER and open educational practices and specifically by raising awareness about benefits of OER but also providing some guidance on how to create OER how to assess OER and Open Teach has mainly three pillars we have online courses this is a MOOC or different modules which we offer practical guidance on specific points like creating OER, modifying, sharing, assessing quality and so on the second pillar here is offering through the platform that we have a place for teachers to share the the resources that they have so we're offering this collection of peer assessed resources and also offering also a possibility teachers to connect to share experiences to exchange best practice and so on to to be able to to put this forward this movement of open education so basically this is Open Teach and in the as part of the needs analysis of this project we wanted to understand the status of OER readiness in these countries and we wanted to see whether the obstacles that research were there as well and as I said the here put the research aims was to identify the needs that we had there in the specific group of teachers also the challenges to OER adoption in these countries we also wanted to get a better understanding on the link between digital competencies and OER readiness so the questions that we were asking here is first the current status what's the current status in terms of OER readiness what are the teachers perceptions in terms of the obstacles for adoption and the relationship what's the relationship between teachers digital competencies and their OER readiness for that we had like a no-line survey that we had it run from September 2020 to August 2021 the beginning of the Open Teach project the we had the items with these variables OER readiness that were composed of perceived benefits of OER in education plus the familiarity with OER and creative common licenses we also have a new instrument for digital competence and for the perceived obstacles to OER adoption all these were created for these projects the items were all in English at first and then translated to the specific languages and where participation was voluntary the teachers were reached out by the network from the partners in this consortium so in terms of demographics what we had here from the collection of data was a higher amount of teachers coming from Portugal then from Greece and then the least amount of teachers coming from Germany it keeps wanting to go ahead so in terms of the subjects we had mainly physics chemistry foreign languages and mathematics we had 210 participants in total and here you can see their profile in terms of age mainly the majority from 40 to 60 and some more experienced teachers we didn't have we couldn't tap into these younger teachers and also in terms of education level that they were teaching mainly secondary level with some other representatives there but mainly secondary now in terms of the results that we we got in terms of digital competence when we're comparing the countries we found that Greece the teacher the great teachers were scoring more in terms of digital competence there so Greece was different in terms of and in comparison to Germany and comparison to Portugal in terms of familiarity there were similar the three countries familiarity with OER but in terms of familiarity with creative common licenses Portugal was Portugal teachers were scoring less there so I have another graph that can show how the situation is like you can see this is a count you don't mind it's not proportional right I mean these differences here should be seen because there are more people in Portugal but still what we can see is that in the three countries we had a large amount of or considerable amount of teachers that didn't have any knowledge at all about creative common licenses so this means like no not familiar at all to totally familiar and in Portugal you see the situation is more dramatic the proportion is dramatic there perceived benefits of OER in education in general they all had a positive view on the benefits of OER but German teachers were more skeptical they were not so in comparison to Greek and in Portugal German teachers were more skeptical about the benefits of OER so now going to the results in terms of obstacles that they perceived this aligned with previous research mainly time consuming finding is troublesome editing is difficult which also relates to digital competence here and also problems concerns about quality this was not a surprise after all and then the relationship from between digital competencies and OER readiness we found like a clear linear correlation between these I can turn so the more digital competent the teacher is the more OER ready the teacher is here we don't know the relationship in terms of the causation so on but still it's a clear relationship the medium effect size so going to the conclusions related to the first research question that we had in terms of the current status of teachers uh we could see that um in terms of familiarity they're relatively familiar but there's still room for raising awareness about OER especially in Portugal concerning licenses perceived benefits of OER we could see that teachers are more lenient towards the positive attitude towards OER but the Germans still skeptical um in comparison to the other countries uh research question two um teachers perception on the main obstacles it's in line with the past research there time constraints finding editing and quality concerns and uh going to the third research question the relationship we found this positive linear relationship and then digital competence could be like a limiting factor in a teacher to be open to OER so we have a need to understand more deeply this connection of these two variables and then coming to this I think this next this last slide here just important to acknowledge some some limitations to this study we had an equal sample sizes in even though we chose some tests that could make you know compensate for that but still the the difference in sample sizes could somehow uh change the the the the profile of the data that we got um also here the representation the focus the majority were from secondary education and this age range is also um perhaps is the relationship from uh concerning the age of teachers and and digital competence this is also something that's needed to to be further investigated in the future and uh here one limitation is concerning the nature of these instruments that were self-assessed not objectively assessed so future research we could investigate better the relationship between OER readiness and digital competencies trying to understand causal relationships and perhaps other variables there mediation and so on and um to understand better these cultural differences there are these different results that we got we could also go deeper into some other factors that could affect these differences cultural differences or differences in the education systems and so on so yeah this is it for this presentation thank you for your attention thank you for questions all of you right so we thank you for the presentation you've covered quite a bit comparison between countries i'm also interested in the absolute patterns for each country so where each of these items on a liquid scale going from one to five or what's the absolute scale there yes uh it was a liquid scale from one to five but then for example from for this for items like um perceived benefits um it was based on icard scale but it's like seven items also so uh this was this your question yeah for example if we're looking at vertical um how do i know is this high or low here yeah like what's the scale of the perceived benefits what's the max what's the minimum okay okay okay i understand yeah so it's it's from one one to five i'm sorry yeah you're right yeah it's not zero here it'll be one two yeah but it's one to five yeah then five is the maximum yeah you have a question yeah i would like to ask uh if on your first slide you tell us uh some reasons why we are and why we did this research and one of the reasons was the they couldn't trust the quality of the we are and now uh the your story was uh about uh competence of the teachers so i was curious did you um did you did any research about the other confidence that you talked about earlier uh what what other competence you told us that one of the reasons was that the they will ensure they trust the quality we are did you do something to address that or only did the research about the competence yeah yeah you're right we didn't target this at this point um basically this this study was also to orient orient take the course or the projects what should we do or should we have perhaps different approaches for different countries there and um yeah that's why the obstacles and so on but at this moment we're not measuring or we didn't measure the uh the perceived quality or how do you proceed all the hours are they proceeded but we have another study actually that uh but with the different population that we explored that and it's very interesting because we we also were wondering whether the near fact of thinking as OER has been free would this affect the perceived quality you know because this is free so this the quality might not be so good or so so what and uh would they do this in in uh the students in Ghana and we really found the difference there that people are influenced by this notion of you know when I pay for something in the quality it's higher maybe now we can discuss uh from the Netherlands and we have a quality model okay and we have one of our main goals is to increase the quality and use uh with uh OER and use the quality amazing yes yes was the competencies more related to to OERs like create I feel comfortable creating materials I feel I feel competent in um adapting educational materials from my teaching this kind of competencies that we're taking from um I guess DigiComp you know I think and then brought to the context of of your presentation and great okay I think that would do for the questions that's just right on time thank you very much again for all of them okay yeah hello I'm I'm in my professor of educational technology at Salomon University and this is a little study I did with Paulo Santos in a project as he was already pointing out open teach and you just might want to take note of open-teach.eu there you can find more information about our projects you can also find a platform which if you would use it I would be most delighted the only thing you have to pay is information because it's not open completely you need to register that's all and there you can find OER like know your platform and also a way of assessing OERs and that is what I'm going to talk about because OERs clearly building on a certain philosophy of knowledge and education being public goods for instance and these are really like amazing values that OERs representing here but it's not only values so values only exist not not only exist in a like a vacuum but really need to be realized by activities and most and even more so communal activities like for instance quality assurance assessing OERs joining participatory culture of of sharing OERs and building community by doing so and and we think that that there are different levels of OER activities very plainly but because when we're asking teachers so what about your OER experiences we get that the majority are using OERs finding OERs and using OERs that's that's really we're already there I would say mostly but creating OERs not so much it's only a very very select sample of teachers who would also create OERs and we believe that in the midway of these two extremes there is this notion of assessing OERs and sharing these assessments among colleagues and that is what we're about we here want to or what we did was basically we looked at different OER platforms and looked at how they asked users to assess OERs and then compare these different ways of assessing OERs this is a different assessment tools and looked at how they could be aggregated to make it more user friendly that's the basic notion more user friendly in terms of actively using the assessment that is doing an assessment and more user friendly in terms of the passive use that is to you know get the most out of the assessment of others and and well as I said the notion was to go from comprehensive to comprehensible because when we looked at these different international OER platforms we found a large set of criteria that they were using and in the first step we aggregated these and came to a much smaller list of criteria which we found out these are criteria that they would be kind of using together if we would kind of like say okay the best of all these different assessment kits would be resulting in this but then we're coming still to a list that is rather long or would be typical of of any one platform and what we wanted to do in terms of you know making it more aligned with common practices common online practices of assessing anything like on amazon if you like is to reduce that even more and then see if that would work or not so so this is what we did we came to these four criteria one is pedagogical and technical quality or let's say overall quality which is of course like if you're thinking amazon the main set of stars that you would give to any one product you want to assess they're doing it the same but if you're diving into an assessment and they have different sub criteria and so do we and that is modularity or adaptability inclusiveness or accessibility and curricular alignment to what extent does OER work for my curriculum and of course in this process of compressing all of these categories into this small set we then needed to lay out what these categories contain hopefully not just you know opening the range again so for instance the overall quality would comprise things as are there any typographical errors in there up to what the technical aspects like does it work at all and and is it good pedagogy or the accessibility is it in a format that can be changed at all versus one that cannot so think PDF versus Word document for instance and so on and so forth and on top of that we then wanted of course to have a visualization of that or a design that is reminiscent of as I said before these common online practices of assessing something like restaurants or goods on Amazon as I said right and this is what we came up with this star based a list of course having an open item here as well in terms of explaining the stars being given and also setting a title and such and some background information about the review which is of course optional what we then did was to compare this OTAT the open teach assessment toolkit with an existing established well investigated learning object preview instrument the lorry which contains again a larger or typical set of criteria to see whether it works and whether the users would be coming to similar results so these are the questions how do the teachers feel about the number of criteria that the OTAT covers to what extent do the OTAT criteria cover and correspond with the lorry criteria is there a match is that is that also visible and to what extent lead OTAT and lorry to the same assessment so are the OERs assessed differently just because you're using different scales hopefully not of course and then how is the open teach assessment to the OTAT usable in comparison to lorry so the people like to the people feel confident when using the OTAT in comparison to the lorry so to do that we used a system usability scale the s us which has items like i find it easy to use i find the rating system is to use i feel confident using the system i felt it too complex or unnecessarily complex there was too much information and so on and so forth and this is again a relatively well introduced usability score which then results in in a score that is if it's below 50 percent regarded as poor 63 to 68 percent is average and anything 80 in the buff is considered excellent so let's see research question one what about the number of the assessment criteria that OTAT uses is it too much is it is the number too high is it is the number too low and we felt like this is this is quite right on target that none of them none of the teachers we asked was having any serious concerns about the number of criteria and felt rather comfortable with with with the with the small set of criteria which again is less than half of all of any other set of assessment tools right and here we have this match between the lorry and the otat criteria so the lorry content quality matches most with the overall quality but of course you can see here a little bit of a problem it spreads out quite a bit we don't go into detail here the only thing you need to look at is how much range does it cover because it shouldn't it should basically match uh very precisely but but of course with content quality having being spread out across the range is is of course yeah well um somewhat more probable than with other criteria for instance learning goal alignment is perfectly aligns with the alignment with the curriculum of the otat or feedback and adaptation perfectly aligns with modularity adaptability and then motivation again of course larger range here but mainly again uh corresponds with adaptability presentation design now this is how does it look and such and this is of course covered mainly in overall quality so the the users the participants of this study we asked kind of found it rather uh or we're rather seeing the correspondence for instance between interaction usability and accessibility or inclusiveness or here the accessibility matches with accessibility that's good and the standards compliance also matches with accessibility here so there is a a rather clear relationship between the categories it doesn't spread out too much when it comes to comparing the assessments themselves along these criteria we found that there were no there weren't any differences which is good so regardless of what assessment toolkit you would use you would arrive at the same conclusions about the respective OERs that we handed out to the participants of the study uh to be assessed with the different assessment toolkits when it comes to the uh SUS scores we were happy with finding that lori was was regarded as average or just average almost poor and and otat was regarded as very good and of course otat also takes less time than lori which we hope would then contribute to lowering the threshold of analyzing or assessing OERs so that leads us to the conclusion otat beats lori in usability with a smaller number of categories while matching the respective lori categories and of course also arriving at the same assessments of OER and hence otat may be a contribution to making OER assessments feasible supporting a bottom-up approach to OER use and assessment and the whole idea of of this project is to really create OER communities and we found in this many European projects that participation has become the most valuable thing yeah it's uh people are not waiting for you but their time is the most precious thing so so this is where we are now to create OER communities of course through this this uh platform and hence your interest and your attention is most appreciated thank you very much and i guess i'm within time or you're more than i i think the slides it briefly showed that it was Luxembourgish and german school teachers in their work with yeah yeah and like how did you recruit the participants and what context were they looking at this scale in where did they have complete examples to evaluate yeah so in this very first um regarding research question one there was a separate pilot study where we just asked that one question whether the number of criteria are sufficient or or too many or too few and and that was basically in the context of a workshop so not like a random sample of course but rather as a way of introducing the the platform in or this criteria in the first place there were some open there was basically geared towards an open discussion about OER and then the question of how to assess OER and the feedback we got was that basically the the smaller version is very convenient to use and that is actually also shown by by this this result here that that would kind of hit the right mark in terms of the number of criteria which is maybe a no-brainer the less the less work the better in a sense or it's it's not so effortful but then seeing that that that doesn't lead to a quality reduction in terms of assessment was then very satisfying and this other this other study then was a sample of a 26 participants I believe also teacher students and that was then well not randomized in a sense of like we wrote in letters to all the German and Luxembourgish teachers and that's not really working but in the in the sense of of asking for for participation among student teachers and practicing teachers students or teacher or the institution or all three but we weren't thinking about the institutions as much because we kind of are a whole own institution building that platform which is free to use so so we were not looking for institutionary alignment of sorts which is interesting because that's a whole discussion actually but we were interested in the active and passive use so active use in terms of the people teachers basically they're doing the assessments as well as again teachers were reading the assessments also not so much in students students are not excluded basically they could join in and also assess or we are but we haven't considered so much to address them as in terms of students of specific subjects other than education so student teachers student teachers teacher students so so pre-service teachers would be well within our scope but not students of any sorts that we haven't targeted and institutionary integration is of course super interesting because in the course of this project we had established contact also with other projects who took a very different approach which is maybe particularly pronounced in Germany seeking institutionary alignment let's say so so basically seeking ministerial blessing and that results then in first of all the notion that there is not so much knowledge really about what is good learning material meaning also what are good OERs it's not so clear of course any any ministry of education within Germany and Germany you have to understand is here in terms of education totally divided so so the education policy is is is done by by the respective federal states and we have quite a few of them so so so they can of course and they do apply certain criteria typically asking the publishers to to take good care of specific I don't know interactive designs or certain quality criteria having an additional checks of sorts that is all very separated within Germany so they can come up with a list of criteria so it's not at all like a national approach but certainly the the ministries who are you know buying asked for certain things with the publishers now with OER it's a whole different game sort of and and of course the main concern is then the quality as well which is not really controllable so the so so many other projects took the took this other high road if you like to say okay um we're trying to establish a set of criteria that the ministries can work with to assure quality of OER which is kind of like a top-down approach and that's valid and good however we and that's the main message of my presentation we took a bottom up approach why so one thing is that these online practices of assessing something are really common we do all have an understanding of what these star values mean and we can work with them also teachers can work with them on the you know active side but also on the receiving side so that's good that works the other thing is however that we are a bit skeptical about the bottom what the top-down approach of of kind of like establishing quality criteria without taking context or specific needs specific preferences of teachers into account so um we we've seen a study of of another study of OER or teacher perceptions of OER and such and and it was quite pronounced that the teachers um felt their perspective missing in this kind of ministerial perspective on learning material so the whole idea of of this here is to say okay as a teacher I need something in my domain so and so and what is important for me is because I want to get to a certain point is that it's modular and adaptive because I need to tweak it I need to tweak it anyhow so then I can look at this particular value and I can make sense of it and can make my own OER because that is one I don't need to tell you one of the guiding principles of OER that you can remix and and you can make use of what is in the written review to to make the most sense and most use of the OER and then sometimes the teacher to teacher communication is much clearer to the respective teachers than any ministerial guidelines so so the whole idea here is without devaluing a top-down approach the whole idea is to fully use the bottom-up approach to to build communities of teachers to make teachers talk with each other and using OER as a sort of mediator to do so that's kind of like where we're coming from again not saying that any other approach is is worth less or something but but but but we also truly believe in this bottom-up approach here that's a question we've got we've got excited I don't want to keep you out of there any longer in relation to that's a question probably an observation I'm looking at it slightly slanty here so I might have misinterpreted it and the categories there are subjective in that if I'm a if I'm taking a resource I want to reuse it and I'm physics instructor teacher and then if an engineer teacher takes the same resource they might get a different result in this in each of these ratings so the ratings are only relevant within a context either a culture or in a discipline area and so how and this is probably getting I maybe miss something how this would be used so if our resource is assessed by 10 people they all assess it quite differently how does that manifest itself in a system that makes it usable for something coming along and saying I want is that a is there a step it's not the case the person who assessed this is a professor of physics or is that is that professor of economics and it's been assessed in that view in that context in that country for that purpose because modularities you know that's even subjective it might be modular for one subject and not for another it might be you know we see other ones that are inclusive in one country's context or one culture and maybe not another that's one thing the other thing is this is a this is a reflection from my some work I was involved in we developed a repository years ago in our institution one of the things we wanted to put on it was a star rating that created a whole big argument about star ratings about what do you feel like when somebody gives you one star for your thing how do you feel when you get one star so you and then here somebody else gets five stars the okay star ratings maybe not the best the best way to go because it's such a range there and you get one star a five star so then we went for the thumbs up the thumbs down well that was even worse because you could be all thumbs down and no thumbs and that just looks terrible and in the end we couldn't put any rate we couldn't rate anything because there was so much argument about what's the perception of that thing and like I said one person's one star another person's four star because you've got to be able to assess it and I think I told like this as an individual user to as a framework for assessing an object for re-using really within the context you want to use it it's a great thing but I just I can't get my head right and how you use it as a thing that that that amalgamates a lot of different reviews and I know it's a bit googly a bit of Amazon s you know it's power at the crowd and eventually if you get enough people doing it there will be some sort of this thing is always inclusive this thing is always modular and some of those things are but there'll be some of those fields that we will swing considerably and you would need a power power approach you need a big crowd to get up yeah you need a big crowd that's true that's the whole point of the wisdom of the crowd approach that it's a crowd but yesterday I've been to a restaurant that I used of course the star rating so it was very very useful and and of course if you're if you're um um competent in using the star rating as a passive user you make the most out of it because then you see okay how many people have voted um and of course you would look into the details the written reviews there's no doubt if you would ask me before I go shopping I look at the written reviews this is the most precious of course all of this takes effort takes time and and then the question is does anyone have time for the written review and we're hoping that that contributes to them taking the extra time to also write some qualitative thing the other thing you said was basically okay anyone can write anything uh the internet is a dark base and oh my god and that's exactly the reason why we ask users to register so you know who is assessing the OER that's kind of the whole point of it because when you do not ask for registration then you're creating dark places online yeah we're not doing anything else with this information but just to make sure that people can understand that uh if you're having like biology material and the physics teacher is is assessing them you might you know uh um think that the overall assessments of that teacher could maybe uh you know scrutinize to some extent so you know who is reviewing and and we're taking care of that through that that's the short answer I think just one final question there's what's quite interest of this I think it's a great thing a lot of the time we don't apply that kind of scrutiny to our standard teacher never mind our open education I don't think we go through that kind of reviewing and rigor in the stuff that we develop for everyday teaching and nobody's coming along and start rating it like that either you know it was a suggestion in our institution that everything should be star rated you know everything in our VLE every VLE module should have a star rating this this module is being rated by the students they've taken the module so that's the perspective students coming to look at that you can see how uh how well I don't think we do in that but I think it's a great idea that we do it because it is a big as you say quality is the big the big blocker for staff when they go quality reusability modularity right I just wanted to add to your question there it's very pertinent in terms of how reliable is if you look at terms like modularity and so on in this new iteration that we're having now we're actually adding some diving questions like for example when we have inclusiveness we're having some guiding questions for example if it's a video does it include you know closed captions if there's a photo so you can have some standard for voting so you can get more standard response so this is one thing and the other point that you also mentioned which is very interesting this uh you know what do you do with the one star you get the one star and so on and we also looked at this how these star systems are working and we see that many people or vote or give stars when the they're very not unsatisfied or we're very happy so we miss the the middle point and then one thing that we were considering but we're not implementing its moment at least but we were discussing this at one point and we're saying when someone gives a one star then the person needs to put there necessarily why is this a one star because for some person I know I don't like the colors of this and then when you look at one star I don't so this is not relevant to me it's just a color and it also helps the other one because this is an OER if you put the problem I don't like I cannot understand the language so you can give hints for the next iteration how to make the material by it great that's great right probably right on the bottom thank our speakers who told me for the actual presentations really join them you're the one thank you yeah and maybe one shameless plug again www.open-tgu you can use the platform right exactly and there's also learning or material on you know very basic stuff about OER use and assessment and so on so if you're ever in need of you know teaching someone about OER there you find very basic stuff and very extensive stuff I would recommend to you if you want to plug it a bit more if you go to the discord group the ruminant and just put it out there just see what it publicizes stick it out there all right yeah that will do then so then thank you very much thank you very much