 So, hello everyone. Welcome to another event organized within the European Distance Learning Week, which Eden has started and for the first time, based on previous very good cooperation with the American Distance Learning Week. And I'm happy that through all of this week we have a number of webinars. We aim to raise more awareness about the issue of distance learning, about the digital technologies, modern education, all things which surround us and which are very much the importance of today. When we are talking about education. So today webinar is titled, as you can see, validation and recognition of non-formal open learning. And it is the fourth webinar in the row since this week. So I hope you are still eager to participate and join us today and tomorrow for the last webinar we have. Today, I have three speakers. Andrea Inamorato dos Santos from GRC-CV from European Commission. And Irina Vietciana from Botatus Magnus University in Lithuania, who is hosting me for this webinar as well. So we are room to room. Now, and Gordana Jugo from Croatian Academic and Research Network, Carnet from Croatia. I'll be moderating today this webinar. Okay, and I'm saying hello to all our participants in the chat and hello everyone. Great to have you today with us. We are going to talk about the issue of digital non-formal open learning, but it means not only non-formal informal learning as well. And before starting with the first presentation, I'll just ask colleagues to give us questions. We prepared two questions for you to try to see what you think about them. So let's just have questions from the pool. Yes, the first question is what is the difference between online, formal, non-formal and informal education? So you have a number of possibilities here to choose. We will later, you can answer the questions during the webinar. We will at one point try to discuss the result. And if we can have also another question as well at the same time, I hope they will be both together at the same time. The second question is who can issue certificate award of qualification for online non-formal and informal course? So here are the questions for you to think about a little and we can discuss it later. And now I'm going to give a floor to Andrea Inamorato, who will present the European Commission work on this topic. So Andrea, the floor is yours. Okay. Thanks very much, Sandra. I'd like to say hello to my colleague speakers and to the entire network that is joining us today. Thanks very much for the opportunity to be here today again, showing a little bit of our research work on different aspects of open education. I can see that some participants also joined yesterday, so it's nice to see your names there again. And so I participated yesterday more talking about open education as a whole, giving an overview of our understanding of open education here at the GRC, at the Joint Research Center in Seville. And today I'm going to talk a little bit about a specific study that we ran between 2014 and 2015, which deals with the recognition of non-formal learning, which is called, amongst ourselves, the Open Cred Study. Some people that carried out this study are joining us, such as Gabby Whithouse. Welcome and please feel free to collaborate and bringing your knowledge as well, Gabby, because you were one of the main authors of this study and we welcome your contributions. Okay. So, just a second, I need to rearrange my screen. Okay, so here we go. I wanted to say that recognition or validation, and perhaps it would be interesting to discuss a little bit about this terminology, recognition, validation, informal, non-formal. I think it's important that we touch upon these aspects later. But validation, recognition of non-formal learning has been in the European agenda for a while. So, if you have a look at this slide, we have the council recommendation dated 2012, saying that by 2018 we should have some sort of arrangements for validation of non-formal learning. Apart from that, something else that supports our studies on open education and on recognition in particular is the 2013 European Commission's communication on opening up education, which steps upon open educational resources and the importance of pushing the field forward. So, all these studies we carry out here at JRC in Seville are also in order to support this communication a little bit more. And this open credit study is one of them. So very briefly, I will try and present these three things. First of all, the open edu project and locate the open credit study within the open edu project, which is the overarching project hosting this study on recognition. And then specifically show one of the, I'd say, most important outcomes of this study, open credit, which is the traffic light model, which is a sort of a visual representation for open learning offers, MOOCs, but mostly MOOCs that universities can use to help aid, let's say future recognition of learning, of open learning. So, very briefly, the open edu project. I think I've given you a brief introduction. It started in 2013. It ended in 2016, beginning of this year. And we had a number of publications related to this project in which open credit is one of them. And now we have a follow up study called open edu policies in which we are dealing with the more overarching level of policymaking at both national levels and regional levels in Europe in relation to open education. So this is the most recent project that we have. But in order to be able to carry out all this research in open education, we had to have a working definition of open education, because open education, as you know, means something different to different people. You know, no matter who you ask, you may have a different answer. We've done that before many times here just to try it out. And some people will focus on open educational resources, others will focus on MOOCs, others will have a more broad perspective in overview and try and combine different perspectives, OER, MOOCs, technologies, recognition. And what we wanted to show was that by means of our evidence in research, was that open education for us is understood as something that goes well beyond OER and MOOCs. OER and MOOCs are part of open education as much as recognition. This thing that we are discussing today because recognition can be a strong enabler of open educational practices and also an enabler of modernization of higher education in Europe. So as you can see from the definition that we had, we discussed with many experts along with the open edu project. And we included, we talked about open education being a variety of access routes to formal and informal education, bridging them, bringing them together and making this bridge. And here is the importance of the subject that we are discussing today. Okay, so here's the overview of the open edu project. So you can see above we have four studies being part of this project. Two of them are on a qualitative basis, which is open cases and open credit, the one I'm talking about today. And two of them have a more quantitative perspective, the open survey and the MOOC knowledge. So the open cases study is published and it's a collection of nine cases studies exploring open educational practices within higher education institutions. And one enterprise in Ireland in particular, open credit, I'm going to talk about in a minute. Open survey is a representative survey of five countries, Germany, Poland, UK, Spain and France, in which we asked these countries, these universities, what were their practices in relation to open education, what sorts of things they did, whether they offered MOOCs, whether they had an open access policy, whether they were simply trying to increase the use of technologies to widen reach. And open survey is also published as much as open credit. And if you are interested, you can find more about it. And we also included a question on recognition and I'll show you the results in a second. And MOOC knowledge is a study on MOOC learners. It's ongoing. Now, with all this information, we created the open edu framework. Apart from in-house research, consultations, workshops with university directors from 19 countries, online consultations with experts. And this is all what has aided us to come up with the framework itself. And I think it's important to show the framework because one of the dimensions of open education that we have identified is recognition. So the work I'm talking about today, we like to see placed here in the recognition dimension with all its importance. But not only that, I think it's important to state that when we create the framework with this visual representation, we wanted to show that these dimensions do not work in isolation. They are depending upon each other and much better when they work together. So if you're talking about the recognition of non-formal learning, we may also talking about developing university strategies for that, having new technologies for that, making sure we have quality on assessment and make sure we have leadership on that type of project. So I just wanted to emphasize that they all working alongside each other. Okay, so moving on. The open credit study here is the formal reference to it. It's called validation of non-formal MOOC based learning. So in the end, it ended up focusing more on MOOCs. And from our side here at the IPTF, I'll show you a little bit of our main motivations to have this study. At the time in 2014, when we come up with the research design of this study, we wanted to know MOOCs are still popular, but at the time perhaps even more like fashionable talking about MOOCs. And we wanted to understand what sort of impact having a MOOC certificate could have to a learner. That was the main motivation. And we decided to triangulate the research methods and investigate the perspective of the learner themselves, the perspective of the professor or the lecturer who was then going through changes and developing new ways of teaching, going through the challenges of designing and offering MOOCs, seeking for university approval and all what comes with it. And if you're involved in MOOCs, you must know that it's not always that easy. But we also wanted to know the perspective of the employers. So if you have a non-formal certification, what's the acceptance of it by the employers themselves? So we wanted to triangulate. That was perhaps very early days, and we did do that research on all the 28 member states trying to identify cases that would highlight those issues for us. We didn't find information from all the member states, but we did find from some of them. So we carried out interviews with the professors, the lecturers that were creating the MOOCs. We interviewed two learners studying those MOOCs. So it was interesting because we had the perspective of the lecturers that created and offered the MOOCs and then of the learners who took those MOOCs. And then we talked on a different perspective with staff members of employer bodies in terms of how they perceived this type of accreditation. We have cases studies in the report, and this is the cover of the report itself. Okay. As I said, in some countries we didn't have any relevant information at the time. Now, I mentioned a little bit about terminology before. And one of the things that came up very strongly and more towards the end of our research process was that we needed to differentiate credentialization and recognition because very often we are using terminology interchangeably and we're not quite sure what we mean by them or you may say, I actually mean credentialization but I'm talking about validation and recognition. So we made sure and try and bring a little bit of a discussion of these items. They may not necessarily be seen as definite. Someone else may have a different perspective on it and criticize it. It's absolutely fine. But we did try and make it a bit more explicit and consulted other types of literature like state default and other European sources to come up with this. So when you talk about credentialization, we are actually talking about the act of issuing accreditation to the learner. So the learner studies and the university then issues a credential to certify that learning and that can be a badge, that can be a certificate, that can be some sort of credential. Okay. More formal or more informal. Now, recognition, however, is what would come afterwards. Our understanding is that a learner who has the credit credential could then seek recognition of that credential and that could be inside of the same institution which issued that credential. For example, for moving from one course to another, from one faculty to another to compile if it has credit towards the degree or it could be externally seeking recognition with another institution or even another institution in a different country. And when we are preparing to offer MOOCs and open learning, perhaps we should be thinking of all these possibilities of all that learners could perhaps want to do with that certification, with that credential afterwards to explore that a little bit. Okay. One important thing that we have is the traffic light model, which is this one. I'll come back to the other light in a second, as I mentioned before. So this is a visual representation of what we thought would be interesting for a MOOC, for example, to have on its very front page. Okay. When a learner is searching for a MOOC or is searching for a subject to study, what we realize is that sometimes there's not enough information on the course itself. It's not clear how assessment will take place. It's not clear what sorts of recognition can be sought afterwards. It's not many things are not really clear to the learner. And so having this traffic light model as a quick way for the learner and for the institution and for people who will in the future perhaps recognize that credential to help them think through the different aspects that were contemplated in that course, in that MOOC. So first of all, the colors there indicate, the green color indicates the more likelihood, a more strong presence of these aspects. Here that we say, for example, green is strong in terms of identity verification or supervised assessment. Yellow, to some extent, those aspects are present. And red is when those aspects are not present at all. And so these items here, identity verification of the learner, for example, was one of the most important ones, a very important issue. So recognition then is very much linked to assessment practices. So this is another important finding for us. If we have proper assessment practices and that we can show in a transparent way how assessment has been done, it's much more likely that in the future that credential will be suitable for recognition. So I'll come back a little bit to the previous slide and show the six elements that support recognition according to the open credit study. So as previously mentioned, the identity verification of the learner, we think it should be very transparent the way the identity verification is done. One of the interesting things is that it was also mentioned in the report that sometimes proctoring and this type of monitoring at a distance of assessment does not always perceive that something reliable to employers or to the external world. One of the learners actually mentioned that specifically. He says, people perceive that a course that was assessed online has a lower value than a course that was assessed face to face. And this is why there is a model in the open credit report in which the learner actually could pay and go to the institution itself and see the assessment on the presence of the actual lecturer who offered the course online. And it was interesting because in this particular case, the learner said it's expensive so that may be a barrier because it costs more having face to face assessment of an online course. But on the other hand, for me, it's good for two reasons. First of all, because I really wanted to meet the professor face to face. I really like the way he teaches. I am a fan of his work and I think it's an opportunity to meet him, but also because it will increase the value of my certificate of my credentials. So this is something for us to think about and perhaps discuss a bit more. I will have only two or three minutes more and I'm done. And that relates particularly with suitable supervised assessment. In that case, suitable was identified to be more face to face than online, although online technologies are every time more and more reliable. So I'm not arguing myself for one or the other. I'm just showing some of the things that came up in our study. The importance of offering badges or digital certificates and also because of all the metadata that can be contained in those types of credentials, which can be more easily transferred from one place to another. We are nowadays every time more thinking about blockchain technologies for this type of accreditation and recognition. So metadata is really important. Quality assurance, absolutely. It was present all the time. We have to make sure that the assessment has quality. The award of the CTS credits was seen as something important and partnerships and collaborations. So just to mention one item, one type of data that we have from the open survey or the survey of the five countries. We asked about collaboration in MOOCs in those five European countries in terms of recognition of the learning. And out of five countries that we investigated, that we surveyed, 41.4% said to us that they pursue national recognition of their MOOCs. So those credentials would be valid inside of their countries. And 3.9% of those five countries said they were already pursuing cross-border international recognition of those MOOCs credentials. So just as an information. And just to finish, we had some recommendations towards the end of the report. And I mentioned some of them that we need to have more transparent information. We need to disseminate good practice. And I think this webinar already serves no to a certain extent meet with recommendations. And here is the open cases report. Also, because we have examples in there that deal with recognition. Thank you very much. I hope I didn't go too much beyond my time. But I'm willing to get your questions if you have. Thank you, Andrea. Can you look at the questions in the pool and give your first impression on the answers? For the first questions between different. Okay. Between formal and informal learning. Okay. We had to look into that for this report. And we took the perspective of CEDEFOP. CEDEFOP has a publication which is about terminologies used in education. And they define formal, non-formal and informal. And we think that was a very coherent way of defining those three things. So, by formal we understand, you know, there is usually an institution accrediting or certifying or stemming that credential. The non-formal side, the non-form, a non-formal course, for example MOOCs are very often, depending on how they are offered, they can be seen as non-formal. To the open learner, for example, they can be non-formal course, non-formal course. To, when it is embedded within the curriculum and this part of the assessment in final grade, it can be formal. So, it depends. Non-formal normally has some sort of curriculum path. The content has been sort of by somebody and put it in some sort of order, let's say. It doesn't happen that naturally, as much as in the informal type of learning. The informal type of learning, normally is that learning that we do when we are talking to friends, that we acquire at work, doing different tasks, you know, when you're helping your child to do the homework, for example, you're learning a new recipe, watching TV, reading newspaper. It's more informal, but something that you can build upon, such as professional experiences that you acquire at work, but not necessarily with the final intention to learn that particular thing without a structure, without a curriculum or a fact structure behind. That's really what we understand, normally what we understand by... Okay, thank you for the moment. We'll move on and then go back to the questions, to more questions. So, now I'm asking Irina to present the new project, the new Erasmus Plus project, which has just started. And Vitatus Magnus University is the project leader in this project, in this re-opened project. So, Irina, can you tell us what we are aiming to do in order to enhance the recognition of non-formal learning? Yes, of course. Hello, everyone. I'm very happy to have this opportunity that we have this issue on the agenda at Eden Organized European Distance and Peel Learning Week, because when we hear our discussions among member organizations, I know that higher education is a little bit more active on the issue. However, more and more we open the question how to involve companies and how to establish collaboration and how to agree and how to have practices and experiences and cases on already existing good practices of linking these different types of learning, especially in the context of open learning. Our project is called Re-Open, and I'm sorry, didn't put the title, which is official, and it sounds like recognition of valid and open learning. As Andrea presented, the project was based exactly on the recommendations of John's Research Center study on validation of non-formal MOOC based learning, which appeared just in February 2016. However, it helped us immensely to finalize and shape actually our ideas that we have been developing with the partnership, and then after the appearance of this study, we just clicked everything and everything seemed to be in their places. So we gained more self-confidence to finalize the application to submit and our application was successful with funding. So now we have, of course, a lot of questions and challenges, and just currently we had our partner meeting and here we have partners in Participant List and you can see them under the Re-Open project team in Participant List here, so behind the neighboring room. So we addressed priorities of open and innovative education, training in youth working, in digital era, transparency and recognition of skills and qualifications to facilitate learning, employability and mobility. So those issues are definitely addressed by these topics, as well as access to training and qualifications for all. We targeted at CVEC because actually our target group is teachers, trainers, adult educators, like-long-learning educators, so we move out of the area, out of scope of higher education, but actually our bigger aim is to establish collaboration among different sectors of education. So in terms of opening up initiatives, of course, we focus on organizational change and on curriculum opening up. And we agree that we do not target specifically MOOC, but we target different types of curriculum designing models and scenarios that would be meeting opening up initiative as well. The consortium that joined, you can see now on this slide. So we have experienced partners who are very experienced, for example, in open learning, but also some partners who have long tradition and long-lasting experience in recognition of prior and non-formal learning. Also, distance learning. We have companies, we have lifelong learning providers, so we think that we have representatives for our aims and objectives. So now you see clearly the study address and what we especially highlighted and grasped from the idea of the study is that we are aimed at establishing validated open learning practices. I don't know if this is exactly actually how Andrea and how Gaby can interpret now, but this is exactly how we put it in our project. So in other terms, we want actually to create validated open learning space to experiment it and to have it as a case and to have internal decisions, whether we are ready and how we are ready to mainstream with different types of organizations. So we really are interested in playing with it. Another thing that we addressed directly is offering learning credentials. And today we agreed with our consortium representatives that actually we want to prepare a very short survey to be distributed among education providers, asking them what existing practices they have in terms of authorization, verification of learning. Whether they have experiences in cases of learning agreements, what are the elements of the learning agreements that they have, and also if they have these instruments in place or if they plan to have it. And then we want to have interviews and to approach for what solutions to accept in the project. So the third item would be establishing digital badges for recognition of learning achievements, but after discussions again with the partnership and following the study recommendations, we think to look at the term of recognition and a little bit in the broader sense so that we don't take it only for certification and already recognizing learning achievements, but going through recognition of learning path. And this is also what we identified in the previous Le Manson study and that was very useful for us. Now we identify that it is very rapidly and it is very rapidly developing is that different platforms already offer monitoring of learning results tool that can be applied through the whole learning process. So in other words, we don't only want to see the learning result, but also to measure the learning process. The discussion went, for example, if we are in the context of higher education, we know that sometimes universities, even in the exchange agreements, in the collaboration agreements, they do not offer very easily recognized learning outcomes achieved in a different organization, not even talking about open learning environment, but also within a closed virtual learning environment. Why? Because, for example, in mathematics, we can have different solutions, different approaches to the solutions and that is why it is not only the result that matters. So this is addressed in the project. And of course, the biggest target is establishing a collaboration with institutions of different types to provide transparent information on potential recognition of open and online learning and to help them to prepare to design curriculum for open and online learning for recognition. So as I mentioned, we have very broad target, but it is within the interest of our experiment, which is the teachers and trainers from different levels of education institutions. So now to concrete aims and objectives, the open aims to create instruments to develop validated open and online learning for recognition of prior and informal learning. And then we will be reached to the following objectives. The first objective is to design a platform for non-formal open learning curriculum and purposefully choose whether it will be for MOOC or whether it will be for not massive open online core development with learning validation and recognition instruments in place. So we are going to implement learning credentials for authorization of learners, for verification of learners, for tracking their learning process. Digital badges will be one of them, where we plan to have other tools like learning progress measurement tools and maybe information system for the learners and other things. Of course, we discussed today that actually European organizations have very, very different scenarios. They have very different traditions from let's say application of already existing e-government tools whenever we can have a request for our learners in universities to log in to their, let's say, e-card and e-identification data until very simple tools. So we will see which solutions will be applied in the project. But then, of course, the next item which is very important is to train teachers and trainers at CVET organizations, companies and higher education institutions and adult learning organizations to design how to design a validated non-formal open learning curriculum. I think this will be one of the major achievements because it will involve application of digital badges as an example of credentialization, tracking one's learning path and then to recognize non-formal open learning results in formal curriculum. So for this objective, it is very important that we have around table discussions with all stakeholders involved involving companies and then we will all benefit from this because we will somehow find the ways to introduce companies at their own stage negotiating competencies or learning outcomes that are usually embedded in online learning environments. And this way maybe the bottom-up approach will be to approach program committees in formal high education institutions also stakeholders in VEC to review their programs in terms of agreements with the companies. We'll see how we will manage. The third objective is to explore the new platform and to design on formal open learning courses for continuous professional staff developing. So these are short-term courses and then, of course, to establish future partnership for collaboration. So we have now when we started only kicked off our project we have a lot of questions, pending and issues raised already and we are now already discussing how to approach them and we have agreed upon surveys and interviews about round table discussions and dissemination events in order to collect as much information as possible as many recommendations and existing cases from countries involved. So first of all, recognition of learning results versus recognition of learning process. So how are we going to measure that? Validation of learning results versus validated learning environment. Recognition of open learning and non-formal learning in formal learning. So how we can compare the templates for this to move slowly through them. Multilateral agreement among education providers and companies on recognition of learning. How can we go with this? For example, if we have, let's say, a very regular virtual learning environment within an organization, maybe we can start with that to involve companies and to review already learning outcomes and competences that are inserted by default in the virtual learning environment and have validation and recognition of a company. Then verification of learners in open learning environment versus verification of learners in traditional virtual and online learning environment. So comparison of those two. Is it really the learning outcome that we recognize? So this issue is raised for several times. So actually up to now we recognize the learning outcome. But this was actually a long way until we realized that open learning is something more than what we can imagine because we in traditional universities in traditional that organization we recognize outcome, competence but we don't care actually a lot how this was achieved. How can we measure recognition of achievement? And that verification of learners solve the issue of assessment. What do we assess? So actually when we have very, very simplified discussions on cheating on how we can be sure that our learners are those that are actually participating in the learning process and very often talking about distance learning we have these questions raised in traditional campus-based universities and also vet organizations. We usually have methodological answers saying that it's up to methodology how assessment is being prepared. But now with the verification of learners I think we will be one step advanced and we will be more secure and we will have, I don't have any doubt that our departments of IT and also departments that are responsible for verification of learners in virtual learning environments will be advanced after we have cases and we share them. So we start, we will reach the answers we will find our solution but at the moment we are at the key cost and of course plan step-by-step how we will implement that. So I'm not presenting today to you the intellectual output actually not the meet the objectives that I just mentioned to you but one intellectual output which will be very interesting it will be cases and scenarios how this is achieved. So then we of course will arrange awareness campaign and we will introduce it to you. Thank you very much. Thank you Irina. Can you also comment on the first question here in the pool what is your opinion if you see that validation got 94% and assessment but for example the quality or duration provider are lower in the percentage of difference what is your comment on this? I agree with it very much I think the quality must be the same actually in any type of education service and I think quality requirements may differ in terms of measured indicators like for example the length, duration so yes duration maybe can be different but you know also I agree that we must pay attention to how we recognize and I don't see actually recognition now here but maybe this is the most important difference in those types of learning so I agree that quality is not the difference we may compare quality of non formal courses themselves but maybe not the same. Thank you. Now we will go to the case study to see how one course a non formal course MOOC in this case look like and what is the experience after doing such a course so today with us is Gordana Juga from Creation Academic and Research Network and she will present the course MOOC on Moodle they have created so Gordana Flor is yours Thank you very much I'm very happy to be here today at this Eden webinar and I'm very happy to have the opportunity to share with you our case study about Moodle MOOC that we developed and delivered this is about non formal open learning opportunity for development of digital competencies of creation teachers I'm going to talk also about validation and recognition but I'm going to give you a bit of some context so you can understand what actually happened I need to tell you just a few words about Creation Academic and Research Network I work for because also to give you a big picture so actually we are National Research and Education Network and we provide connectivity for all primary and secondary schools in Croatia there are about 1,400 schools in Croatia we also provide connectivity for all higher education institutions in Croatia about 250 of them this is only the basis for all other services or all other support that we actually provide for our users which are all those institutions and individual users like students, teachers and other employees we support integration in teaching and all other processes that go on in schools and universities I just mentioned two examples like e-class register which is provided in many schools in Croatia that means that teachers and other who need to use it, they actually don't use paper, class register but they use web application instead also we provide a model for our users at the moment we have about 5,800 courses of different teachers that we host and also there are about 120,000 users at all including students and teachers we also provide a lot of teacher training in development of digital competencies this model MOOC is one of the courses that we actually provide for teachers and I just want to give you a little background like what we did before we developed the MOOC we actually delivered online courses for participants they were like very small groups of maximum 15 participants and they had a very extensive support by moderator they were actually creating their own model MOOC their own project and the moderator was giving them support and feedback all the time but this model the learning outcomes were very good but this model was very costly it cost about 30 euro per participant and we actually in 4 or 5 years from 2009 until 2013 we actually managed to educate 360 participants in 24 groups and it looks like we need a lot of time to educate all our users I already told you that we are hosting at the moment more than 5000 courses so we were looking for another model and we discovered MOOC it was a very good solution to our problem because we could include many participants at the same time but in order to do so we had to change the course itself the design of course and earlier we had very extensive support by moderator and now we had to kind of share these moderators with participants and also with the system that means there are a lot of automated activities but the costs are very low about 2 euro per participant and also we managed to educate 1150 participants in just 3 groups in 2 years our target group is primary teachers at all educational levels in Croatia but also in region because Croatian language is spoken or understandable by people from region like from countries like Slovenia Serbia or Bosnia so many participants from those countries also attended our course and they were very thankful for us because they don't have opportunities like this in their countries so we are very proud of that we actually we also wanted to kind of the advantages of MOOC to use that because we could use this networking issue how to say a lot of people at the same time at the same place discussing the same thing so we wanted them to share their knowledge so we changed our course in a way that we offered 3 different learning paths for users so some users who actually wanted to learn how to design and develop an online course they could join the course but also those who maybe don't have enough pre-knowledge or they don't want to do that they just want to see some possibilities of model or maybe to get experience in attending that kind of course MOOC we also gave them opportunity they had some light version of our course and also we wanted to attract people who already are teachers who are using model and who want to share their knowledge with other users so we actually managed to do so and we have 3 different kind of users and I was very happy to see that some experienced users shared their knowledge with beginners and this was a very good thing and kind of bonus for our course I'm sorry this is not very good visible but I'm going to tell you that we had more than I said more than 1,150 participants and about 20% of the participants actually got at least one of the pages I'm going to tell you more about that so the participants were very successful which is like more than actual average of MOOCs in general we also did some research on who were our users most of them 70% were women but this just reflects actually the whole population of teachers in creation schools and universities which is about the same and this is very interesting that the average age was about 40 years and about 60% of participants they worked in schools more than 10 years because we often hear in Croatia that people who are young that they are more eager to use technology but this is not the case in our MOOC but it looked like that more experienced teachers tend to use technology more so we had about 37% of teachers from primary schools about the same from secondary schools 15% of teachers from higher educational institutions but also there were about 20% of other participants like some government institutions or even commercial companies we of course we wanted to give some kind of credentials to our user we wanted to also we thought that it was a good way to motivate them but we are institutions that is credited for adult education but in Croatia there is a pretty complex procedure if you want to for your course or program to get formal recognition so we decided to use badges which we thought they will be a very good motivator for our participants but also that they will be kind of with the badges they give insight in what actually they have learned so each batch except they have of course name and issuer which is kind of in this case also they include all learning outcomes that the participants have actually achieved so we had three different kinds of badges I already told you that there were three different learning paths the first one was Karnat Moodle Mook participant and Karnat Moodle Mook designer and Karnat Moodle Mook distinguished participants for those experienced users who wanted to share their knowledge with others participants could share their badges for example, a system but also via open badges in their other I don't know maybe LinkedIn or some other their accounts or network and we also the participants also use these badges to get recognition from agency in Croatia this is education and future development agency who actually recognize those badges as one kind of prerequisite for teachers who wanted to get promoted this is kind of maybe a great zone because the rules for promotional features are very outdated and these kind of credentials like badges they're actually not included but since we are government institutions and we are accredited for adult education actual evaluators who evaluate teacher for their promotion they actually in most cases recognize the badges participants success was pretty high about 20% of them they successfully finished the course but they actually earned about 590 badges it was pretty mystical to us because as you can see that each participant earned 2.5 badges on average it means that they actually wanted to earn all three badges no matter that those learning paths were aimed at very different learning groups but they were very very eager to earn the badges so maybe so they were also very satisfied with the course 3, 4, 5 day correspondence like rate in schools 73% of them were very satisfied about 10 to 4% were satisfied with the course and since within that this is a very good model for educating the training our users we are going to deliver model MOOC for two groups in our eSchools project which is now actual which is running right now this huge structural project which is aimed at development of digital maturity of 150 creation schools at the moment we are running the pilot project until 2018 and then we are going to to run the we call it the big project for let's say about 50% of all creation schools and we think that MOOC is one of the modes that we are going to use for digital training for development of digital competences in this project but also in our other training programs thank you very much for your attention you can contact me if you have any questions here's my email thank you Gordana for a very interesting case can you tell us a little about how teacher reacted about the possibility for validation of their badges was it important to them how much did they ask about it it was very very important for them I already told that it was a huge motivator for them and sometimes there were some errors in the system and they were calling us they were sending an email where is my badge I did all the activities I was required to but I didn't get the badge so it was very important for them and also they for example it was of course digital badge but they asked us to how they can print the badges to have something like physical to have like a kind of award for them yes some kind of certificate let's say and I would also like to comment on the results of the questioner I'm very glad to hear the results that for example that most participants think that any institution can issue certificate or award because we as we are not a formal institution and sometimes this is some kind of disadvantage but advantage of Karnat as a non formal provider of education is that we are how to say on the edge of new technologies we are always trying to to give our users something to try new approaches and this is something that actually teaches in creation need because the initial training was a long time ago and the technology and the teaching methods and everything else they need they are now very different than then and I think that the gap between actual requirements from employers and from actual knowledge that possess that we can kind of be there to to manage that. So the participants can ask questions in the chat for our speakers maybe Andrea you can comment on this second question as well about the certificate for qualification you see the results should there be a difference on institution or person who can issue some kind of certificate for qualification of some non formal school certification and then with the certificate how they will be recognized by the some kind of employer or some kind of agency who is going to use this credential certificate for recognition. I was actually thinking about it a bit more now don't reading the answers or the possible answers for this question and I must say that more than one answer should be allowed let me see can I do that probably I can right I can click more than one answer yeah great why is that because first of all I think we need to I think to have things clear and I can only speak from our perspective here. Normally we don't have certifications for informal education because we understand that it does not exist a such course we cannot have an informal course what you would because if it's a course it has some sort of structure of content behind it some sort of path of learning path no matter what it is but there is some thinking behind it with the intention to learn so it's all about intentionality so we wouldn't use an informal course as such but so we would say perhaps the way we understand it that no for informal learning we don't use an informal course because there is no structure behind and there is no intention to learn and to achieve learning outcomes specifically okay now for a non-formal course which is perhaps more suitable in that question then I would say that more than one answer is possible because as already happens an accredited educational institution can offer an informal course like a for example an open educational resource learning path that is not linked to their course offer on any degree course or is not leading to any final certificate or degree certification it's totally a free non-formal course and can they issue certification of course they can any institutional company of course they can you know at work you can take a non-formal course and receive a certification of participation that you achieved the learning outcomes of that particular course although it's non-formal it perhaps doesn't have any formality outside your work environment for example an individual why not if an individual is an expert on something and decides to teach informally gets together a group of people that are interested in getting the knowledge of this individual why not so I think non-formal certificates can be issued by pretty much everyone really the problem is how to recognize them later on I agree with you we have to think very much about recognition and do you think Irina this recognition should be clearly stated in the some strategies of the institutions or in the international level or not only national but European or the global level because what we have at the moment is with the qualification framework that we need to adjust the qualifications for the formal learning through the old on a global level in order that people have this mobility so about this non-formal learning to be to make clearly what are the what is the framework when someone applies with certification with the certification of non-formal learning actually what has already happened and we want it or not I do want it a lot is that the open learning and the open learning initiative affected already without any chance to go back all formal learning high education that any level of education and formal learning and I'm very happy about it and we have already saw this actually proving that the biggest impact is upon the improvement of curriculum so now we are discussing already the educational offer from formal learning and formal education provided when we speak about high education we see that and we already discussed during this weekend in other days that actually in Europe sometimes at some point in the history we went too far but actually maybe there was a setting and the context that required that what we have in high education is for a year of programs based on learning outcomes then master programs and now we already discussed the need for short term programs for open learning for open curriculum and how to integrate that and recognize that and that is already inevitable we will never go back so we want it or not it will need to happen it will take place but it just think how to do it so they don't go and develop necessarily all open courses but they think how to integrate open courses and how to use them in formal curricula and how to recognize it and it is very good so actually it is in the process I think we need to discuss now only how to do it and of course it can be done both ways we have a top down over time but as in case of high education I believe that universities have a high autonomy and they will need to discuss it first before even reaches some maybe national regulations unless we have very, very good example coming from one or two countries on how to implement and mainstream it and then we can suggest to other countries to pick it up and maybe apply some context are very diverse and member states have very different experiences and priorities so I don't think if this can be then unanimously and in one way we can't speak about one way in Europe I think we have two diverse so my suggestion would be to to discuss on any occasion how to do that but that it is coming and of course the future of universities will be that we will have to mainstream in one or another way what we have now as mainstream education however open services and non formal education will be in the context of the same institutions and they will need to do it and they are already doing it but I think we are in search of the best scenarios that need labor market needs and companies and I think what is still missing we approached in many cases and they were initiatives from life long learning program and from Erasmus program and from other initiatives on how to strengthen the collaboration between enterprises between companies and universities but still when we come back to the curriculum you know those quality assurance requirements from you know sometimes regulations they allow less flexibility than we wish and in this aspect universities administration but also teachers become very tight even before they start introducing these flexibility and innovation and I think from on one hand they want they are motivated they are they want to introduce it but on the other hand before they are able to do this they are already tight so what I would start maybe discussing also as one important item is how we could a little bit minimize the requirements that restrict us from flexible introduction of the innovation you have right the flexibility today is very important because the environment the the situation in some way dictates the things how things will go on for example the immigrants are a completely new target group which has also which has recently become very huge amount of people and something very necessary important to think about but going back to the preparation of such an non-formal courses and flexibility can you share your experience what was the framework or some qualification standards in preparation of this course of yours what was your guidelines when you started with preparation of this MOOC course in kind we take some kind of practical approach we want our users to learn by doing or to do some project work so this is something that is most important for us so I don't think that we followed any standards but the learning methods we wanted them to try and to learn in that way and we also give them some kind of model for them to use those kinds of methods in creation schools maybe the audience is not familiar with the situation in creation schools there is how to say the teaching is very traditional and theoretical and there is a huge gap between what students learn in schools and what they actually can use in everyday situations and on their work in places so we want to take this approach that we actually have hands-on activities we also use some new approaches like connectivity we want them to share and learn from each other I know that this is not the answer to your question but this is actually what were our the guidelines when we started this yeah that is very important because you have to think about the target group to be able to follow the course so you have to think of a special in which way to prepare the course in order to be more friendly for them okay do we have some questions from participants maybe then maybe we have mentioned several times the learning outcomes so should the methods for validating learning outcomes required through open education resources be the same as the methods for learning outcomes in formal learning I give floor to any of the speakers who want to comment on this but Sandra can you please tell should be the ways of validating learning outcomes the assessment methods the if you look at the Bloom taxonomy how the learning outcomes have been designed should these methods be the same in non-formal courses or in formal courses for example you can say that for some learning outcomes in non-formal course it's enough that you understand something but in formal course the learning outcome can be for example some analysis or synthesizing synergize something you know they can be much higher so can we compare these ways of validating learning outcomes in this different environment I could comment here a little bit I think you know actually what unites education providers in bad in higher education doesn't matter in non-formal or formal learning is that after now organization that validates learning outcomes or competencies if we speak about that we all request for just for good one word which is best we all request for evidence for evidence based proof so whether it is competence or skill we request the learner who comes with prior learning or with non-formal learning to prove that actually those competencies and skills exist I give you an example even before I read the report on validation of non-formal learning in our university we have for example many experience of seven years when there is a regulation that people can come and register for the study program and then they should bring evidence that they have certain learning outcomes or competencies you know achieve and then they have a discussion with the professor who usually request the evidence for that either you know portfolio, certification anything so actually this is in all cases in terms of recognition and these things unite all different scenarios I think all education providers also companies if you come to a company and say you know I am the bachelor of economics actually maybe the boss of the company would ask to prove through the duration of several months that the person is suitable for the job that he or she has the competencies needed for the job but now I think what we are opening up is a little bit broader definition of recognition talking in terms of the learning path talking in terms of the process of the learning and how we already integrate recognition elements at a very early stages and try to monitor the learning process so that we are able to match recognition and validation in formal learning with recognition and validation in open learning in non-formal learning and sometimes even maybe until maybe in informal learning but if systems are in place and if the tools allow us to do that with a verified learner identity I think this would be maybe a solution to facilitate recognition from informal and open learning into formal education sorry for this complicated answer and I don't think we may discuss the levels of cognitive taxonomy or psychometric or other type of taxonomy effective taxonomy because I think this is already the object of the curriculum design and even though we touch upon curriculum design in any of the cases but the definition of learning objectives or learning outcomes or competencies is actually the issue of the system we have one question here in the chat have you met skepticism or opposition to these initiatives and how have you deal with it maybe Andrea you can comment it speak from the research side because we are not we don't provide certification here at all it's skepticism no actually what we've encoded quite a lot is the discussion of how difficult it is actually to come up with a path for credentialing credentialization of informal learning and for recognizing it so normally universities tend to say look no matter how much we try sometimes we are stuck because the existing system has to do with recognition of prior learning RPL and what we try to do is to create a fast track for that because normally RPL is something that is very slow in time you know it takes a long time for a student which decides to apply for RPL to go through the whole process and get if they are learning recognized or get new credentials or they are learning recognized I wouldn't say skepticism but what has appeared in our research is that some lecturers may have for example for example in the RPL context if you apply for it and then you may come across to having to go through a particular faculty and speak to a particular professor because he is the one who is the expert and that subject that you want to get your learning recognized and then that professor sometimes thinks okay if you didn't study here our teaching is better here therefore you are going to have to do complementary courses to be able to get your learning recognized or it's a little bit about ego it's a little bit of a matter of trust they sometimes don't trust the other university I mean I'm not really sure they have done it as well as they should have done it or you know so this sort of thing so we discuss the trust matter in the open crisis report as well and trust is important for recognition as much as transparency this is why we see some transparency the more transparent we make our non formal courses in terms of what has been studied, the content, the ways we created assessment the roots the identification of the learning, the more transparency we upfront make available the easier it is for recognition later on so I would say that trust perhaps is one of the items that came up in our research lack of trust yeah yeah okay you mentioned also e-portfolio for recognition of learning path and I think it is a good idea but also the formative assessment is in some way recognizing the learning path because when you are following the learner through the whole course then you actually know how he is progressing during all the time in the course so following learning path it is not easy when you have huge number of students in the course but in the smaller courses I would say it is much it is much easier okay so we are coming to the end of today's webinar stay tuned with us tomorrow we have another webinar on digital skills my favorite topic although I won't be able to participate as I will be traveling but my colleague Lisa Mariblashke is going to moderate it and there are really good choice of speakers there so I am inviting all of you to join us again tomorrow at one o'clock and follow the webinar I am thinking I wish to thank my speakers today for time and energy and willingness to participate and share the expertise so thank everyone for participating and thank you all for joining us bye thanks very much