 Hello everyone, I think we can start it's one o'clock central European time So hello from Zagreb Here in Croatia. I'm happy today that we are having the event organized by the University Computing Center CERCEN and European Distance e-Learning Network even we didn't open Education Week Which is celebrated every year with aim to foster and hence the open education in Cold education through all levels of education Today we are having a topic of How to promote academic integrity in online education. I think very high topic very highly on agenda and With us today speakers experts in this field and I'm very happy that they have found time and Join us today. So today's speakers are Frederick little professor emeritus from University of Saulo Paulo in Brazil and president of Brazilian Association of Distance Education Dr. Irene Glendining academic manager for student experience at Coventry University, UK and Chris Envarts lecturer at the Institute of educational technology at Open University of UK so Why we are having this topic today? because Immersal internet and available availability of lots of materials and resources We have to talk also about issue of a possible plagiarism or fabrication or dishonesty and the question of academic integrity which has brought these possibility of Using lots of materials Available on the Internet Also with online education. We have to set the goals and the policies so lots of educational institutions are Facing challenge how to solve and prevent issue of plagiarism and how to make academic integrity highly valuable standard in the academic world again so Today we have three presentation which will provide different focuses and The different part regarding this topic we have in the first presentation a little bit a little bit more about what is academic integrity and Issues about this in the second presentation. We will have the results of research done within the European Area of our not topic of plagiarism and the last presentation We'll hear experience from the Open University UK also about the project Tesla who is being with one of issues within academic integrity So at the beginning I would like to announce Professor Frederick Lito professor emeritus from University of Saulo Paulo Who was professor there of communication there for till 2005 and then has retired Professor Lito has really Very fruitful and productive career and still is very active and I'm very happy that he has joined us today Within his numerous achievements. I would just Emphasize that he has done lots of books, but maybe to Distinguish to emphasize the book on learning at the distance We which was awarded the price for the best book in informatics and technology by Brazilian publishers association and Maybe that in 2014 the open courseware consortium Selected him to be a recipient of the prize for lifetime achievements in recognition of his efforts to expand access to learning Title of his presentation today is plagiarism in open and online learning. So Professor Lito floor is yours Thank you very much I am delighted to be participating in this seminar It was delighted especially to have received the invitation because I had the opportunity to visit Zagreb in 1969 and I found it a delightful University and delightful city And so it's a pleasure to be back in contact with you. And of course I'm grateful also for the opportunity to talk about a subject. That's very close to my heart I've been studying the question of plagiarism in the academic world for Some five years now because there's very little on academic plagiarism in the Portuguese language I've lived here in Brazil now for 45 years and taught at the University of São Paulo and We have serious problems probably no no greater than in other countries With plagiarism in the academic world. And so when it deals in plagiarism distance education I am doubly interested. So let me move ahead We have an association of all the participants the professionals of distance education here in Brazil and that's the we'll call it a bed Brazilian Association of Distance Education So Let's see to advance my slides. I'll go to the next slide We'll talk about plagiarism in open and online learning. This is a photo of our capital in Brasilia First I'll talk about some definitions and types of current academic plagiarism I'll address the question of is there more plagiarism in higher education now since the advent of the internet What are the factors which are encouraging the practice of plagiarism despite? so many rules against it and this plagiarism detection software a solution with no Drawbacks whatsoever or rather some drawbacks indeed and finally do interventions like Ethics training really work. We'll talk about what the literature says about that and then finally there'll be a bibliography Which permits people to? Follow up on some of the remarks that made here in the simple basic Say in English a boilerplate Definition of plagiarism that is in the most simple terms It's an act using the work the ideas the words or graphics of another author and They're thereby Submitting it as being your own without attributing the original source or having the first author's permission It must be distinguished from copyright because copyright is a legal matter with possible commercial and most likely commercial interests plagiarism on the other hand is an ethical matter Not involving money necessarily or not directly Perhaps indirectly involving acknowledging one's debt to another author through attribution to his or her work in the realm outside of acadene in literature journalism music cinema The motivation is to play for plagiarism is for pecuniary gain to make money to make money In the world of acadene want the world the rules are Around the question or the the mission is word of certification of knowledge certification of competencies and original work and hence the motivation is to try to appear more competent and original than one really is and hence its dishonest and Consequently it makes the certifying institution into a liar, which is not very good You know if one plagiarizes Now there are different degrees of plagiarism the most serious of what we call hard plagiarism as opposed to soft Begins with serial plagiarism Recidivism where a plagiarist Continually produces work based on plagiarism This is the most serious crime of all and it has to be noted in the literature so that people are aware of Chronic plagiarist also most written work where one pays To have a thesis or a paper written for him or her either contracted through or purchased online from a library of Prepared works Also very serious the reproduction of an entire text verbatim Line by line perhaps word by word for word the fabrication of references when you may write your own Paper but to make it look stronger than it really is you fabricate bibliographic references To boost your prestige with a reader of your paper Sentences or paragraphs copied verbatim also a no-no something not to do quoting or paraphrasing or citing improperly Consciously because work what they're talking about all of these in the serious line Is intentional plagiarism The idea is to fool your reader and as we go towards the more softer plagiarism It's usually innocent and maybe punished initially But then a person should not Commit an offense of plagiarism thereafter patch writing also Moderately seriously weaving together pieces from different sources. It's also called mosaic plagiarism or pastiche plagiarism Cut and paste mash up and mouse click plagiarism But basically not your own work. We're just putting little pieces together from different places a double submission This is serious in the sciences the hard sciences and in medicine Where they think because it's so expensive to get an article published for the publisher They don't like the idea of the same text being submitted to more than one journal at a time Self-plagiarism where you repeat works you've published in other places again. This is a more serious crime offense in the hard sciences and Medicine but not so in the humanities in the humanities where there are problems of linguistic few Pfeifdoms where Spanish and Portuguese for example are similar languages But not the same that sometimes in the humanities an author can publish the same paper But it's good that he cross-references as that he's published this before but it's not considered an offense Collusion especially especially between students without prior permission of the professor can be considered formal plagiarism as well courtesy or gift authorship where somebody who never participated in the research Suddenly becomes one of the co-authors of a paper. That's considered a gift and it's it's it's wrong It's it's because the person did not participate institutional authorship where the institution gets credit Although it did not sponsor either financially or contribute to the work that being presented. It's also called bureaucratic Salami slicing dividing up the results of the same research in different publications Making it difficult for far researchers who follow to discover what was actually done They have to put it together from different places Meta plagiarism, which is one of the delightful things I've discovered. They're not a good number of historical cases plagiarism of plagiarism I wish I could talk about that more, but we can't Cryptomnesia false memories confusion of sources You don't remember where you got that source But you the idea maybe the words and you put it in we call that unconscious plagiarism to it It's not too much of an offense Slightly and finally the less the least serious one what we call soft plagiarism journalism which is not at the academic world but As far as I know, but in the journalism world where the journalist depends entirely for his content on press releases from other places He doesn't write anything. He just Mashes up our press releases and finally honest mistakes done by students who were not fully involved in the Rules of our plagiarism in referencing the use of court marks unintentional errors And I remind you plagiarism can occur not only in the text but also in footnotes and bibliographies Is there more plagiarism and higher education since the internet? Well, the literature really is not conclusive Most claims of a significant increase are based on studies which depend on self-reporting data from the students now Students, you know our normal human beings. They are not necessarily prone to Convicting themselves admitting to having cheated on exam. So the literature is not Conclusive on this and it's not entirely reliable with the literature also said we need more exisions more rigorous studies They are required in order to decide this issue. There's a characteristic failure Among professors and institutions to report places of major cases of plagiarism for example Professors fear to report plagiarism where they might appear to be ineffective as teachers Professors fear not having the support of the department chairman or hiring institutional authorities and finally Professors fear that the institution will hide the fact that there was plagiarism either Individual or especially if the whole class Does it because it means bad public relations in society, but in a study of 386 doctoral dissertations dissertations written between 2009 2013 and the references in my bibliography here in programs that are doctoral programs online and face-to-face Using turn it in the conclusion was there is no Significant statistical difference in occurrence of plagiarism in dissertations online and face-to-face programs. That was a rigorous job of research and reliable, I believe What are the factors that encourage plagiarism today with the internet all over the place? Major the major factor. I would say it's not technological. It's a paradigm change caused by postmodernism Leo Leo tired one of the Celebrated authors in his book the post-modern condition says we are at the end of a grand single Overreaching overreaching narrative. We need to have an assault on norms We need normlessness or moral deregulation Because we have the power to defy norms and to determine the veracity of competing claims all truth and morality are relative Relativism so it's not such a crime to pleasure plagiarize this it's minor and so forth Truth is not found but made the post-modernist say and another author Seth Godin says one cannot write without using the ideas the Metaphorses styles that tropes the process of the concepts and examples with successes that came before another factor essay meals existing on the internet Contract ghost writing for academic purposes and the delightful thing is in their advertising these commercial essay bills say ah You can get from us Text that are plagiarism free when they are what that way in what they are selling is full of plagiarism It's a plagiaristic act. Let's say Also available on the internet for free now our paraphrasing tools Which are rich and synonyms that'll take a text that you send them the software will take out a certain adjectives Nouns and so forth change them for other words and create a new text which makes it difficult for plagiarism detection software to discover the fact what are the techniques for identifying online plagiarism that is what should the professor look for when When he senses that a text might be plagiarism an extreme change in writing style within the document a noticeable change of type funds type size type color Identifying uncommon phrases or keywords which the students let's say a first-year student in the university Quite unlikely that he will use them that will might raise a sufficient beware of non-native speakers international students who Suddenly are blooming in excellent programs in the text that they in Australia They have an especially great problem They have a great number of foreign students and suddenly there's a lot of plagiarism going on there They publish a lot of research on plagiarism with foreign students coming out of Australia turn it in it's now promising for 2018 a new software instrument called authorship investigation, which is based on machine learning and forensic Linguistic best practices and it's designed to be personalized for each student. Why personalized and what do I mean? Well, one of the methods that I would suggest for combating plagiarism at the beginning of every course Ask your students to send you a sample of their earlier work in digital form a five page minimum I would say why for later verification of the writing style They handed you one thing at the beginning of the course Let's see if what they hand you at the end of the course is done in the same style If not it raises suspicion advise the students of the rules concerning plagiarism and it's non-acceptability Students to confirm in writing their understanding and agreement with the term Professor should become familiar with websites that sell papers and automated editing services and let the students know that you Know about these services that will frighten them a little bit, which is good Assignments should require original thinking and going beyond the first page of a Google search Consider using the plagiarism knowledge survey, which is a a short quiz It's on the internet free with one you can download it send the bibliography on the web as Reinforcement of what you already taught your students about plagiarism combating pleasure and throughout every course Require more than one written assignment and each of these assignments be submitted with the following decoration I know the rules about plagiarism and this is my own work followed by a signature then that that makes it very Potent for the professor or the institution to punish the student if it's been a serious case are case of pleasure Or you can develop an honor code for all students and teaching staff to follow with regard to academic work Detailing the obligations of academic integrity the honor code is not a popular thing not even in America University It's only about a hundred which have them I have another talk which I gave about honor codes and very much in favor of them And there's reasonably good evidence that they are a successful in their Mission is plagiarism detection software solution with no drawbacks Pleasure and detection is not a magic bullet it requires other things in the environment to reduce plagiarism Online detection service should only serve as a mere diagnostic tool to highlight possible cases of plagiarism Which human judgment always needed to investigate further now in the coming years we're going to be Encountering another problem the rise of the visual internet We're what what we're witnessing according to the literature is the decline of the importance of text and the rise in the power of Audio and video of podcasts of means of talking assistance And it's going to be very difficult to accompany this to discover a plagiarism Google in research will be used more and more by all of us do interventions finally like ethics training really work a study in 2016 led by Anna Marusek of the University of Split in Croatia I'm pleased to say and Elizabeth wager a celebrated lady from the United Kingdom the leader of cope Organization devoted to integrity in academic work this study found little good evidence that it does It was a meta analysis of 31 studies 15 randomized control trials with 9500 participants. What was the focus to discover since With ethics training What came after it in the knowledge the attitudes and the behavior plagiarism or no plagiarism with regard to plagiarism by these 9500 participants training indeed had some effects on attitudes, but minimal effect or short lived effect on their knowledge of plagiarism the evidence the Researchers reported was in general a very low quality the studies used different measures of success So it was difficult to compare and we're not well designed to reduce bias in the questions that were asked Few details were given about the number of sessions that were held in the training the length of the training with a curriculum The training and so it was impossible to repeat or adopt this research All areas of integrity Luckily plagiarism fared the best perhaps because it's more technical or because software detection is easier to quantify They didn't know what caused it. Here's my bibliography. I Invite everybody to come to the next International Congress on distance education here in Brazil It'll be in early October of this year. You're all welcome We usually have more than 2,000 people attending and we have a lot of fun with dinner dances and so forth Thanks for your kind attention. I hope I've raised some doubts. That's it Thank you, professor. We thought a very very good presentation emphasizing the all possibilities of plagiarism. I wasn't aware of all of them They are really quite a few of them. We started some discussion here in the chat Maybe I opened the the floor for some questions in the chest if they are maybe Termiting like software Prevention of plagiarism, but actually Will be software Prevent or detect the plagiarism because it's different if we want to get someone Doing it or we want to prevent Students to think about it even and I'm sorry to hear that from these studies that there's a low Improvement regarding the the education about the Plagiarism at something wrong What would you say? Would be the best maybe not best but possible solution to make students more aware that This is wrong. Can you hear me? Can you hear me? I Can you hear me? I Think the things that I've listed there The training is important, but it has to be done the training although we don't have good information about What is the best the research that has been published is not conclusive It's a training is necessary. I mean even if it's a weekly done if it's if it's not excellent But it's good. It's still important, but I think after the training the professor of a course has to use certain techniques to Close down as much as possible the possibility of plagiarism first letting the students know That he will he or she will be aware That means the the software detection What before software how did a professor? Know or suspect plagiarism He knew the literature in the field He could detect plagiarism with his own individual knowledge But now with the the avalanche of published information Production of doctoral thesis and so forth. It's it's almost impossible for an individual professor to know the literature Intimately enough to be able to Feel a red light going off in his mind when he reads a student paper So you have to ask the student to give you An earlier example of the work and then compare it in style And there will be software to help you compare this from here on in To see if what he hands in here and now Is similar to what it was three years ago six months ago and so forth We cannot eliminate training training has to go on it has to be repeated It has to be done in high school. It has to be done at the beginning of college and in every course Every core syllabus distributed to the students should not only have the program of the course What's going to be studied but the warning about the avoidance of plagiarism? I think that's about it in response to your question Okay, we have here in the chat one question regarding cultural differences This cultural difference influencing the different Idea what is the plagiarism? So how how to deal with that? perhaps what's something What's wrong in Croatia? Maybe it's not wrong in Japan. So how how to Deal with that how what are the experience, you know something like that? Yeah, may I may I address that point you've just made? I did not put into my talk my my Sentiment of five or six years of of intensive research about academic plagiarism. I'm convinced that The rules about plagiarism, but more importantly the punishment Given that is what is considered plagiarism what is not considered plagiarism differs from country to country Based on its academic traditions and its culture. It it differs from university to university from professional Counsel, you know medicine biology Humanistic studies and so forth it depends on the department it depends on the Prestige or no prestige the political situation of the professor in the department if he or she is Intuned with the thinking of the department He or she is protected if he or she plagiarizes and if he or she accuses a student of plagiarism if the professor doing the accusing is Intune the political Culture of the department The department will support the professor if he or she is not tuned Sorry, the professor is playing with danger. It really depends on on Environmental culture academic culture. Yeah, and maybe just one more question or comment that we have to think about the role of the assessment Design can play in reducing plagiarism So if you reduce the opportunities for students to plagiarism through assessment design like reflective exercises Portfolio it can also contribute to reducing plagiarism. Do you agree with that? Sorry, so the last question was We think to think about role of assessment design. You can look at the presenter area Can you can you see this Laura Costello? I ask About a role of assessment design in reducing plagiarism What is your opinion about that? Can you hear me? Can you hear me? No Okay, no, I can't hear. No, I didn't need a question. Okay. I Please look at the questions here in the presenter area So you can maybe through chat answer to some of them and now we will move on to the to the next presentation Okay, thank you very much professor Lito and now we are going to Dr. Irene Glendening Academic manager for student experience at the University of Office of Teaching and Learning at the Coventry University Irene has been working in high education for 27 years Taking lots of positions there. So she has quite valuable experience In this field, but maybe to emphasize that she Was principal investigator and project manager for the Erasmus funded project impact of policies for plagiarism in high education across Europe's which will She give some results today about it Topi a title of her presentation is academic integrity distance learning programs the curing authentication and verified Authenticity of assessment. So Irene floor is yours. Thank you very much. Can you hear me? Yes Right. Thank you. Thank you very much for inviting me to speak to you Um, I I haven't been to Zagreb, but I do love Croatia I think lots of parts of creation and lots of parts of former Yugoslavia And I'm going to slip from that slide What I'm going to try to do is I'll give you a summary of my recent research But I'm going to skip over that very quickly because I look at that in more depth as we go through the bits of relevance Most of my research has not been into distance learning But I think a lot of the things that that we found in general higher education apply to distance learning and some of them are even more so And so I'm going to actually focus on the results that we've had more recently than the IP project We did a project in in South Eastern Europe actually in the in the Balkans region And so those results are actually of direct relevance to Zagreb and and perhaps other people who are listening Uh, I'm now going to try to apply some of those findings to distance and and uh, I'm learning And then and there's some recommendations at the end and because time is short I might skip some of the the um the slides and not do them in so much detail So I'm not going to go through all of that But if you want to have a look at that I've been involved in lots of different types of work And my most recent research that I'm involved in is it is a global study for and it's actually looking at the role of quality assurance agencies and accreditation bodies in in tackling academic corruption And that's a really interesting study that I'm doing Um, but I'm I'm also involved. I'm still involved in the area of kind of plagiarism and contract cheating So there's lots of things I've been involved in and particularly I was part of the organizing group for the international day of action on contract cheating which happens in october of the last two years and hopefully we're running to gain this year So I wanted to start off by giving my definition of academic integrity as opposed to plagiarism And I think what I'm talking about when I talk about academic integrity also the terms that are used can be research integrity education integrity or just plain integrity I'm talking about practice. I didn't get the highlight quite right there practice in post culture strategy values and policy and It also occurs on an individual basis. It applies to everybody But we can't think about these things as being institutional national international So it's it's it's important to reestablish that clearly. It's the converse of academic more practice And I like the international centre of academic integrities Six words for describing integrity honesty trust responsibility respect fairness transparency You'll see me using some of these terms But also the courage to act on your on your values and to defend your your your rights and your values And and I do like is I no idea who said this but I do like this expression whether you have integrity or not Perhaps you can measure that by how you behave and that applies institutionally as well as individually how you behave when nobody's watching you So the challenges to authentic assessment and that particularly apply to online distance education Clearly plagiarism is a big one and it is still the biggest one But as we heard from Frederick, um, there are different types of plagiarism and there is hard plagiarism and soft plagiarism and so on Completely agree with everything you said Um, it's actually a very good introduction to what I'm going to present and see some overlaps In appropriate collusion to talk back to contract cheating. I'm going to talk about a little bit more It doesn't it's highlighted there because it is a big problem Self plagiarism, uh, Frederick mentioned because I'm shooting is a big problem. It's a big problem that we found in particularly in parts of europe Impersonations a problem particularly for Distance learning. So how do you know it's the person that you think it is who's doing the work? And that applies if you're doing If you're doing examination for a lovely that they're not controlled in any way Certainly the data fabrication manipulation selectivity and all the things to do with research and how and how research can be corrupted in different ways Private I'm going to talk to you about more in a few minutes But also things like coercion bullying to accept Things like the lecturer insisting that the student has to buy the textbook that they have written from them Otherwise, they won't ask things like that. We can cross. Um, but also aiding other people to be honest We do come across students who who are working as contract cheating And many companies who are supplying information to other students, but there's lots of things there So the highlighted ones are the things um And that that we identified in problematic particularly within the south southeast european region and then start by Just defining what I mean by contract cheating and this is defined In in a document from quality assurance agents in the uk that came out in in october 2017 so what we're talking about contract cheating Is not just about about paper mills or s a mills at all It's also about ghost writing by different people. So they might be family friends. They might be Colleagues at university. They might be older students with the same institution But clearly it's a big problem Because there are a lot of these contract cheating companies that are really profitable commercial organizations And it's not easy to keep track of them So the research that's being done in australia and in the uk Where various people have identified Over a thousand of these companies and the global they're they're everywhere And if you get rid of them in one country, they'll pop up somewhere else. So they're very very difficult But there are lots of things on that slide that I haven't got time to go into but they will It's not just about essays. It's about lots of other things as well They will also these companies will also provide Kind of cradle to grave service for a student where they can do all the assessment work for them throughout their their career student Including impersonation just as an exam and so And they will also produce drafts of work so that if you If the student needs to show that they're making progress They can do that as well and it all depends on how much you you prepare to pay for it So it's a big complicated, but it's also a very serious threat to the integrity of higher education So I'd like to spend a bit more time on that slide I'm going to have to move on otherwise I'm running out of time with the other things I want to say The project that that I was involved in last year as a 16 17 With mental university in Brunau with who worked on the earlier project that I led And was funded by the Council of Europe now this what this project here was it Extended the research that we did on the 27 European Union countries In that case it didn't involve Croatia And and it brought in six more countries I'll show you those countries in a minute, but basically for me because laughing countries in that area plus Albania um And so what we did was we slightly modified the survey that we did for the first project But not so much that we couldn't compare the earlier results to the later results So and and we also had in the local languages as well It's worth saying at this point that Council of Europe is extending the research that we did on the originally the project They're providing funding so that eventually we will have surveyed 50 countries so that covers all the council of europe countries and We're involved in the next leg of those those those that were searched probably for about september this year So these are the countries that we that we surveyed A year ago we finished about a year ago And that was quite interesting funding to do a little bit about that now certainly It's not a secret that the um correction in in in the um in the Balkans region is quite high Croatia has has the lowest of the of the six countries that we looked at and that's 57 in the 2017 Corruption perception index by transatlantic transparency international Um, but on this on this latest one, which I only discovered yesterday. I was using an older version of it Macedonia is really down in the 107th position out of the 180 countries survey So there is a cultural corruption in the region, which actually does impact on higher education as well. She'll hear and on the next one The universities tend to be very large The they don't function very much as an institution because the faculties are quite autonomous So it's very uncommon to have institutional policies within the area And that's one of the one of the factors that we see as being important in tackling Corruption and and malpractice There are very big differences between the university and the region and there are some really Good examples as well as some poor examples of the practice But what we found is generally a very relaxed attitude to quite serious form of cheating and But there was awareness and willingness to discuss the problems in most of the institutions and Interviews that we did but not everywhere and some if there was a little bit of the And kind of ostrich mentality kind of head in the sound in some areas particularly where Where perhaps a senior management will not aware of any problems with the students from the same institution telling us about them so Here is some evidence that's quite important and most of this applies to the kind of distance learning on my own situation There is a perception that corruption in society Makes it difficult So the fact that there is a high level of corruption in society people think that there is no opportunity for changing things in education, which is very sad is actually There is there are lots of Well, they need to do something about it those writing essay mills But all those things were very common in the area and they were in other parts of the earlier survey, but they were particularly dominant within within the region that we surveyed last year and There was that the students and the staff were telling us in many institutions There is a lot of rote learning and the students were particularly to upset that their their their own opinions and their critical thinking was not necessarily valued by professors They just wanted them to repeat stuff that they've been told And that really in that context is very difficult to separate out plagiarism Because if you're just expected to repeat stuff you've been told then plagiarism is okay And you know, I think we all know that it isn't um There were major problems that we found cheating and a lot of it was just to do with lack of vigilance The students knew which lecturers didn't really care and they knew that they could could misbehave in in in Formal exams when certain lecturers were supervising There are lots of repeat assessments So the work had been shown around several times and lots of standard solutions out there In certain exam questions as well that the students could draw on so they weren't really being challenged by the assessment that would have been told Worryingly we were told that that some professors were taking bribes from students to change marks to influence their results And that's really quite quite quite concerning because these are bordering Malpractice bordering on corruption, I would say and I would certainly bribery. I would put something under the banner of corruption From the positive side almost all the people we talked to all the talk to managers We talked to students and we talked to to professors teachers tutors Almost all of them said they wanted more training in guidance and we're very interested in in Fredericks Take on that and the information about that Certainly sustained training is very very important in my opinion and it's no good doing it just once it just won't work There were lots of example to the professor before and a very keen interest by most to improve and to To make a difference So this probably won't mean a huge amount to you in in the context, but basically on the earlier research that I Was involved in that I led I created a tool called the academic certificate maturity model and it Used the survey results to Create some metrics. So on the basis of nine measurements We were able to compare the results for different countries based on the Answers to questions from from the different levels of survey that we did and those are the six ones so you can see Albania, Bosnia, Herzegovina, Croatia. It's the Four of you because I've been I do Macedonia Montenegro and And Serbia are the countries in that odd on there And I'm going to move on to the next slide which gives the same information in slightly different formats and using a radar shot So there you can see the strengths and weaknesses So the point of this of this measurement is not to judge As much as say these are areas that you need to focus on where we've identified weaknesses And there is a SWAT analysis for each country is it in the reports that's now being published by the Council of Europe So those are the 33 countries surveyed in order in terms of the academicity maturity model results And you'll see round about the middle. We've got we've got Bosnia Herzegovina Just on the positive side um, Macedonia, Croatia and so on so The the Balkans region doesn't Stand apart from the rest of Europe. It actually integrates quite well in terms of how they stand the score the maximum score incidentally on the on the vertical scale is 36 So even the uk that had quite a high score Have got a long way to go before it could be considered excellence. There's a lot more improvement there needed in most of those categories And I believe you'll have access to these slides if you want to look at that in more detail We have used the survey in countries outside Europe as well And I've also used the survey to to assess individual institutions And which is quite interesting So in terms of the online distance learning and I'm now wearing other times, so I need to be very quick on the last few slides Um, clearly one of the problems is identifying identifying the students If they're not on campus and if the assessments are not controlled But also making sure that distance learning students have this education about the training they need about academic integrity um Creating assessments that the challenge but also accurately measure learning and achievements and don't just measure those what the students have learned by roads And how to control the assessment process and I've got more questions here than answers for you I'm hoping in Chris's talk that's coming up soon. We'll have some more of the answers um ensuring the students are doing the assessments themselves Um, but most importantly at the bottom is also about the tutors and it's about how tutors approach the students and their consistency of approach and certainly what we found is There's quite a lot of autonomy in terms of the way students, the tutors are allowed to penalize students and so on um Based on my current research I'm doing with chair to the Chair international quality group The oversight and quality assurance is actually very very important to ensure that Things that sound but also not just in terms of the students but also in terms of the way the marking is taken place the way the professors and the tutors are are behaving towards the students um, and I know it's not the norm in every country but certainly a combination of strong internal and strong external examining and an oversight and help with all of these these things So in terms of the institution there should be an institution-wide strategy with consistent consistently applied policies and procedures for both discouraging this kind of but also handling accusations making decisions on this kind of particularly such proportional and fair sanctions and penalties and making sure that those are applied consistently and fairly to the students um Maintaining records of misconduct and the outcomes of those cases is also quite important and and I think quite often that's not done because it's done on a very local level And in that way you can you can have a measure of whether you're making And again, I've said consistently from a series of tutors is very important And it's quite difficult to know how much academic misconduct she's not detected or not been mined Um, but also you need to bear in mind the the integrity of academic staff in all of this to make sure they are are behaving in an appropriate way towards the students because if you're not such a good example then the students you can't expect the students to behave um I'm going to skip this slide but clearly the teaching learning and assessment These methods are really very important and I have mentioned some of these before Different ways that can be done. I'm just going to briefly mention Extracurricular activities and I mentioned earlier on the international day of action that we've held in the last two years In conjunction with the international centre for academic integrity. It's a great day And we get students to involve themselves and it's possible to make it as much as we can student-led But that's what that sign is all about. So if you haven't been involved in that process before hopefully you'll sign us now but The 2018 Votions that which hopefully will be not so coming up um So these are some some things I want to emphasize. Am I okay to go on Sandra? I know we're running. I know Yeah, you're a little bit running of time, but if you can summarize during two minutes or maybe it's important that that students are seen as allies not as our enemies because The honest students are as keen to have malpractice stand out as as the As the manager into the university as well The education both the tutors of the students Whistleblowing policies quite important So if anyone wants to report malpractice, they can do it in in safety and provide opportunities for learning So I've said all those things earlier on And there are very serious consequences if it's not addressed Not least the institutional mutation I'll leave you to leave that slide in at your own time But there are some global recommendations here which I'll leave you with and there are there are lots of references on the slides That I won't show you that are coming later and including Which which Frederick mentioned briefly as well establishing a common understanding of acceptable and academic practice Because there are differences in opinion across the world And but also make sure that quality assurance and accreditation bodies Know what their role is to help with this process There are there's also legal issues that can be brought to their particular contract cheating And some countries have already done that so we're working on that with the UK at the moment But the very last point I will point out here Certainly, I agree with with Frederick that we need to start much earlier on. It's it's not something just for higher education It's something we need to start right from the beginning So thank you very much for listening and those are some of my references. There are a couple of pages of theirs. There were so many of them Thank you very much Thank you Irene Very good presentation and I I I'm gladly glad that I was able to see the results of this project that was done in for me with slaving countries Maybe to come back to to one of two issues you have mentioned is a Academic integrity issue which is built between Teacher students or or should it go to the high level to this institution or the ministry on the national level? So can a teacher by itself? Give some Regulations about it or it should be much more wider on a high level So what do you think? That's a very very good question and most Most institutions will tend to have a panel That is that will function either at the faculty level or at the institutional level My institution doesn't have and there are a few institutions in the uk that don't have that we have what they call academic integrity officers and our academic conduct officers and they are they Are trained people it's part of my role to train train the staff They are trained to have an interview with the students and to and to deal with the issue But they are working on a very strict basis and their The outcomes from their decisions Are communicated directly to the student, but they are monitored and moderated And centrally so that we have a consistency right across the whole institution so we devolve our Our decisions for many of the The accusations of discomfort to these academic conduct officers, but we also have panels So where we have where we have very serious cases or repeat offenses And the student will not be seen by an academic conduct officer At what will be we'll go to a panel and the panel consists of a number of academic And it will be dealt with there And in this way we have a consistent approach and fair approach that's devolved The value of having academic conduct officers who are trained Is that their role is not just to penalize it to support the student and to try to correct the behavior and It will vary between different institutions. So for example in In sweden they have a A kind of institutional level Panel that's Used to be chaired by the vice chancellor and it will have people like judges on the panel And that is quite a scary thing for a student to go through because the student has made a mistake They need to be corrected That won't always work, but they need to be given a the opportunity to Make good work done It should not in my opinion It should not be dealt with by an individual tutor Because you'll end up with You'll end up with all sorts of Problems because it's been the tutor has a relationship with the student It is about educating the student therefore this tutor should not Who deals with the misconduct Tutor might present the case But will not make this decision on the panel Um, I hope to answer the question. Thank you. Thank you for this good advice And just one last question looking at the panel and uh the questions and comments The the participants are provided. We were focusing very much on the students But what's about academic staff and plagiarism? Who is Because the academic staffs are not only teachers. They are also researchers and they are producing papers And who is looking upon them? How to deal with Sorry in terms of my own institution is quite a separate process of the misconduct What does that because staff can also be students as well Um, but there is quite a different process and it's a disciplinary issue if the staff is it's about to have Any issues that need to be answered So they go through the staff disciplinary process rather than through the same procedure as the students Okay, thank you are in very much For your presentation and now we will move to the last presentation and last presenter Uh, Chris Chris the works at the open university uk. He's lectured there And he chairs the master module openness and innovation e-learning Uh, and also he's researcher on the project tesla So he will say something about that in his presentation But also he will say something about how the open universe uk is dealing with this issue. So Chris, please floor is yours Thank you very much Sandra. And thank you for inviting me. It's a it's a real pleasure to to be here and to be able to speak with you and to to get some Uh, questions and hopefully some new ideas as well that I think I can pick up on Um, I've been working at the university at the open university for Um over 22 years now and that's always a shock when I think about that. But um, and There's been a lot of change over that time And we we've developed in the way in which we we teach students and help them learn and And in the way in which we we think about curriculum. So I'm going to just talk about some aspects of this over over to the session today And um, I'm just seeing how I can oh, I thought I'd be able to Can you tell me how any oh, I know so I've got it So The contents of this presentation I try to keep it quite tight. Um And to talk about first of all some aspects of our teaching model that that we've used and that we've developed And then to look a bit about authentication about what we do currently and about some aspects of That are developing in the authentication and that's where the tesla project comes in And lastly just to look at some other possibilities that are for ways that are helping support academic integrity and so if I Just move on again A title of this um, you know, you teach your model and and add a communications app because there's obviously lots of ways in which we can We can think of our model of teaching and there's lots of representations of it um, and one one of the things that Uh, uh, particularly important about academic Integrity is to do with the communication And and the path. So I've just drawn this out and it it helps illustrate some of the ways in which the university has changed over the years as well So we have our our module materials that generally created by a team and and Then presented through the internet through our virtual learning environment. So to the students And to the the tutors or the academic and the associate lecturers that we have that do the tutoring There's also there's always demand for students that want hard copy whenever we carry out a survey and ask the students They they would much rather have hard copy and have things delivered through Or solely through the internet. I think a lot of them realize that um electronic delivery through the internet actually allows A much wider variety of assets to be produced and opportunities for them. So I think they appreciate that But they they're very reluctant to give up and paper all together So we we still offer that and I think we offer print on demand for a lot of other things. So there's that corner um So the students will will um communicate with with their tutor and with other students also mostly through the internet through the forums that we have set up within the vle and during that time we The students will start to learn about each other and the associate lecturers will start to learn a little about the students um And students also have the other opportunities that are speaking by telephone Or there's also face-to-face opportunities that we still have and through through day schools or through workshops and And and and maybe even residential schools and some subjects still so I think When we when we think about the fact that we are a distance learning organization and we might have Getting on for 200 000 students at any one time that the model that we have is is A bit more complex than just dealing with sort of providing for students an online experience um, so if I move on again if we look at um Or what we do now in terms of of authentication in of student work and and of what's going on so Being a large institution. We we have some modules that have perhaps up to 10 000 students a lot don't most don't But the approach that we have is scalable which means that that every tutor will have their own group of about 20 to 25 students and they're the people that they Interact with and they teach and they were teaching their tutor Tutorials and most of which will be online, but there might be some face-to-face ones as I mentioned um And so even it doesn't matter what size the module is in terms of cohort That's the model so that every student or or relate to a tutor and um And so most work when when students assess they will submit it through through our own secure systems And and it be marked by that tutor that it started to get to know them and so I think Picking up on something with Frederick said The start was that one of the things you look out for is any any sort of massive changes in students work or any sudden leaps in their abilities that that they present um But tutors will start to know the kind of language that students use and the kind of not just from the work that they submit but from the way in which they speak in the forums and which they they communicate with other students so There's that aspect of it The the electronic system for managing the work is is something that students have to log on to and that In theory it is only they that can do that but of course all these things could be open to to abuse um and Last of all that we have um submitting work to to turn it in which we do routinely and We have an anti-collusion system which we call copycatch and so both of those Systems are diagnostic in that they they flag um quite a lot more um Possibilities that for plagiarism that that we have to then look at and filter through and make individual decisions about and And as as we've heard before that that these are tools that are diagnosed that they don't make the decision on our behalf um And and every module will have uh someone who's allocated to to look at each assessment as it's handed in and make their own uh judgment in terms of reviewing that data and then if if any piece of work seems to be plagiarism Either deliberate or or accidental then then it will be dealt with through the university systems and uh, and and the right action taken mostly I should say in the module that I run I find that the tools tend to pick up um poor practice poor academic practice in which students would tend to um Quote another piece of work and they will then cite that piece of work as we'd expect in um, but they wouldn't necessarily use quotations so when When the software looks through it would it would pick up the quotation and see that it wasn't quoted in the correct way and flag that so If most of it is to do with academic practice, then that's something that we can deal with with students on an individual basis and that help them improve Obviously if that's repeated then then we have to then think about some some other action So that's essentially what we we do now and um What we might be doing in the future or what we have the opportunity to do Is then is pick up and a lot more of the the tools that have been developed All around the world by all sorts of groups and there's multiple versions of lots of these tools but for the last year or so I've been working um Uh with a team in our university as one of the partners in the tesla project And the tesla is essentially trialing five different tools and I've listed them there There's an anti plagiarism tool Which actually when you look at it is much more of an anti collusion tool In the way in which it works And there's also a facial recognition tool Which Does just that it uses a webcam to confirm that we have the right person sitting in front of the computer So it might not be recording them in a sense that that might not be recording their assessment or their interaction It could just be used to just check that that's the student that's sitting at the computer doing the typing Or or producing something else or working through an activity um Forensic analysis Is that it's another tool we we've been is we're trialing within this project And that looks at the kind of style that a student uses in their writing and I think it It sounds um like it might be similar It's to the to the work that um turn it in a producing in their new tools that they're going to be offering later this year um But as far as we're concerned It tends to be a black box that the students work is put through and Like most of these tools it needs to be Seeded if you like or or primed with some authenticated data from the student And so the facial recognition tool needs to have a reference sense of images of that student So then it can compare all the subsequent Times instances against that reference set and it will give A value Or return a value to to the Student we could we could have it going to the student but also to to the academic that's involved in the tutor Which would then give some indication of the the confidence that that we have or the tools have that that's actually the right student So just going on with the other tools that keystroke dynamics tool actually checks in real time Um, so student would have to not be typing in word and that they'd have to be typing in the web interface Which can present its own own issues But but if if student is typing an answer to something or doing some writing some drafting Then then they will type it into into the the web browser interface and What was going on there is again a reference set of their typing would be made at the start of their studies and This particular instance of typing would be checked against that and it's Looking at the rhythm of typing looking at how keys are pressed in conjunction with other keys So it's not bothered about what words Um are actually written it could be nonsense, but it's the way in which the keyboard is used And so that again would return A value of confidence And last of all, there's the the voice recognition tool which Libre facial recognition tool will make a recording although this time just a voice recording The student in fact it has several instances of voice recordings that are set up at the start And then those are compared to any later instance So the idea is that these tools could be be used um on their own or in conjunction with one another depending on on the work that the students doing and the importance and the value of that work and That the idea as well that this is all transparent so The last thing that um the tools are set up to do is is to make students feel spied on You can see that this is a real risk and it's um, I think with with the Way in which we we read the media in the accounts that we we see on the television that we I think feel a natural skepticism to to having allowing these tools to to work even in the background, but then If if students completely understand why these are used and if they completely Trust the institution that's using them and understand where the data is going And they know that they can turn them off or when they need to then Or they can they activate them for a particular piece of work then That that becomes something that that might enable them to feel that they um They're not actually just being spied on all the time and it would allow The institution maybe to to look at different kinds of Assessment that could be used that they they've been reluctance used till now because they have a higher degree Of confidence that they they can be sure that it is the students work that is being presented So that's a brief run through those tools um There is an issue that I think we're finding um, that's kind of fundamental so these and and that's for the reference sets you need to properly authenticate the reference set and you You are if you just trust that the students setting up their own reference set then at a distance then You're obviously going to be checking against that for if for all this to future study And so if you're not absolutely sure that that set belongs to our students um You you can't be absolutely sure that you've got the right students Subsequently, and so I think that would present issues for us in how how do you Establish your authentication of those initial data sets those reference data sets um Obviously a face-to-face institution. You can have your students run through These perhaps over a few days in the beginning of their study and then say okay now we can We don't have to have all your assessments handed in on paper But we can allow you to do all your assessments or a lot of your assessments In a variety of different ways over the internet in different locations And we can be sure or fairly sure sure enough The that it's you that's doing it so um So that's what's going on there I put the url for the tesla projects and page on on that Slide and you can look and there's a video There but it it runs for two and a quarter minutes roughly which um I won't show it now because it would take up a huge chunk of this this presentation But but do have a look at it and there it lists all the 18 partners as well that are involved in that project So if I just move on To the next slide which um, I wanted to to highlight a couple of other tools that have been developed that um Perhaps not quite so directly related to academic integrity But I think you'll see that they are as I talk them through and um one of them's called called open essayists and It's um, it's a tool that's been divide and designed in In collaboration with with with oxford and with others and and the idea of that is that it analyzes students writing and Gives a report back that partial graphical display, but also an analysis so on on written on their work um Comparing that was showing giving an illustration of the quality of their writing and It's best if I just show you the image of that in a minute, but there is um The one of the papers on the most recent papers available is is Uh written there so you can follow up on that and the other Um development is called open mentor and that is actually to give feedback on tutors feedback. So um, so this perhaps is is um The least closely related to to academic integrity in the sense that we're which or authentication But it does it does lead into to the kind of best practice in terms of a feedback that the week we can offer students and both of those tools and Uh A developed to a point where they they can be used I think as an institution we have yet to make the decision that we're going to resource to use those More widely although I've really liked that. I've used them both in the module that I run and that particularly with open essayist open essayist, sorry, I was and Really encouraged to see how students when when they first used it they they found they met up with some degree of skepticism How can this tool possibly Should tell me something useful about my writing and and then by the time they used it once or twice on their Assignments they began to see well actually it does offer something and it and it's um, they they became quite Willing to just submit their work to this and get the feedback Which is only for them to use so it helps them build their academic practice and their their skills in writing Um, and it didn't form any part of their their assessments per se so if I just move on Oh, I think I went two slides, sorry and um This I'm afraid it is not particularly clear here, but um You can follow up again if you follow up on on the the publication and you can also contact you'll see that in both those um, if I just go back in both those also Publications, Denise Whitelock is the is the the lead author In fact, it was Denise that was meant to give this talk originally So it's a late apology in terms of presentation for them, but I'm not Denise and um As she couldn't be here. So um, just just moving forward again So open essayist actually if we look there there's Four different images. This is the graphical output and um, if you put 50 identical sentences in into the system It will give you this kind of cloud where everything's Well, all the nodes are quite well dispersed And at the other end of the scale if you put um a prize winning essay in from stanford university, it gives you this very condensed set of points and um, when you you work with a tool when you start to look at what it's offering you you start to to look at the way in which the these nodes are being um drawn of your work, so And it helps you to look at what you need to target So it might be that the way in which you write the conclusion or it might be the one to where you You bring up themes and their response of themes or carry things through um alone mark alone, um marking scoring essay What is it at the bottom left hand corner and a high marking essay at the right hand side So not quite up to stanford level, but they're getting closer. And so this is a very simple Uh way of offering feedback and once the student understands how to interpret this then it can give them very powerful Ideas about how they can improve their their work The next slide Um again, this is low quality because it was just taken from another slide and Actually shows that the submitted work and it's just been Carried over into a text a plain text format And then the text particular elements of the text have been highlighted with annotations So that a student could look at that That overview and and get some ideas about their their work from that So this is one tool that I think has has value. Um, it has value in improving The integrity of our work and the students work as as they develop into their academic practice But it's not directly related to authentication the um Open mentor is the other one that because I wanted to just Say a little about and that's to do with them The feedback that students tutors give on on their students work. So Students tend to just submit their work in word Although there's a whole variety of formats that they can use depending on the kind of work It might just be an image for some things or it might be a video or an audio recording and open mentor wouldn't work On on those but for those assignments where their work is submitted in word um And tutors will tend to use the end, you know the annotations that you can use within word to Said to put in their feedback comments about particular parts of the students work As well as later on giving a summary, but these these feedback comments, I think tutors Don't get a lot of support Uh, because it would it would mean they might get some support as they start out on tutoring in terms of what So things to kind of say and pick up on but um, but from day to day as they're marking work They they're not scrutinized closely and and we wouldn't want to do that um But a tool like this would would look through all their comments and it uses a well established Framework to to compare those against and and it again it will give a graphical output on each of the elements in that framework shown where they they've um, you know done really well or Not so well compared to an ideal kind of set of feedback and and that can be again something that tutors could just use And they could use to help develop the way in which they give feedback without having An outside scrutiny of their work and so that can help develop their practices Okay, last slide I just thought that partly what we're talking about in With with this use of data is is trying to find The structures and the patterns that we can we can use to the good We're within these and then I thought that this this image helps represent what I felt about that um The the other images I used I've just given references there as well. So I'll stop there and Sandra if you've spotted any questions that I need to answer first of all, then please do say Thank you Chris. Yes, please The question is are these tools open mentor and open essays generally available or just for use within the opening Okay, I'm pretty sure they'd be available for anyone to try um, I think I'm not I think there's been an issue about the the servers that we've stored everything on And and and how well supported those are but I think if if anyone wants to So to have a go at using those then if you contact me or Denise Whitelock um, then it it we we should be able to arrange it and um And if you do then it all it helps us make a stronger case as well forget and better support So it might help us to provide a better service So I will mean to just contact us if you if you're at all interested in trying Okay, thank you. And the second question is regarding this copycat software Uh, how much information uh additional information is provided within this uh software Okay, no, that's a that's a very good question. I I haven't got an example I can easily show it readily show you um, not least because it would involve real students work, but um So what we do is when we when we Set up our assignments and our assessments we define What we want this software this copycatcher software to compare each submitted piece of work with And it might just be to pay depending on that on the assignment It might just be that we want it to be compared to all the other pieces of work that the students in that cohort Have been submitting Or it might be that we want it to be compared with all the work that's ever been submitted On that module or it might be that we feel that there's another module in our curriculum that has a very similar type of question or Do we want to compare it to as well? And so we can define that when we set it up Where the output from from the software shows the um the students work and it shows where Uh words have been repeat Well similar combinations of words are also found in any of the those other work that's being compared And and it's color coded to show the sort of the the severity Of that and it then gives a summary As well And so you can anything with a particularly low value you obviously can ignore But anything where the value is well over what you deem is significant for that piece of work Then then you you start to filter to look through them and start to make judgments over whether you think it's And these are just the kind of words that students would normally write when they're writing whether it looks like it's something You need to follow up more details. So again, this is flag just helping filter What we what we have to focus put our focus on Thank you. Great. These are very good examples How can we prevent and ensure that the students have all additional information? In order to avoid possibility of plagiarism, and I think this is a very very good example So we're getting to the end of our today's event And maybe for the end I will ask each of the speakers just just with one or two sentence to Summarize or maybe to give opinion how to deal With the today with the with plagiarism with the Academic integrity, how can we enhance academic integrity today in relations to to misbehavior of the students but also So Frederick will we can we start with you? Did you hear me you have to turn on the mic don't forget Okay, did you hear me? You you cannot hear me Please try to talk Can you hear me now? Yeah, hello. Okay. I just Oh, yes, I wanted to say we are going to summarize at the end today with one or two sentences Uh, the topic of of which we have discussed and maybe uh, what is your opinion and or your advice? How we can avoid And misbehavior In academic community regarding the plagiarism, uh, or it's honest on so May I we can hear you please cutting out Uh, can I can I speak? Uh I think two things to reduce To reduce plagiarism two things Education that is the use of new technologies like face recognition And uh, what are the other things? Keystroke all of that is necessary as well as teaching the students What they what the rules of the game are? I was delighted to learn new things from irani and chris about studies being done about academic integrity in other parts of the world The problem of integrity Is not only treated as an overall overarching question and with research published in in generalist journals, but it's also To be found in subject areas like chemistry and history and and so forth so I'm just to to follow to be able to accompany research done in europe Asia and other places in north america Uh, a seminar like this is really wonderful because I I've learned from the bibliographies and the words Presented by my colleagues on the table here has been very educational And I hope that some of the things that perhaps I reported on was interesting for them And it'd be because bibliographic control in medicine and hard science is very well done since most of the Publications are in english, you know, everybody wants to publish English so as to be seen This is not true in the humanities and and social sciences Where they're still linguistic linguistic and cultural things So we need some kind of way to be wonderful to get really good bibliographic control over Worldwide global efforts at reducing Offenses in academic integrity. I'll stop there. Thank you very much Irene your point for summarizing today's event Yes, it's been it's been really interesting And as Frederick said it's it's always it's a pleasure to hear other researchers talk And other experts talk in this field And and there's always a there's always a lot to learn from each other. So that's amazing I would say that the two things I would say is It needs to be institutional there needs to be a holistic institutional strategy Act by the by the senior management so to take from the top with strong And resourcing as well To tackle the problem but involving students. It's really important that students are involved in it In the uk we have students unions and they are They're always very interested in getting involved in this kind of initiative. So That's the thing I would say but also within the institution strong consistent policies across an institution and And make sure and communication about those In clear language and communication about those Not just to catch the cheeks But really to change behavior and to to instill The joy of learning that's really why we're all here why we all care so much about our education And is because it's important for everybody. It's important for one's society as well Thank you Irene. Yes, it's very important that I agree with you And for final Chris Your last words summarizing Okay, I think I'd probably just suggest two things that in Which I think overlap with what's already been said, but I think firstly The culture within the institution is it's critical and I think the Or the culture particularly amongst the academics academics and the institution and because they're Involved in developing the courses and the teaching and the assessments and they set the culture within the courses, so That's one thing the other thing is as has already been said Is to ensure that students understand And so that means providing them information information at the right time the right level of detail And I don't think we'd expect A first-year student to to deliver the same kind of Expertise and these skills that we would a postgraduate student to studying their masters But we'd expect them all to be developing and their skills and have their understanding of these issues And to know that it's important they're important and that the university treats them as important as well So I think they're just they're the two that I'd highlight Thank you very much Chris. Yeah, if I can summarize so issue of Education and providing understanding Among students and between students and teacher and mentor then communication at such Culture which have to take for the culture as well not only within the institution, but within the society Often in online courses, we have students from very different Cultures and you have to combine and now how to approach this culture as well The policies at the institutional level or at the national level also can very much influence or even a european level or global And what we have to think about is prevention more than punishment And in order to be able to find right solution Which is very important and If changing the results and know how Also is very important. So I would like to thank all of you today for participating in this event To my speakers frederick, iran and chris and all the on you for participating and commenting Giving the opinions and asking questions The recording of the prayer of the event will be available at the web pages of even and And I hope to see you in some another event in future. So, thank you again and bye from zagreb