 Hello everyone. My name is Aaron Haim and I'm going to be talking to you about licenses, terms of use and services, and private policies. Now, what we hope you take away today is that who we are and who we are not, what is a copyright and what is a license, the basic levels of licensing and how likely you need the lawyer, and some resources that make it available on what you can use and how you can use it. So, who are we? My name is Aaron Haim. I'm a doctoral student in computer science at Worcester Polytechnic Institute who usually does research in seeing whether or not work is reproducible or not. I have my two advisors, Stacy Shaw, who is an assistant professor at Psychology and Learning Science, and Neil Heffernan, who is the professor of computer science and is also my main advisor. Now, who we are not? Well, we're not lawyers and as such, I have to provide a legal disclaimer. What we're providing you today is not legal advice and is intended just to educate you on what these subjects are so that you can use it. Center of Open Science does not endorse anything we're saying, so as such, just contact a lawyer if you need specific legal advice or do your own research to be able to do it. Please don't sue us. So, to start it off, you have rights, or at least you have rights in the US. Specifically, we're going to be talking about one form of your rights, which is copyright. This is a type of intellectual property. Some may be familiar with patents and trademarks. Copyright is essentially the exclusive right, typically for a certain number of years, to publish, reproduce, adapt, or distribute some creative work. Usually copyrights last a while. In this case, US lasts until 70 years after the author's death. Now, for those who don't know what a creative work is, it is essentially a expression of the idea, but not the idea itself. So, let's say that you were to copyright addition. You can copyright your implementation of addition in some language like Python or Java, but you can't copyright the concept of addition itself. That's a method that other people's years. Another way to think about it is that you can't copyright the methodology in your paper, but you can copyright the paper itself, as that is an implementation of that methodology, or the source code that you wrote to write it. Of course, there are exceptions to copyright such that other people can use it. One of the most common ones is fair use and fair dealing. Fair use in the US, fair dealing in other countries. This is what allows us to use research papers and their resources for nonprofit or educational purposes, and why, let's say someone accidentally unlicensed something or did not license something correctly, then we can still use it and not get into legal trouble. Now, what are some common examples of copyright ownership? Well, first is an article you've written, such as a research paper or maybe even a journal article itself. Other things can be source code you developed, like we were saying, you wrote a program that does, that acts like a calculator. You can copyright the program that acts like a calculator. You can also copyright infographics like figures, tables, and even you can copyright expression or ideas like the Mac Flash burnout inventory, which they basically have the methodology of burnout which they can't copyright, but they have this implementation that they are able to copyright and make money off of. So now that we have a basic understanding of copyright, let's get into level one, licensing paper and research products, where you can probably get away without allowing. Now, we have all these rights that are available. Because we have all these rights, we need a way to let other people use these rights. This is what's known as a license. Licenses essentially governs the use or distribution of a creative work, such that if you wanted to, let's say, have another person use it or have another person distribute it, they are able to use it in some capacity and release it to the world or maybe even just use it in their own software. Of course, there are many different types of different licenses for different files because of how specialized license are. They're the common ones that are specifically for software, like MIT and GPL, the general public license. There are those that are specific for datasets, like the open domain commons, public domain and attribution licenses. And there's even ones for just general files, which is creative comments. Now, depending on what license you have, other people have different ways of expressing what that license is located. Some of them may be on the actual paper or the actual file itself. So in this case, a research paper has this work is licensed under creative commons and license that they can use. Others may have it within a group of files to have it revert everything. So in this case, on GitHub, this project has it licensed under MIT because it has a license file within it. Others may have like a drop down menu where you go through and choose which specific license you want to put it under. In this case, for preprints, we have it under CC by attribution on commercial no derivatives. Now that we have all these, there are actually different types of licenses and we can break these down into how open they are. So these first licenses are what's known as free and open licenses, meaning they can be used without any issue in any kind of software for open science or open source purposes. Or they are considered to be available for open science or open source. First we have the least restrictive on the mall public domain, where anyone can use modify or distribute without restrictions. These are in cases where you want to release something that's so common that everyone uses. So in this case, maybe an addition thing and have anyone else use it without having to cite anybody and just implement it within their own work. The most common ones of these are CC0, which is the Creative Commons zero public license. And then there's also the unlicense, which is another form of public domain dedication. Next we have permissive licenses. These usually only have minimal restrictions on modification distribution, but in most cases they only require attribution, where an attribution is just saying who wrote the code, where does the code come from, and what license is it under. In these cases the most common examples are the MIT license, Apache 2.0, and VST, which is the Berkeley software distribution license. Getting even more restrictive, we have a copy left, which is essentially we have the use modification distribution must be free and open, but also any kind of use modification or distribution of that work itself, such as in other projects or as a library, must also be free and open. The most common ones of these are GPL, which is the General Public License, or DSL, the Design Science License. Of course, in the cases where you don't want a consumer of the work to be free, so in many cases these are for libraries, let's say, because libraries are typically used for commercial purposes, they have what's known as weak copy left, where only the modification and distribution must be free and open, while if you're using it, you do not necessarily have to make the using product open as well. The most common of these are LGPL, which is the lesser general hold of license, essentially an addendum to the General Public License, on you can use it without having to be under the same license, and MPL, which is the Muzzle Public License. Now that we have those, we have some licenses which are considered to be non-free, which are normally not open source for open source usage. One type of license like these is actually non-commercial license, where they cannot be used for commercial purposes. These are typically used with papers or data sets in research such that other people can use it, but that's not typically considered an open source or open science license because of its restriction. These licenses include the Java research license, or JRL, and the CC by NC, which is a non-commercial Creative Commons attribution license. Of course, they're also licensed for commercial proprietary, which typically restricts how it's used or distributed. You can usually find these on games, on software, on anything a company typically produces that can be used in a capacity, as they have some kind of commercial proprietary license that markets how they can use it and if they can distribute it, usually not in those cases. Next, we have software which may be released publicly but is considered unlicensed, which is all rights reserved. In these cases, this is known as default license, where no rights are granted to anyone else, so you can't use, you can't distribute, you can't modify. In these cases, people may not put a license on it, and as such it's considered all rights reserved. So unless it's specified, you can't use it. Of course, educational or fair use and fair dealing still apply, where you can use this as an exception, but if you want to use it for other purposes, I suggest talking to the authors to get it licensed or to have an agreement between them. Finally, we have those licenses which are considered trade-seek-it and are not made public. Any other questions before we move on? Okay, seeing and hearing none. We're going on. Now of course, this is just talking about licensing your own work. There are cases where you have to contribute to a licensed work as well. For those who are unfamiliar with this, if you contribute to a licensed work, the work you commit is still considered your work and as such is all rights reserved. In those cases, you need what's known as a contributor licensing agreement. Now, CLA's are essentially the terms of how a work has been contributed to a product and saying, I'm going to give you this under this license for you to use in your thing. Usually this comes into play with outside contributors where, let's say, you're communicating with somebody else in some university or just for open source projects having people work on it with yourself. You need to have some kind of irrevocable license that says, hey, I'm allowed to use the work that you created for our stuff so that we can distribute to everyone. Now of course, there are also a different type of contributor licensing agreement known as copyright transfer agreement. This is something different because it's basically instead of they're licensing it to you, you're basically taking the ownership of the copyright for the project itself. So they no longer own the copyright, it's responsible for you. The most common use case for copyright transfer agreements typically falls around patches or something where you're modifying another thing which may adversely affect another person's license. So in this case, I think the most common example would be let's say that you're working on somebody else's project and their stuff is not licensed, but you have permission to use it. In that case, if somebody else wanted to contribute to your work that uses that specific project, then you'd have to take copyright ownership such that there's not really an issue if they're modifying the work that the other person is creating because it's all yours. One of the most common examples of CLA is the one from Microsoft which has basically an entire section of how you can contribute to their stuff so they can use it. Others ones are like GitHub, Idea, IntelliJ, there's a bunch of them that basically have this on there to be able to allow people to contribute to it in some capacity. So we now have a basic understanding of licenses and contributing to projects with licenses. Let's level up a tiny bit. We're going to proprietary programs and online services which you should definitely have a lawyer for. Now the most common ones that people are familiar with is the terms and conditions. Has anyone ever... I think everybody here has signed at least one's terms and conditions because we all own a phone of some capacity. We all own a computer. We have signed some form of terms and conditions. It's essentially an agreement between the provider of a program or service and its user where basically if I say you can use this software you can use our service you are subject to listen to these terms and these conditions to be able to do so otherwise we're not allowing you to use our stuff. These usually happen when they let when they sign away hey you cannot sue us because you're using the product and it fails or something along those lines. So there are numerous different names for terms and conditions. It's not just one thing. Terms of service is another one. Terms of use. End user license agreement. General conditions. Legal notes. There are so many different specific versions of how you describe what you're licensing to other people that there are different names. In most cases it's typically just depending on what you're using it for. So in this case assessments which is the example you have here has the terms of conditions which essentially say how you can use assessments for its stuff. GitHub on the other hand would have the terms of service because it is a service providing information on top of it rather than an actual software that's saying you can use it. Apple actually has a terms of use itself to be able to use. Now of course this only governs what conditions you're giving away for what terms. There's actually another section which is how they handle data which is known as the private policy. This is specifically how the program or service plans to gather users and releases a user's data in general. So in that case whenever you type in your password or you sign up for an account they have a private policy gathering. What is specifically going to happen with your data that they're collecting? Will they distribute it to third parties or will they keep it completely internal? Will they use it for some other purposes like advertisements or what they're planning on handling it? If you want to think about this in terms of research private policy is probably roughly equivalent to consent forms or basically you have to say hey we're going to collect this data from you and we're going to use in this case and we have this data strategy of where we're producing it for. In this case all softwares and implementations have some form of this. Assessments once again has a private policy because it collects data from user by logging into accounts. Similarly with GitHub on what they're planning on using as well. And actually as a side note you have some information on consent forms Stacy. So Aaron mentioned these things look often like consent forms and human subjects research. Aaron if you could just advance the slide. Again the purpose is human ethics for consent forms. This is to provide information for participants about what they're being asked to do. This clearly identifies the terms of participation very rarely are you using deceptions of people should know what they're sort of going into and this is the whole idea that participants can really freely enter into the research study knowing what's being asked of them. Yep and there is some legality that can actually be involved in special cases with consent forms so just a blanket statement always talk to your IRB. Most of the time this deals with medical stuff but often we think of consent forms almost like some of these things we've been talking about right being up front with how we're going to handle your data if it's if it's going to be made open how we're going to protect your identity things like that. That's all I got for this slide so talk to your IRB. For an additional information the IRB sort of functions as a lawyer in this case because what they're doing is they're actually going through all the laws to say what do you need an IRB for how are we going to accept this how are we going to reject this and make sure there are no issues or no adverse effects on the university itself when they're releasing this information or allowing you to conduct your study. Now we're now going to level up a lot more where you there are other unique circumstances where you definitely need a lawyer to be able to do anything and move that mouse again. Ah dang it a mouse can't believe it's there. Now there are numerous other circumstances where there could be different kinds of licensing for different types of things the most it's like it's not even a specific type of license words just a piece of paper that's on a location or you're signing some agreement there are a number of different things like writing custom licenses assistance has its own custom license for some reason they made me write it but I'm not a lawyer but you have custom licenses or contracts between it two parties another one that most people aren't familiar with is non-disclosure agreements those are a type of license that actually restricts what you're allowed to talk about in exchange for working at the company other ones like having a university own the rights to the employees or students work that's a license that students sign when they accept admittance to be able to go to the university for all these cases where you're basically where there's no specific thing of a lawyer streamlining the process of writing hey this is what you need to do this is what you're going to happen and this is what's the end result you need to get a lawyer there is no other circumstances you want lawyers to write your stuff for you such that you're not running into trouble in the future where you're collecting things or you're running into other situations where legality and suing issues come into play and this slide should actually read get a good lawyer get a good lawyer is probably nice there are specific lawyers for licensing so I suggest those I don't know which ones are they because it depends on where everyone is but always have a good lawyer at hand if you ever run into a very specific circumstance that you need a custom thing for so yeah that'll be all if you have any specific questions of situations that you have currently I'll be happy to answer them now of course I have some resources here as well which goes over just all the licenses that are currently available some of them which are approved for open source or open science usage and additionally some websites that have been made about going over which license and when you want to choose it and what rights you have and what rights you're giving away and what liabilities you have as well so are there any specific questions for your specific circumstances that I can provide an answer for and yes I can go first because I think there's plenty of time for everyone to ask questions so yeah um are there I know what kind of downsides are there from using a non-commercial license so a license that would prevent others from using your work for commercial purposes for any like well regular output that you would have as an academic so data called figures in papers so a lot of the issues usually don't come in academics per se usually comes after the person graduates from academics let's say that you create something for non-commercial usage that you want to make a profit on the future some people actually have brilliant theses or works that they create but they're like oh hey this is useful for a general community I want to sell this as a product to other people after giving out my prototype and those cases what non-commercial is very not is very not useful because you're essentially saying hey no one else can you can't make this for profit I can't even make this for profit because that's the rights I'm giving to other people and it also has issues in terms of uh how do I say this it's in other locations where you're trying to sell your thing to other companies as well in most cases in most cases what you can do to be able to get around that is that if you are the original licensor you could license your stuff to another thing to another person but in that case those take a long time to get back and forth and most people are usually dissuaded by the first license they see on a document or on a specific implementation so in those situations if I understand correctly it's still possible um to have exceptions to the license or to make some modifications later like what if futurely it's going to realize that nah past me did not make the right choice so you can change the license in the future but it depends on three things the first one is has anyone else contributed to the project if you have contributor licenses agreements you have to get every single person who contributed to sign off on the license change because they own their code and you're changing the license saying that hey we're doing this under something else now which they have to approve it's one of the reasons why most people don't change the license after first distributing it the second one I would say would be do you still own the copyright to it if you own the copyright to it you can change the license or you can distribute it to other people under a different license to your aid of commons actually has a specific carve out that says hey if this is not subject to any specific licensing that you're given to third parties so if you say I'm distributing it to this party under another license you can do so without any issue the third point I would probably say would have to just relate to the nature of the work in general in the case where you're using the work and you're already stripping it to other people you have to be careful that you didn't already give the copyright away to somebody else or that the or that if you give it out to something else you're not you're not violating any other copyrights that you have with other people that you're working with okay and briefly and actually going back to Matt's question in the chat so that means that for example if in the example that Matt described there would be some work under a ccbi nc license and someone else wanted to include some figure or something in a book there could be an exception given only for like that book okay you can use that particular figure but then the ccbi nc still remains for everyone else yes okay it's just that the person writing the book will have to contact me ask I would have to say yes so there's a bit of back and forth yeah and most likely and most likely they'll have you sign an agreement that says hey we're giving you a non-vocal a non-vocal license and all stuff be careful to read those sometimes they take sometimes they're specifically that takes the copyright ownership away from you or that they're saying they can use it and then sub-license it to anyone else which is usually in the case of most books because they're trying to get it from other publishers or have different editions so you have to make sure that they're only using it for your stuff in particular okay yes Neil um I just wanted to say something which is um a little bit back you said hey go get a lawyer for any of these cases and I just wanted to share something actually called the federal the federal demonstration project and so um when I wanted to actually share data uh with with other universities um my my my uh office of sponsor projects uh says we just use all these uh forms that are created um for all these universities to share that basically there's this nonprofit that group got together and so all the osp's can use the same forms that are created which has massively reduced actually the cost of having other universities sign these forms like if I make up forms and that other university needs to actually hire lawyers to go look at this but actually I wind up sharing a lot of data because I I wind up using the same form and I just and then we just change the appendix and I'm like militant about we're not going to change any of the license terms because otherwise the other universities have to go pay lars to figure out what what what words did this person change that is in their advantage and so um feel free to take a look at the ft the f the fdp okay uh I'm looking at matt's comment in the in the chat and he's going there is cc by attribution 4.0 and cc by attribution share like so it really depends on what the service provides in general usually most open source or open science centers don't provide uh non-free licenses because they're trying to say that this is open source or this is open science so it may be harder to find a non-commercial one that's specifically for your use case if if you can't find one that's non-commercial you'll probably find one that's probably non-commercial share alike or I think in the one that I showed with stacy's it was non-commercial no derivatives if you want to know the difference between share like and nothing share alike just says that the license itself is or the paper and any distributions that use it must be under the exact same license or under a valid sub-license of it so there are cases where you may want to share the work but and that work that's being shared is not under a compatible license compatible licenses I feel like it's not a good idea for creative commons as they're very difficult to specify I feel like a better one is like MIT or Apache where basically as they're both permissive licenses you can have one project license under MIT and you can use it into the project licensed under Apache where they're essentially compatible there right that can be used with each other okay can I um reframe my question I feel of course how I how I ask it like when I'm at the doctor's office if you were creating something where you wanted to your primary goal was to minimize the amount of future paperwork that you would have to fill out but maximize the potential spread of your work that you're creating like you want the world to see it because you created it so you want attribution but you want to allow as many other people as who would ever want to use it to use it very easily and it was probably either a figure or a paper which uh license would you select so for figures or papers those would be one of the creative commons content licenses in the case where I don't care about what it's used for I would just pick creative commons by 4.0 because it's used for attribution when I get used if in the cases where I didn't want it to use for commercial purposes or I wanted to have other people use it with me I would probably I would probably choose creative commons by non-commercial no derivatives or basically they can't modify the figure whatsoever this figure can only be licensed for non-commercial usages in the case where people want to license it for commercial usages you can have a document that's written up by some lawyer that says and you give it to the publisher or they have something that they give to you that basically says hey we're licensing this to you for this specific purpose and these specific implementations within your own work so in those cases you would have either one document saying that anyone can use it and you're not going to worry about it anymore or you have two documents where you say anyone can use it and you can give out the second document for anybody who wants to use it in the future all right thank you every situation like in math's example we're like I don't know what this is gonna end up happening so I'm gonna take my figure and I'm gonna publish it first on my website on whatever I'm gonna give it the license I think it should get and then when I publish the journal article I'm gonna say like used with permission from state like I you could use it as like you're you're citing yourself I guess but you've given yourself permission to use it with the publisher that way the publisher can't just say who owns it because it comes from another primary source does that make sense yeah so yes you can do that without without much of an issue usually in the case of publishers what they have is they have sort of three different scenarios where they either have the copyright for the paper and the license itself which in that case since it's only specifically for the paper you could argue that the figure is a separate entity that the paper uses as an expression so they don't have copyright over the figure itself right there are cases where you were where they retain the copyright where you retain the copyright but they handle any issues with trademarks or licensing where basically you're licensing them to use it for any scenario and then there's cases where you own the copyright and how it's being distributed in general in a lot of different cases it really depends on your specific circumstances and what you talk about with your lawyer but generally the argument I would make there is that since I'm putting this in a paper I am saying that this is under this license that I'm pulling from this location and I'm just giving copyright to the other person at the paper itself but not what is used inside the paper right so the primary source is something that's independently yours and you have ownership over and then you're giving the paper or the publisher the permission to use it so that therefore that's not the primary source and they don't have like jurisdiction over your image or something yeah thanks any other questions oh thanks matt yeah uh I had I work on open science products on the on occasion and I usually document a bunch of resources so I had to be very aware of licenses as otherwise I would have issues where people wanted to use the documentation or use the software and run into scenarios where we're having this back and forth of going we need to use it for this but we have this and otherwise sources one of my current projects is actually known is actually reproducing other people's research projects and since I'm generating patch files that handle use cases where they have code but like something else I have to handle very specific circumstances where hey this can only be these things can only be distributed to these people for other people these things can be used by other people only under the license that the original user had and then dealing with copyright transfers so that I have the blame if anything goes wrong or that I have that's everything licensed under my specific license that I have available Erin is trying to catch up the reproducibility efforts on learning analytics side to the giftedness and talent development and special education folks who've already done a lot of really really great work um Erin can I drop your email in the chat if maybe people have questions later sure I don't mind answering them all right there's even Erin's email um Erin knows a lot uh and I would encourage you if you ever have questions to email him because he probably has an answer or a resource oh yeah thank you for going all right any other questions all right thanks for coming I do have one question I was just thinking that maybe Ella or Laura something so similar question to what Matt just asked but for codes that you generate so what's so sorry if you don't want other people to make money out of it so there are non-commercial licenses that are specifically for software I think the one that I said was the Java research license which I believe is specifically written by Java to basically to basically prevent the non-commercial implementations of it there aren't as many non-commercial licenses that are written for streamlined use particularly because most people who write licenses are doing it as sort of like a open framework that other people can use without much of an issue even my work that I have released out in the wild is typically under a permissive license that allows people to use it for commercial usages just purely because of the maintenance of making sure it's not used for commercial where other people have probably developed something similar is most likely not worthwhile in the end but then with a copy left license not be enough I mean if my work is basically open and free for everyone and whoever uses it also needs to be open and free then that kind of seems to prevent commercial use right uh no it doesn't prevent commercial use it just says that the source itself must be open and free they could still use the thing for commercial purposes it's just that you can see what you can see the original what the original what the work is they're doing in it okay all right so you can see the source but while the actual thing can be sold against money yes yeah it's one of the ones that I think is probably an interest that is probably true is IntelliJ or our visual studio code is actually probably the more interesting one where there are where it's like the source is visible but you they still sell something above the product that for money itself okay and one final thing and then I'll let everyone enjoy their weekends I was also thinking at some point of just trying to reproduce results in other papers and I'm wondering is it possible for people to try to get away and have people not be able to make derivatives out of their code analysis code by putting a license on it that forbids because for me to verify if the analysis in a paper is correctly done I'm going to start by using the data and code that the authors share and if I notice a bug or a problem I'm going to make modifications to that analysis code so is there a loophole there so in general you should never make the software non derivative because if you ever want to have somebody else contribute or update to the project then I'll run into issues to get around that you could claim since you're providing it under free and non-profit that it's part of whatever the fair use or fair dealing laws that you have in your country are an additional thing that you can do is instead of distributing the entire source itself you're only distributing a patch file which means they still need access to the original but you're not modifying the code you're essentially providing a file that does modify it so okay it's it's a bunch of legality arguments that you get into that scenario you have to talk about what specifically you want to do in that case I would really recommend talking to a lawyer for that yeah Anna I just wanted to actually brag uh for Aaron so the paper he's presenting this week actually where he looked at 80 odd papers at the learning learning analytics conference and tried to reproduce each one of them uh so when you say you want to reproduce stuff so he's gotten serious about this uh and trying to figure out how to how to do this he's also actually succeeded in getting the educational data mining conference to add open badges uh to papers this conference season and we'll see over time if the rate of open science increases the rate of open science and the four major conferences that I attend because I'm like I'm in this ed tech space it's maybe five percent of papers actually do anything in open science uh and so way to go it's disappointingly low I'm not in um so I'm in a different research field but I yeah I definitely think that we should check things more often because I'm not a fan of just having badges for the sake of having I don't it looks like being in kindergarten and having stickers for being a good kid a gold star but no one really really checks and we should check before we give the badges the um um you might have more experience than we do but actually uh we just started this badge thing this first uh this first time uh and so reviewers then could could could make a statement about like did they actually claim open science um they clearly could have decided to check what we did not require reviewers to go uh download the code can you reproduce a number in the paper uh that that that would be that could be that could be hard uh to try to get a bunch of reviewers to do that do you are you aware of a community that actually has figured out how to add that so let me see I'm in marketing I'm not that that's my field but the American Economic Association so all of the journals that are part of the American Economic Association uh for papers that are conditionally accepted they have I'm going to paste this in the chat they have a uh a team of like data editors of reproducibility editors so they have a team of people specialized in checking numerical reproducibility so they will not check whether the statistical method makes sense or whether you implemented it correctly or anything it's just like okay does your paper send five then I need to see five being produced by the code you have and that they're pretty nice like they also have online events where they explain how they're working and how they're doing things um marketing journals do not do anything close to this so yeah but there's hope wow that's especially cool so so the reviewers review the paper it's concept conditionally accepted and then somehow this society actually has gotten because like it's funny because Aaron went and did this for 80 papers for to publish a paper about like the open science but here they're smart enough to be like let's get you to actually well but they somehow got somebody to spend their time to go be able to verify this right is they are employed that's their job so they have paid people not the reviewers read the reviewers focus on the um well the reproducibility is still part of the science but they focus on the non reproducibility part of the science the soundness of the arguments the theory the method whatever and then this team of employees from the American Economic Association is really yeah checking that your code executes and that the output matches what you have in the paper and they might also offer advice on how to improve reproducibility because they always they have a lot of experience so uh yeah yeah it's funny because like we just in our society we had a hundred thousand dollar sure plus actually at the educational day mining society and everyone's like what should we do with it and we're like let's let's give out scholarships so people can attend but maybe your response might be no no let's actually pay people set up a system by which actually people could verify actually some of the numbers in the paper and yeah good idea i also know of an initiative management science so that's another journal in management but people from marketing also published there they had recently an initiative where they incentivized other researchers to verify papers published in this journal and so you would sign up as part of this initiative you would get data and code for one of the papers published you would verify that the results reproduce or not and as a reward there's going to be like one of those big team papers where people get credit for well checking other people's work that's another method and in marketing there are a couple of people marketing and psychology there are a couple of people who might do checks more targeted towards catching fraud cases and the outcome that usually gets published on like specific websites that might protect the identity of the people who check because well if you find suspicion of fraud you might be a bit reluctant to put your name on it yeah a lot of things like people are like this is too good to be true and then they look under the hood and it's like oh yeah no that's why and our field is still dealing with like this is obviously wrong and the journal still won't retract the paper so yeah and by just contacting the journal sometimes you get nowhere and that's why these like basically the data police have to be for data bags yeah um so that but I don't know other ideas I had over the time but I don't have a journal to implement them um this can be very useful for students for graduate students um you could have basically like internships or I don't know some other way class final exams people last final exams that could be coupled with like journals in and then you could at least check some of the papers so but there are ways and that can also help the community because then students learn how to do with how to properly do it how to prepare for submission and yeah okay Ella says we're at time yes thanks so much for coming thank you Erin for doing this thank you Ella for moderating