 So, yes, sorry for the horrible title. I changed a little bit. I'm going to talk about reproducibility. And I'm mostly talking, actually, about the sociological aspect, because I think that's the most important aspect today. So I'm going to introduce briefly the issues in statistics and software. And then take two examples for presenting some solutions. I don't think there is yet a solution. And then indulge a little bit myself and say, you know, let's think that we are in 2050, and what will we be describing of the history of the field at that time? Statistics and methodological. So a couple of examples that are not specific to neuroscience, just like a scientific example. And the first one I want to show is this one that is exactly an example of one of the studies that started the field of imaging genetics. It's an interesting study in the sense that it had the reproducibility aspect. So these guys were looking at the bold effect in the amygdala when the subject is presented with fearful faces. And if you look at the bold effects, you have like an increase of the amygdala. And this increase is different for the short allele of the carrier, the short allele for the serotonin transporter than the carrier of the long allele. That's an amazing thing. And the interesting aspect to it is that you see that the court of that study where 14 subjects, two replicated ones. I can tell you it has not been very well replicated after this time. What's the problem? The problem is that you're looking at a short group size, and this is in this figure. You see looking at the group size is the size of the circle here. And the early studies show some high effect size, quite variable, but high effect size. And then as you look at the years, with the bigger cohort, you see that the effect size is shrinking. And that's for the distance for the link between the hippocouple volume and the BDNF allele. That is the problem. That's one problem, sorry. And that's been seen again and again. So first in this Justin Stein's science paper about the hippocouple volume, he was looking at the literature before and says, OK, I can actually not replicate any of the steps that have been found. And the problem is the power. Basically, the more the power problem, the most significant and powerful, the biggest effect size that you find in this field is the epiuree for Alzheimer's disease. And that power, that effect size in terms of coincidity is 0.2. And that's the problem. A very recent paper, it's not been published, but it's on archive already. We're looking at the median sample size of ephemerized studies. And you see that today where the medium sample size is about 25 subjects. And for 25 subjects, what is the size of the effect that we would have to detect it with a reasonable power? And that's about an effect size of one. I can tell you, and I can show you, the effect size in ephemerized studies is not one. The effect size, and that's the median in some various paradigm, motor, working memory, emotion, and so on. The median is actually around 0.5, 0.3, 0.4, 0.6 maybe. So basically, we're fundamentally underpowered. Very interesting, switching a little bit from the power aspect to how do you analyze data? So that's a very interesting study, because it's a serious study. In the sense that if you look at this paper, which is looking at how students can have some precognition of a future event, it's very well done. It's a group psychologist who are doing that. I can't remember, I think he's from Yale. So you're thinking, OK, that's not possible, right? This guy has done nine studies where he replicated a kind of a precognition effect. Nine studies. And it's very hard to find, you're thinking, OK, that's not possible. Well, there's a very good blog from an Italian colleague that says basically, why good researchers publishing good articles in good journals can still get it totally wrong? And it's difficult to find it. And there's a number of reasons why, you know, it's probably a peak hacking aspect. I don't know if you know peak hacking, but basically it's doing small methodological errors or repeating analysis so that you get a significant effect. And there's a number of things that were done in that paper are not obvious when you read the method section. It's not obvious why this paper is probably wrong. I'm not saying it's for sure wrong. I'm a scientist. I want to see the science of it. A little bit of a software aspect now. Who hasn't heard what Poti and Baggagli, have you all heard about the Poti and Baggagli study? No? No, you haven't? OK. So it's absolutely a must if you want to learn about reproducibility and the problem of like publishing in science and so on. It's a must study. You have to read about that. So what they did in the year 2006-2008, they chose cell lines that are most sensitive or most resistant to drug. They take the array profile of those cell lines, look at the genes that were most differently expressed, and then they, using those genes, they build a model that can actually predict whether a patient will be responding to the drug or not. So what happened is that there were so Baggagli income, so two biostatisticians, look at that study and try to replicate it. And the reason why they could is that the data was actually available. And basically, I'm making the thing short, there were many errors, many errors. And it's not clear now whether there are some errors where in attention or not. But let's say they were not intentional. That's a completely different problem. It's very good, meaning researchers trying to do the best they can. Well, they got a very poor documentation, irrepositability, even well-meaning investigator may argue for drugs that are contraindicated to some patients. Actually, there was a clinical trial going on. That clinical trial was actually giving the drug to the patient that shouldn't have gotten the drug, given the result, the actual result. We've had some effect on society, right? It's not like just a pure, you know, my little research in my corner somewhere. And they actually look at those things. It's called the forensic analysis, because it was really hard. It's a painful paper to read. It's all the minus details of what happened. But what they say is that the most common errors are simple, for example, row or column upsets. And we see that over and over again. And the most simple errors are common as well. And how do we make sure that we don't do the simple errors and the common errors? Joffrey Chang, a very last example, maybe. Free science paper, one PNIS, one, and I know I'm any more, was a crystallographic looking at the structure of the protein. A homemade data analysis program that flipped two columns of data, iterated from another lab, and passed to other labs. And all those papers have retracted. And again, well intentional, this is fine. This is like researchers trying to do their best. And I have to put a figure on that paper. It's a very recent paper in PNIS. But the test of a cluster size in fMRI data, how do you detect an activated region in the brain? And basically, I put that in the method, because I think the method is actually not bad. The paper, I must say, in my opinion, is a little bit hyped. I mean, when I say a little bit, I think it's a. But basically, I think it's the problem there. So they show that there's a very large false positive error rate when you analyze data where they shouldn't be activation. And you find some significant clusters. That's what they show. In fact, what I want to say on that thing is that there is a software which had a very bad default in terms of how you set up that test. And everybody was using the software. And that was, to me, the main result of that paper. And I can talk more about that paper, because I've studied it extensively. So lesson learned. Software, please set the right threshold. That's a very simple thing, right? Train users so that they know what the tests are doing. An interesting comment on the Chang crystallographic problem. So Locker says, I think he was on immense pressure to get the first structure, the first structure. And that was made him push the limit of his data. He could have been more careful, but immense pressure of publishing in high-impact journals. Buggerly, forensic possible, because data, time, and expertise was there. Time was there. It was very interesting to see that Buggerly couldn't actually publish his stuff in a life science journal. He tried to publish in the Nature Medicine Journal that was publishing the particular papers. He couldn't. He had to go for a statistical journal. And they took it right away. Donoho, a little bit of a conclusion. Donoho is a professor on Stanford about computational statistics. Publication is the advertisement. The scholarship is in the code. That's my message from all these examples. So solutions. There's a lot of people trying to mend that problem. It's not an easy, menderable problem, not at all, because of the incentive structure that we have in research. So improve data and code sharing. That's one of the key aspects. And improve the culture such that data and code sharing is just a norm. That's what you do when you are a scientist. And I don't actually like the name share for the data, because most of our funding is coming from public agencies or governments. So that's not actually sharing. It's just doing the publishing that you should be doing. So let's talk about publishing data. And if you've got Nature and your scientific data, and many papers would actually accept data paper, because that's a fundamental block of science. You could have a quick look at the top principles for the publication as well. I think that's a good in terms of code and data publication. And I'm going to talk briefly about two projects that are trying to improve reproducibility in neuroimaging specifically. And then maybe a word on teaching, maybe a word on race awareness. So Repronim and CRM shows that they show that basically the funding agencies have started to acknowledge the problem and think about the problem. And they're starting to fund a group of researchers to do reproducible science in their domain. That's new. That's not a... I know that the United East Matrix Center, for instance, is already a couple of years old, right? And the Open Science Framework is already like a few years old and so on. But that's new for NIH, at least. Repronim is a decentralized, recent P41 grant, and basically trying to set up a set of tools that help researchers find data, launch analyses, and document results in a reproducible way. And CRM is also a Stanford-based and I've got the luck to participate to those two projects. And it has a clear vision of the incentive of it, which is basically give me your data, we'll do some analysis for you, and you'll get back some results. So there's something to buy in for you, right? So those two projects are based on two standards. And the development of standards is completely critical for the efficiency and the responsibility aspect of our science. So NIDM is the standard that is underlying the Repronim project. And it's based on the grant vision, I'll show you. And it's interesting because it is embedded in the most common software in your imaging. It's starting to be embedded, sorry, like it's work in progress, but it is now like an NFSL as an exporter, NSPM as a native thing to export the results. It's a work in progress with AFNI. So really, it's actually there. Bids is a very simple directory structure and layout on your disk, on your file system. And that thing is the basis of the Center for Repositional Neuroscience in Stanford. And also, those two things were actually seeded by NCF, which is interesting to think about it. That's not seeded by NIH, that's seeded by NCF, it's interesting. All right, so what is Repronim? That's the grand vision of it. You have your data, your workflow, and the result those things produce. And all those things can be encoded or written up on the disk or as files with a certain model that can interact between those things, that can make the interaction, the interoperability between those things. And once you have all those data on the disk that are coming from the description of the data, the expression of the experimental design, the expression of the workflow and so on, you can have apps or applications that are, okay, this is neuroblast trying to find and alert and query where are the data, where are the workflows and so on, where are things. This is another application that would say, okay, let me help with the design of that model, of that workflow. Let me validate that I'm refinding the same thing for different data. Let me test, if I've changed the software, let me rerun some of the tests on the data that I've run before to see whether I've found some similar results. NiceMan is the actual computation, how, which platform do you use? Do you use AWS, do you use your local cluster, do you use anything else? And so how do you do that in that context? And all those things and what's the information that you need to get from this computation to be able to reproduce or at least to document? I'm not saying that you may be possibly to take it all, take the Docker container and push it again and reproduce it, that's fine. But just documenting, at least documenting, sorry, this project is doing to do that as well, but documenting what has happened is completely critical. And the whole interaction is always on the same layer of communication, like computer communication based on linked data. CRN, the Stanford project, I mean, it's more than Stanford, but based in Stanford. Put your data in a very simple data structure, I'll show you the format of that quickly. Contrast some apps, so ask all the application in your imaging and construct an app that knows the format. It knows that this is how the data will be. And then use Docker actually singularity containers. Singularity is much better for the security aspect than for your cluster aspect. And then use some other platforms to relate with the data and the clusters and get something, get the results, get the thing that you want to publish. So what's 9DM going back to the format of 9DM? It's really based on that vision that you want to have a communication between the experimental data, the raw data, so the experiment itself, what is going to be done, the raw data, the analyst workflow, the derived data. So there's a lot of transformations. You're not going to have like just one raw data and then the result, right? There's a lot of steps in transformation. All those steps are deriving some data that could be used for another workflow and so on. And then the publication. How do I relate a fact, a finding that I have in my paper? How do I relate that to the whole process of finding this thing? And that's the key aspect and what the 9DM is trying to, and it's a provenance model. It tries to say, what did I do with the data? It's based on the linked data technologies and the probe specification, W3C specification, then it's built on the part of the core vocabulary that if you find the vocabulary somewhere, of course you'll be using the ontology that exists and that are solid, but that's a difficult thing to do, but if you have a term in Stato for the statistics aspect, first of all, you will be using that term because Stato is stable and you can use it and people will be able to know what is a t-test from Stato. And then it's trying to painfully and model all those words, what's the experiments, what's the workflow, what are the results and so on. And we did a lot of work first on the workflow and the reason why we did, on results, sorry, the reason why we worked on results mainly to start with is that most of the analysis that you find in literature are done either with SPM, FSL or AFNI and that's about 80% or maybe, so trying to solve for the 80% before trying to solve for the 100% is one of the thing that we need to try. And that model, it's a model, it's actually encoded and one see it's implemented and it's not treated, it's a triple store somewhere. And that model is valid for both, for all the SPM and FSL in AFNI. It has some friends and it's going to describe all the data using that model such that if you're doing a meta-analysis and you're trying to grab what are the voxel activated in the brain from SPM in that analysis and what are those activated from FSL and one those from AFNI, you're doing to do one query on the same model, on the same exported data. And that's critical for the efficiency aspect and that's also critical to basically standardize what is that you've done and what is that you get. Communication, somehow. That's a computer communication but that's also like our little neuroimaging society communication. So the meta-analysis aspect, we work with Tom and Camille Moumet in Warwick to work on how to do meta-analysis and meta-analysis is one of the aspect that is an answer to reproducibility. You're thinking, okay, if there is a lot of studies and some of those studies are not too well powered, maybe if I take all those studies that are talking about something, I will get a more solid answer. I can tell you that's not always the case. There are some meta-analysis of 30 less not well powered studies and they show effects that they shouldn't. But anyway, that's a little parenthesis. So we work on that model and now we have something that can work we can work for meta-analysis and that's what we're doing at the moment and Camille is doing at the moment. Now the experiment, it's a much more complex thing. So how do you encode all the experimental aspects? So the, and that's using again, using W3C Pro, so it has a specific language of how you do that. What's the project? What is the specific study for that project? What are the acquisitions? All those layers have to be encoded and that's a very long and painful process to do. But starting with some of the examples of big databases or that's a possible thing. And how do we engage, how do we make that successful? How do you engage all the community to work on those models and say, hey, okay, I want to also model my thing such that I can interoperate with you and find things and so on. And so we use these social tools, the GitHub of course, and also try to get some viewers. For instance, a viewer now can just look at the results that have been produced by either of those software and then load up those results and view those, right? And there's no specific viewer for one software on the other or the other. You could have version of that, different version of that viewer that is because people prefer one way or the other, but the underlying data will be the same. What is BID? So I'm switching from opening NIDM things to the CRN BID aspect. It's much simpler. So NIDM is kind of difficult and it's a linked data model and those people in the room that have worked on with that, it's not an easy sell. It's the Python libraries are not always there and so on, right? BIDs, it's very simple, very extremely simple. So I've got some Dicom data. I'm going to specify how they should look like on the disk and that's just a directory layout. And it uses fine-name forming rules. So you know that if you have a T1 image, it should be called like a sub dash, so key values sort of aspect to it and plus an add something and so on. So there's a rule for naming the files and there are a number of JSON files to describe what's going on if you need to something more than just a file name, which is always the case. And the experimental data themselves, the time series, not the time series, but the events recorded in the scanner on those things are TSV files. And very rapidly this spot, like we cover up with an ecosystem that is kind of a large ecosystem now. The major databases wanted to export in BIDs because that's simple or they can read in BIDs from BIDs because that's simple. So that's working progress for that. It's been extended very quickly to EG, MEG, PED data. BIDs are exported to the NDA, NDA is the NIMH data repository and you can easily export to NDA and some of the grants now are required that you push your data to NDA and so you can easily export to NDA with that and so on. There's a BID validator. So if I have a layout and some JSON files and so on, you add those thing BIDs and you have like a little JavaScript validator that Chris Gorgreski have written to do that and Chris has been leading that very energetically and very successfully. There's a lot of BIDs apps now. We had like a week in Stanford where we have like 25 people working on making all those tools to analyze data compatible with BIDs such that you can have a little Docker container. You push it like a BID directory and just run the results and then you push that to the cluster and then you get the results. So that's going to happen. It is happening now. So what are the lesson learns? I mean I just don't want to find that but what are the lesson learns? So I think there's, I was trying, so I'm not talking about this other thing here, but lesson learned that 9DM is a long-term, very interesting, very principled way of doing things. It's going to, and the scope is very large. It's going to take some time. It's a long-term thing. It may not be fully successful because it's a long-term thing and we know that research is never too much but long-term. It's for the next grant, right? It's for the next paper. BIDs is extremely successful very rapidly. How much long-term would that be? It's questionable in a sense because you can't put every information in a principled way in a JSON file in a BID directory. It's hard. It's going to be harder and harder the more information you're trying to encode. So those things are trying to think about those projects as what is the first minimal product that you can get with it? How is the adoption going to happen? The adoption with BIDs is going to be very easy. The adoption with NIDI is always going to be very hard. I'm not saying it shouldn't be done. It's just like a fact. And if you embark on those kind of a neuroinformatic project, you just have to think very clearly about those things. Is that, you know, have enough chance of success? Anyway, okay, that's my lesson for those two things. Do two standards that are actually the basis of two reproducibility projects. I think that's interesting to think about that. And there are some first success already. Like there's already from the repro-name an analysis that goes from a data set that was selected through a query that's run with a workflow that has a provenance written up and the results are written up as well with the same provenance model. It's like a toy example, but it already is happening. That's interesting. With BIDs, there's a great number of data set, the adoption is very fast. So, you know, both will make it easy to record what has happened. Both will make it easy to actually redo exactly what you've done before. And there's also like a new rule for ICF organization or like to, or you know, other things to make sure that we, we don't, you know, we just, we don't want really to compete on those things. We just want to participate in the other project that is doing the right thing. And I think that's the case. I mean, you know, we both think it like a, both project out trying to do the best to coordinate. And that's also an interesting aspect in terms of sociology of the science that we're doing. So, let me talk very briefly about the teaching and statistics and method aspects of a rapid disability because I think that's one of the key things that we have to change the culture as well. And I wanted to, I don't want to, you know, do long things. I don't have the time for that. But I just want to remind you maybe some of you code and some of you have read the code complete. It's a, it's a very good book. It says, I just, you know, and the science, the science and for the scientists, and I think that's, that's absolutely true. It was, it was true for the programmers what he said on the slide. It's now true for the scientists, for many scientists still. In the years of programming, a programmer was guarded as a private property of the programmer and so on. It was basically a love letter from the programmer to the hardware, right. And I think, you know, and the, and the conclusion of that is that such programs are unintelligible to those outside the partnership. You actually don't want to look at the love letter of someone else, right, you just don't, right. And, you know, and if you're a postdoc in a lab, you, you're basically, you know, you're writing your little script to do your little analysis. You don't want your colleagues to, you know, mess up, you know, oh, what's that code and what's that for? And, you know, what do you call that variable? You know, I won't tell you what the most, the most common name for a variable in FSL is because Steve Smith has chosen a very funny name. But, you know, it's, you know, it's Steve Smith's variable name, you know, that's fine. But today, I think things are changing. We're not writing code for ourselves and for our little project or our little science. We're writing code to communicate, right. We're not writing to, you know, our hardware. We're writing to other scientists. So, there's a lot of code review and publishing code, things that are happening. I'm just, you know, laying a few of those here. I think the, I mean, I know that GigaScience, for instance, is doing some code review and things like that. There's the natural neuroscience that I think we'll be doing code review. And so, things are happening on that side as well. The code is not really your code. The code is just to publishing code. I mean, it's your work. Training scientists, completely critical. There's no magic bullet. There's absolutely no magic bullet. There's a training to be done in terms of the NTP hacking, anti-file drawer aspect of things. That's, that should be really like the key aspect of the training. There should be some training obviously of the software script, you know, how do you write provenance and how do you test your things? Our testing is so critical. How do you minimize the, you know, Excel, click and new things? And that's not an easy sell for biologists. It's a difficult thing for neuroscientists. It's absolutely urgent that we talk to our funding agencies, to our politicians who say, hey, just we have to raise the awareness that if we want to do proper science, we have to take the time to do it. We have to move the culture to be a more open science culture. Think of your job as not as a, like a, you know, 95% researcher and whatever 5% of, you know, think of your job, you know, if you want to do science, you have to spend at least 50% of your time advocating for how we should be doing science. That's part of our job. So let me conclude and finish with like a, what will be the time of the years 2020 will look like if we were in 2050? Like, okay. At that time, you know, in the 2020s, you know, the funding agency implemented the fair for data. That was the thing that, you know, they did that, right? The battle for that actually was basically won. Okay, you know, people got in the culture. That was the time where the culture changed. It was, you know, it was prompted many by the sort of a pushback by and often by the clinical societies. That actually, there's very specific culture in the clinical societies. That was the time where the tools started to be reusable. So the output of a grant or the output of research was not the paper anymore. It was the tool. It was the code. It was the data. And that was the output. It was also the time where biologists couldn't major without serious methodical training in stats. And they all knew GIT perfectly. They knew the model of GIT, right? Bioinformatics just became neuroscience because that's just neuroscience. So the informatics effect was still embedded in the biology. Bioinformatics just became biology. It was the time where the publication aspect, when it just slowed down, you know, publishing with more less money, you know, things got more open, but things got more replicated and on several data sets and the generalizability of a research finding was actually possible to test on other data sets with other methods and so on. Publishing of scientific data collapsed. You know, basically new models of online publishing happened. The automatic publication processes started. So, you know, once you've designed the whole experiment and the whole analysis and so on, you had reports. It was a human readable report, but that was entirely automatically generated. And that was the first time it happened, 2020s, right? The OneLab research almost disappeared. Again, it was not funded anymore. OneLab was not funded anymore. It was always projects across the community. Collaborative web tools became standards. And I'm just going to show two quick example of historical example of that. How it is this, for instance, tagging with metadata and the, you know, PDF also website and so on, such that you can re-find things that are tagged with ontologies. That's one thing that happened in the 2020s, actually, you know, 2016s. Collaborative segmentation. People started to collaboratively hack on the same segmentation of the specific part of the brain. And that was entirely on the web, entirely completely distributed across many labs. So that's what happened at that time. And I would like to acknowledge many colleagues and good friends that have worked with, and I was privileged to work with along the past years. And thank you Alain for the invitation and for the organic entity. And thank you for your attention as well. Thank you. Questions for, JB, I have a concern about the whole BIDs initiative. You recall where we were with Nifty and Nifty One? Yeah. This was adopted as a fast, lowest common denominator that everybody could get into quickly. And then the reality is emerged over time that we needed something more complex to take account of the metadata. What, aren't we doing the same thing with BIDs? I think we are doing the same thing with BIDs. But I can, yes. I think, I was actually specifically saying that, right? It's a little bit more of a short term. But during those years of Nifty, I'd say waiting for something else maybe to take over and how that is going to happen. So the question is, are we going to be stuck in a local minima? Yes. And how much energy do we will need to get out of that local minima? I don't have an answer to that. I think you're right. There's a little bit of that problem happening, possibly. But it's going to be so as a step, so useful. Like Nifty was hopefully a step, right? But it's been extremely useful because people could take different software and use the same image format and play with that. But yes, I think there's a danger of a local minima and the only way that can be overcome is by getting the funding agencies to think more in the long term. That's a hard thing to do. And so the advocacy of the, you know, on those things is just like a critical thing. We just have to, and the social aspect, how do you get the social aspect going such that we can be more long term as a community? That's the key problem. Tristan, I think you had your hand up. So thanks for our very nice presentation. There is obviously a lot to digest and to think about. Maybe one question about the preprints. Could you summarize or explain a bit better why preprints are important and how pre-producibility? I think they don't help with pre-producibility directly. They help the open science and they help the psychology of a researcher. That you're putting your research out before, you know, as it is, before it's actually reviewed, and you're sort of breaking the fear of being scooped. So that's the psychology of it. Actually, some people think, we're actually putting a stamp stamp. We were the first to do that as well. I think it can work both ways. The preprint is important in a sense that we need to get away from the power of large publishing companies. And I think that's one way, because they have vested interest in the statu quo of how we do science at the moment, because they make a lot of money from it. So that's also another aspect of it. And there's more to discuss, but yeah. I'm totally aware of that. One more question, I'm afraid, Steven. It's been on Nithyan did the... I wanted to put a positive spin on Nithyan did. So during the Nithyan one process, there was an attempt to move to Nithyan two, and it was clear the field was not ready because the major app developers basically said, you can do it, but we won't touch it. And I would say, Bids is the direct successor of Nithyan two. And it's just, it's a nice example of the sociology has taken, well, 10, maybe 12 years to get, I don't know if you see it that way or not. I do. I'm not saying, when I'm saying we stuck in local minima with Nithyan a little bit, I don't, I think it was, it is still very useful. I mean, I don't know how to do better, right away. It's just like a, so if I put a negative spin, I really, I want to remove it. I mean, it's just a fact that it can't do anything that we need. It just, we'll have to beat the last one. And the question is, how do we get out of it? And that's a difficulty, but it's positive. I'm afraid we're gonna have to move on in a busy schedule. So thank you very much, Jean-Baptiste. You're welcome. Fabulous talk, as always. Thank you.