 So, what I'm going to try to cover today really is rather than just going over what the data sharing efforts are of the Thousand Functional Connectomes Project, I'm going to try to really go over more of the scientific driving, the driving force and motivations for data sharing and for the initiatives you see the Thousand Functional Connectomes Project doing. Five, six years ago, if you told me I'd be heavily involved in open data sharing and all these things, I would say, no, I would have to do any of that. It's a scientific model that really drove the thought of sharing and made us realize the need for sharing. And so I'm going to try to talk a bit about the model that, you know, what we're after, where we're going and why data sharing and not just data sharing but other open science initiatives are so important for the functional imaging community. And to be clear, the examples I'm going to be using are resting state fMRI in several instances, that's what my expertise is in. At the same time, it applies to just about any other sort of imaging you're going to go to look at with many of these concepts, so it's an easy thing to do, I know the disclosures. Okay, so my background very heavily is in child and adolescent psychiatry as well as in cognitive neuroscience and so forth. So that, you know, it's coming at imaging from that perspective, the question is what's the end game? And from the psychiatric perspective and the neuropsychiatric perspective really, the end game is really about biological tests. If you speak to any psychiatrist and also varying aspects of neuropsychiatry, folks will tell you that we live in a world where we're the one branch of medicine that has no tests, no biological tests, nothing to guide us in everyday practice. And when it comes to psychiatry, where everything is based on history, if you're a child adolescent psychiatry, you don't just have a history from a child, you have a history from parent A, parent B, sometimes parent C and D depending on what the constellation is. A teacher, there's a real loss of objective reporting. And psychiatry lives with an osology that's really derived from the identification of constellations of signs and symptoms. And this is really a major challenge. Not for trying to diagnose everyday garden variety ADHD, that's simple. If I showed you a kid with nothing else but ADHD, put them in a room, you would pick them out. But once that kid might have OCD in the picture or maybe there's depression or maybe there's psychosis or mania, it's going to get a lot more complicated. And that's just one example. When we talk about imaging and medicine, sorry, and psychiatry, it's not that we want to sit there and use imaging to sit there and check every child and population. What we're talking about is using imaging in a way that will help us find, with those tricky cases, where we need information to guide us in decision making or to guide us in treatment decision making. And here is just a sample of every psychiatrist stream. So you have someone goes in for an image, you generate some connectome-like image, and you wind up with a report akin to a laboratory test that will then be another piece of clinical information to guide a psychiatrist in their day-to-day practice. We don't have this. There are many people who are trying to claim to have this in varying ways. People who come to see me because of my imaging expertise clinically, I tell them, the best thing I'm going to be able to do is tell you don't use any imaging. None of it's going to guide you in clinical practice, unless maybe there's a brain tumor or MS or so many other things, but not for the bulk of what we see. So really the question is why don't we have biological tests? There's a prospective piece by an AMH director, Tom Menzel, which I'll talk about a little bit, which I really speak about many of the truths as to why we don't have these tests, and the thing you'll notice is I'm going to speak about the research culture, rather than the actual science itself. And also there was a paper that much of what I talked about draws from, which is a paper that we really adapted a lot of our thinking as well as some of the insult commentary and brought it to functional imaging. Okay, any time we get into these conversations, first thing we should all agree on is what is a biomarker? So the one of the best lessons I can ever give anyone I ever train is Google. It's like the first thing you should ever do when you have a question is check Google. So this came up the first time I checked Google and I felt good and was able to move forward in life. NIH biomarkers working group. So basically a biomarker is a characteristic that is objectively measured, evaluated as an indicator of normal biological processes, pathogenic processes, or a pharmacologic response to a therapeutic intervention. So what are we talking about? We're talking about an indicator. It's not necessarily meaningful as far as what it is, but it's indexing some of the processes and that's what we're after. You know, when you talk about, well, what are potential biomarkers, everyone has their favorite. Me, I like resting state fMRI, so of course I think it's God's gift to everything, but the thing is someone else looking at genetics is going to point to that, someone looking at morphometry. The bottom line is there's a very broad array of potential candidates for biomarkers and we really need to avoid honing in on anything prematurely and this figure here by Builder and Poldrack is one of my favorites because I think it is a wonderful job of putting out the many ohms and how they interact and I love that it's all within the environment and this is something we need to keep in mind and there are many levels that we're working out. So it comes again, what exactly is a biomarker useful for? The first thing everyone thinks about is diagnosis. In psychiatry, that is a little difficult because of the, we don't have a gold standard you're actually going against. We don't have any biological ground truths to say this is the exact right thing you're going to predict biologically. We don't have any ground truths as far as I can sit there and on one hand we've established increasing reliability in the psychiatric community by having DSM-4 and other diagnostic manuals but there's questions about their validity in several instances. So you have an imperfect raters at best. But then also comes other uses that folks don't think of as much. For instance, staging of a disease. If you look at cancer, there are multiple examples in cancer where they couldn't, they didn't have a marker to say whether or not the cancer is present for screening purposes but they do have markers that can guide them in the staging of a disease. And incredibly important determinations of risk or prognosis. Child with pediatric anxiety, much higher risk being an adult with depression. Child who's not verbal by five years who has autism has a much worse prognosis in the long term for end outcomes. But what if you knew who that child was going to be at the age two? Of you could have intervened. When you hear, when people talk about early intervention or prognosis, it is these determinations of risk that they're so important. Prediction and monitoring of clinical responses to an intervention. Right now you make decisions based upon, you know, on what you've seen, you know, an experience on some papers that might be guiding you, lots of anecdotal things. You don't have clear things to guide you that in this case a stimulant will be better than going with an alpha agonist when I'm treating a child with ADHD. In some other case, no one better off going with the alpha agonist. And you could go back and forth. There have been vignettes in their pieces of knowledge we have. But there's nothing objective that we could test to help with that response. And when you take, look at medications that have horrible side effect profiles like neuroleptics where you get weight gain, metabolic profiles, everything else. This would be incredibly useful. Now this here is always a tricky issue. When it comes to what makes a biomarker clinically useful, you hear lots of people talk about, I got a biomarker, and you're very excited about their biomarker. And then you look at it and you think to yourself, well, is it hitting the core profile or is it hitting the core criteria of any clinical test in any other realm? And there you talk about validity, reliability, sensitivity, specificity. How often in a literature that's coming out of the imaging world when people are talking about biomarkers, how often are people putting their tests to these standards? Every other domain where you talk about laboratory tests, this information is provided. In imaging, there are many times we see people bypassing this or misinterpreting and overselling. And of course, everything is widespread availability. I think it's the world's greatest test. If no one else has it but my one center, it's not really going to have an impact. Effect sizes. Now this part here is the most sobering. This here is a diagram that we put in where, you know, it's a standard ROC profile, but the thing is the least impressive curve there has a cone effect size of 1.5. The most impressive is 3. Now the least impressive 1.5, if you look at things like PSA, prostate-specific antigen, you'll see some net looks like that. And it has usefulness in screening. It also has its downsides. The one with the effect size of 3 being the highest of them, that's being much more useful for diagnosis. But the thing is you've got to ask yourself, where are imaging? When you hear people saying, I think I have a putative biomarker, how many of your biomarkers are going to have a curve even near the 1.5? And this is where you see a lot of misusage of the word biomarkers in the community. And this is something we're really trying to get people to think about. What is it that you actually have? I promise the whole talk won't be depressing. At the heart of this comes down to the issue of everyone's chasing a P less than 0.05 significance level. And the thing is, that could be informative. I can learn about the brain from seeing these differences. I am not going to be able to sit there and do classification, prediction, or anything else from a clinical application point of view with P less than 0.05. Yet that's what the literature chases after in the insult paper to refer to this as significance-chasing. And they get into the issue of what people will do to get P less than 0.05, which just makes it all the worse. Because if you're having to hedge and make these little trade-offs to get to P less than 0.05, you really aren't going to do anything that's going to be helpful in the clinical realm as far as an application. Maybe not a scientific meaningful, but not from a clinical application point of view. Also, it's important to keep in mind that biomarkers, they're associative by definition. They're not necessarily causal. And that's good and bad. That means that we might have a biomarker which gives services as an indicator, but it's not going to help us inform us scientifically as to what the process is or the nature of it, just that it's there. But the other thing is, a biomarker does not necessarily have to convey nor a scientific meaning. And you'll see this fight go on in the imaging literature where some people want to model with a very limited number of features and nodes so they can interpret what their classifier is doing and feel very good about that. And then other people will say, you know what? I got some incredible number of features. I'll do some selection. I'll get some meaning. I'll get some classifier, which maybe also has nonlinear properties. And I may never be able to interpret what's going on there. But if the classifier works and can change someone's life, then who cares? Each side of this has merit. And once again, no best modality for biomarker discovery, no matter how much we all love our own. Now, here's one of the big challenges to that constantly of seeing, which is validation. Now, leave one out is good for a sender and testing proof of concept and feasibility and so forth. But you should not be talking about changing someone's life meaningfully in a clinic based upon leave one out or even K-fold replications. It's a difference between trying to convey an idea and a principle and say something's worth a future study and larger samples than to sit there and say, let's go with this. And I've been at conferences where I have a clinician who does some research, come up to me and say, OK, I got this classifier and I get 95% accuracy. It would leave one out cross-validation. And I feel really good. So I think I should start using it with my patients. What do you think? My comments, no, don't do it. Please. Start to investigate. Create larger samples. Start to look at then more stringent forms of validation. So of course now the ideal is independent replication samples, which if you're in the genetics community, you're looking at the imaging community and thinking, of course, you guys should know this. But the other thing is you also have to be careful. Because even when you go and get that independent replication sample, it's not a guarantee. Because if you constructed that sample the same way you constructed your prior samples, you could have different biases in sampling. There could be lots of different aspects that your data may not be as representative as what you think. So it still can be confounded. So it's actually a large problem. The challenges here are significant, especially when you look in a field that in recent years has always been talking about 20, 30 subjects in each group. So then comes the excitement about connectomics. And it is the connectome better positioned genetics. If someone is into connectomics, of course, I would point out, well, the neural level is closer to the cognitive, symptom, and syndrome levels. So when you look at shared variants and from one level to the next, we're closer up in the tree and that we can go in both directions. People could go back and forth about this. Like I said, I'm someone from this literature, so I'm going to talk with some examples from this. But everything I'm saying is pretty much more broadly applicable. As far as MRI-based prediction, it is a growing reality. People are starting to get to there. There is a growing number of papers emerging, some with issues like with the resting state one to the left by Dawson Bucket out, trying to look at brain maturity. And then on the right, you have one with using T1-based imaging, looking at their index of brain maturity. Now, you could see one is inferring that things are linear, the other non-linear. One could go back and forth as far as differences in their sampling of the age range. And one can also point out gaps in the sampling of the Dawson Bucket one. The point is, we're starting to get there. The ADHT 200, years ago, we had a global competition where we gave away the ADHT 200 data. We gave folks, at first, about 80% of the data to use for training. And then we had kept out about 150 data sets and gave those away unlabeled and had people go forth and do their best. The good news was, well, a couple groups had data that was highly specific, about 94% specificity. The downside is that group also had 20% sensitivity. So if they were calling you ADHD, they were generally right, but they would miss many cases. That's from two years ago. Over the years, just by giving out the simple data set that was not really meant to be, this was an aggregate data set, it wasn't really meant to be used in any sort of way like this, we were actually able to really inspire a lot of folks to bring analytic methods in. And I'm always impressed, each conference I go to, I always want to see someone who's bringing the numbers up for their ability to classify ADHD using data that was made openly available regarding the consortium. At the same time, this is, once again, a reason for sobriety. And the Castellanos paper that we did, we actually have table that as of 2013, we looked at every resting state study doing prediction, put the other comprehensive table that for each one shows you what's their sensitivity, specificity. We also give positive and negative, the real world prediction values. But one of the other things is for every one of them, we also indicate the validation techniques used. And it's very sobering when you look at this, that there's some signal there and there's something meaningful, but there's also lots of caveats because this is a very early field. Okay, when you look, and now going with resting state, the one of the questions you have to ask yourself is, are we being critical enough, which is funny because if you do resting state fMRI, what you feel is that you're constantly criticized and we're like the noisiest signal on the block. But the thing is when you look at any of us, you're gonna be asking yourself, what level of validity have we met? Are we having face validity, concept validity, criteria validity, these are key concepts that other scientific disciplines speak about constantly, yet in the imaging literature, we actually tend to shy away from this. A lot, when you look at the initial BISWA work, that would be an example of face validity. It looked right. He seeded a region in the motor system and the motor system lit up and that was very exciting. But the thing is when you start asking about criteria validity, are they considered with independent gold standards? That's a much higher standard to hold yourself to, especially when you struggle with what are independent gold standards in which you're supposed to put your modality against. I'm gonna talk about this more in a minute, but the other thing I'll point out is reliability. Reliability, on one hand, myself and others have really pushed the importance of test-re-test reliability for our measures. And for resting state, that becomes incredibly important because you have a state where people are relatively unconstrained. And so researchers would sit there and say, well, if you can't control the person's thinking on two different occasions, you're never gonna get meaningful test-re-test reliability. And we were to debunk that idea years ago and show you do get reliability, but reliability's not enough because artifacts can drive reliability. And that's a major concern, and I'll show you one piece of information that should really make you consider that. But going back to validity for a second, if you take something like resting state fMRI where folks can say, well, it's hard just to establish validity, well, it's yes and no. For example, here, resting state fMRI, this is one of my favorite studies, this is by Steve Smith, where he basically took the brain map database, and what he was able to do is run an independent component analysis on the brain map database and retrieve a set of functional networks that come out. And he ran an independent component analysis on resting state fMRI from 36 subjects. In one case, you have thousands of contrasts from over a thousand studies coming together. In the other case, you have, I think it was like 36 subjects. And the bottom line is you get highly consistent networks emerging from these two analyses. That is something that really excites someone when you see that and think to yourself that this is two independent modalities coming together. Also, when you look at MEG literature, there's an increasing number of studies trying to compare resting state fMRI and findings in networks from MEG, once again showing a high degree of concordance between them. Here's a study that Corey Keller and some of our other collaborators, we all work together on, we're using cortical evoked potentials looking at the millisecond scale. We were able to basically link patterns of cortical activation from stimulation from neurosurgery to what we get in the same subjects from resting state fMRI and show concordant patterns. This is one of the means we could go by of establishing validity. Now it comes back, of course, to artifacts. And you hear people constantly talking about we need to standardize, standardize, standardize. Yes, we should as best we can, but we also need to be humble. No matter what you do, you will never have the perfect standardization you think you have in your MR protocols. You get the same sequence, you get the same scanner and the bottom line is you can't control everything. So here's a table from a paper we did a year and changed back where we basically go through acquisition-related variables, experimental-related variables, environment-related variables, participant-related variables. You can't control sleep deprivation. You can't control whether or not someone's anxious in a moment. It's hard to even control whether or not they drank coffee before or took some sort of caffeine in because you tell them not to and they go ahead and they do it anyway and it's whether or not they tell you. And anyone who needs an extreme example, look at substance use. When the person who says they don't use substances comes up positive on a cocaine test, done during the scan, well, that's telling you something. So instead, it does push for the idea that we need to find different approaches for how to correct for these. And there are classic examples of artifacts that live that we need to be aware of and there's a whole literature that's come out about head motion in resting state fMRI and the reason why I say calmly in that is because the resting state fMRI community is not handled artifacts calmly. It's handled it somewhat hysterically at times. But there is a large literature emerging for how to handle artifacts and as someone who's in the field who reviews papers, the challenging thing is when you get a paper that doesn't take into account any one of these corrections. There's no perfect correction but you should at least be accounting for it and there's definitely a need for folks to be adopting more and more. And then of course, going back to the world where you don't have complete biological truths, you have the global signal argument with a resting state and the significant literature that's come out there. These are two examples of two of the most common artifacts but these are things that folks need to be thinking about when they're looking at their findings. One consortium that we, another focus that I mentioned is reliability. In that regard, one consortium that we recently put out is something called CORE, Consortium Reliability and Reproducibility. Now, as I said years ago, we established test retest reliability for a variety of measures but that was with a subject, a data set of like 25 subjects, image twice in the same session and once five to 11 months prior. And there are a series of other studies by other groups that emerge showing reliability and so forth but this is still very much an open question as well as asking what improves or decreases reliability for measure. So with CORE, what we've done is we've pulled together data sets from about 30 samples from 18 sites and pulled them together about 1600 subjects with over almost 6,000 scans combined from a variety of test retest designs and we've made them all available through the Thousand Functional Connectomes project. These have been up online since June and openly available and anyone could go get them, pull them down, there's a variety of designs. It could be that you want 30 subjects who were scanned 10 times over the course of a month, three days between each scan with ASL, diffusion and resting state fMRI or it could be one or two subjects who were scanned, I think it was like 45 times each. There are different designs there. It's definitely worth taking data from. Going back to issue of artifacts, just one little thing that I'll point to folks out is you constantly hear about motion and one question that, and this is something that stills up a suit of flavor that's always known but it's not really pointed out enough, motion itself which is a potential confound resting state fMRI can be reliable and actually you're seeing that in this graph right here. If you take a mean frame wise displacement at 0.2 millimeters, which is actually a decent amount of motion, but you go 0.2 millimeters and down, your correlation between session one and session two for motion is about 0.6. Now the interesting thing is you take really noisy data, motion mean at fDs above 0.2, well there your correlation goes down. So I don't know if that should really make folks feel better that if you have really noisy data it doesn't seem to be correlated but the bottom line is even when you have low noise it's still something you need to be accounting for. Going back to the question of biologic applications, I just want to point out everyone is talking more and more about diagnostic prediction, you constantly see folks doing it, but you have to keep in mind there's lots of questions about the validity of the diagnostic system we use. It doesn't mean we should be throwing it out because the reality is what's been built up allows for reliable communication between clinicians which if we didn't have that we'd go back about 20 years or so and it'd be a wild, wild West version of practice but we do have to think is that maybe the best goal or not? And so that's where there's been increasing push for stratified psychiatry where you deemphasize diagnosis and you start asking about risk, prognosis, things that we can quantify. And so if I follow subjects over X number of years and see just the question of does someone with this marker where are they 10 years from now, that's doable. Or if the question is if I give someone a stimulant versus an alpha agonist what's the response gonna be and it's observed? These are the kind of things that are more tangible and we can really make concrete and have more quantifiable results. Okay, so for the final part of things I'm gonna do is just go back to that in so paper I mentioned. So the research challenge like I said lack of gold standard, that's a big one. But the bigger issue it comes down to research culture and there, one thing is the significance chasing which I already mentioned. Another thing comes down to replication. In the imaging community replication is far from a norm and often when people say they replicate someone else's findings, what do they often do? They often say, well it was in singulate. Singulate's pretty big, we're in singulate. And how specific are you being in your reporting of findings? Usually not specific. And then extreme comparisons. This is another thing that came from the answer article which I thought was really something that was well said and we don't say enough of which is everyone is going with, okay I have autism versus controls or schizophrenia versus controls. Now first of all, do you really need imaging to figure out whether as child with autism is typically developing control or not? I think if you just spoke to a child for just a little bit you'd get the idea. So the question really is, is this autism or maybe this is a child with ADHD and expressive and receptive language disorders or maybe the child is psychotic. There are lots of different, or has very severe OCD. There are lots of different things that creep into differentials. And if we're gonna ask of what's more meaningful in imaging, we should be comparing disorders to one another, rather than just single disorder versus a typically developing population which if you think about it, let's say you come with the best classifier in the world for autism for autism versus typically developing children. Have you established something that's useful? The answer is who knows? Maybe I could put a child with ADHD in and that classifier is gonna work just as well in saying the kid has ADHD, or sorry, or not. The problem is you have an established specificity. We need to understand specificity and that's a major challenge. All this goes back to the issue for, we need big data. The idea of significance, I said you can't do these studies with 1530 subjects. Going back to why would someone like myself who initially had zero interest in data sharing six years ago and all this open science and so forth be such a strong advocate nowadays. The reality is at that point I started building a much larger resting state fMRI data system than what most people had and I quickly realized they're useless unless I start combining them with other people's data unless I change the scientific model. And it's not that I have any compounds with the utility of the method I'm using. I think it could be very powerful. It's the scientific model that's the challenge. Going back to the issue of what to do about validity and of the psychiatric system we use. So the NIH has been pushing the research domain criteria project. The idea here is that we should break psychiatric illness down into a set of different constructs and domains and within each domain, construct would be working memory, whether it be negative valence and so forth, you come up with these grids that have the genetic layer, molecular cells, circuit, physiology, behavior. This is just a tiny piece of the RDoC plan and the second you look at this, it's overwhelming. And the question of what kind of data model is needed for this, in my mind, that's screaming big data. I mean, if that's not gonna be big data, I don't know what is. To fill in those grids with any sort of completeness is a massive challenge. So on the one hand, the RDoC is very good in exciting us and making us realize the weaknesses of what we've been doing and basically saying what many of us have said for years. On the other hand, it can be misapplied in different ways, which it's not all a compensating answer and there are lots of caveats and so forth, but it is pushing us more and more to rethinking scientific approach. And what's the scientific approach becoming more and more? It's data-driven analysis, which as someone who's a background in cognitive neuroscience, that was not a, when I was in grad school, that would be sinful. If I'd say, well, what about this or that? I'd hear, well, would you have a hypothesis? And it's like, no, well, then it doesn't matter. I don't think, what do you mean it doesn't matter? It's there. But this here is one of the challenges that we face for years. I love pointing to this paper from Damian Fair, which actually has zero brain imaging data in it at all, though this has also recently been done with brain imaging data, where they sat there and took about 250 subjects with ADHD, 250 without, took neuropsych profiles for each, did a data reduction, and then built a graph where each node in a graph is actually a subject. And what he did is he carried out subject community detection. So he looked for communities of subjects in the sample where half the individuals had ADHD and half didn't. Now, the first thought that many would have is, okay, great, we're gonna find this ADHD cluster and versus this typically developing cluster, and then within ADHD, maybe we'll find some heterogeneity. What they found was actually quite different. They found that there is no ADHD cluster. Instead, what you get is six cognitive profiles that emerge across kids with ADHD and without. And then when you look in each cluster, ADHD winds up modifying the profile within that cluster. And that would, if you think of ADHD and the many presentations that comes with it, that would actually partially explain why is it that we see what we see clinically. Is that something that someone would have thought of just sitting around and came up with a hypothesis? Probably not most of us, maybe someone. But this is the potential value of what we can do. And as I said, this has recently been done with imaging Zhiyang and Zhuo. They recently took an ICA-based approach and wound up showing that they could differentiate individuals with schizophrenia from those without using a default network connectivity and using subject-community detection. And then they showed with another network they were able to differentiate positive versus negative symptomatology. Keep in mind, what we're talking, it's a heart of all this, what we need is knowledge. Knowledge is gonna be essential. Here's an example from the NCAT Rockland of what our phenotyping looks like for a lifespan. This is two days' worth of phenotyping we put folks through. Where we have these comprehensive assessments, what you're seeing on the right in a given line is what questionnaire is given based upon age. But there's a huge amount of information collected. Going from an informatics perspective, you can see the science, if you adopt these models, you're mandating use of powerful informatics. Otherwise you will very quickly cripple yourself. Here in this example, what we went doing is building everything in with coins from my research network. And what we did is we have their assessment tools do all the capture of all questionnaires, of all items. So we have individual item data for everything and we were able to make this more tractable in our operation. And actually once we switched away from paper and pencil, despite an initial few months where I was a very unpopular person for doing it, ever since the team wouldn't think of doing it otherwise. Going back to big data, the reality of big data is what you're trying to do is you're trying to understand complex systems through application of data intensive, data driven approaches. The thing is you're not coming up with hypotheses, but you can't just look and say, big data give me an answer. I mean, you hear these people, people are getting really magical with this. So what you need is a range of questions. You need a specific question that you have in mind and when you build a big data resource, you need to have anticipated a range of questions people may want to ask. If you don't do that at the beginning, then you basically lose out in the end and even when you do the best of jobs, you find that one thing that you're like, I really wish we had that in our sample, but if we don't. As far as our not congruent with big data, like I said, it's absolutely consistent with the model of trying to break psychiatric illness down into all these pieces and building up. So I spent a lot of time getting beat on by folks over the issue of big data and their science. So this year was my attempt at trying to find a way to explain to people. It's not that we're saying hypothesis driven science should die, it's far from it. If you have hypothesis tested, but the thing is, as you see here, frame with me with AA, this really helps a lot of people with disorders get through tonight and I think it could help the imaging community. So really, this oppositional thing between big data and hypothesis testing really needs to calm down. And I've seen pockets where it is and pockets where it doesn't go so well. We're going into the fight soon of what's big data. Is 1,000 subjects really big data from a sample size perspective? Probably not. Is 1,000 subjects with thousands and thousands of observations? Probably you're getting there. And you could argue about where to put the number. But we also have to be practical and expectation. You look at genetics community, they're not even talking about 10,000, they're talking about 100,000 plus. And you kind of hope from a connectomics perspective we could do better because we're closer to the behavioral level and so forth. Maybe that's true, maybe not. But even one of the last few points I wanna make is representative sampling strategies is a huge issue and I'll talk about those more in a second but you can't speak beyond your data. So if your data don't sample a particular population in an appropriate manner, you can have all the data you want, the bottom line is if you left out a particular population or you over biased things in ways of some subpopulation you're not gonna get informed of results. Because sometimes I think people think if I have a lot of subjects then that's the solution. So when you get something like the Generation R study happening here in the Netherlands, that's a gorgeous example of having an epidemiologic sample which that, from someone who's sitting over in the States, I'm just beyond envious of that initiative. You need representative sampling strategies and then you need everything else. You need a data intensive hardware, you need automated quality control metrics, which is tricky if you notice I put the word consensus there. Good luck trying to get consensus in the imaging community. We're getting better, but the fight continues. And you need sophisticated data driven exploratory techniques. On the one hand, people have been doing data exploration for years, it's called ICA. On the other hand, there's a much broader range of techniques that need to be considered. And the thing is, as a psychiatrist, I may have a background for my younger years in computer science, but at the same time I'm pretty humble to sit there and say I'm not gonna be able to go and compete with the best machine learning theory guy on the block because when I was in college, we weren't learning machine learning theory. And the thing is, I don't expect myself as a psychiatrist to turn into that, I expect myself to find collaborators. And the reality is the guy doing the best SVM on the block, first of all, maybe SVM's not even the best approach, but if it is, he doesn't know what questions to ask because he's not a psychiatrist, he's an ordinary scientist. So it's a collaboration that's what's needed. Last part of what you wanna talk about is data sharing, which is put the numbers in perspective. If it cost me $1,000 on average for a non-clinical sample, which, if I take scanner time and really expedited phenotyping, maybe I could pull off, to my mind it's for 1,000 subjects, it's a million dollars. Once you start talking about a low difficulty clinical sample, all of a sudden you're talking at least 2,500. By the time you get to a high difficult population like autism, by the time you go through the assessments, by the time you have someone who's trained with research reliability, by the time you have the support staff to guide that individual who's going to text your staff a lot more to get through the magnet, you're at least looking at about $10,000 a subject. I base these numbers on our own grant applications as well as some of others. 10,000, 1,000 subjects is a huge investment. And the thing is for, to ever accomplish that and have five or six papers come out from that sample and have that as, need to be the life of the sample is a huge travesty. When you look at genetics community, back in 1996 with the Human Genome Project, the Bermuda Principles, they said, release everything as quickly as possible, don't put these delays in, make things public, go for transparency, these base principles, they're talking about making things open and aggregating and making it a game of leapfrog, which is what open science should be. Not to sit there and play the research silo game, which the imaging community has definitely dragged on for many more years. Here I'm gonna finish up with is looking at the Thousand Functions Connectomes Project. What it represents is grassroots data sharing. No one paid us from beginning to to share data. Try to know when really pays us anything much still to to share the data, would be nice. The bottom line is these are grassroots initiatives where we wind up forming consortiums of people who all have a like-minded interest around a particular disorder or topic and say, would you give your 30 data sets, by the time we're done, if we have 1,000 data sets or even only 500, you all will do better as a community. What you're seeing on the right, so there's the initial release back with the Thousand Functions Connectomes Project, that was Rob Biswell and myself that co-founded and directed that one, and their data we had released with AgentSex, so very quickly we went with a bunch of papers looking at AgentSex, and then realizing the need for deeper phenotyping and more in which samples and so forth, I went and founded the International Neuroimaging Data Sharing Initiative, and under this we've had the Autism Brain Imaging Data Exchange, the ADHD 200, the Rockland sample, which is actually a slightly different model. Here we do pre-publication sharing. This here is a sample that I've been generating for the last four years, and we just give it away on a regular basis. Just to give you an idea of what's the impact of that kind of philosophy, we just checked the other data, 46 publications since 2010 using the Rockland sample data. I've had seven of them. So does that mean I should be working more? Maybe, or it means that the community is doing pretty well and that sharing really has huge merits, and I don't have to, if I was to sit there and think, what's it gonna take for me to get 46 papers in order to then really, it would have been horrible. So people really, if you give them data, they will be excited about it, and with the ADHD 200, there have been over 60 papers generated by that, there's only like six or seven so far, but there's a huge number I've been seeing at conferences, a byte was just released a year and a half ago, and then Core is the newest on the block. Just to give you an idea, this is at any point in the last five years, sorry, four years, if we go and look at Google Analytics, we will get a map that looks like this. This is what the audience for 1,000 Functional Connectomes project looks like. These are example papers generated by our group. These are announcement papers. So this is 1,000 Functional Connectomes project. This was the consortium paper, and we actually did look at sex effects, like I said, that's one of the things we could. Then comes Indy looking at subtypes of ADHD, in the HD 200, which you couldn't do with a small dataset. It, do you abide consortium, resolving a controversy as to whether a brain's hyper-connected or hypo, turns out it's both. How is this kind of data used? Here's a quick survey that was done. You can see doctoral, these these papers, pilot data for grants. This is the way you want to community operating. It's, you know, there's plenty of room for any person. Standardization of data, here's a paper that we had done, Chaganyan and all in our group, where because we had this aggregate dataset, we had to come up with how do you count for all the variation, and what could we do to take into account groups, site-related differences? So we, you buy on the borrowing from the gene expression literature, a variety of different kind of post-hoc statistical approaches and showing their potential utility. Once again, if the data wasn't available, these kind of questions couldn't be asked. The same thing with dirty data, like with data that has poor quality. People often think, why would you release that? And my answer is because I want us to be able to evaluate quality across sites, I also want us to be able to actually motivate folks to come up with more advanced corrections. And of course, as I said, phenotyping is key. One last note on representativeness. With the Rockwood sample, we've gotten through a great lens to try to sit there and do a community ascertained design so that we're seeing a representing different zip codes. And I will admit, it's very hard. Like I said, that's why I'm so envious of the generation R for what they pulled off. I mean, it is hard getting folks recruited and going through all this, and it costs a lot of money, and it's absolutely what's needed in the field. On the horizon, this here, you buy preprocessed initiative. Four different pipelines in resting state community. Each of our pipeline representatives had taken the data and had agreed to take the abide data and preprocess it. So now, this is actually readily available for download. Pre-processed data and derived data from these four initiatives using the same dataset so you could compare the initiatives. Also, back when we did abide, we had taken the same dataset and spoke to each one of the folks who are producing databases in the resting state community, each one of the developers, and they all took the same dataset and put it in their respective databases to allow for comparison of databases. I think a lot more initiatives like this are needed to really get people to understand what databaseing options exist. A lot of people don't realize there are actually some very good databases out there, and they really present good options and solutions. And last thing is, everything I spoke to you about is inhuman. We know resting state and the functional imaging approaches for morphometry could be used for monkey and translational. I strongly suggest we start thinking about sharing that data, and I'll skip the last two and leave it there. Thank you so much for your talk. This is great. One thing that you didn't really address, or I may have missed it, is the provenance of each data sample as it moves from one database to another, which this data really has done. So from where I sit at NIF, it's very hard for me to figure out is this actually a different sample than the one I've got over here? So if you could address that question just a little bit, that'd be great. Oh no, so there are many levels of provenance. One is just even when you start with the MRI images that are being shared, you could get into trouble. Most of sharing we've had has used NIFTY, where folks have gotten rid of most of their metadata, which is challenging. And the DICOM would be more optimal, though it's more difficult getting folks to share and handle the DICOM format. But even at that grain there are challenges, which we've been working to try to get folks to give us dumps directly from the magnet and so forth. Nothing's perfect. As far as going metadata beyond your images, that is a key issue. One of the things I keep saying about the phenotyping, right now people have a tendency to only to share the bare minimum information, even in their papers frankly, about their sampling strategies used and who they've actually characterized. So I'd say there are two levels of problem there. First, they themselves need to better characterize the data sets, and then you could talk about how do we share those descriptions. With the NKI Rockland, the thing that's made as possible for us is really the usage of a sophisticated informatic system. If you're doing electronic data capture, your life is not so difficult, then you just need to make sure you've captured the information and it can be maintained and then shared. There is the issue of course, linking one database to the next, which that there is between the database developers and I know some of the different initiatives are working to try to create interoperability between the databases. But the other thing is the fact that a lot of this information is just not logged and I think that's where the biggest challenge in my mind is to get the data generators to essentially start using informatics, to start using something more than an Excel spreadsheet. If you send a seed of the thousand functional connectance project in Indy, we see everyone's data because we get these files and CSV, XLS, SPS, and we see a lot of the dirt at the labs and we appreciate that people are opening up and sharing and then we spend a lot of time going back and forth saying what did you mean here? Could you check this here? And we go through this. So I think it's a working problem but it starts with getting the labs to change their behaviors and it filters up towards the databases being able to communicate efficiently and switch data dictionaries, all these kind of issues. Michael, thanks a lot for the talk. It's wonderful to hear you on the... And there's so much that you've done that is actually shaping a lot of the data sharing in neuroimaging. It's just amazing. And a lot is going to happen because of those initiatives and the... So apart from that, there's so much... I mean, many of the points you raised around the whole talk, but there's one point you've never raised which I was a bit surprised is that there is a classical excuse for not sharing data which is the ethical approval. And I've got my answer on that but I will be pleased to have yours. Yeah, that's an important question and with one of the multiple levels. At the end of the day, the first thing I would say is it's going to always come down to the local IRB or ethics board. And so for 1000 Functional Connectomes, we've always asked folks to go and get approval from their ethics board. Generally, what happens is if it's a data, the first question is where folks consented for sharing? If the individuals in a participant study weren't consented for sharing, now you could jump into... It doesn't necessarily end there. Now it comes down to what kind of information is being shared? Do the risks outweigh the benefits of the sharing and also how are you sharing the data? So if it's like 1000 Functional Connectomes project, where we were just giving away agent stacks, no IRB really required... Actually, that's not true. Three IRBs said that they would need to go and consent. Everyone else said, look, risks are outweighed by the benefits here, just go ahead and share. As we've gone through the Indian initiatives, we've seen variation where in some of these initiatives, the IRBs have said you should go ahead and re-consent and for a buy, there were several sites which did re-consent and then other sites said, no, that they disagreed with that. We had one site where two investigators going to the same IRB had different answers, so one re-consented, the other one didn't. You go through this process. When you start sharing things like the NKR Rockland data set, there we have something called NKR Rockland Lite where it's just images, agent stacks, and handedness, that anyone can grab. But if you want that high-dimensional data, which contains all the diagnostic information and everything, you have to go through and sign a legally binding data usage agreement between the two institutions. We in no way restrict your analysis in any way other than you can't re-identify. And then comes the hard part, which is right now, if you have any idea you might be sharing, and we all do, then the question is, if you're looking at what needs to change quickly, every person entering a research study should be given a right to know whether their data may one day be shared. And so it is a complex issue, but I don't think it's a good excuse not to share. If anything, they should be able to sit there and say, when is it my IRB? And my IRB told me no way. Or my IRB said you have to go re-consent a bunch of people that couldn't possibly re-consent. Beyond that, I don't think there is so much weight in the claim. Yeah, what I'm getting at is putting the burden of the ethical use of the data. I mean, it's something different from getting something which is ethically acquired. And somebody could actually get the political approval for the use of the data. And that's the other thing. And the other thing is possibly putting the ownership of the data to the subject themselves, which is another, you know, like a... Yeah, well, so you can get into, you know, and I know just last week in light, and there was an intriguing conference actually going through these issues with ownership of data. And you know, my comment on this is we need to look at other literatures and not act like we're making this decision in isolation. If you look at the clinical trials literature, they've been working on for years who owns data in farm trials and things like that. And it typically does go with the idea that it's not the investigator. Yeah, and there's a variety of variations.