 Hi there. Thanks so much for having us to co-present. My name is Jen Goldsack. I'm the CEO of the Digital Medicine Society, or DIME. For folks who may not know DIME, we are a 501c3 nonprofit organization dedicated to advancing the safe, effective, ethical and equitable use of digital technologies to improve lives. I am delighted to be co-presenting with my colleague, my longtime collaborator and great friend Jesse Lindan. Jesse, do you want to introduce yourself to everyone? Sure. Thanks so much, Jen. My name is Jesse Lindan. I'm an assistant professor at Duke in the departments of biomedical engineering and biostatistics and bioinformatics. I run the Big Ideas Lab at Duke and I'm thrilled to be presenting at this venue and presenting alongside Jen. Fantastic. So the way we'll run this today, I will start by presenting a little bit on sort of current state of the field and specifically digital clinical measures. Then what I'll do is I'll hand over to Jesse Lindan to take us through some of the science that we use to ensure that when we are using digital technologies, digital tools to measure health, we can be certain that those measures are trustworthy. Then Jesse Lindan will hand back to me. I'll keep it short at that point and we'll just reflect on what needs to happen to take all of this to scale. And importantly, Jesse Lindan and I actually recently collaborated on some research. We have some data on the state of the field and in particular security research that we thought it might be interesting to share. So that's how we'll close out. So with that in mind, let me go ahead and start sharing my screen. Jesse Lindan, can you see that? Yes. Fantastic. All right, so let me go ahead and dive in. So we're going to be talking about digital clinical measures today and specifically what's the current state of the field and how can we anticipate them evolving and sort of what hopefully will soon be a post COVID era. What's interesting is, if we look to the press, for example, we'll see that many of our sort of terrific scientific sort of journalist colleagues will say that health care is undergoing a monumental shift towards remote patient monitoring. During the last year, three of the five biggest companies in the world have announced new remote monitoring products. We've seen Amazon put their halo out. They launched last last summer. Google now owns the Fitbit Cent. They are actually have a clear technology. So in some ways they are a regulated device. Their ECG app is a clear product, although that is the only part of their sort of measurement suite that is cleared. And so many people who have developed a phenomenal amount of health measures measuring capabilities through their smartwatch. They also have a clearance specifically for pulse arcs monitoring. Let's think about how these tools have been used recently though, and I think this is really important for us to pause and think about. In a lot of use of technology and health care, but the digital clinical measures piece in particular, where is that in its development during the pandemic at its height, we saw telehealth increased by nearly 1200%. At the same time, of those telehealth visits, only 11% of those relied on remote patient monitoring. Now, there's a couple of things there. First of all, I actually don't think by any stretch of the imagination that every single telehealth encounter needs to be based on or reference sort of digital clinical measures data. We don't have to be reliant on remote patient monitoring for every telehealth visit. However, 11% does feel low. QDOS to our clinical colleagues, all of this care during the pandemic delivered by telehealth, they turned on a dime to be able to deliver that care. So there's definitely some aspects there that perhaps those tools and technologies weren't deployed and weren't available, but that 11% number does seem low. It shows that there's a lot of work to do to integrate these tools safely, effectively into clinical workflows. It's also worthwhile to look at some additional data we collected and published around how those digital data are being used when remote patient monitoring is happening. Really interestingly, when those 11, when we're sort of leveraging that data, we found that the vast majority of times, it wasn't that clinical care was actually taking advantage of the continuous nature of those data streams. Rather, they were being referenced and they were being used in exactly the same way that we would treat a stock data point captured in the clinic. What this tells us is on the healthcare side there is enormous potential, but we haven't yet harness the power of these technologies through a redesign of the clinical workflow flow through truly monitoring patients outside of the clinic during activities of daily living. We're tracking the monitoring to see if there's a spike or within patient change that should trigger some outreach that should trigger additional care. We're not there yet. It's certainly a powerful way that these tools will transform the way we think of healthcare, but it's important to realize where we are today. So that's over on the healthcare side of the show. Where are we in the sort of clinical research side where are we specifically in terms of medical product development. If we go back just a year and a half in October 2019, my organization dime launched a crowd source library of digital endpoints being used in industry sponsored trials of new medical products. And we were absolutely delighted to see that there were 12 sponsors who were sort of actively collecting these measures and that 34 unique endpoints were being used to answer clinical questions about new therapies. We were so excited. In fact, we went ahead. We published an op ed in stat news saying, you know, this is not the stuff of the future. This is here today. But when we actually looked just 18 months ahead, the growth has been absolutely astronomical. What we can see is we've gone from 12 to 62 sponsors relying on these digital clinical measures to make decisions about whether we take a drug to market. We can see that over 50. So over half of these measures are being used in critical phases of drug development. And we can also see that the growth in the number of measures themselves has gone from in from 34 up to 207. Absolutely extraordinary growth, not only in the number of measures, but also in the trust essentially by many, many more organizations to rely on the data from these digital clinical measures in their decision making in their product development pipeline in their R&D. If this data is interesting to you, this is all open source and you can access these data on the dime website. These slides will be available and we'll make sure you get all of the links. I do think it's worthwhile pausing for one second though. What's interesting as you mine the database is you'll see that every single one of those 207 measures is unique. Yes, while the growth is impressive. This does give me pause. We've got to think about measuring things that matter. I don't think that there are 207 unique measures of health that are critically important. We see over 50 unique measures of sleep, certainly sleep is a complex complex sort of domain of health. It's unlikely that there's just one measure that will account for it perfectly in all therapeutic areas. But the idea that there are 54 measures that warrant development, I have questions about the same as true for physical activity. I think we've got over 70 measures of physical activity in this library. And I think it's really important that we're focused, not only on our capability of measuring health, but actually making sure that we are measuring things that matter, and we'll return to that in just a little bit. And so, you know, Jess Lynn and I talk about this a little bit, but you know, the reason that we're excited about sort of the field of digital clinical measurement generally is we imagine because these measures because our understanding of health can be developed outside of the clinic because it can be developed during a patient's everyday life. We can imagine a future state of health care where health care is good because we keep people healthy and we keep them out of the doctors. Health care is no longer measured by how well are we able to dive in and add the assist once someone is sick once they need to come to the clinic or come to the hospital. And I wanted to give a couple of examples just to underscore this. So, first of all, in current state, if a patient needs a bone marrow transplant, in the two weeks preceding that visit, they have to come into the clinic, roughly five times, simply to have vitals measured to make sure that they are stable and healthy enough for surgery. There is absolutely nothing right now technically stopping us from measuring all of those vital signs with a high degree of certainty while that patient is at home without dragging them into a clinic while their immunosuppressed without asking their care partner to take more time off work and bring them into the clinic without sort of adding to the stress and the time and the fatigue that these continual clinic visits rely on. To me that's actually something that at some point it becomes unethical to choose not to use those tools to keep bringing that patient in for fee for service lab visits as opposed to simply monitoring their status at home as they prepare for that surgery. Another example is in Duchenne's muscular dystrophy for folks who might not be familiar with the natural history of Duchenne's. It's a pretty awful condition that affects primarily boys and young men. The natural history is diagnosis is typically around seven years of age. Unfortunately, these boys are usually in a wheelchair, struggling to sort of walk independently by the time that they're 13 and unfortunately life expectancy is in the early 20s. So how we currently measuring this, we use the six minute walk test that systematically excludes 60% of the population of those boys and young men from participating in any kind of research. That means that we aren't developing medical products that target those individuals and whether in research or in care we aren't thinking about the pieces of their condition that maybe matter a lot to these boys we're not thinking about whether they can text their friends play video games whether they can feed themselves whether they can do their homework whether they can take themselves to the toilet their dignity. We have the capabilities to measure that using things like smartwatches smart rings, and yet we still rely on the six minute walk test. And the last example that I wanted to share to really underscore why what Jesslyn and I are talking about today these digital clinical measures of health are so important. Parkinson's disease. Unfortunately, the number of people affected around the world by Parkinson's is growing rapidly in the US. Most people with Parkinson's are diagnosed incorrectly four or five times before they get that correct diagnosis and then once they are diagnosed due to the maldistribution of specialist neurologists and the patients who need them 40% of Parkinson's patients pass away before they ever seen a neurologist. Imagine the power of a suite of sort of digital clinical measures, measuring things like tremor freezing of gates measuring things like cognitive decline, measuring tenor and tone of voice. All of those data could be collected in the home of the patient and then we could use telehealth to connect those 40% of patients who never see a neurologist to a specialist who can more appropriately care for them. It would also allow us to develop a disease modifying treatment, which to date we've had no luck doing. That's why I think this is so important. That's why we have to focus on those measures that matter. We can harness all of this data right now those sort of continuously captured currently invisible data points using using these tools. But what we have to do is make sure that we are generating powerful meaningful signal and not simply adding to the noise. So to make sure we're doing that I'm going to go ahead and stop sharing and Jess Lynn, who's lab absolute experts in doing all of this work. She will dive into some of the tools techniques and science that they use to make sure that these measures are trustworthy. Thanks so much, Jen. This was a perfect T up to what what we're diving into next. So I'm just going to turn on full screen and hopefully you're seeing no notes and just slides. Perfect. Great. All right. So, Jen said it perfectly there is immense potential for these digital technologies that is currently going on tap for the most part. So in my lab, we're called the big ideas lab it actually stands for something which describes what it is that we do which is biomedical informatics date integrating data engineering and analytics. And with that we're trying to take these existing technologies and develop new infrastructure to capitalize on that data for early detection intervention and prevention of disease. I don't think at this point in time we really need a reminder about how important this is worth living in a global pandemic. And what's fascinating is that there have been many existing and ongoing efforts to try to harness and leverage our existing digital infrastructure to curb the spread of COVID-19. There are a lot of different areas that digital infrastructure can be useful for so I just give a few examples here on screen. And where, you know, our current standard of care or standard of exploring who might be infected is contact tracing. So getting on the phone and calling up people and finding out who they've been in contact with. There have been a lot of efforts to try to automate this process because we all carry smartphones those smartphones have the ability to know when they're near other smartphones or other technological systems and so this is one area where we've we've realized the power of the tools that we use passively in our everyday lives. There are many, many other areas and today I will give an example of how we can use wearables. And, you know, wearables can be useful also in a variety of different ways from the detection and monitoring of infectious disease or chronic disease, all the way to the interventions to deliver information at just the right time to the right person. Again, there are so many opportunities to apply this technology I'm going to focus here in my talk on early detection, but you can imagine how Jen mentioned the possibilities for at home monitoring for a variety of different disease types. We can also expand this to public health monitoring, because if we have a huge population already wearing devices or carrying devices. There's a ton of information that's being passively collected that could be leveraged to realize where there are hot spots of infection outbreaks before people are exhibiting symptoms. And so again, so many opportunities for this technology. So first I want to explain a couple of terms to dive into some vocabulary when we talk about digital biomarkers what we actually mean. So, first, I like to back up and just revisit this concept of a biomarker, because many folks who are working in this space, maybe you don't have sort of the traditional view of what we think of a biomarker when we think about a clinical use case. So biomarkers typically have diagnostic capabilities and can also help us to learn new things about physiology. So you might think about an existing clinical biomarker like a blood glucose measurement. And that allows us to know who may have diabetes or not, how bad that diabetes maybe. And if we give somebody a treatment like metformin we can track how well that treatment is controlling the diabetes status. We also through many new technologies allowing us to measure large quantities of biomolecules can discover new biomarkers that are associated with disease. And that allows us to learn something new about physiology. So for example, we might see that there is a gene that is associated with diabetes onset in young people. That is a biomarker that can then be used to predict who might later come down with diabetes. So we can extend this concept of a biomarker to a digital biomarker, which means that rather than taking a physical sample of blood or urine, we're actually taking a digital sample. So we're measuring something about physiology, for example, heart rate from a vital sign monitor that we can transform into an indicator of health. And what that looks like on a very pseudo mathematical level is that we have some outcomes, and we develop this function of these digital measures. And so these outcomes might be more of our traditional targets, like blood glucose, for example, or other measurements. It could be survey data. It could be illness events. And the input data that we have here can be anything that is collected digitally from a smartwatch, a smartphone, from digital online traces. You name it, we have a ton of passive sensors in our environments, and that information can tell us things about our health. So this isn't just a futuristic vision. There are a lot of digital biomarkers being developed in practice. And this slide is a little bit old. So it's just an example of what already exists today. But we and others have been developing digital biomarkers ranging from infection detection, all the way to chronic disease detection and monitoring. And you can see that a lot of these digital biomarkers have commonalities in the types of sensors or measurements that they use. Many surround heart rate and blood flow, many surround temperature measurements or movement measurements. And a lot of the machine learning algorithms that we apply to that data also have commonalities. So we use basic regression strategies we use random forests neural networks and we bring these common elements together to develop these new predictors of health and disease. So with that I want to pivot for a moment and introduce a little bit of vocabulary that will help us to kind of get through the rest of this talk. So we worked extensively with Jen and the team at Dime and a team of experts kind of from around the globe on developing a common language that we could use to describe these digital clinical technologies that are enabling us to capture these kinds of measurements. So, this idea of a biomet is essentially a connected digital medicine tool. And when I say connected I mean that it has some sort of wireless, or even a wired connection but it sends data from its own location to somewhere else. And it processes data that it's captured by a mobile sensor, and that there are algorithms that are on board that translate this, these sensor measurements into measures of behavioral or physiologic function. So this sounds like a very, very detailed definition and it is because when we try to think about these different technologies that exist in our world today. It's hard to know what we classify as a wearable or a mobile device it's hard to know what's regulated by the FDA versus categorized as a wellness technology. And so by introducing some of this very specific language, our hope is that we can start to develop standards surrounding these biomets, which I'll talk about in a second. But one of the things that I like to mention here is that a biomet is not just what we see on the outside right it's not just a Fitbit or an Apple Watch or a Garmin device that's sitting on someone's wrist, but there's a whole lot going on under the hood. And for this audience in particular, it's really important to recognize all of the data transformations that can take place on the device itself and then in the process of transferring data from that biomet to wherever that data is headed. And so there are a lot of considerations for what is going on with that data are there transformations that are taking place. Are we compressing data are we down sampling data. Are there security and privacy concerns at these onboard transformations or at transformations or processing to push data to somewhere else. So, there's a lot happening under the hood. And in order to make accurate digital biomarkers that can outlast the product lifecycle. It's very important for us to understand how each of these stages of data processing affect the final digital biomarker. So then that begs the question of how do we evaluate these biomets and digital biomarkers. So if we see that a smartwatch company is advertising their heart rate detection algorithm as clinically validated. What does that mean. So, one of the major challenges here is that there really is not a standard right now for for the terminology surrounding the evaluation of these, these biomets and these digital biomarkers. So with the time society, this is what we set out to develop. We, again, develop some of these more systematic terminologies like the data supply chain, which I was showing you on that previous slide shows us how data flows through the biomet. And then the these ideas of evaluation processes. So we have verification and validation that we've organized into a really nice framework that we refer to as the three that helps us to determine whether a technology or a biomet is fit for purpose. When we say fit for purpose we mean that the final experiment or implementation in the clinic fits the way that device or that biomet or that digital biomarker was evaluated. So, so what this v3 framework entails is essentially a verification process and analytic validation process and a clinical validation process. An example we like to give in this case is of an accelerometer, where we might have an accelerometer that we would place on a shake plate in the lab, and we expect that the RPM that that shake plate is set to is what the accelerometer itself would measure. Now, ultimately, our goal might be to use this accelerometer to distinguish between health and disease of a specific disease. In order to do that, we want to know what is the movement that we're trying to capture. So let's take MS, for example, and say that we want to use step count to capture who's likely to have MS and who's not. Well, we want to make sure then that this accelerometer can actually accurately capture a step. What we do is we have a gold standard evaluation process, like a person actually watching a video and marking steps, and we compare that to the accelerometer, and that's the analytic validation, it means that when we put the accelerometer on a human being, it measures what we expect it to measure. And finally, we have that level of clinical validation, which is, if we put this technology on a patient, are we able to distinguish between a disease state and a healthy state. For example, can this accelerometer distinguish between people who have MS and people who do not. So what's really tricky about this process is that we know that these technologies are not static. So there are firmware updates, there are hardware updates, there are new generations of devices coming out constantly. And so how can we ensure that this v3 framework is followed for all of these different iterations of these devices. And so really the take home message here is that steps need to be redone when any change is made to the biometric monitoring technology. And so if there are changes to hardware if there are changes to firmware which often occur over air and many people don't know about them. That changes the data. And so it's very, very important that that gets taken into consideration when any of these these biomets are being employed in practice so that we know that they're fit for purpose. So I want to switch gears for a moment and talk about an implementation of a digital biomarker so that we can see an example of how this would actually be useful. And so there was a lot of press around this sort of first example of how we might be able to use wearables to detect illness before a person themselves might know that they're sick. So I always give the spoiler alert before I actually explain the details of what we did and so this was work that I did alongside Mike Snyder and his team at Stanford, where we collected a lots of data from a cohort of people there were about 100 individuals wearing these smartwatches for between three and five years. This was an interesting study because a proportion of this population was pre diabetic. That's not what I'm going to be talking about today today I'm talking about infection detection, but that's initially why the study was launched. And so there were samples that were given from these folks in the lab and four times per year. And then we also had people come in to give samples when they had anything kind of anomalous happen in their lives. So viral infection stressful events dietary changes. And so what we can do with this is we can capture what it looks like on a biomolecular level to see health and to see disease. So this was the setup for this study where we developed these digital biomarkers of infection. So what I'm showing here is kind of just an overview of what that data looks like we have data from 43 people who are wearing this Intel basis watch, it's measuring three different things heart rate temperature and accelerometry, and you can see here that I'm showing in blue the skin temperature data and then in these red circles the heart rate data. These black squares here are showing the number of days monitored. And so what you can immediately see is that there's a big difference in the variation between the skin temperature measures and the heart rate measures. It makes a lot of sense because we know that temperature cannot vary that much and still sustain life, but heart rate we know can actually vary dramatically between different people and within a particular person. So this becomes challenging when we think about how we can detect anomalies within an individual. For example, if we have two people, let's take this person here, and this person here that have very similar resting heart rates. The measurement that's out here for this person would be very anomalous. It does not fit within their, their sort of confidence interval of their data, whereas for this person it would be very normal because it fits within their typical variation of their own body. So, when we're developing these anomaly detection algorithms it's very important that we have a personalized look at this. I'll show this even a step further if we look at some of the circadian changes that occur with heart rate which I'm showing here in pink or skin temperature which I'm showing here in blue that at nighttime we know when people are sleeping their heart rate tends to dip down their skin temperature tends to increase. But again, this is true to different extents for different people, and even for some people the opposite is true. It's really important that when we're trying to detect something that's abnormal in an individual, we know what normal looks like for that particular person. So all of this is leading up to how do we detect an infection in a person using just smartwatch data. So the method that we developed here is called the change of heart method. And the idea is that we have outlier periods of time that we want to try to detect from an individual baseline. So the algorithm that we developed here essentially goes based off of the individual mean and standard deviation. And you can imagine that we could have sliding windows where we take a sliding mean or a sliding standard deviation. And what we're looking for are measurements that fall way outside of what we need to be normal. What's tricky about this is with a smartwatch that measures one measurement per second every day, we get 86,400 measurements per sensor per day. And the likelihood of any one of those measurements falling well outside of the norm is pretty high. That's due to miswear that's due to all sorts of things that can happen. And so what we want to do instead is figure out the proportion of measurements that are falling outside of the norm. And that we're smoothing over these sort of outlying events that are not due to a change in physiology. And so that's what the size of these circles is showing essentially the size of the circle is showing how much the, the proportion of measurements is deviating from the baseline for that particular person on that day. So, employing this change of heart method we were able to show in this, this data set from this iPod study at Stanford that we were able to detect infection in all of the participants where infection occurred. The thing about this is that this was a sort of real life study where people were getting sick naturally and so we didn't have an intervention where people were actually infected, and we kind of had to rely on what naturally happens in the environment. But we were able to develop these, these outlier detection algorithms that did indicate when people had some sort of influenza like illness. And the way that this worked was that we have these individual baselines based on that we developed these digital biomarkers, and that leads us to this early identification of acute and chronic illness. So there's obviously a situation now where there are a lot of people getting sick. Unfortunately, we have a huge widespread respiratory infection. And this is enabling us to really see in real time, if we can detect these signatures of infection from smartwatches alone. And I'm showing here on the screen just a sampling of studies one of them is run by my group at Duke co identify. But there are many other studies that are going on in this space, trying to understand whether we can actually detect infection before people themselves know that they're sick. And the idea there is that if you can tell somebody they're sick, even if they're asymptomatic or pre symptomatic, you can prevent them from going out and spreading the virus. So that is the goal of this infection detection digital biomarker work. All right, so with the last couple minutes here, I want to talk about the goal of digital biomarker discovery some of the open source methods that we're working on in this space, and where all of this can take us. So, so with digital biomarker discovery really our goal is to extract useful information and knowledge from large volumes of bio met data in order to improve medical decision making or provide insights for health decisions. And some of the specific questions that my group is looking at our can we develop digital biomarkers to detect or predict improvements from sleep or exercise pre diabetes prior to symptoms, issues around infection how contagious a person is or how susceptible they are to contracting illness, which becomes very important if we think about sending out warfighters into field setting so so knowing different baseline physiology is really useful. So we are looking at circadian disruption and shift workers and risk of preterm birth and other women's health issues. So what's interesting about this if we think back to kind of the beginning slide that I mentioned where we are developing these different digital biomarkers. A lot of these are using very similar sensors, and very similar machine learning methods. There are some commonalities amongst these digital biomarkers that we can capitalize on to make the downstream process easier and more standardized. And for those of you who have worked in the bioinformatics from before potentially in genomics. This might feel reminiscent because a lot of this is akin to the early days of genomics where there, there were a lot of common methodologies being developed in genomics where there was a realization that groups could actually come together and do a better job. And so we've been working on this initiative called the DB DP, or the digital biomarker discovery pipeline, which is an online environment that aggregates open source software algorithms data sets, and is working to develop best practices for developing new digital biomarkers and evaluating them. So I'm just going to throw this out there that you can go to dbbp.org. And you can take a look this is actually an older snapshot but since since the snapshot we've added more modules and actually our hope is that the community continues to come together and contributes more and more modules to the dbbp. Right now this does require a level of coding experience because all of the digital biomarkers are written as code modules on GitHub, but we do have some really great documentation of resources for people who are new to wearables or in biomats or digital health and want to figure out where to get started. I highly recommend checking out the user guide and and some of our sort of getting started guides to understand where to begin and, and how to become hopefully a user and contributor. And one of the other plugs that I'll push here is, as Jen mentioned, and there's a huge need for work in privacy and security. And that is an area where we really have not done a lot and could really envision some beautiful contributions. So, finally, you know, looking to the future there's there are ever expanding options of measurement for biomets. We have smart textiles we have smart ink. So the ability to collect information about our physiology and behaviors is only going to increase as time goes on. So the hope is that this data gets used for these four purposes, and and nothing nefarious which is hopefully what everyone here is working to ensure, but leading to patient empowerment precision therapies, just in time interventions and improved access to care. So I'll pop this last slide on here just to say thanks to lots of folks who contributed to some of the work that I showed today. And I will turn it back over to Jen for a nice wrap up. I had to do it at some point. I was going to say that was terrific. And I'm lucky enough to sort of see your work up close often but to see it sort of presented and see the body of work you've done is absolutely phenomenal. So go ahead and, as you said round us out with a few thoughts. What you showed us just a little bit was that, you know, these technologies, the measurement capabilities using digital tools are not only extraordinary but they're actually largely ready for prime time the science is there. I, I think this place is a place is acid across roads right how now we have these new tools in the toolbox now we have these capabilities. How do we ensure that we actually use them to improve individuals care right do that sort of precision medicine and not violate individuals and privacy how do we make sure that we're taking a data driven approach to public health just like you showed us Jesse Lynn, and make sure that we're not just facilitating harmful surveillance how do we make sure that we use these tools that can extend by plates that can extend by capabilities that can connect clinicians and experts with patients wherever they are people during their daily lives, and make sure that we use that to improve access and improve health equity, and not just sort of exacerbate existing disparities along the digital divide. And I think there are a few issues that it's really important that we start to address. Some questions that we're asking ourselves here at dime that I want to share with folks is, you know, should we be thinking about broadband access as a social determinant of health. You can see here a snapshot from the Dartmouth Atlas of Health where you health, where you see red health is typically better, and where you see dark blue over here we can see that broadband access is much higher. These two areas correlate so if all we do is roll these tools out to folks where the technology infrastructure already exists, we're simply helping the privilege you rather than improving care for all. So the point that extends well beyond sort of just broadband access we have to think about other issues of health equity digital literacy and other social determinants of health. And there's a really terrific paper that was published through a collaboration between the Ohio State University and the National Digital Inclusion Alliance, but I would highly encourage everyone to take a look at the link is here and it's absolutely terrific. So like here at dime and specifically through our collaborative community data CC, we're exploring the idea of a new term. We're thinking more less about sort of health disparities, it seems like a statement of fact, and trying to take us towards a more action oriented Let's think more about digital inclusion. Let's think, you know, and how that applies in digital health measurement. We would say that it refers to activities and practice throughout product development deployment evaluation and commercialization that are necessary, not optional but necessary to respond to the needs of all populations and to enable all individuals communities to benefit from these digital health measures. So go back to the power of these technologies that Jaisalyn shared and think about how we can sort of deploy them for everyone. Data rights are increasingly important we have to build trust in the health data ecosystem and there's another really terrific paper here around privacy protections that encourage the use of health relevant digital data. And I think that's the other piece to we are awash with data right now, not only are we making sure that these data are being used to help not harm us, how are we making sure that we're not just adding to the noise. There's a concept and actually we spoke a little bit about this last year with my colleague Dean Mendelsohn, you know, sort of this idea of redefining patient safety and the digital era. And this is something we've continued to explore here at Dine. The definition of patient safety I actually think is excellent, but there are new kinds of harm, there are new kinds of risks. And the definition of safety is contingent on sort of current knowledge of resources available and the context in which cares delivered. Current knowledge is limited in this interdisciplinary field clinicians knowledge of cybersecurity, for example, is limited. We're not collaborating well enough yet, there isn't sufficient knowledge sharing we need to do much better at knowledge exchange in forums like this. We need to think about all of the different resources available. Jaisalyn mentioned some we produce a lot at Dine, but there's a lot of silos of resources unfortunately. We need to think about new contexts of care. Care isn't always delivered in the clinic anymore and that means the data isn't always confined to a locked filing cabinet in the clinic either. There are all sorts of harms that now exist and we want to make sure that we read all of the benefits of digital clinical measures and limit those harms as far as possible. There are other challenges around the sort of lack of standardization the challenges around integrating these data into the traditional clinical data sets whether that's an EDC, an electronic data capture system on the research side or whether that's the electronic health record on the care side. And perhaps one of the most important things and something we focus on a lot at Dine is, we have to build the digital workforce necessary to actually support the field. We have to think about this sort of big tent thinking to make sure we have all of the technical skills necessary alongside the traditional clinical competencies and importantly be making sure that we're now attentive to the patients we're all here to serve. This emerging sort of group of powerful citizen scientists and ethicists are pretty critically important in all of this work. Now, how we're actually training the next generation intentionally we can't hope is not a strategy we really need to start upstream with the future workforce and we need to do some reskelling retooling of the current workforce. And perhaps, you know, most importantly as we think about making sure we're ameliorating as far as possible health disparities, we have to have a commitment to diversity in this workforce that goes beyond just lip service. The clinical side of the shop on the clinical side of the shop these workforces are not terribly representative of the patients we serve right now. We have to make sure as we combine them we improve that not exacerbate. There's a sort of lack of representation in the experts who are working to develop digital health as a field. I'll look a little bit in the last piece of data I'll present was from an enormous collaboration that Jess Lynn and the lab were part of and that we were lucky enough to host the digital clinical measures playbook. And really this is the essential guide for a sort of the development and successful development and deployment of digital clinical measures across public health, across clinical care and across research. And what we sort of what we came up with was several hundred slide playbook that gets into every facet of thinking about these measures from the sort of sort of the v3 process that just Jess Lynn mentioned, all the way through to the ethical considerations, and a variety of different operational and technical issues in between. We really dive into this. And also 110, I believe action oriented tools from educational curricula all the way through to checklists and tool kits that I encourage you to take a look at. And we also did some work really thinking about the key archetypes the key experts who need to be involved in this process for it to be successful. I recognize that our academic colleagues were particularly important cohort in advancing the field of digital clinical measures and making sure that not only as we develop them but we deploy them. We can be confident that they are trustworthy. And so, really, there's some fascinating data though I want to share in closing this is data extracted from a preprint currently in a gym at the Journal of Medical Internet Research. And what you can see is the kind of work that Jess Lynn discussed around verification analytical validation clinical validation that sort of that piece of the text that they're doing great work there. Currently we're doing the work to make sure that we're measuring things that matter. We're starting to get sophisticated around how we operationally sort of deploy these. But there are some frankly alarming gaps when we look at the last two years of published literature by academic researchers, as we can tie security research specifically to digital clinical measures there was one study. There was only one published study really diving into issues of data rights and governance. There was not a single paper examining the ethics of advancing these digital clinical measures. We also saw a death of work trying to establish standards. There was limited work ensuring that these things economically feasible. And when we think about the operations of deploying this in care vis a vis research. We saw that that was a lagging sort of indicator to and if we think back to how these measures are being used in care right now without reorganizing the workflow that shouldn't be terribly surprising. And the last slide before we close out and I think that this gives us a real call to action here is how are these studies that are published how is this research in the field of digital clinical measures being funded. And I think that there is robust government funding, most of it is going into operations of research, but we can see that government funding is supporting these areas of v3 and identifying measures that matter. However, they are not supporting them in quite the same way as we see other areas of sort of medical research, we're relying heavily on independent foundations, and also on the hard work of academic colleagues and labs like Jaisal and to find the resources that are needed from investigator funds to do this work. We can see that 100% of the security researchers government funded but when that's one study that's not enough. It's critically important that we see funding agencies start to identify how important it is that we move to plug this gap quickly, we need to start having requests for applications for grant funding that address security that address data rights and governance and address ethics and until we get to that point, the promise of this field to deliver on everything it can bring to improve health and health equity is going to be limited by these risks. So with that, that's why Jaisal and I have been so excited to come and join you today this field is absolutely mission critical to take all of this to scale. Yeah, I'll stop sharing. I'm about to get the sun right in my eyes. I can see it setting it's I'm about to go blind so the timing is perfect. Thank you so much for having us Jesslyn would you like to close out with any final remarks. Oh, this was great. Thank you for this community and hopefully this inspired some some interest in this work because there's a lot that's needed. Thanks. Thank you.