 So our next speaker this afternoon is Piers Keen from the University College London and Moorfields Eye Hospital by the NHS Foundation Trust. If you are a reader of nature medicine, then over the last few years may have noticed a number of very interesting papers on machine learning in oftalmology. And Piers is a driving force behind this work. So there's a big collaboration between Moorfields and Google DeepMind with exciting results. And I'm sure we'll hear more about these in a minute. So Piers is a medical doctor by training. He's now an associate professor at the Institute of oftalmology at the University College London and holds an appointment with the Moorfields Eye Hospital. He has won a number of awards in 2015, a clinician scientist award from the National Institute of Health Research and this year, a UKRI Future Leaders Fellowship. So we are very excited to welcome him here and to learn more about his research. Thank you for joining us, Piers. Thank you. Thank you, Karsten. It's a real pleasure to speak to you. And I'm gonna talk to you, I'm gonna talk to you about what I believe is the potential for machine learning to transform oftalmology. I think it's fair to say that there's a lot of hype around artificial intelligence and machine learning for healthcare. But even despite all the hype, I think there is potential to do something quite exciting if we approach things in the right way. And one of the things that I'd like to try to make a case for over the next hour is that oftalmology is one of the medical specialties that's at the forefront of these transformation. And I think that there's a lot of our experiences we can share with other medical specialties about not just the development of machine learning tools, but also the validation, the implementation and the adoption of these systems. So Karsten, we've already summarized my background, but I have a dual appointment. So I'm a consultant oftalmologist. So like a senior oftalmologist at Moorfields Eye Hospital in London. And Moorfields Eye Hospital is the oldest eye hospital in the world. And one of the largest eye hospitals in the world. At Moorfields, I specialize in the treatment of retinal diseases. So conditions such as age-related macular degeneration, which is the commonest cause of blindness in Europe. And diseases such as diabetic retinopathy, which is the commonest cause of blindness in working age populations in many countries in Europe and around the world. As you said, I'm also an associate professor at University College London at the Institute of Ophthalmology. And I lead a clinical research group at UCL, which is focused on AI-enabled healthcare. So we have a lot of work that we do with industry. So in particular with DeepMind and with Google, but we also have a lot of work that's separate from industry that's purely in an academic setting. And I'll tell you a little bit about both of those things during this meeting. And then the other thing is that I'm very privileged to be funded by UK research and innovation as a future leader. And this is a relatively new funding scheme and funding organization. And so it gives me the rare opportunity in that I'm a medical doctor, I'm a clinician, but essentially I have 80% of my time protected for research. And I think that that's the thing that has really facilitated my ability to work with machine learning collaborators because it would be very hard. And I'll bet some of the people listening would have experiences working with clinicians where it can be challenging in particular when the clinicians are working 100% of their time in clinics and surgeries and other things. The other thing that I would say is that I have acted as a consultant for DeepMind. I'm an act as a consultant for pharmaceutical companies such as Roche and Genentech and I'm involved in some machine learning work with them as well. But I think given that I'm gonna be talking a lot about the collaboration between Morfields and DeepMind and Google, it's important that I volunteer that financial disclosure at the start. Now, in terms of an overview of what I'm gonna discuss over the next 45 minutes to an hour, probably more than 50%, maybe two thirds of the talk will be about the Morfields DeepMind collaboration and some of the research outputs of that collaboration. Karsten, you mentioned some of the articles in Nature Medicine and I'm gonna talk about some of the results there. But then in the last third of the talk, I'm gonna focus more on work that has come out of that collaboration but is independent from DeepMind and from Google. And so in effect over the last five years, we have gone from at Morfields, we have gone from knowing nothing about clinical AI to having a lot of expertise and infrastructure to do clinical AI projects. And one of these is something that we call the Insight Health Data Research Hub. And I'll explain what that means. Another thing that I'm particularly excited about is called the ELTS Eye Study. And this is the idea of using the eye as the window to the brain, sorry, to the rest of the body. And then lastly, I'm gonna speculate a little bit about an emerging ecosystem for clinical AI. And in particular, I'm gonna give some of my thoughts about the potential for automated deep learning and automated machine learning. And I think that I'll be very interested to see, to hear what people's comments and criticisms are on that, particularly given that this is an audience with machine learning experts. And so I may try and say some kind of controversial things to just provoke a little bit of discussion, but I'm very happy to be pushed back against in some of those comments. So first of all, let me just set the scene. What is the problem that we face in ophthalmology? Why do we need new technologies and innovation and why do we need AI? Or maybe this is just hype and AI is just another way for us to publish some papers and get some grants and things like that. Do we really need it? And so this figure is something that I love showing to clinical audiences, particularly when I'm speaking to doctors from other medical specialties. So in 2017, ophthalmology overtook orthopedics as the number one busiest of all the medical specialties in terms of clinic appointments in the National Health Service in the UK. In fact, nearly 10% of all clinic appointments in the NHS are for ophthalmology. That constitutes nearly 10 million appointments per year, and that's a number that's going like this. If you plot orthopedics over the last five years, it's like this. If you plot ophthalmology, it's like this. So in some senses, and maybe this is not an appropriate metaphor, but in some senses, we're drowning in the number of patients that we need to see. And if you go to an eye clinic, whether it's in London or Vienna or Boston or Beverly Hills, it's going to be very, very busy. I can guarantee you that. And it oftentimes standing room only in the waiting rooms. Now, the upshot of that is that there are some patients who are potentially losing sight and even potentially going blind because they cannot be seen and treated quickly enough by human experts. For example, a retina specialist like me. Now, this was particularly illustrated in January of 2020. There was a lot of media attention in the UK because there was a very tragic case towards the end of 2019 that I appreciate it's sort of anecdotal evidence, but nonetheless, it highlighted the challenges in the system that we face. And so this was a case of a woman in her 30s. I think she was 36 years old in the south of England. And this woman was pregnant and she also had advanced glaucoma. Glaucoma is an eye disease where you have high pressure in the fluid of your eyes that damages your optic nerve. And it's a common cause of blindness. Now, what happened with this woman is that she needed to have urgent glaucoma surgery performed. And for whatever reason and for whatever chain of events, there was delays in her getting the surgery and she lost her sight completely. She went blind. Now, I cannot think of a more tragic scenario than a young mother who has lost her sight and perhaps it could have been prevented if the intervention had been done in a timely fashion. And so I believe, as I say, and it's not perhaps not a surprise, I believe that the latest advances in machine learning can at least begin to address some of these problems that we face. So essentially, this is the reason why I initiated this collaboration between Moorfields Eye Hospital and the artificial intelligence company DeepMind. Now, I should highlight to people in this audience, particularly if you're asking any very detailed technical questions later, is that I'm coming at this from the background of a clinician. I don't have any formal machine learning or data science or computer science training or anything like that. And so I contacted DeepMind just because I knew that there were people going blind because they couldn't be seen and treated quickly enough. I knew that there was massive advances taking place in the machine learning community over the last five, seven, 10 years or so, in particular with regard to deep learning. And I knew that we had huge data sets of atomic images and I'll tell you a little bit more about those. And so I was looking around for collaborators who had deep learning expertise. And so initially I was looking for some academic collaborators back in sort of 2014, 2015. And I didn't really find anybody that was particularly interested at that time, although I have subsequently. And so actually what happened was I was reading, I was reading a profile about DeepMind in Wired Magazine in July of 2015. And I had known about DeepMind because I've seen the media attention when they had been acquired by Google. But what really resonated with me was that two of the three founders of DeepMind are alumni of UCL, and which is of course the university affiliated with Moorfields and two of the three of the founders are from London. And of course the other thing was that their headquarters are based in the King's Cross area of London. And for those of you who know London, that's only two stops away from Moorfields on the Northern line on the London Underground. So it's like a very, it's a very short distance between the hospitals. And so I approached them and Moorfields then sort signed a formal research collaboration agreement in July of 2016 with them. And essentially what we wanted to do is develop a deep learning system that could look at these eye scans, these scans of the neural tissue at the back of the eye, which is called the retina. And these eye scans are called OCT scans or optical coherence tomography scans. And OCT is kind of like an ultrasound but it measures light waves instead of sound waves. And in doing so it gives very high resolution three dimensional images of tissue. So the axial resolution of an OCT scan is approximately five micrometers. So if you think about it, that's an order of magnitude higher resolution than a CT scan or an MRI scan. And OCT has become the dominant image in modality and ophthalmology. In fact, we do more OCT scans than all other ophthalmic imaging combined. And then we do more than a thousand OCT scans per day at Moorfields. So it seemed to me that this was a nice overlap between a pressing patient need and a situation where we had large amounts of data that could be amenable to machine learning. So the research collaboration agreement actually, I mean it was signed in the middle of 2016 but it had actually begun when I had read the article in Wired in July, 2015. And in the article, the part of it described, one of the co-founders of DeepMinded, Mustafa Silliman, who's leading their applied division. And he said, I'm interested in applying AI to healthcare, to climate change, to energy consumption, making the world a better place. And so he was the person that I contacted to initiate this collaboration. And the exciting thing for me was that, just a few days after I sent him a message, I found myself sitting in his office and telling him about the potential of this collaboration. But I remember going in and saying, we've got so many patients, we're doing thousands of scans per day or per week. People are going blind. And he said to me, well, how many OCT scans do you have in total? And I didn't really have an answer to that. And he said, what are the different systems? What's the quality of the scans? What proportion of the scans do you have labels for? You say that diabetes uses a common cause of blindness, but how many of the scans are from diabetic patients? What are the file formats? Are they proprietary file formats? Are they open source file formats? All of these questions. And to be honest, if I look back now, I feel embarrassed because I was so clueless. I didn't really have any answers to the questions that he was asking me. And he asked me equally important, he asked, well, what are the ethical and the governance requirements? Because this would involve sharing these scans with the company that's owned by Google from the NHS. And again, this was a collaboration. I just knew that there was this urgent clinical need and this technology could address it, but I didn't know all that would go into such a project. And so actually it took a lot of teamwork behind the scenes at Morefields to begin to figure out at least some of the answers to those questions. And an interesting thing that I would say actually that I've observed over the last five years is that deep mind and Google are very commonly approached by clinicians who are in a similar situation to me and they know that they've got some big data set, they know that they've got a pressing clinical problem, but in the meeting with them, you know, deep mind or Google or sometimes me would say to them, okay, well, can you answer some specific questions about the data set? And they usually can't. And they say, at the end of the meeting, they say, okay, we'll find out the answers to those questions and we'll get back to you in about a week's time or two weeks time with all the answers and then we'll proceed. Nine times out of 10, you never hear from them again because they find it very challenging to get the answers to those questions. So when we, I think we were good because I think we were kind of tenacious and when we did answer all those questions we could put in place a data sharing agreement and we ended up sharing back in 2016 about 1.2 million anonymized historical OCT scans with deep mind and that began this collaboration. Now, the thing is that, you know, this is a sensitive and appropriately sensitive area because this is, like I said already, this is sharing NHS data in the UK with the company that's owned by Google and a multinational tech company. And so from the start, we wanted to be as transparent about what we were doing as possible and to make sure that we did everything right or do our very best to be cautious and careful and respectful of the use of people's data. So from the start, if we put a section of the Moorfields website, which is dedicated to the collaboration and it's still there now and you can go and have a look at that and there's videos and there's FAQs and there's things like, you know, if you want to opt out of this research, here's the email address of our information governance officer, giving as much information and being as transparent as we can. If you go to the waiting areas in Moorfields, you can see that we've got sort of flat screen TVs to try and see people happy while they're waiting, while they have long waits for their appointments sometimes. And one of the things we do is we have screen savers that come up intermittently and one of the screen savers is about the collaboration and it tells people what we're doing and it points to people to where they can learn more about the work that we're doing. The other thing that we did was that before we had done any sort of experiments whatsoever, we published our protocol in open source machine learning or open source peer reviewed publication and then we could finally sort of get to the science and it was very exciting for us then in August, 2018, Karsten, as you've mentioned already to publish our work in the Journal of Nature Medicine. You can see that there's more than 30 different co-authors here and this reflects the fact that we had experts from UCL, experts from more fields and experts from DeepMind and the key point is that this wasn't just giving them the data and then coming back to later with a publication, this was very much a research collaboration from the very start and I don't think it would have been possible if any of the three groups were not contributing. Now, it was also very exciting for us because it was published on the cover of Nature Medicine and of course this is AI so there's a lot of hype and it was very exciting that it got global media attention. Now, you know, it's thrilling, I think for me it's a thrill to see your work getting widespread attention. It's also a little bit awkward because the fact of the matter is that we have not saved the sight of millions of people yet and in fact, what we've published here in Nature Medicine is a proof of concept that a machine learning algorithm can do this and in fact, if you were to email me after this talk and say, can I come to more fields and see it in action, you wouldn't be able to because it's not in action yet. We're not using it in the real world yet and what I've come to learn in the last few years is that going from the research publication or the conference presentation to implementation in real life is really, really hard and in some ways, doing the first step is kind of the easy step and before this, I always presumed that the hard part was publishing the paper and that's not the case. So, in any event, what we did was we created you know, a deep learning framework that consists of two neural networks. The first neural network was a segmentation network. We'd feed in the raw OCT volume and we trained it with about 900 manually segmented OCT volumes and that allowed it to create this intermediate representation that could be fed into a classification network that we trained with about 15,000 labels and ultimately would give a classification output such as urgent referral, semi-urgent referral or a diagnose up to 10 different retinal diseases. So anything that you would expect an ophthalmologist to be able to look at and see on an OCT scan. Now, we created the tissue segmentation map as an intermediate representation for a variety of reasons, but one of the nice byproducts of that is that it gives some clinical interpretability to the model and also it gives a range of kind of interesting quantitative metrics for clinicians when they're treating these patients. So this is just an example of the video of the model running. And so at the bottom row, you can see an OCT volume and the bottom right, you can see the figure legend of about probably about 10 or 15 different anatomic parameters that are segmented. This is a patient with diabetes. This patient in his 40s got poorly controlled diabetes and you don't have to be an ophthalmologist to see that the retina is very swollen to waterlogged. This is called retinal edema. And in the top row, you can see referral suggestion is semi-urgent referral and diagnosis probability macular retinal edema 98.5%. Now, we wanted to evaluate the performance of the model. So we got 1,000 new patients that had presented to Morfields that had had a macular OCT scan done at presentation. And let me tell you, this was a retrospective evaluation. So this wasn't perspective clinical trial and that's an important distinction in terms of validation of these systems. And essentially, we ran the algorithm on those 1,000 cases and we looked as a primary outcome at the errors on the referral decision. And what we found was here that it had about a 5.5% error rate on the referral decision. This is the triage decision because remember our use case is not replacing the retina specialist. Our use case is targeting those people with cytraine disease to get them in front of a retinal specialist who can actually make the diagnosis and get the treatment. Now, we want to benchmark the performance against human experts. So we got eight human experts and experts one to four are senior retinal specialists at Morfields and experts five to eight are optometrists at Morfields who have a little bit less experience in looking at these scans. We locked them in a room and said, look at all the 1,000 scans and give your triage decision, your referral decision and give your diagnosis and all of those things. And we looked at their errors and the gold standard that we took was when the patients had seen a retinal specialist and they had not just had an OCT scan but they had, you know, fluorescein angiography and a range of other tests and then they were treated and followed up for a six month time period. And that was our reference standard. Now, how did the human experts do? Well, what we saw was that the algorithm did better than all eight human experts except for experts one and two. When it was better but it wasn't statistically significantly better. Now, one thing that I've learned over the last five years is that whenever you see the hyper and AI and you see superhuman performance or a headline like beats the best doctors, when you dig into the details, you often find that the doctors are not doing a task that they do in their day to day life or they're doing a task with one arm tied behind their back and it's not a fair comparison with the real world. And indeed, this is not a fair comparison because in the real world, the human experts would never make a triage decision based on an OCT scan alone. They would know the visual acuity of the patient. They would probably have a retinal photograph and they might have, they would likely have some history, some details of the patient. Now, to make it fairer and to simulate that, we repeated the experiment with the human experts and we gave them all of that additional information and locked them in a room again and asked them to reanalyze the stands but in a different order and after a time period. And it wasn't a surprise to us then that the human experts, that improved the performance of all the human experts and it got to the point where expert number one was able to equal the algorithm using the OCT scan alone. And let me tell you, experts number one and two are world famous ophthalmologists who have more than 20 years of experience each. Now, interestingly, the algorithm did as well as it did with just the OCT alone but adding that additional information didn't really improve the performance of the algorithm significantly. So what did it get wrong? Well, I'm not gonna go into the confusion matrices here but what the BBC picked up on was that AI did not miss a single urgent case. Now, from my perspective, what was interesting is that when I looked at the expert number one, the human expert number one and saw that they had a 5.5% error rate, well, what did they get wrong considering that they are one of the world's leading experts? Well, actually the cases that they got wrong were ambiguous cases. And an actual fact, when we looked post hoc, maybe our human experts label was correct and our reference standard was incorrect with the benefit of hindsight. But of course, we can't change it after the fact. Now, interestingly, when we looked at the errors that the algorithm made, that was also the case. So a lot of its errors were not really errors when we looked at it after the fact. Now, where are we going with this? Well, at the moment, we're trying to bridge what I call the AI chasm. And this is a term that I read, I read an article in TechCrunch a couple of years ago and I was talking about this in the tech world. And this is the idea that if you're a startup, it's sort of easy potentially to do a cool demo of a machine learning system, but it's a very different thing to have a scalable deployment and meaningful human AI interactions at a large scale. And if it's hard in the tech world, I think it's extremely hard in healthcare to bridge that chasm. And so that's what we've been trying to do since we published the paper in 2018. So what are the different ways you need to do that? Well, one of the things is working on the technical maturity of the algorithm. And as a non machine learning expert, I think it's fair to say that most doctors don't realize that the piece of experimental code that is reported in a research paper is not really something that can be deployed at scale in a piece of software around the world. And so the team at Google have been working to essentially rewrite the algorithm so that it can run with a fraction of the computing power in a fraction of the time and make it into a cloud-based API. And the cool thing was that we did a live demo of that in Wired Health in March 2019. And we actually got a patient with AMD, we got their scan and live on stage, we sent it to the cloud, we ran the algorithm on it, we logged in with a different laptop and showed the algorithm giving an output. And so that was exciting but also a little bit risky because it would have been quite embarrassing if we had screwed that up. The other thing that I think it's important to emphasize is the need for better clinical validation. And this is a paper that came out in the Lancet Digital Health that I'm a co-author on in September, 2019. And it was a systematic review and meta-analysis of essentially deep learning papers in medical imaging. And really the take-home message is that the quality of those papers is very, very poor in terms of clinical validation standards. And only a tiny percentage of papers are of a good clinical standard. And so we are very aware of this. And so we've been taking, I think, a very measured approach to validating our algorithm. And so one of the things we've been doing now is trying to validate it in OCT scans taken from eye clinics all around the world. So we want to make sure that an algorithm that we've developed in London works as well in Ghana or Brazil or the US or Asia or any other countries around the world. And it turns out that there's lots of complexities to that. Now, the other thing with implementation is that it's not enough just to have an AI system or a machine learning system in isolation. It has to be able to fit into pathways. There has to be billing within the healthcare system to be able to support the machine learning system. And so Moorfield is where I'm a consultant is building a new hospital in King's Cross in London beginning in 2021. And we're trying to anticipate now what flexibility we'll need in our clinical pathways so we can take advantage of these breakthroughs in machine learning technology as they come through. The other thing is that we're increasingly seeing OCT being used in the community and community optometry settings. And so we're trying to think about how we can link the community optometry clinics with the hospital system and then put a machine learning system on top of that. And of course that's the considerable challenges associated with that. And then there are in fact OCT devices which are being built for home use and a number of companies are in advanced stage in developing those. So ultimately we see a chance with this technology to bring world-class expertise into the community and into people's homes. Now, just to say that was the main thing I want to talk about. I'm gonna tell you a little bit more briefly about the follow-up study from the Moorfields DeepMind Collaboration. And then lastly, I'll tell you much more quickly about some of the other studies that we're doing. But just to say, we're really proud that our work has been patient centered from the start. This is a patient of mine called Elaine Manna at Moorfields. And Elaine lost her sight from wet macular degeneration more than 10 years ago before there was any good treatments. And in 2012, she started to lose her sight in her good eye and she got an urgent appointment to see an ophthalmologist but the appointment was like six weeks later. And imagine if you were a situation where you're at home and you're losing sight in your good eye and you're told that there's a treatment now available but you need to wait six weeks for it. Well, if that was my mother, I would want her to be seen and treated within six days and not within six weeks. And so that was the motivation for the first collaboration I've just told you about but it also got us thinking about something else which is this idea that patients with eye diseases are really worried about their good eye being affected. And so they may be coping when they've lost sight in one eye but if the good eye is affected that's when they really get stressed about it. And so we began to think could we use machine learning to predict ahead of time the involvement of the good eye relying on the fact that if patients are being treated for their first eye, they typically would have scans done on both eyes for a period of time. So essentially it means that we have a situation where we have lots of normal data or data before a catastrophic ocular event happens that we can use to do a prediction task with deep learning. So if we could do that, then maybe we can protect the good eye with kind of novel treatments and maybe that means we stop the event from happening or we can mitigate its effects once it does happen. So in May, 2020, we published a second paper in Nature Medicine and just to highlight that the joint first author on this paper is a PhD student of mine who's at UCL is an optometrist called Rena Chopra and really was the brains behind a lot of this paper with the amazing team at DeepMind and Google and UCL and others. So what we did with this was this idea, the setup was that you have patients with wet AMD, exudative AMD in their first eye, they're receiving injections into this eye and having regular follow-up visits and at the same time, they're having scans done in their first eye. Now, we created essentially a prediction model that was an ensemble with number five here which is an end-to-end prediction model based just on the raw OCT scans plus also in number two, three and four here, a prediction model that leverages the segmentation outputs from the system. Put them together in an ensemble and then essentially the output of the model was predicting what we called imminent conversion which was development of wet AMD within six months, the site threat and form of the disease. Now, I'm gonna go through this quickly but just to say that in terms of our data set, it was big by AMD terms but maybe small by machine learning terms. It was about 96,000 OCT volume scans and in our test set we had about 5,000 scans and we wanted to predict imminent conversion. Now, the main figure from the paper really is this one, the main and essentially we've got the blue curve and we've got the orange curve and the orange curve looks better than the blue curve but actually the blue curve is the one that's clinically meaningful. That's actually predicting the conversion point and for that we had an AUC of 0.745 and of course that's not spectacular when you compare it to the sort of 0.99 AUCs that you see in a lot of papers but the fact is that the AUC is not clinically meaningful. Really what you have to do as you all know is select operating points on that ROC curve to determine your sensitivities and specificities and those are the clinically meaningful metrics and so we chose what we call a liberal operating point and then more importantly a conservative operating point. So our intended use case for this was that if we could predict imminent conversion then maybe these patients would be able to get a treatment such as an injection into their eye to prevent conversion. Now for that use case we wanted to optimize for specificity because we didn't want to have, we didn't want to be injecting people false positives essentially we don't want to be giving them an invasive treatment if they're not going to converge. So in that scenario we had a with a 90% specificity we had about a 35% sensitivity. So we're not so much worried about the false negatives because we're monitoring those patients pretty regularly anyhow. Now the other thing was that we wanted to benchmark it against human performance and this was actually not a task that human experts had ever done before and they did better than we expected but the fact is that they were very variable as you would expect compared to a machine learning algorithm. Just to highlight the subgroup analysis and this is mainly for an off-talmic audience but the key point is that of the 5,000 OCT scans in the test set the only about 4% of them actually developed wet AMD within that six month time period which sort of highlights the challenge of the task that we set ourselves. I'll just skip ahead through some of those. Even this is just to show that even in the highest risk group it was only 6.7% developed imminent conversion. So where I think this is going is the reason that I'm excited about this is because in 2021 we would like to use we would like to begin a clinical trial which uses this algorithm or some improved version of this algorithm to identify high risk patients for fellow I conversion to recruit them into a trial. Now, I think that that's going to be a very powerful use case for machine learning because it mitigates against a lot of the issues with deep learning in terms of its brittleness, its fragility, et cetera when it encounters data that is different from what it's been trained on. So in a clinical trial setting we can control what type of OCT device we can control the quality of the images and we have a lot of other controls that sort of allow us to maximize the likelihood of getting a meaningful answer from the system. And so I think that's one of the first use cases that we're going to see in the real world in the near future. Now, I know that I've talked a little bit too long some of the things and we've got about 15 minutes left but to include some questions but just in the next five to seven minutes I just want to tell you about some of our other spin-off projects and what building from what we've learned from all of this work that we've done with DeepMind in Google. So the first thing is at more fields we've come to realize in a granular form exactly how much imaging data that we're generating. And so just from one or two different vendors we're generating about 1.2 million of pelvic images per year at more fields. And we've greatly expanded our research database now and we've also moved it to the cloud. So we have a more fields research cloud and essentially we've got a BigQuery database and we've got a lot of these scans in the cloud. And one of the exciting things that we're doing then is that we've got funding at the end of 2019 from Health Data Research UK to share all that we've learned about the aggregation and curation of aftalmic data with other NHS trusts. And so the idea would be to try and create a national bio resource for aftalmic data. And then this bio resource would be available to both industry and academia all around the world to get access to this data to hopefully lead to patient benefit in the future. So that's something maybe I'd love to talk about at some point in the future. The other thing that I'd like to talk about our learnings is we have learned a lot about how to approach the ethics and the governance of using clinical data. That has allowed us to do a spin-off project which involves using the eye as a window to the rest of the body. So this is of course an old idea, but since about 2018, we've come to learn that using deep learning you can potentially see a lot more things looking at retinal photographs. So with this image we can say, this is a woman just from deep learning and no retinal specialist or no ophthalmologist can do that very well. We can say she's 58 years old. She's not a smoker. She's not diabetic. Her body mass index is around 25 and her blood pressure is 150 over 85 approximately. Now, this is kind of mind-blowing for ophthalmologists that this is possible. Now, if you're skeptical, you may say, well, what's the point? It's easier ways to tell if it's a man or a woman or the age of a patient than applying machine learning to a retinal photograph or to measure someone's blood pressure. You just put a blood pressure cuff on or you measure it. But I think what we've come to learn is that if we can get the appropriate data sets and apply the appropriate expertise to them, maybe we can uncover some interesting stuff. And so one of the things we've done using our experience now around governance is we've linked about 3 million OCT scans plus 3 million paired retinal photographs with a national database in the UK called Hospital Episodes Statistics. And it took us about two and a half years to get the permissions for this. And essentially that means that we know every patient who's had a retinal scan at Morefields that has gone on to develop a heart attack or has gone on to develop any kind of dementia or a range of other diseases. And in fact, the numbers we have are two to three orders of magnitude bigger than a lot of other cohort studies like UK Biobank study, for example. Now, if you're interested in learning a little bit more about this, this was featured in The Economist in December 2019. And interestingly, this was featured at a point where we didn't have, and we still don't have actually any breakthrough results from this work, because we've just started it. But the emphasis in the article in The Economist, and this is a tweet from the person who wrote it, how Hodson was our attention to detail around the use of health data and the governance of health data. And this is what has allowed us to get to this point. And then lastly, just in my last two or three minutes, I just want to say some interesting, maybe stimulate some conversation, which is that I feel that there could be an emerging ecosystem for clinical AI. And so I think that much of our work has been with industry for the last five years. In the last two or three years, we're developing really good links with academia. But I'm really interested in actual healthcare professionals beginning to explore machine learning, and particularly those without specialized expertise. And so this is inspired by an article I read in The New York Times in 2017 about automated deep learning platforms that are now available from many companies. And these are essentially drag and drop interfaces for the most part that allow people without coding experience to develop a deep learning classifier. If you've got a thousand pictures of cats and a thousand pictures of dogs, you can upload them and you can develop a model to be able to distinguish between the two. So we got excited about this and we got some five publicly available medical image data sets, skin lesions, chest x-rays, retinal photographs, OCTs. And essentially members of my research group without coding experience, were able to train classifiers and get pretty good results within days. And that was tremendously exciting for us, particularly because we've been kind of like looking and all at some of our machine learning collaborators over the last few years and kind of jealous of their capabilities in some sense. And we published this in the Lancet Digital Health in September 2019. And one of my heroes, Eric Topol, said, this will be regarded as a classic paper when AI becomes part of every medical school curriculum. Because whatever you think about whether this kit is actually useful in the real world, this is certainly I think a good tool for clinicians to be able to learn about how to do these things and what are the mistakes you can make and where you need to be careful in a variety of ways. Now an interesting thing that I can't help telling you is that the editor of the Lancet Digital Health said this in the revision process. She said, one point that we hope is that you will be able to address is to emphasize that these tools will not necessarily replace AI experts. Now this made me smile because for the last five years every time I give a talk to doctors, people say, you know, is this going to replace us? And I'm excited about the technology, but I'm realistic. So I don't think that healthcare is going to be solved in the next five years. And I don't think that artificial general intelligence is going to be in solved in the next five years. So I think that there's always going to be work for AI experts, at least for the foreseeable future and the same for doctors. So why am I excited about this? I think it could be a bicycle for the mind. And this is an advertisement that I love from August 1980 about the Apple II computer. And Steve Jobs said, personal computers can be like a bicycle for the mind. Now what's interesting is when you read this, you forget actually that back in the 70s the idea of a personal computer was not widely accepted. So you had these famous quotes like from the head of digital corporations saying, there's no reason why anyone would want a computer in their home. Like essentially at the time, IBM made mainframe computers, digital made mini computers, and they were used by the military and they're used by universities. But they said, well, why would anybody want a personal computer in their home? They can't really do very much. They're very limited. But actually Steve Jobs and Steve Wozniak and others had the vision that if you give these things, even if they're crude, people will come up with lots and lots of applications. And I think the same could happen as we begin to automate some of the machine learning tools that we begin to have. So when I'm talking to clinicians, I say, if not you, then who? So I say, if you're interested, go and download some public image data sets and play around with these things. Now, of course, before the audience start throwing things at me, let me say just there's lots of caveats. So the first thing I say to clinicians is, don't use images from your own institution unless you have the appropriate approvals. And the second thing is that I say is, nobody is suggesting that these could be used for direct patient care anytime soon because if we have to be cautious about the clinical validation of bespoke state of the art machine learning models, we need to be even more cautious if it was an automated system. But I think that this will be exciting because I think clinical researchers can look for a range of ideas, begin to dip their toe in the water and find some interesting things, do a proof of concept, and then if there's potential, work with real experts to be able to actually develop better systems. And Eddie Korat is the clinician for my team that is leading that work. My last two slides just to say, I think of themology is there's a real potential to reinvent the eye examination for the 21st century using machine learning. And if you'd like to read a little bit more about it, there's a short essay that I wrote with one of my heroes, Eric Topol in the Lancet in December, 2019. And we need to move towards implementation, AI assisted clinical trials, AI assisted scientific discovery and I think democratization and industrialization of the technology. That's it for me. This is my email address. If anybody wants to get in touch, I always try to answer. Thank you very much. Thank you so much, Piers, for this excellent talk. That explains like the realities of bringing machine learning to relevant clinical problems into the clinic. It was very exciting for us. So let me see whether they are raised hands already. Yes, Sukhi from the network, Sukhi Lee has a question. Please go ahead. Hello, hi, Piers. Thank you, just gave a very interesting presentation. My question is about your model evaluation. So you said, as you mentioned, like not all the true labels are actually true. So there are some mistakes. So how do you measure and address this problem? So I think there's no easy answer. I think that what I've learned is that for deep learning projects with medical images, the major blocker is the governance and the permissions, but the other major blocker is you need to have meticulous attention to detail for labeling the data. And you have to have a proper strategy and protocol in place to label the data. So typically we would get automated labels from the electronic health record, but those are gonna be garbage a lot of the time. We would actually manually review all of the labels and we would typically at least have two graders, one of whom is a junior grader, one of whom is a senior grader. They would have a protocol that determines and then in areas where they have discrepancy, we would have some other person who adjudicates. So you have to have a systematic approach to labeling those. And even with that approach in place, oftentimes the gold standard that you have is gonna be still ambiguous and this is just a challenge for the field. Yeah, so do you also take like a consistent protocol for this kind of labeling? Some, what kind of protocol? Consistent, like for all the data you have. Yeah, so yeah, we like I get my group to prepare protocols where they describe two or three typical cases of each label, they identify edge cases for each label. We also look at the reproducibility of our labels to see if we're consistent with each other. It takes a lot of attention to detail to get that right. Okay, thank you. Thank you. Do you have an estimate of how transferable to other populations outside the UK these models are? I mean, that's the million dollar question. It's about the generalizability of machine learning models. So I don't even really know how well it generalizes within the UK yet and we have to do the study. So our big thing that we're interested in is will the model generalize from a hospital based setting within the UK to a community setting where the disease prevalence is very different? And then will it generalize to other countries? And we are doing, I briefly mentioned it, but we're doing a global validation study where we're trying to test that if we have a patient that's an African patient with diabetes who comes in London, will it work as well if the same patient was in Ghana, for example? And I hope that we'll have some results of that in the next year. I feel quietly confident that we'll do okay because of the fact that London has a very diverse population in terms of ethnic group and in terms of disease severity, but we need to prove it. And until we prove it, we cannot assume it will generalize. I'm trying again. Leslie, does this work now? Yes, I see you are muted, please. Did any of my question go through? No, not at all, it's not on my side at least. Okay, I wanted to say thank you for your talk. It was very interesting to hear this all from your perspective. My question is about when you mentioned that you had substantial hurdles to getting access and approval to work with the data, you mentioned that you were also very intentional on making your efforts known to patients by having it on the website and having information available in your office. And I was curious if you were aware of any patient reactions to your work, if there were any? So the patients, to my knowledge, have been unanimously supportive of the work. One of the things that I didn't say is that when we launched, when we announced the work, we also brought the support of the Royal College of Ophthalmologists plus all the major eye disease charities plus our patient groups. And we've had a number of talks to the public and to patient groups where I've talked about it. And so when I spoke to the Macular Society, which is the main disease charity in the UK for Macular Degeneration, I spoke to more than 200 patients on one occasion. And at the end of my talk, the chief executive said, what do people feel about this? Because this is sharing your anonymized OCT scans with Google. And she said, hands up those people in the room who would be happy for this to happen. And every single person in the room put up their hand, except for one person. And the chief executive asked her, what's your concern? And she said, I'd be happy as long as my scans are anonymous. And of course, we do anonymize the scans. And so I think that the patients, in my experience, have been very, very supportive. I think that there are members of the public who I think quite rightly ask some questions around sharing of NHS data with the company, like Google, and making sure that people's privacy is respected and a range of other questions around that. And if I was in the same position, I would ask the same questions. That's great, thank you. And thank you from all of us peers. We sent you a round of virtual applause for this excellent talk. I hope it's the start of a beginning of a dialogue between us and you. So this was wonderful. Thank you very much. It's my pleasure. Thank you, Kristen. You're welcome.