 The study of how to produce machines that have some of the qualities that the human mind has such as the ability to understand language, recognize pictures, solve problems and learn. Welcome to artificial intelligence in radiology and imaging. This talk concentrates on the very basics. I'm a radiologist with about 30 years of experience post-MD and what am I doing in the field of artificial intelligence? I am what the AI guy calls a domain expert. Projects at the intersection. AI can do lots of things. Radiologists obviously do a lot of work and there is some intersection between these two fields and that is where I come in. I'm part of a cross-functional team. Let's define certain terms. Narrow AI and general AI. When we say narrow AI, that's what in general AI exists right now. The algorithms, the software, can perform a specific defined task like say play a game of chess or to read a chess text or diagnose a pneumonia or diagnose a skin melanoma from a photograph of a skin lesion. Now this trained algorithm, this bit of software cannot play another game type like the one that plays go cannot play chess or the one that can read a chess text rate cannot read a CT scan unless it is retrained in that task. Now even to come to this level required one breakthrough which happened in the year 2012 in about 50 years of work in the field of AI. At which point let's turn our attention to general AI, artificial general intelligence which is the theoretical ability to mimic human cognitive ability like something that will replace a human brain and that is why we call it a theoretical ability. Now if I have stated that it required one breakthrough in 50 years to get to where we are now imagine how long it will take if at all for something like general AI to come about. We are a long, long way away. So basic terms, within AI you have machine learning. Now machine learning is very programming heavy. Let's leave it at that at this point of time. Within machine learning you have deep learning or the neural networks, the convoluted neural networks, the recurrent neural networks and so on and so forth. Now deep learning or the neural networks have the ability to self-learn by back propagation. We'll come to this point a little later. Data science has parts of all of these and plus something more. So now let's elaborate. What's the basic difference between machine learning and deep learning? Traditional machine learning methods require hand engineering. That is, it requires an extreme level of programming to extract features from inputs. For example, if you were to teach an algorithm to pick up a pneumonia on a chest x-ray, then radiologists and the computer science specialists would need to sit together for a long period of time to try to work out what are the exact instructions to be given in a program so that the computer can pick up the opacity. In contrast, deep learning methods like neural networks learn these features directly from data without any explicit programming. Attended a course in AI by Andrew Ng on Coursera and this statement of his AI as the new electricity really hits the nail on the head. It's an AI powered world today. AI is like electricity. Anything that you can do with one second of thought you can now or very soon automate using AI is what he said. In radiology, I would explain it by saying that it is spot diagnosis which is now possible. For example, on a chest x-ray is there a pleural effusion? Is there a cavity on this radiograph? Is there a fracture on this fundoscopic image? Is there a diabetic retinopathy? Yes or no? On a photograph of a skin lesion, is this a melanoma or is it not a melanoma? On an endoscopic study, is this a colonic polyp? Is this a significant colonic polyp? These are the kind of spot diagnosis that we doctors make and today AI has the capability to make these spot diagnosis. Let's analyze the difference between the human brain and digital computing. Device count. Human brain has 10 raised to the power 11 neurons. But each of these neurons has 10 raised to the power 4. There's 10,000 connections. A silicon chip approaches the device count. It also has about 10 raised to the power 10 transistors. But the limitation of the silicon chip is that it has very sparse interconnectivity. Device speed. Digital circuits are now approaching 100 picoseconds as compared to human biological clock speed, which is about 100 microseconds. So in terms of sheer speed alone, the computers excel. But then the human brain kind of comes back because of its massive parallel computing and adaptive connectivity, which is because of those synapses and it scores a lot over the computers there. Digital computers are more capable of sequential information processing rather than parallel computing. Now in capabilities, digital computers are very good at going from A to B. So they excel in math and in simple processing as compared to the human brain. But the human brain is much better at solving ill posed problems such as those in speech and vision. So you could consider that the several cortex is like a table computer or like a tablet, which is a layered sheet of neurons about six layers thick. It's about one eighth of an inch thick in terms of measurement. And this thick plate contains about 30 billion neurons. And each of these neurons is about 10,000 synapses making 300 trillion connections and a very complex network. Similarly, now you can understand that what is an artificial neural network or an AI algorithm basically consists of many, many small perceptrons or artificial neurons interconnected. So a neural network is a group of artificial neurons, each of which computes a relatively simple function. So how do you train this network to a radiology application? Now this training is done by adjusting the parameters of each node. And these parameters are called the weights and the biases. Basically the algorithm as it is made consists of a random configuration of parameters. Now these parameters are adjusted using optimization processes to find out a numerical loss function which quantitatively measures the initial error. That is you use a fresh algorithm and you run all your images through it. The training set of images and you get to a number which is the error. That is by what factor did the algorithm or deviate from the actual diagnosis made by human beings. Now by a system of back propagation each parameter of the network is then adjusted by small increments in the direction that minimizes this numerical loss function. So you keep training it that is you iterate it or repeat the step of training and number of times till you get the smallest error margin and the parameters approach values of accuracy. Super now we know what a neural network as we know that there is something called a deep neural network a convoluted neural network recurrent neural network which are all parts of the kind of neural networks that we have described earlier big deal. What does all of this have to do with radiology and imaging a lot as we shall see. Look at the sex rate. Somebody has very helpfully annotated saying that it doesn't have a blunted CP angle. That there's no opacity. There's no cavity blah blah and has said that this is a normal study. Now this something is actually not a human radiologist but a computer algorithm called QXR and this is how AI functions today. Here is another example on this x-ray it has boxed the pathology that it has picked up. There is no blunted CP ankle the AI saves but the algorithm points out that yes there is an opacity and not only is there an opacity but there is a cavity as well and it goes on to say that it would advise a tuberculosis screen in this patient just as you or I would do. Yet another example here is a radiograph and here is QXR saying that there is an opacity that yes there is a cavity and yes there is a consolidation and obviously stating further that it would advise a tuberculosis screen just as a human radiologist would do. So a brief history of AI. In the 1950s Frank Rosenblatt was in the US Navy developed the Perceptron which was a single layer of neurons. The team Artificial Intelligence was first coined in 1956 at a conference at the Dartmouth College in New Hampshire USA. The 1970s and 80s saw Hinton and his post doctoral students do pioneering work on neural networks that we have described earlier. So here is one of the first papers that I could look at when I was researching this field. Now this paper was published in the October 1999 issue of CHEST predicting active pulmonary tuberculosis using an artificial neural network. Now mind you this did not study the diagnosis of tuberculosis from X-rays by the artificial intelligence because that kind of technology did not exist at that point of time. What they did was they input radiologist findings and the clinicians findings into a neural network and tried to predict if that meant that this patient was likely to have active pulmonary tuberculosis and they compare it with what a human group would do. And at that point of time their neural network had about three layers and even at that point of time the neural network performed admirably in that particular task. And then and then nothing. The period called the AI winter where practically nothing useful came out of all this thought which had occurred in the 1950s, 60s and early 70s. So why did it take almost 18 years for the next big step, the ability of machines to actually start reading chest X-rays or to kind of diagnose melanoma from a photograph of the skin or to diagnose retinopathy from a fundoscopic image. And that is because of the requirement of enormous computational power which simply did not exist then. What you require for this task is something called parallel processing. And so where did all this sudden power come from? That was because of the developments in the gaming technology. Along with this was the availability of the software itself. That is the availability of convoluted neural networks or neural networks since 2012. In 2012 a type of deep learning called DCNN won the ImageNet LSVR competition and all winning entries since 2012 have been using neural networks and the availability of this neural network ensured that the classification error rate dropped from 25% in 2011 to some as low as 3.6% in 2015. In other words, the ability of computers and computer vision to make accurate diagnosis improved remarkably because of this one breakthrough. Now apart from the availability of the machines or the raw computational power is the requirement of large data sets. How do you train this algorithm? So the NIH's release of the chest X-ray 14 data set which had more than 100,000 chest X-rays was another catalyst. So it was a perfect storm. The availability of technology, the availability of algorithm and thoroughly the availability of training data sets all coming together in the early part of this decade which led to a rapid progress of this field. So the next big step, please read this article, Learning to Read Chest X-rays, March 2016. Ho-Chang-Chen from NIH published a study, Learning to Read Chest X-rays and I quote from this article, in this paper we present a deep learning model to efficiently detect a disease from an image and annotate its context. We employ a publicly available radiology data set of chest X-rays and the reports and use its image annotations to mine disease names to train the neural networks. Now this study used around 7,000 plus chest X-rays. The years 2017 and 2018 marked the publication of many papers and studies validating the use of the neural networks in diagnosis of pathology on chest X-ray. For example, in August 2017, Lakhanyan Sundaram at the Thomas Jefferson University Hospital used data sets totally consisting of 1,000 plus chest radiograph to evaluate the efficiency of the neural networks for detecting tuberculosis on chest X-ray. They used a metric called the area under the curve to assess the model performance and they found that the best algorithm or the best performing classifier had an AUC of 0.99 and this was an ensemble of the AlexNet algorithm and the GoogleNet algorithm. I'll come to this later and they also found that the AUCs of pre-trained models were greater than untrained models. In contrast, they reported that prior machine learning approaches which did not use neural networks had areas under the curve that ranged from 0.71 to 0.84. Now see the marked difference in performance of the neural networks. Let me explain a few of the terms that we have just used. An ensemble. The authors reported that the use of ensembles improved performance. Now what is an ensemble? It is simply a blending together of two different algorithms. For example, we often see X-rays or CT scans being reported by one radiologist or sometimes by a group of two readers where maybe the weightage in this whole process went to the senior radiologist or to the one who has a more sub-speciality interest in that particular application prevails. So similarly, the computer guys can blend different algorithms at different percentages and find out which such blend or cocktail works the best and that's an ensemble. They also found out that pre-training of the neural network with everyday images, not necessarily radiology images, improves the performance and this is called transfer learning. Transfer learning concept reduces the need for very large training data sets. For example, we already had at the turn of this year the ability to detect tuberculosis on chest x-rays. When the COVID pandemic broke out, these same algorithms trained on detection of tuberculosis could very quickly be retrained to diagnose the COVID x-ray features and this is how transfer learning helps. Also note that an interesting feature they pointed out was that the neural networks are functionally called black boxes. It is difficult to determine how the network arrived at its conclusion. On 25th of December 2017, Pranav Rajpurkar et al., let's call them the Stanford group, reported better than radiologist's performance in the detection of pneumonia. They used a 121-layer network which they called the ChextNet trained on chest x-ray 14 data set which we have already referred to. The input was a chest x-ray image, output was the probability of pneumonia, along with localization of the probable pneumonia. Further, in July 2018, from India, Preetam Putha et al. reported this study wherein a deep learning system that they called the QXR was trained on 1.2 million chest x-rays and their corresponding radiology reports to identify abnormal radiographs and the following specific abnormalities. This was an ensemble, incidentally. Blunt costofrenic angle, calcification, cardiomegaly, cavity, consolidation, fibrosis, hyalurinlargement, opacity, and pleural effusion were the following specific abnormalities that it could pick up. Now, the Stanford group again in the year 2018, in November 2018, published their follow-up article. They developed something called a ChextNext algorithm, a CNN to concurrently detect 14 different pathologies in frontal view chest radiographs. And they compared the performance of their algorithm to the performance of nine radiologists using our now well-known metric, the AUC, or areas under the curve. The experience of the radiologist in this study was from four to 28 years. And they claimed that ChextNext achieved radiologist-level performance on 11 pathologies that they evaluated and did not achieve radiologist-level performance on three pathologies. The other take-home message was that the average time for the radiologist ranged from 180 to 300 minutes and was the average of 240 minutes, whereas the AI labelled the same 420 chest x-rays in 1.5 minutes flat. So to conclude, at this point of time, we can state that AI or deep learning systems or these convoluted neural networks, they have enormous potential to firstly increase access to radiology interpretation services and to markedly reduce the turnaround times or reporting times, because as we have seen, they are very fast. WHO estimates that more than 4 billion people lack access to medical inmate interpretation and so automated chest x-ray reporting has great potential in underdeveloped areas of the world and this could be called as a replacement approach. Even in developed countries or cities which have easy access to radiologists, AI can be used to prioritise the workflow. That is, if you have a hundred images lined up or a hundred data sets lined up consisting of CT scans and chest x-rays and you need to know what to report earlier, then in your workflow, AI can help now. Thirdly, so that can be called as the triage approach and thirdly, AI will be very useful even to individual radiologists as an associate or as an adjunct when you could call it an add-on approach. Now, I have oversimplified this, but this was my take-home message from a CT conference called CTBuzz where the speaker was Dr. Klaus and the topic was pulmonary nodule and AI. Given a pulmonary nodule of any size, the human detection rate was say 50%, but for computed ADA diagnosis or AI, it is as high as 75%. Obviously, with increasing size of nodule and increased density, that is, as you progress from a ground-glass nodule to a denser nodule, the human detection rate markedly increases. Now, while the human detection rate may be only 50%, and the CAD alone rate may be 60%, when you add human plus CAD, it goes to 75%. And if you like an ensemble, you add one AI to another AI, then it is about 65%. I think this slide gives you the whole idea of how augmenting human ability with an AI tool can markedly improve the efficiency and the accuracy of humans. Here is another article on similar lines in the European Journal of Radiology which studied whether instead of the normal of two radiologists reporting a mammogram, whether a senior radiologist and an algorithm, that combination can perform as well. So, I refer you to this article for further reading. Here is an example of detecting fractures on head CT scans. As you can see, the algorithm which was developed for reading the head of CTs very nicely picks up the fractures at different locations. It is also very useful in detecting bleeds as these set of images shows. I was extremely impressed by the ability of the algorithm to detect this extremely thin hematoma. And you can see the pickup of the fracture and the associated hematoma. And there is another example of the fractures being picked up. Now at this point, let me also point out to a term called edge deployment. Now edge deployment is when the software resides not in the cloud, but rather on the device. For example, if the algorithm is within the machine or within the console of the machine, so to speak, then as soon as, for example, a chest x-ray is taken, at that very moment an alarm can go off by the algorithm stating that the exterior of that patient has pneumothorax or that the CT scan of that patient has a cranial fracture or a hematoma. And already FDA approved algorithms exist for pneumothorax detection and AI triage for brain bleeds. We doctors use deductive reasoning, the selective use of our experience and the data from the patient's history and clinical parameters to solve complex interpretation challenges or diagnosis. And this thought process cannot be reproduced by AI at present. So will there be an AI winter again? As Anju stated in his AI course, it's the Goldilocks rule, being neither too optimistic nor too pessimistic. AI today is creating more value and there's really no need for unnecessary distraction of fear. AI automates tasks, not jobs. Should radiologists oppose AI development? As a recent ESR white paper on AI says, that would be futile because AI development is taking in many many fields at a very rapid pace and AI algorithms developed for different purposes can always be repurposed for radiology. The other thing on a personal note is that remember that a lot of this work is done either in India or is being done by Indians abroad. So why should we oppose something which we seem to have an innate strength to do? Rather the real danger is not that we will be replaced but is that we will do what computers tell us because we are awestruck and over a period of time as this AI powered machines replace the machines in the existing radiology and other hospital departments, medical students and residents may become dependent and not confident because everything is automated and also note the downside. Remember that this is after all computer software and it will be vulnerable to the kind of virus attacks that are prevalent today such as adversarial attacks and deepfakes which are the adversarial attacks of the AI world. For more links, thoughts on the subject of AI in radiology please visit my blog site www.irrad.guru and you can always write to me milind at irrad.guru Thank you very much for watching this presentation.