 will also be get affected. And we are focusing on a very small part of the health care applications of deep learning. And that is what I'll be discussing today. So before going to computational pathology, let me briefly touch upon what the clinical practice is for pathology. I'll be talking about cancer in particular. So whenever a patient comes to a clinic and he's detected that he has a cancer, he or she has a cancer, then if the biopsy is ordered, I will be talking about invasive techniques, non-invasive techniques like mammography, MRI scans. People are working on those things, but I'll be talking mostly about this, tissue sections in a pathology setting. So the tissue sections are taken out of the body, and they are embedded in the paraffin waxes, embedded over paraffin waxes, and then they are stained with certain chemicals. And those chemicals give them certain colors. And those colors are very important for pathologists to basically identify different structures present in the tissue, for example, nuclei and their substructures and all those things. And the pathologists visually inspect. So as in this slide, we can see that H&D staining, this is hemotoxylene and eugen staining, this is the gold standard in pathology, and it colors the nuclei with blue color, and cytoplasms get different shades of pink or red. And these stains are used to visually inspect the biopsy of the patient and then make the decisions about the prognosis and the treatment planning and all those things. And that is done by visually inspecting it through a microscope. And that is the standard clinical procedure which pathologists follow. And what we want to do is that we mount a camera. This is an example. We mount a camera on the microscope and we take a digital picture and we try to develop computational algorithms that can do at least a similar kind of inference as the pathologist and how better they can do that is what our goal is. So first, the clinical pathology has certain limitations due to which computational pathology is required. The foremost limitation is the inter-observer variance. If the same tissue section, it is being observed by multiple pathologists, they might differ in their opinions. And the data is there to support this claim. And then there is also inter-observer variance. It means that if the same pathologist is given the same pathology at the same pathological tissue at the different time instants, his opinion may be varied. And it's not because of the expertise level of the pathologist. It's because of the limitation of a human brain. We can only keep track of few variables and that too up to a limited time frame. And that's why we need certain sort of quantitative and objective measures that can also be reproducible so that we can make the inferences in a certain objective way and a reproducible way. And that's why we move towards computational pathology. So throughout this talk, I'll walk through a case study to introduce the concepts of computational pathology and how deep learning is being used in this particular context. And this case study is basically of prostate cancer. And whenever a patient comes to the clinic and the PSA test is performed, and if this prostate-specific antigen has a level beyond certain threshold, then prostate biopsy might be ordered. The pathologist might say that biopsy is required. And then pathologist determines, by observing the biopsy, this is the tissue microarray. So there can be whole-slide imaging and tissue microarray. So for this particular case study, we will be working with tissue microarray. If someone is not aware of what those things are, just consider that these are images of this circular shape or something like that. So by observing, visually observing this image, the pathologist assigns a glycean grade. Glycean grade is, again, a gold standard in prostate cancer. So there are two types of glycean grades. So let's say glycean grade 1 is 3, and glycean grade 2 is 4. So the total glycean grade comes out to be 7. Let's say, assume. So what does this mean? The pathologist has certain knowledge that, OK, if glycean grade is this, then this might be the state of the cancer and this particular treatment might work. And they recommend their treatment accordingly. But what's the problem with this thing? Problem is, when they decide about the treatment, so sometimes they have to make very serious decisions. And it might be taxing for the pathologist himself as well that whether he should remove the prostate or he should prescribe chemotherapy. And what if, after doing either of these two things, the cancer comes back after a few years? How to infer that from the glycean grade only or from by visually inspecting the biopsy only? So that is a challenge where deep learning or computational techniques can be very useful. So this is the question which we want to answer, that if these two things are done, then will it come back? Can pathologists infer that by visual inspection or can we do it in a better way by using computational techniques? So this is the case study which we will walk through. So let's start with the first step. So we slightly twist the question and we ask that, OK, we take the data set in which two biopsies, both have the biopsies from two patients, have the same glycean grade, OK? So let's say biopsy, a patient one and patient two both had the same glycean grade and both had done, had undergone radical prostatectomy. Simply, the prostate was removed from both the patients. And now, even though both were on the same stage and same treatment was prescribed for both the patients, then it might be possible in one patient the cancer might come back and in another patient the cancer might not come back. So what is that distinguishing feature that makes the biopsies of these two patients different? So this is the question, how do their visual appearances differ? And what is not accounted for in the glycean grading? So this is the other side of the same question, but we'll try to answer it this way so that we can get insights to the first question which we asked. Now, for the data set, for this kind of study, we use this CPCTR data set. It's basically consortium. And it's not an open source data set. We had to get the permissions to basically get access to this data set. And in this data set, the interesting feature is that we have patients from those patients in which the cancer didn't come back after the radical prostatectomy or after the particular treatment, after the removal of the prostate from recommendation of the pathologist. And we also had the patients where the cancer came back. And the most exciting feature was that both the patients were matched on the grade. So both set of patients had same glycean grade. They were at the same stage of the cancer. They were matched on the race. They were matched on the age and all other factors that can be taken into account by a pathologist. So given everything else constant, what is the visual differences that pathologists can see and tell that, OK, in this biopsy, the cancer will come back. And in this, it will not come back. So that is what we wanted to answer. So a naive approach to this is just sample the patches from the given biopsies, feed it to the convolutional urine network, sample millions of patches. You have normal patches and you have tumor patches. And within tumor patches, you have recurrence and non-recurrence. You just ignore the normal patches. These are sampled by visual inspection or by the help of pathologists. And from tumor patches, you build a convolutional urine network and say, OK, in these patches, cancer will recur. And in these patches, it will not recur and create a heat map out of that. And you'll get very nice results. But the problem with this is what kind of inference we can get? What kind of visual differences? So understanding convolutional neural networks. So the first pointer here is the theoretical understanding of convolutional neural network. Even if it gives very high performance, then what does those features mean? How pathologists can get inference from that? That what is it there in the biopsies that is making them different? So the question is, can we trace back the morphometric features characterizing recurrence by using this naive approach? For that, we have to move backwards. It's not just getting forward, getting the labels correctly. It's also moving backwards that what this CNN is learning and how the characteristic differences are there and what the pathologist should write in his report that, OK, these are the visual differences which I can see. And that's why I am recommending this kind of treatment. So this kind of naive approach is limited in that sense. It may give results. But for the computer vision guys, it's OK. But for the pathologist, we might not be able to tell them that what we have learned. So let's discuss this is one limitation. Let's discuss a few more limitations of this naive approach. So in cancer, cancer is a heterogeneous disease. So it must have some field effects. It's not like every pixel is independent of each other. Then you can use very nice probabilistic models of IIDs and build up the models. But field effects are like generally pathologists look mostly at the epithelium. So the tissue compartments I will discuss later. But I'll just give a pointer to two tissue compartments. Let's say epithelium and stroma. So epithelium contains the most of the glands. And stroma is the less active tissue. I am studying it in very simple terms to get the point clear. So field effects are like if we are observing the cancer only in epithelium, there might be certain features in stroma we might be ignoring. And our heat map, whatever we get from naive approach, it might say non-recurrence in the stroma and recurrence in, let's say, epithelium. But we might not be able to see the correlation between these two or what is the changes in the stroma when there is recurrence in the epithelium, when epithelium says it is recurrent. So that kind of inference cannot be done. Or it requires a lot of effort to do if we want to do it through direct naive approach. So a phenomenon like EMT, like epithelial to mesenchymal transition, I'll not go into the medical terms. But these phenomena, they are difficult to detect by using this naive approach. And the next thing is this is an example of breast cancer. So in breast cancer, we have these particular subtypes, UDH, ADH, DCIS, IDC, and ILC. Whatever data I will be talking about, I am talking it from NIH sources. It's not in the Indian context. So 80% of the breast cancer falls under IDCs. And within that, we have different subtypes. And can we get inference about something, about to which cancer subtype it falls? And we can train the CNN for each of these. But there are very small differences. And if we have to capture those very small differences, that comes the problem of texture recognition. And therefore, that CNN, we require very big architectures, very deep architectures. And even by getting very high accuracy, what kind of inferences we can make out of that? That is the problem. So what is the approach? How we are approaching this problem? There might be other ways. But this is how we are approaching this problem. So we sit with the pathologists. And we ask them, OK, what kind of features you are looking at? So they say that, OK, first of all, we look at the different compartments. So we annotate them. These are the manual annotations, which we do by our own hand. Or initially, pathologists was there to do it for us. And then we got trained and we tried to do it ourselves. So the green ones, the labels are epithelium. And then this is the stroma region. And then red one is the debris. There are other regions as well, which I'll show in the next few slides. Like we have lymphocytes and all other things. So we extensively annotate the biopsies. And this is the most time-consuming step in our approach. So we spent a lot of time with a lot of care here. Because once these annotations get wrong, then our entire effort is collapsed. So we carefully annotate these things and prepare the training data for CNN for tissue compartment identification. Now you can imagine that this is the input. And our CNN should give us an n-ary map. If there is two classes, we will get a binary map. And since there are a lot more classes, only three are shown here. But we work with six to 12 classes. I'll show the entire object hierarchy. So we will get a map of 12 colors, which will say that it's a segmentation problem. This area belongs to epithelium. This area belongs to stroma and all those things. So first step is identifying the tissue compartments. And secondly, within those compartments, we identify the type of the nuclei, like whether the nuclei are normal nuclei or they are nuclei of irregular shapes and all those things. These are the features which pathologists look at normally. So for that, what we do is we do these kind of annotations. So we have inside the nuclei, it is white. So on the boundary of the nuclei, it's red. And everything else we term as black. So we have three class segmentation problem. Why we do this three class segmentation? We can directly do a nuclear detection. But nuclear detection is not important for us right now. Exact morphometry of the nuclei is important. That what's its shape, how much round it is, what kind of eccentricity it has, whether it is elliptic elongated to which axis and all those things, those are important for the pathologists. So that's why we want to segment out the nuclei, not just detect the nuclei. So after doing these annotations and preparing the data for CNN in the first stage for global segmentation, like tissue compartment identification, we get these results. So we feed in this biopsy, we have to train CNN and we get this NRE map. So there we get epistoma, epistoma, confused region which our CNN was not able to detect to which class it belongs. Then inflammation, we have lumen, luminal debris and we have lymphocytes, all those things. So we get a very nice map out of the CNN and you can think that how much effort we put in training this kind of CNN. And it's not really so straightforward that training a CNN and getting these results. And after that, so this gives us regions of interest for nuclear segmentation. So after getting this thing, okay, if we want to say that, okay, I want to only concentrate on the epithelium region. So I can segment out epithelial nuclei very easily now. And if I want to see the field effects in the stroma, I can segment out the stromal nuclei and so on. I can segment out lymphocytes and all those substructures present in a biopsy. So we take a region of interest. This is epithelium and few larger patches taken. So some stroma is also coming. So region of interest segmented out by previous CNN. We feed it and we get the binary map. Now, as I have already said that nuclear detection is not important. Nuclear segmentation is important. The problem with the binary map will be that if there are overlapping nuclei or touching nuclei, then those will be seen as one object, not as two objects. So in that case, we have to create n-nary mass. For each nuclei, we have to create separate mass so that we can segment out the touching as well as overlapping nuclei. So our final results are this thing. These are nothing but these are created from the binary mass but by using some clever processing techniques. But the final result is by feeding this thing in CNN, we will get this mass where we will have nicely separated out. We will not get 100% separation of the touching and overlapping nuclei, but we are doing it very well as compared to whatever pathologists are using. Open source softwares are available. And there is one company, Definiance is also available, which pathologists normally refer to. And we are doing very well as compared to them. Okay, so now we can extract tissue compartment level features from the previous n-nary mass and we can also extract the very micro features which are these nuclear features to create final discriminative vectors. And once we have these discriminative vectors, we can use any very simple classifier. We get very high performance even by using L1 logistic regression. So getting up to this point is time consuming and requires the effort, but going beyond this, it's very straightforward. So, but before that, I have only touched that we get the image, we do epistroma classification and we can get epithelial nuclei or stromal nuclei. But what we actually do is this thing. So we give the image, we get these object levels. First, the gross objects and then for each object, we get into the details. And for each of them, we extract the features. And there are a lot of problems which we can solve using this approach, but let me first complete this prostate cancer case study, then we can move to the next one. So we use CNNs for segmenting out objects in the class hierarchy for each object. And after that, we can extract the features and these are all hard coded. We don't use libraries available because we want to develop our own software kind of thing using for this particular kind of problems. So we get nuclear morphometric features like area length, all those geometric features and everything. So these features we can easily extract, but what we are lacking is we want to extract the graph based features. Those are the things which we think that will be very important. So graph based features in the sense that the nuclei or any of the object is not independent of the other objects. So they might have relations with each other and that you can think of in terms of nodes of a network. So nodes are not independent of each other. So if there is some activity or spike in one node, there might be, it might induce spike in the other node and all those things. So those graph based features are what is pending and we are looking forward to code them. And those, once those will be coded, our system will perform better. I'll show you the performance of this system. But once we get these features, we do simple feature engineering and normalize the data just to get everything between the range zero and one so that we have different ranges for area, we have different ranges for eccentricity and all those things. We normalize everything so that we should get everything between zero and one and get a very nice model out of that. Very nice, simple, elegant. We require simple model because we have to give the inferences to the pathologists which they can relate to their own understanding that what kind of visual inspection they have to do while looking at the biopsy. Okay, so these are our results. So we started with 2,400 to 3,500 variables and these are pretty old results. Now we can work with 10,000 variables and all those things. So we have moved up to 10,000 variables. So then we use recursive feature elimination. Recursive feature elimination means all the features are not important. So we only keep those features which are very relevant for predicting the recurrence versus non-recurrence. So we boil down to 74 variables and that gives us 95% performance, let's say AUC. And when we go down to 18 variables, we get 81% and these are way beyond what humans can do. And we had around 57% human performance and what we presented in American Association for Cancer Research and the human performance was 57% at that too by the expert pathologist. The pathologist who have given their for at least 30 to 35 years to this field, to the prostate cancer experts. So these results were well received. The important thing, what we can tell the pathologist is that now we can go back to the features and say that, okay, which features are important, which features have discriminative power and which features are irrelevant for recurrence versus non-recurrence prediction. So I'll just illustrate, touch upon one feature and which was very powerful. So these are the biopsies given to us and when we zoom in and we see the effect of our feature, feature vector which we have found and we identify that this particular feature, nuclei hole, so we detect the holes in the epithelium, we see that if there is this hole present in the epithelium, so there's one hole, two hole, three hole, then there are stromal holes as well, then it is, we can say with 85% to 90% accuracy that it will come back, the cancer will come back in these kind of biopsies and it will not come back in these kind of biopsies. So this was very striking insight that we gave and from this, we moved from this and we started working on different cancers to get similar kind of insights, okay. So this was a case study for prostate cancer. I'll quickly touch upon another problem that is related to breast cancer but we use the same pipeline and I want to keep the same pipeline. The reason for that is we want to develop a generic kind of software that can be used in different clinical settings or different pathological problems. So cancer is a heterogeneous disease that everyone knows that but breast cancer is the most challenging one for the one which we dealt with is the breast cancer which was the most challenging one. In that, let's say we have a biopsy and there we have dominant subtype. There is dominance of clone one, dominance of clone two and then mixed dominance. So I already pointed out that, let's say we have four subtypes, HER2, BESEL, LUMINAL-A and LUMINAL-B and pathologists generally say that okay, this biopsy belongs to a LUMINAL-A class, let's say HER2 class and they will give a standard drug which is known as herceptin. So standard drug is prescribed. I'll show you in the next slide. So whenever Bleslam is detected, mammography is ordered and if it turns out mammography or biopsy is ordered, if it turns out to be HER2, then a standard drug is there which is herceptin, which is prescribed. And the question is what we ask is will herceptin be enough? And there was a clinical trial and which suggested that HER2 patients, 25% of the patients, they never respond to herceptin, even if they're detected HER2. So we took the dead data and we tried to analyze it using our computational framework and we identified that we can figure out those differences by which we can tell that, okay, these patients will not respond to this particular drug and they will need alternate treatment. So should we prescribe the chemotherapy? Chemotherapy is generally prescribed for BESA-like cancer or more aggressive cancer than HER2. So how we came to this hypothesis is that so these are molecular subtypes. These are not types of cancer. These are molecular subtypes of cancer. Means that certain molecular tests are done, let's say ribonucleic acid count and all those genes available in that they are counted and their expression levels are measured. So in that, what it is seen is, let's say this, we have plotted using some statistical analysis by ranking of the molecules present identified by that particular test and this is the two types of cancer, two subtypes of invasive ductal carcinoma, that is one type of breast cancer, most prevalent type of breast cancer. 80% of people suffer from that and within those that we have four subtypes, HER2, BESA, Luminal A and Luminal B. To keep things simple, I'll illustrate with two but I'll present the results of all the four types of subtypes. So if we talk to a genetics expert, if we give this RNA data, he'll say that okay, both these are well separated, very nice linear boundary can be formed out of this and these cases are HER2, these cases are BESA, prescribe her subtypes to this and chemotherapy to this. But as we move towards different, there's other molecular tests like reverse phase protein arrays, DNA methylation, microarray, we'll start seeing certain interesting phenomena and that is some misclassification is there and that is because, that what we hypothesize is because the patient doesn't have only one subtype of cancer, it may have, he or she may have both subtypes of cancer, it may be that one type may be more dominant than the other but he or she has both types of cancer. So can we verify it using imaging? So we designed the similar classifiers, we designed one classifier for BESAL, BESAL like means it's, it classifies BESAL as zero and anything else as one, anything else means HER2, luminal A, luminal B, any other subtype as one and from the results we can see that and other classifier was HER2, it classifies HER2 as zero and anything else as one. So we can see that those patients, so these are very well matched, those patients which were purely HER2 or which were in all the maps, they were correctly classified as HER2, they had to, the classifier response was also very high for that and the BESAL classifier response was low for that as expected and similarly for the pure patients for the BESAL case but those patients which were confused, the classifier was correctly able to detect that, okay there's 50% chances that it is BESAL and 50% chances that it is HER2. So that is what we gave the inference to the pathologist and this work is quite new and we are conducting clinical trials of this work because we need a lot of data to basically support our hypothesis. This is just with 20 patients but we need more data to support our hypothesis and that's how clinical trials are being conducted and we will soon have data and we'll run similar algorithms and establish this more strongly. Okay, so this is all about what work we do. Apart from that, this is the sporting work which we do that is nuclear detection. As I told, nuclear detection is completely different problem from nuclear segmentation. There we don't care about the boundary or anything shape of the nuclear, we just want to detect each and every nuclear. And tissue feature extraction, that is I told you that graph-based features that work is pending and we'll finish it soon and then I'll walk through this thing. This is an important problem. So quickly, so for this detection of the nuclei, we use distance transform approach. We use this as input and distance transform is nothing but we have hand annotated nuclei and we take the binary map out of that and then we take distance transform of it. So distance transform will be very high at the center of the nuclei and it will go down as we move towards the boundary of the nuclei. So it has very nice bell shape and by using the distance transform as the output and the biopsy as the input and using CNN in a regression framework rather than in a classification framework, we can get these results. So we are able to detect most of the nuclei very accurately. So even very close ones. So this can also be used in the segmentation part because sometimes we don't get proper segmentation so centers can be used to basically mark the two different nuclei when they are overlapping each other. So this is another dimension to this problem and this is not current clinical practice but we think that in future people, few labs are pushing that this might be the next coming thing into the clinical practice. So instead of capturing the image and then staining it with only three wavelengths like getting the image in RGB, we can get the image in multiple frequency bands. So we can shine in simple terms. We can shine the light of different frequencies and we can get the chemical responses from the image and these chemical responses can be used to infer the structures present in the biopsy and then those structures, any deviation from the normal chemical response can be used to infer the disease state or any other kind of inferences which we want to do. This is a standard procedure used in material identification in astrophysical or some other domains but for medical, for the medical, the main problem is the signal to noise ratio. First, the cleaning of the data is very difficult because these responses, they are very close to each other and to get very clean data, it's a challenging task. So first, we have to remove the noise and then we have to clean the data and then we can process it if we want to get these false color images. So what our job is, first one is cleaning the data, second one is annotating this data is very difficult because RGB annotations is easy. So pathologists, what they did is they gave us few annotations and they were reluctant that, okay, we cannot spend much more time on this so you please work with these annotations. We thought that it is a good opportunity to work with semi-supervised setting. So we came up with a technique which can give us, which can, from partial annotations, we can get full annotations in automated way and we wanted to verify that, we are going to verify that how closely it matches to the actual full annotations once they do it. So with an algorithm, we can do these full annotations and this might be very helpful for like time saving for annotations and getting correct annotations and then doing inferences on top of that. So it's related to semi-supervised non-negative metric factorization. We have also used convolutional neural network in this some stage, at the stage of the processing, we can discuss it later, but this is a completely different algorithm which what I have talked already about. Okay, so far so good, we have worked with images but that is just one part of the whole story which goes into the clinic and that's, we have the genomic data, we have proteomics data, we have multiple imaging modalities, one first mammography is done and then after that, let's say H&D stain slide is there and then we have concurrent section of immuno-historic chemistry stains, then can we club those multiple modalities and get some better inference and then we also have the patient records and that is the problem of natural language processing. We have all these factors recorded in a document and can we process their document and use it along with these other data to get better inferences about the disease states and the treatment and progress, treatment and progression. And so these are our recent results. I got this news yesterday that we won this her to scoring challenge. Recently it was organized by University of Warwick and we competed against University of Oxford which has very strong bioinformatics center and Carnegie Mellon University and IIT Kanpur. So by using all these techniques which we have developed in-house at IIT Guwahati, we have got very excellent results in this her to scoring challenge and one other challenge is lined up and we hope that we'll be able to make a mark there as well. So a few acknowledgements. So we are working at IITG, we are working with Dr. Amit Sethi and these are my colleagues Abhishek Vardhanay and Ruchika Bhama and then these are the expert pathologist which we consult for annotations and hyperspectral data or what kind of data we need. So Professor Peter at UIC, he's a senior professor and then Dr. Michael Walsh. He was a postdoc where I interned at University of Illinois at Urbana-Champion and Professor Rohit Bhargav he's the director of Illinois Cancer Research and then this is our colleague Andrew Beck at Harvard Medical School. So we are funded by MHRD, Indo-U.Science and Technology Forum and NIH and Microsoft Research India. Thank you. We are thankful for our partners and collaborators. Thanks a lot for listening.