 Thank you Dan. Thank you everyone here today. A big thank you to the North Carolina School of Science and Mathematics where I got my start and this is part of my continuing effort to say thank you and it's going to be ongoing over the years. I am very excited to get to talk to you about the particular topic today of addressing a question to try to predict early on whether a patient is going to be a responder to therapy. And this this problem the the work I'll show you that addresses it uses exponential functions, logarithms, transformation of functions, horizontal asymptotes. It is literally pre-calculus that that can be used to address a question like this. There are some more advanced topics as well and what I will do, I understand there are many different classes that people in the room are teaching. You have students with different backgrounds. So I'm going to talk about those that are very understandable to a pre-calculus student. I will also mention things that are a bit more advanced. If you have students who are excited to go and learn some new techniques or something to look forward to in the future, I will point out some of those today. I like to get my motivation right out in front. So here is a picture of me and my husband Colin on our wedding day and our wedding was in a grove of redwoods. That's our grove of redwoods in the hills above San Francisco. And it was a beautiful day. Now looking at this picture, I have to say Colin's suit looks a little baggy. He wore his job interview suit for our wedding, but I couldn't say anything because I wore my high school prom dress. That was a science and math dress. It was new when I bought it though. Unlike the clothes today. So anyway, but Colin looks good here. A month before this photo was taken, he did a hundred mile bike ride and he came home and he bragged to me that he passed a lot of people, but nobody passed him. And one month after that photo was taken, Colin was diagnosed with stage 4 gastric cancer and he's not given very long to live, given a terminal diagnosis. So the motivation here for the problem and the mathematics I'm going to talk about is you're in a situation where you have an advanced aggressive disease. So you don't have a lot of time to test treatments. Right now, many of our treatments are lacking diagnostics to predict if the patient would respond. So we actually give the treatment to the patient and see if they respond. So you would like to figure out as early as possible, will they have a long-term good outcome if they stay on that drug? Or should we switch them to something that actually has a hope of working if this one's not going to do anything for them? Now at the time of Colin's illness, we were not testing immunotherapies in gastric cancer, but so he had chemotherapy and in fact, so he had seven months after his diagnosis. He passed away in February 2012. After that time, the immunotherapies were tested in gastric cancer and were found to have some efficacy. So there are some patients who can benefit. And I'm going to talk to you some about the differences between a couple of approaches to therapy. So a chemotherapy, very broadly, and in this talk I will use some simplifying terms. This is a very simple explanation of chemotherapy. In general, a chemotherapy will try to kill cells that are dividing. So it will kill more cells in cell populations that are dividing rapidly. That includes cancer cells, but it also includes hair follicles. Colin lost his hair on chemotherapy. It includes the gut lining. So he received treatment for nausea and it affects the immune cells that are rapidly dividing. So he actually had low white and red blood cell counts and they can give a therapy to try to address that side effect. But that's a chemotherapy approach. We hope that we kill more cancer cells than other types of cells and get the cancer gone before the others, before we have to stop because of the others we're affecting. Generally, chemotherapy can often give some relief to a patient, can often shrink tumors to some degree, but it may not work for a long time. Sometimes we do have some chemotherapies for some cancers where we can have no evidence of disease after treatment and that patient can then have a lifespan that doesn't seem to be limited by cancer. But many times the chemotherapy works for a while and then stops working. And that's going to come into play in the models that we'll look at. So that's the chemotherapy. Immunotherapy, on the other hand, is a different approach and it is a treatment that goes into the patient's body and boosts their own immune response to the cancer, which is a very targeted approach. Now, we do have targeted chemotherapies, but immunotherapy is this new modality and it can work really well. So for example, some of the early immunotherapies would work for patients who are now, who were at the time of the study, told they had less than six months left to live and are now five years, eight years, ten years out, seem to be doing fine, but those patients are not the majority in general. So it depends on the therapy. It depends on the type of cancer, but we really don't have something that's going to work for everybody yet. And so, but if it works, it can work really well for a patient. So if it might work, you want to keep them on the therapy. But if it's not going to work, you want to switch them as fast as possible because it might have very little effect for them, might not benefit them at all. So we want to predict as early as possible if the immunotherapy is going to work for a patient. And so here's some work. I'm going to show you how we predict response. And I looked at some data. I was on a project and we were all trying to figure this out for many reasons, not just for the patient, but even for developing the drug more quickly so that we could get it out on the market and get it to patients. We want to know as early as possible if somebody will, in the end, eventually respond to a treatment. So this looks at tumor size. And I'm going to first show you a pattern that I use to help figure out, help predict who is going to respond and who is not going to respond in the long term. And then I'm going to spend some time explaining how I get that picture that shows the pattern. But here's the pattern. So what I'm showing are some data from patients with melanoma who received an immunotherapy that was targeting the CTLA-4 pathway. So it's an anti-CTLA-4 immunotherapy. And the colors represent different outcomes. And we have some criteria. This is according to resist criteria, which I can talk to you about if you have questions. But there are four categories. And one is progressive disease that's shown in red here. Most of those people are on this side. And then green is stable disease here through the middle section. And then blue is the partial response. In this study of 338 subjects that I have data for, we didn't have any complete responders. That would be a yellow color when I show some data like that. But of those three categories, if you look at that picture, you can tell there's a separation. Somehow, however I've been able to draw that picture, I can separate the patients who will have those different types of responses. Now, typically, we are looking to separate progressive disease and stable disease from patients who have partial response or complete response. We just split it down the middle and say, the ones on the right are responders. The ones on the left are non-responders. We would like to try a different therapy that might help them more. And again, this is immunotherapy. So we could do that split here. And I'll talk about how to do that once I explain how I got that picture originally. From this picture you can see, we could also actually divide it differently. We could use this to decide who's going to have progressive disease, which means their tumors continue to grow. They're above 20% of the minimum that they've reached at any point. And then the rest, my husband Colin and I would have been really happy if he'd had stable disease. That would have been a good outcome. So maybe we'd want to split it there in the future. Or we could do both in this type of model. So what I want to do is spend some time explaining how I get these patterns. And I'll start by showing you tumor-sized data. So these gray triangles just scattered across that plot are showing tumor sizes for different patients. Each of the triangles is a summary for that patient for that day. And the patient might come in and might have a measurement at the beginning, which we call time zero here, at the very beginning of treatment. They might come at week eight, week 16, week 24, or however they're scheduled to come in. And then they might come in at other times when they're seeing the doctor just because they needed to come in and see the doctor. The doctor might say, oh, we'll get a scan while you're here. So these are the data. Now behind the scenes, I actually know which triangle belongs to which patient. So I could color these by patients. I can connect the dots. And I use a mathematical model to do that. I'll just say that these are summary measurements. The sum of longest diameters is where we take index lesions that we're following throughout the patient's treatment. And we look at how big they are on the scans, and we find the longest diameter on the scan. And we measure that each time and add up those index lesion diameters. Okay, so I use a model. I fit my curves to those triangles. And here's the first model I'm going to show you. It's an SLD sum of longest diameters. So the SLD we're going to get from taking an initial SLD value, so whatever they had at time zero. And then we're going to use exponential decay, which dominates at first for many patients, and then linear regrowth. So S is for shrinkage, R is for regrowth. And those are the simple pieces in my model. Very nice precalculus function you could present to your students. And I will say that there's something more complicated going on behind the scenes. I'm using a technique where I don't just take the three points from one patient and the ten points from another patient and do, you know, fit the three for that one patient, fit the ten for the other patient independently. I actually use a very powerful statistical technique called nonlinear mixed effects modeling, which is really a wonderful technique for helping us deal with sparse data. We can't get lots and lots of measurements from every single patient. So we leverage, we borrow from the other patients. And so in addition to this mathematical model here, we have a couple of statistical models we add into the structure. And I'm not showing those. But one accounts for the variability between subjects. And so it makes every subject come in more toward the others. And then the other piece of variability deals with within subject variability. So we know there's some error in the measurements. We also know there are other factors that might give a false reading or a slightly off reading. So we have structures, these powerful statistical tools that have been developed by researchers. Thank goodness. That can help us leverage and get more information. So every curve you see here is drawn using this technique. But you can think of it, I just connected the dots for each patient. But I'm actually using a little bit more information when I do that. So let's look at this curve a little bit more. This is called a Wong model. And that's because of a paper that appeared. I'm going to give you the references. I have all those at the end. A paper that appeared, Wong and his co-authors published a paper looking at lung cancer. And used this mathematical model for patients who had received a treatment. Now this is for a chemotherapy that he looked at. And you can see there, as I mentioned, there is this nice exponential decay added to the linear regrowth. And these rates here, sorry, these rate constants, KS and KR help us determine the exact shape for each patient. So each patient is going to get their own SLD0 and their own KS and their own KR. And when we think about the exponential, we can pick some values. I picked an SLD, initial sum of longest diameter of 10 millimeters. And I picked a rate constant K shrinkage, KS of 0.1. And that's my exponential curve. It goes to zero. And then here's the linear regrowth. I should have labeled that for R. So the KR rate constant I've put here is 0.1. And T is the time. In this case, I was thinking of days, but you can use weeks. It depends on the data that you have. And then when we add them, here what I've got, I'm showing that same exponential. I'm showing that same linear function. And here's the sum of those two. And I use this software I really like, Desmos, which I think some of you use. And it's freely available. And we can see the shape. So this is a typical chemotherapy tumor size response. You get some good effect, but eventually the tumor regrows. The sums of diameters regrow. But that captures a lot of patients. Now the other possibility is that, in fact, the best fit for a particular patient ends up having a rate constant here, the regrowth, being zero. So we do have data like that. We'll get a zero sometimes. Very rarely, but sometimes that happens. And so the possibilities are either the patient is cured, zero tumor size, or they regrow. That's our chemo model. That's the Wong model. And so here we're doing that fitting for all those subjects. I do exactly that curve. You see, some of them are still just on the downward slope. We haven't seen any regrowth. They may go to zero. We can project out. I have the parameter values for each subject, so I could just draw the curve further in time. Here I've truncated it to where their data stop. So the technique that we use to get that pattern, I fit each patient, I fit their data, I get their KS and their KR, and then I take the natural log of each of those and then I plot a point. What is their KS? That's going to be the horizontal value. Horizontal axis value. What's their KR? That's going to be the vertical axis value for that patient. And then I put that point right where they coincide. And I color the point by the eventual outcome. So I'm looking at data sets where I know the outcome. We ran the study for a year or two years. We have subjects who've been in it for long periods of time. So I know what happened. So I'm trying to use that information to build a model that I could use for future patients. And when they come in, find a KS and KR for them, I don't know their outcome, but I'll put them in the plot and see which side of the line they fall on. So we color each dot. So here's...what color is there? Here I have a blue dot there on the lower right, and that's a partial response. And that patient had a logarithm of KS equal to a value around here. Can't read that axis, but maybe around negative 1.7. And then their KR, natural log of KR, was around 1.1. So I plot their point right here, and I color it blue, because I know their outcome. That's how I get that plot. So then I still need to figure out, well, how do I divide? How do I separate the groups? I have a line I want to draw. I want to put that line there, and I want to optimize. I want to minimize my error when I draw that line. And I'll show you a picture later of some error minimization. But the technique I used here came from supervised machine learning. So you might have students who are interested in that and have been working in that. There is a free software package, also one I really like, called R. And I use some of the functions that are built into R. I tried linear discriminant analysis. I tried quadratic discriminant analysis. I tried principal components analysis. And in the end, something very simple worked. So I went with the simple one. I used a logistic regression. And then I optimized that threshold for the cutoff. So exactly where to position this line comes from fitting a logistic. I can talk more about that if you're interested. I won't go into too many details here. Logistic regression plus optimizing for this particular therapy and the particular cancer. So it does vary for those. But when I have more data from the same cancer type and same therapy, it fits the pattern here fairly well. Okay, so there's a picture for that therapy. The anti-cetilate for patients who have melanoma. Here's a picture for a different therapy, a different immunotherapy. Anti-PD1 for patients with non-small cell lung cancer. So type of lung cancer, the most common type. And same model using the same Wong model here. And I've got this separation of responders and non-responders. So progressive disease spelled out here or stable disease here, those will go on the side of the non-responders. So you see the red and green, now they're overlapped. My model can't distinguish those two. So that's an interesting feature here. But the blue and then the yellow stars, the partial response and the complete response are separated from the red and green. So I can put a line there. And I optimize that. It's more sloped than the previous picture I showed you. And then I can also ask, once I've got my line there, I can also say, well how good is that? How well have I done? I have the answers. So let me check how well I did. How many errors did I make? So what you're seeing are orange dots where I actually made the correct prediction for that patient. And the other colors show where I got it wrong. So on the left I've got turquoise circles. Wherever I predicted the patient would have a non-response and I was wrong. So my prediction was false. I got it wrong. So I predicted they would not be responders because they're on the left side of that line and that has a high regrowth constant, rate constant, and a low shrinkage constant. So that makes sense. On this side, this region, I would expect that they would not respond. But I was wrong. They responded. They kept them on the therapy, which we did in this study. They would respond. And then on the other side, I have a higher error rate when I'm looking at the patients where I predicted they would respond and I was wrong. So we could ask which one is more important and sometimes you care more about one than the other. Maybe it's okay. Patients over here still at least had hope. Whereas here, we took away their chance to have a response to a really good therapy. But in this case, I actually didn't focus on either of those particular types of errors and we can go through and look at different ways to structure the errors. But I looked at total error. I was minimizing, when I did my work, I was minimizing the total error. So I get about 11% error rate, which is better than anything I have seen quantified. So I was pretty happy with that. And I looked at some other data sets to see how does it change on the other data sets. So here's the one we were just talking about, the anti-PD1 therapy for lung cancer, non-small cell lung cancer. That's about 11% error rate. 540 subjects in that data set that I had data for. Here's the one I looked at at the beginning with an anti-CTLA-4 for melanoma patients, about a 10% error rate I get there. And then I looked at others. I looked at a bunch of PD1 study data, some in combination, and then the CTLA-4 was the other I looked at. So you can see these error rates vary between 6% and 19%, average 13%. Now, I've labeled this prediction using full data sets. And what that gives away is the fact that I've been cheating the whole time. So what we really want is early response, and I'm predicting their final outcomes by looking at their final outcomes. And I have a 13% error rate. But we're looking at tumor size to make a decision about whether they were in a category of responder or non-responder. And I'm looking at, actually, I'm not looking just at tumor size. I'm looking at a curve that I fit to their tumor sizes. Their actual tumor sizes are used to decide whether they were responders or non-responders. But the parameters in the curve that I fit for that patient, that's what I'm using. And sometimes I'm off. So how do we get to the real goal? Which is early prediction of response. So let's take a look at some of these data again. I'm going to take the anti-PD1 therapy data for patients with non-small cell lung cancer. And I'm going to try to use 10 weeks of data. And for the different studies, they had different times when they were supposed to come back and get another scan. And I wanted to make sure people had generally had another scan. So I go till the time when they've had another scan, at least one more scan. And here's the way I'm going to structure it. So for 80% of the subjects, I actually split my data set. Take that data set, split it into five pieces. And I'm going to do something five times. But I'll describe how I do it the first time. And it is a modified cross-validation. Another really interesting topic. Very useful in the sciences when we are testing a model or simulating and want to know how good our model is. And we don't have additional data, external data yet, to validate the model. So we can use a cross-validation. Use the data set that we already have. Hold some back. Use the rest to build the model and then test it on the part that we held back. So I'm going to hold back 20% each time. So for 80% of the subjects, I'm going to use all their data. So that's the cheating part. I use all their data, not really cheating. These are what we think of as patients who might have been in prior studies. We use their data to build a model. And all times, any measurement they had, I'm going to throw that in. Get the model. So I'll get KS and KR values for all those subjects. I'll optimize my line for the cut-offs because I know who responded and who didn't. Then I'm going to take that remaining 20% that I haven't looked at yet. And I'm going to throw away all the data from them except the first 10 weeks. So this is my test. How good can I do if these were new patients coming in and I only had 10 weeks of data, could I make a decision that would be correct? How often would I be correct? So I'll use these early time points, get their KS, get their KR just from their early time points, put them in the picture, see which side they're on. So then to do this cross-validation, because I actually have the rest of their data, I can check how good that response was. So over five times, each time holding out a different 20% of the data to make sure I'm not just overfitting to the particular data, I get an average error rate of 11%, ranges between 5% and 15%. So that's an error rate that I can say is for an early prediction. So then I thought about ways to get that error reduced. And I was looking at the data. I was looking at the data a lot. So I would go through some of these data sets and look at every single patient one at a time and look at my model fit to those patients. Hundreds of patients. And then I go back and look again. And then I look again. And I would tally. I would start tallying. What kinds of behavior do I see? What kind of patterns are there in the data set? And then I realized that the model we were using might not be able to capture the particular patterns we see for immunotherapy patients. That's when I started thinking about a new tumor-sized model for immunotherapy. And I wanted to see if we could improve that response prediction if I got an improved model for their tumor sizes, if I could fit those triangles better. And here are some data that are from a chemotherapy study. This is for glioma, brain cancer. And you can see the tumors are growing until this dashed line where treatment begins and then the chemotherapy is administered and you see that exponential decay and linear regrowth. So that model, what we call the Wong model, is quite good for many chemotherapy settings. So many cancers, many chemotherapies. It's a good fit. And here you see the same type of thing. You see the exponential decay. And if you just look at the data, forget that curve. I'm not going to explain how that got generated. That was another paper here. But if you look at the data, they look quite linear there. So that's a good one for that. But for the immunotherapy, here is a paper that shows some patients with melanoma who were treated with an anti-PD1 here. And you can see a variety of behaviors. One of the behaviors is they go down and some of them go to 100% shrinkage. This is a different type of axis that's being presented. It starts at how much shrinkage or growth there is and negative numbers are shrinkage, positive numbers are growth. It's the change, percent change from the baseline. Everybody starts at their own baseline and then they can shrink, and some of them 100% shrink, or they could grow, or they could do something in between. They might grow at first and then respond. So we see for some of the immunotherapies, we see what we call pseudo-progression. On their scans, it looks like it's not working. But if you look, maybe if you looked at the curvature, you could tell, even though it's concave down, it's not going to continue to grow. It might come back down. And then there are many who have a response that shrinks their tumor size and then it stays shrunk, but it doesn't go to zero. Well, our model, the Wong model, doesn't actually capture that behavior. You can shrink to zero, or you can shrink and then regrow linearly, which means without bound. So we needed a model that could capture that. So this pseudo-progression I talked about and this non-zero eventual asymptote, finite limit, if we project it forward and if things stayed the same for that patient. So that's when I started looking at other models. Now, first I'm going to talk to you about a few other models and then I'll show the new model, which is the one I proposed for our immunotherapy data. So the Wong model is here for comparison. That's the equation. And now I'm shifting to a notation that was used in a paper that I'll reference for you that you might want to look at. And it does a comparison of all these. So I'm using that notation so you could easily read that paper. But now instead of writing KS for shrinkage, they use a D parameter for death. And instead of using KR for regrowth, they use G for growth. So it's completely analogous. So there's the Wong model. The D and the G are now the parameters of interest. And Bonet and Subtle, and that is pronounced Bonet, Bonet and Subtle came up with another idea which was, you know, it's true that not everybody has this linear regrowth. So what they did was they tried to pull down that linear regrowth by putting the quadratic decrease. So we've got a minus quadratic term. Stein and co-authors said, well, let's take that original exponential decay, add on exponential regrowth and adjust it accordingly to get the appropriate start values. So it's like the Wong model, except instead of the linear regrowth, they have exponential regrowth. And then here's the new model that I proposed which is start out the same exponential shrinkage or death. And then there's a different function added on. But it's involved, it's a transformation of an exponential decay. And we can go through that. I'll show you some of that. And what I want you to notice is that I have fit all four of those models to these data. This is one patient who received anti-C2A4 for their melanoma. And I have about a year's worth of data from that patient. And you can see that the model seemed to do equally well. I mean, there's not a really big distinction here. Although I can show you some weighted residual plots that we'll show you there is a difference. That's a nice topic. I used conditional weighted residuals because I was using the nonlinear mixed effects modeling. So that's a nice advanced topic for students who are interested in looking up more. But here it looks there. The differences are negligible perhaps. So let's take a closer look at comparing those. And I'll start with the Bonnet model. And as I mentioned, there is this quadratic that tries to pull things down. And here is a typical curve you could get from that behavior. You'd have, and I'm using Desmos here. I wanted to show it's in Desmos. But I only start at time zero. The model does not apply before that. It only starts once the therapy is administered, which I'm calling time zero. So the previous part is not applicable. And then I have exponential decay. It starts to linear regrow. But then this quadratic pulls it down. All right, so that's it. Something's happening over here. Okay, and that keeps it from shooting off too fast. And it might keep it on the data for patients who have a longer-term response. Now let's look at a couple of features. So I want you to think about, for this model, what is the value of that model at time zero? So just think for a moment. So we're plugging in zero for T in that top line. What do you get? Y zero. So Y zero. So that's the part on the axis. So that matches the Wong model, where we had the same type of thing. And then I also want to think about, well, what happens when time gets larger? And in this case, and there's Mike making some noise. Okay, sorry about that. In this case, we're going to look at the equations up here, and we're going to let T get larger and larger. We want to know what do we get in the limit as T goes to infinity. So just think for a moment about that. You can talk to your neighbors. So let me hear some thoughts. What are you thinking? What do you think might be the limit? Negative infinity. Negative infinity? What are some other thoughts? G over H with a T. Good. Okay, so the exponential is essentially going away. So we really get to GT minus HT squared, and then we can talk about a T-axis intercept. Very good. And others had said negative infinity. Any other thoughts? What dominates in the long run here? The T squared is going to dominate, and we can do some formal mathematics to show that. That's very important, in fact. So, yes, we get a limit of negative infinity. So that's not the ideal model for tumor sizes. Because, first of all, it's becoming unbounded, but it's becoming negatively unbounded. So we don't want negative tumor sizes. But on the time period that we're looking at, we might use it and have a pretty good fit. It's just when we project beyond the data that we would run into that issue. But that's the behavior of that model. And I want to look at the Stein model as well. So think for a moment. If I plug in zero for all of the T's in the equation, think what you get. Okay, what is it? Why zero? Okay, all right. So looking inside the parentheses, what do we get from the first term? One. We put a zero in the exponent. We get a one. How about the second term? One. Minus one gives us one inside the parentheses. So we multiply by Y zero. We get Y zero. Great. And so that has the correct initial value for the model. Y zero means the same thing in this mathematical model as it does in the others. And then we can look at the limit as T goes to infinity. So think for a moment about that. So let me hear some thoughts about that. What do you think? Positive infinity. Okay. Positive infinity because of this term E to the G times T, what happens to the first term in the parentheses? It's limit is zero. The third term in the parentheses has a limit of what? Minus one. Okay, so we have zero minus one so far. And then this has a limit of infinity. Okay, so it dominates it no matter how we do that analysis. We'll get infinity there. All right. That's negative, but it does grow without bound. And that might be okay on the data set that we're looking at when we're fitting to those data. It might work okay because it'll try to be close to the data when we fit, when we minimize the differences between our model function and the data. But if we try to project beyond that, it's going to grow without bound. So it will have a bias in the end. And now the new model. So I want to explore properties of the new model. I want to actually break it down a little bit. And I'm just going to show some of these. You can take some time later to think about this more. But I have the exponential decay part of the function. So that's, if I use the values here, I'm showing over here that I used in Desmos, I've got an exponential decay start somewhere positive and it goes down. What is the horizontal asymptote of that function? So what is the limit of this piece by itself if I let T go to infinity? Zero. Okay. So it has a horizontal asymptote. It is zero. Now if I take the negative of that, how do I change the graph? Okay. Take the one I had before and I flip it. So now it's below this axis. But then I have a one in front of it which shifts it up. So that's my functional transformations. First I started with an exponential decay. Then I flipped it with a minus sign. Then I shifted it up with the one minus e to the negative gt. And then I'm going to have a coefficient in front of that, y1. And if I make a value here, like I have a positive value there, I can transform it by the scaling factor in front, the y1 I get here. Okay. So I'm going through that. Then you can think about it more on your own later. But in general, here's a picture when I take what we just looked at and I combine it with that exponential decay because I can look at them this way. I've got the exponential decay plus the other piece I added on. And here is their sum. I'm adding them. Here's my exponential decay function. Here's the rest. So I can get a picture like that which means I can have the decrease in the tumor sizes and I can have some regrowth or I could have it not regrow depending on the relative sizes of those parameters. But it doesn't have to have to have the behavior of the others. Let's take a look at some of the behavior of this model. So I want you to plug in zero for t and think for a moment what you're going to get. Okay. What do you get? Why zero? Why zero? What happens to this parentheses at the end? Get a one minus one. So that becomes zero there. All right. So the value at the beginning. So why zero in this model has the same meaning that it does in the other mathematical models. And then let's look at the long-term behavior. If you let t go to infinity, okay, the first term goes to what? Zero. What about... All right. I know what happens to one. One stays one. What happens to the second term in this parentheses? Zero. It goes to zero. In parentheses, I get a one. So what's the final result? Y1. So it can have Y1, a finite positive number as its long-term limit. So all I was doing was I was looking at these data sets and I was thinking to myself as a mathematician, what is a function that has a horizontal asymptote that could decrease at first and then have a horizontal asymptote that's not zero. And I could also... I could have just added a constant. So I can do other things. In this case, the value of that function was that it had flexibility. And I'll show you some of that flexibility. Let me compare these models. And I'm projecting beyond the patient's one year of data. And now you can really see a difference in the models. So if I project what's going to happen, I might get... Now I've ordered them in the order they show up on this end. Stein, exponential regrowth. Wong, linear regrowth. Bonet, linear followed by quadratic pull down. And then the new model can have a horizontal asymptote. So you can see a difference there. I could go farther and see more of a difference. But that's in a case where I actually don't know the outcome for that patient. So now let me take the data and truncate to half a year. I've got 24 weeks of data for this patient showing now. And I fit the models just to that. Now on the data set itself, each of the models does fine. That's acceptable. We would use those. But if I want to project out, some of them do better than others. That's where the difference is. And I wanted to make a prediction of what was going to happen in the future. So I actually... The values I use for these constants, D and G, are very important in predicting what the long-term outcome is. So tying the goodness of a model in projecting to the prediction of response was an important piece. That's why I wanted a more accurate model. Had the properties I was looking for. Okay, so I can go out farther and even farther. So I can see the differences in the end. You can see that the Bonnet model actually here has gone negative. Here's the Stein going way up here. And there's the Wong steadily climbing at a linear rate. And here's the new model prediction. Okay, so I've got those different ones. Now let me go back to my data sets. And here was the division and the separation when I used the Wong model. And here was the division for that same data set. This is the anti-PD1 therapy for non-small cell lung cancer. And I get a very different picture when I use my D and G parameters, or in this case KS and KR have gone back to that notation. And I have a little bit of a way to tease apart the progressive disease and the stable disease patients. So somehow I'm getting more information here, I think. And when I look at a comparison of the error rates, I'm looking at the Wong using those error rates. These are the ones I showed before. And here if I use the new model, what kind of error rates do I get in my predictions of response if I use the new model instead of the Wong? Each case I'm decreasing by a significant number of percentage points, usually. Now, again, this is cheating. I'm looking at full data sets. But what these represent are the limits. These are the lower bounds. For anything I could predict if I just truncated data and looked at early times. I can't do better than using the full data set. So this is the bound. Now, I'm going to look at one of these using a new model. I'm going to do that cross-validation, modified cross-validation. And now I'm looking at the CTLA-4 data set for melanoma patients. And I look at 12 weeks of data. Same technique as before. Split up my data set, use 80% to build the model. For the other 20%, use just 12 weeks of their data. See what my prediction is. Now I know the outcomes here. So I'll look at my error rate, compare. And averaging, I get about a 9% error rate. So back to that table where I had the full data set errors. Now I've used a new model. And I want to look at the error just on early data. So that goes here. I know it's going to not be as good as 6.1. Can't be any lower than that. Typically not as good. And it's 8.9. But that's lower than the best I could have gotten if I'd used the old Wong model. Now the other data set I showed where I had done that modified cross-validation was using the Wong model. I had looked at the anti-PD1 for non-small cell lung cancer. And that was higher than the lower bound. So you can see how these play out. Now I would expect to get best, the best numbers would be here if I did it for each data set. It's very time consuming to do that cross-validation. So I didn't have the chance to do that. But I want to summarize this method. Here I'm using the new model. And if you look at that picture from before where I had the Wong model, you can see I'm capturing more of the subjects better. And I fit. I get every patient fit here using my non-linear mixed effects modeling, my math model, plus a statistical model on top of it. I get a KS and a KR for every subject, just by drawing the curve, fitting the curve to their data. And then I take those parameters. I take the logarithm of them. I plot the KS on this axis and the KR on that axis, the log of each. And for this patient, they had about a zero for their log of KS and about a negative 2.8 for their log of KR. So they go here and I know their outcome. And then I can calculate error rates. But that's my predictor model. New patients coming in, if we start a new study and we want to get an early read on the study, do we think we should switch patients to the combination arm instead of keeping them on the monotherapy? For example, that might be a question we would ask. Then I could use this model. And all the new patients, I fit their data, just the early data, and plot their KS and KR. I won't know what color, but I just plot it somewhere. If they're on the right-hand side, I predict they'll respond. If they're on the left-hand side, I predict they won't respond. That's my best predictor that I can do. So just to summarize in the details here, I'm fitting that model, just what I said before. Get those parameters, take the natural log, plot, and color by response to build my model. And then I use data mining to actually find how to separate the group's best. And that's from the prior patients. And then the new patients coming in, I'm just using their early data. Okay, so I'm going to keep this slide up here for the references. I, in closing, want to say a few words. So this is mathematics that is understandable by our pre-calculus students. It's really, it's a real question that we were addressing. It was a real approach that we were looking at. And I actually have a poster that I presented at a research conference on this. What I felt was that my contribution on the team was, I was able to make a contribution on the team because I'm a mathematician. Because I think mathematically, because I've taught mathematics, and I know those transformations of functions. So there are lots of people in our industry who are very quantitatively trained. They are using nonlinear mixed effects modeling. They're using the math models and the statistical structures. They're using model selection techniques, which I didn't show here. But we actually, I have done that on these models. I've looked at 15 data sets and compared all four models I showed you and get outcomes to measure the goodness of fit. So we use information criteria, different types. Bayesian information criterion is a nice one. We use plots of conditional weighted residuals. We use visual predictive checks. We use lots of very complex mathematics and analysis. But to have the mindset to say, let me step back and think, what function do I know that has a horizontal asymptote that's not zero? That I think requires a different approach. So with all that complex training and focus on patients and patient data, sometimes we might forget about, oh, I know I can take an exponential decay and flip it and just shift it up and I'll be able to capture, the advantage of that over just adding a constant is that I can capture not just patients who go down to a finite non-zero value, but I can also capture some that dip down and come back up to a finite value or first go up and then go down. So it's a more flexible model with only one extra parameter. And in fact, because of this issue of the long model showing that everybody decays and is cured or decays and regrows without bound, there was a modification made to that model. So there was an extra parameter added so that they could shift up from the zero. But it couldn't capture. It wasn't flexible enough. Even with the same number of parameters, it was not the right type of model to capture pseudo-progression or the dipping down and coming back up. So that's the advantage of this model. So I want to thank the North Carolina School of Science and Math for starting my career and allowing me to think in these ways that I think can really contribute to drug development.