 Hello everyone can you hear me you had a good lunch okay so for what you liked tell me at least one thing which you didn't like in last two days or you liked everything which is also possible something that you didn't like maybe it's too cold I mean it can be as trivial anyway I was just trying to wake you up okay so I will talk about you know a churn model and this is specifically in the situation where mostly you are trying this for a market leader in any industry and generally if it's a market leader their churn propensities are very very low and when the classes you know are not balanced it becomes very difficult to use the traditional methods to actually model them and thankfully I was working for one such client I mean one such organizations where actually it was a single digit churn at places it was only 3% and it was very very difficult to model that so we went to really tough time but finally we could crack it and I thought why not I share that and probably some of you might be in one of those industries where it might be helpful and with some changes you can literally use it from you know FMCG to media industry to e-commerce industry in different places you can actually use it in this case it is a telecom and because as you know in India it is more than 90% people use prepaid connections so therefore it's a non-committal market where people can choose whether they want to recharge their mobiles or not therefore it becomes way all the more difficult compared to the western world to figure out whether the person has really churned out or you know the person is just slipping and can actually can come back and recharge their phone in some time so what happens is you will the moment you will the algorithms are so trivial the moment you know for say 10 15 days you are out for a vacation you have stopped using the network because probably you are on a mountain hike and therefore you had no network to use all of a sudden when you come back to town you will start receiving calls we will give you this best package and you know start using because their most trivial algorithms has predicted that probably you are going to churn why suddenly otherwise you will stop using and then you will you know you will start receiving this project you know calls or you know SMS that this is a better package please use it while it probably was not really required I mean you are not using for a certain reason and the moment you come back you will start using why they had to actually offer you a better lower priced package and unnecessarily diluting their market what is happening is when they are predicting people not churner as a churner they give a very aggressive package to you by that suppose earlier you were paying 100 rupees for a certain kind of you know schemes or certain benefits they might ask you to pay only 65 rupees for that what will happen next time you will never recharge with 100 rupees because you know that 65 rupees package is available and it is a spiral challenge so over the time you know from 65 they will drop you to 45 45 to 35 and slowly it becomes a very infinite spiral loop because paying I mean making a person to pay higher is way difficult than making a person to pay lower and therefore they keep on diluting their own market lose their profitability lose their they don't lose the market share but just not to lose the market share they lose a huge amount of profitability by that way so therefore attacking this problem was very important where the traditional algorithm was not able to actually solve it now given that's the context now there is a picture of a well and I have said searching a well because for this we actually tied up with a with Harvard Business Schools biostatistics department and we used some techniques which ideally they use for searching wells in the oceans in their oceanography you know studies because it's very difficult you know you don't generally find a single well in the big oceans so you instead of finding single wells you actually try to find a what is called in Hindi the jhund of wells like the group of well right and that's how they move so if I have to actually search them like that can I actually apply it back in our own telecom you know organization also so that's the reason why the wells picture is there and the beautified name so when when the charn probability is only 3% it almost becomes searching a needle in the haystack and if that is the case what is happening is you are spending so much on advertisement there are direct sales indirect sales promotions all other cost that you are incurring is actually becoming a leaking bucket without even knowing whom to target on top of that you are targeting wrong people and diluting your own market so there are two way that you are losing your profitability so what is most important is and this varies from industry to industry it's three key success are you know important there first is accurately predicting who are the charners now that is still possible if you do exposed and all those kind of things but it's equally important that you rightly predict who are the non-charners now most of you and I am assuming you know what is the confusion matrix and you will find it's very difficult to optimize type 1 and type 2 error together now telecom is a situation where they need to optimize both of them together and they are okay to let go some people like they are okay not to reach a very high accuracy here but it's important that they reach this high accuracy which is very opposite to a credit score companies a credit card company or a medical the healthcare organizations for if I am trying to figure out who is a cancer patients it's very important that I figure I mean even if I can identify someone who is non-charners and make that person to go through some more test it's still okay but it's bad if they a person has cancer but I can't detect this is therefore this is important that accurately predict that similarly for the loan market it's okay if I don't give some good people if I don't give loan to some good people but high value loan cases I would rather prefer to not give loan to anybody who has a probability to default but in case of telecom it's opposite it's important that we predict very high this one it's okay that I lose some people because ticket size even if it's a very high value may not be more than thousand two thousand rupees and number of people who are that high value are so miniscule you know you really won't mind but this is so important because it's an infinite spiral loop and therefore it's all the more important to predict very rightly who are the non-charners and the biggest challenge is predicting well ahead and the reason well ahead we have so many telecom player most of the people I mean most of you here probably has multi-sims most of you are using you know different networks if you don't get network on one network player you will actually switch to another one and most of us has more than two seams actually in our pockets if that is the case switching is just a call okay I want to switch you don't need to always depict a behavior that you want to switch so even when I'm getting that okay this is the behavior which is driving the Charner by that time probably the person has already turned out and you almost can't do anything and that is the time when you have to bring back the person it costs way more than if you could have identified earlier so it needs to be identified or predicted so ahead of time that the person is just thinking not had just done it and because seams don't cost and seams are available almost everywhere probably these days seems have started been available at the pun shops or the cigarette shops so people can just go out for a smoke and can switch the seam so if it's that is the challenge then you know it's all the more difficult it's much agile space compared to a health care or a loan because to apply for another apply for loan in another organization is actually much more tougher than buying a seam then we found there is another challenge the most of the cases we try to solve is the active turn active turn means I am taking a call that I want to turn out from this network in telecom there is something called induced turn for example you know that Vodafone probably offers a CUG connection where you can call from your mobile to few numbers free of cost or there are unlimited calls certain benefits you know that most of your evening calls or STD calls like you know people of us like who stay in Bangalore Mumbai away from our parents mostly make a call every day towards the evening time to our parents and generally a longer conversation a big amount of our consumptions goes there right or probably we are staying out of spouse that that's where the bigger consumption goes generally office calls are not long so what happens is when you turn out you also ask your parents at that person he listen you know move to idea or geo because they are offering such new offer so even though the other person will never showcase a behavior that they want to turn or probably there was no reason for them to turn they will turn because you are the decision-making power in that family or in that group because you are turning you are inducing another turn among the other people induced turn figuring out through the traditional method becomes all the more difficult so now all these four challenges that we were trying to tackle so any anywhere anytime when you do any second of turn propensity model I think all of you have learned that no single shoes fits everyone so you always try to do a segmentation now I just want to ask you know because I want to make it a more dialogue I want to ask what kind of segmentation would you choose in a telecom market so anyone of you has any idea in telecom market like worked or see all of us actually use mobile right so how would you if I would have given you an opportunity to segment our consumer base of telecom users how do you have segmented it but segmentation to customer lifetime value first though you will have to predict age okay do telecom player always has our age do you give them our birth year all the time okay okay okay age of one okay how many people use international okay post-paid prepaid but here we are only talking prepaid so let's segment I mean calls and data okay okay so your geolocation okay okay very nice thought hold on to that somebody else was saying data consumption so my mulch is it call and the data consumption and rural and urban okay okay and when you say device used what you mean by that okay so you mean the differentiation between our smartphone versus feature phone or within smartphone the kind of the handset cost okay amount of bills amount the spend okay okay okay okay okay okay so that's the spending capability or how much they're spending okay how do I know that yeah but that's not ethical to use it yeah I know I know but doesn't mean that I can use it for example probably towards the I mean don't don't take my words but probably towards the nighttime many of us are watching porn and I'm not supposed to protrude my nose into your life and figure out you are watching that sorry I cannot so that's not ethically try doesn't give us any try doesn't allow us to even read your SMS try doesn't allow us to figure out which websites you are going to so let's not do anything unethical no I mean we do we can have data but no no not on that so hold on to two ideas when you said where I stay what is the network strength and you said if I can have all their you know call directory the typically call CDR and if I can have that okay okay good so we try to do a very so okay so tell you most organization most telecom organization use it the usage based segmentations okay the kind of parameters that they use actually you already said how much call they do how much consumption how much they paid how much data connections or you know how much of the imbies of data are you consuming where you stay urban rural all those kind of staffs age if I know yes well and good but we found that kind of segmentations actually doesn't help us so why do you do segment to actually identify people who are more homogeneous in terms of their spending in terms of their you know usage pattern but we didn't find that's true for example if I go by our pope probably there are two people who are using hundred rupees recharge there is one person who goes at the start of the month he is a hundred rupee note recharges by a hundred there is another person who every two to three days go and recharge by ten rupees the second person is actually not affluent enough to take out hundred rupees at a time and these two people's usage will be very different even though at the end of the month average spent both the people are hundred many a times you will find that you know if you are spending thousand bucks on phone calls we have lot of our drivers or say the home like the maids who are actually coming from some other places like Bihar UP and actually working they also spend that because they need to call their home but they cannot recharge with a thousand bucks at the start of the month they charge a small small amounts at time now I cannot literally use both of their data together and build a model it will not give me the reasons the reason why she will turn versus the reason why I will turn are so different so what we did we and this is something we borrowed from a Turkish telecom they did this segmentation they forget all those we human being are such a social animals we always prefer to stay and work at a place where people are of the similar cadets so if a person who is a CEO he will never stay at a housing society or somewhere where people are not of the same cadet similarly you know all the you will find that you know all our maids or the you know different different economics straight up people generally stay together and you will find generally work together Turkish telecom so but then there was a challenge what they did they only from the CDR data from the CDN data they could figure out from the network connectivity because whenever you are coming to work you are connecting every day to a certain network latching to a certain network going back home night time you're latching to a certain network it's very easy to identify where you work and where you stay Turkish telecom only did by that it worked magic for them now imagine a Mumbai or a Bangalore just beside a soba you know properties there will be a big slump right so if I just do probably both of them are latching to the same network if I do just by that it won't work so our poor separation you know the spend also we had to take on top of what we borrowed from them so it become a simple three dimensional segmentations and we didn't do any big came in segmentation or anything we did a simple cross multiplication so it's a Cartesian product all of you understand Cartesian products it's like 3 by 3 matrix if there are say 40 different segments here 20 different segments here and 3 different segments here just multiply 40 by 20 multiplied by 3 we had those many micro segments and when we analyzed their usage pattern I mean then we look back they are actually profile we found they are so similar even if you plot the days they go and recharge okay so first thing is we didn't do it for whole India never that will work this has been done in circles because telecom has their own circle like even Mumbai is a separate circle than Maharashtra similarly Bangalore is a separate circle than greater Karnataka so we did it for one-one circle okay within that circle when we had this micro segments we found that they are so similar even the locations the day they recharge was coming to be very similar for example if it is a construction area the construction people get paid probably once every week and there is the only single day that is they get paid by the Venice day or every Monday something like that they get paid every evening they will go and recharge most of them will recharge the similar amount to the time they you know call the kind of amount to do they use on the data everything was so very similar not exactly similar but we found a very very good homogeneity when we created instead of a segments we created this micro segments which has simple Cartesian products okay so that did a big magic for us then we changed the whole concept so all the say the logistic regression or a decision tree or a random forest whatever you are trying to do ultimately you are trying to get for every single person what is the chance probability after this many days even when you are doing a survival model you are checking what is the life for that person so how many more days the person going to live with you right and that is always calculated at n equal to one level this one we said we will not do that it doesn't matter whether it is me versus Arun versus anybody else what matters is this micro segment to me so whatever I will define I will define for that micro segment because we found so now this is the this are the graphs we actually and this is exactly where we used yes exactly exactly okay so I mean those micro segments I just named them honeycomb sorry yes yes yes so it was mostly like you know if you look at so there might be you know 145 160 such segments micro segments we didn't find name for them so we just call them honeycomb it's just my nomenclature then what we looked at so when I say their behaviors are same now just plot a curve in a for a whole month on your x axis at the days you can plot two kinds of curves one is how much a person recharges and after which day then they again recharge the amount of recharge is the highest of the curve the day they recharge there is a upward then there is another way you can draw this is which date and time they are calling and how much amount they are spending or what is the call volume how many minutes they are calling right now this is actually you know forms like a sound curve so you know if they if I'm speaking and if they just analyze that what I'm speaking it will be like you know this kind of pitch and wavelength kind of curve so then we went into the spectral analysis what spectral analysis does whatever be the curve they try to break it down and showcase it as a summation of sine and cosine curve okay so for any such honeycomb whatever be the structure for their their repeat recharge or they are called you know the way they were calling and using the data or the call volume we can always find for different group this in number of this sine and the cosine curves if you sum it up on you know overlay on top of that you will come back to their structure for different honeycomb you will actually have this different curves right now once I have that we then used their call network and that's exactly where you said let's use their call network so now I have this connected graph for all the people there and for within each of those honeycomb so these are these are actually one one honeycombs right and within that I have their normal structure normal wavelength calculated in a certain fashion summation of sine and cosine curve whenever they're you know what happens is probably I am not going to turn but some people around me are probably going to turn what will happen their curves will start changing the moment their curve starts changing the set pattern that I set for them based on their earlier behavior it will not be able to predict the moment it doesn't it's not able to predict and this is they're doing you know probably every day if I'm checking the moment it is differentiating the error that I'm getting the higher error the color will change so it's almost becomes like a hit map so wherever the intensity of the color is red that means whatever is their state pattern either in terms of their research pattern or in terms of their call pattern whatever it is the moment it starts changing this color change so even in that group if I am going on Himala and trek it's not possible that everybody else has gone so what's the point probably my color will be red but doesn't matter to me unless and until that whole group as a whole changing the color so if the whole group's color is red it almost sure this people has already moved out no point calling them no point sending sms they have already moved out of it the best people to attack are this yellow people orange people because they're still in the zone where they're changing their pattern in the position where they're thinking and they probably will be changing so instead of attacking one single person we moved it from one single person to a group of people if the group of people show the pattern and more people going out of it we start calling them and that's how the structure actually changed the green the better green all these things are actually means that those are people are almost you know behaving the same way and telephone is such a necessity for us we generally don't change our behavior much most of the day we'll find your calling patterns are very similar very rarely it changes so therefore if I know that this is their set pattern it's very obvious to figure out then within the group you know how the group is breaking their set pattern so yes yes yes I'll come to the next one so far all these things that we have done was pretty good it was identifying it was able to target people and probably these are the places where there are some people becoming red I'll not target these are the people I'll go and target I have a much higher probability to save them yes yes so what we did to keep it more simple because you know ultimately the call center people has to call if I make it very complex it will be difficult for them we did it simple 10 deciles so whenever these errors are broken down into 10 deciles whenever those errors are going beyond certain level based on that the color codes had been defined the moment those color codes are falling and the yellow orange one so yellow orange one generally will be between 50 to 70 percent we have found you know the moment it is going more than that almost people has moved out no point trying to save them but over the time what will happen is because I am running this whole engine every day over and over this rate colors will diminish because I will be almost starting to attack people much earlier than predicted so expectation is over the time I will find the segments which are completely becoming red rare now it is possible that suppose this is you know a certain area of Kerala because of flood you know my network itself is not working then I know that this is a sudden reason why it is not working and why it has suddenly gone red that's a different challenges right right possible possible that way so what we also tried to check is so when we were segmenting this groups creating micro segments we also made sure that no one segment has more than probably you know half a million or you know some such kind of amount of people that we tried to make sure but what we found when we did this three cross Cartesian products we almost got very small small segments because it's so huge number of segments chances are very very low that there will be segment which has almost you know half of the population that's not possible either it is possible that all of those people are very similar in which case breaking them down forcefully is not a suggestion but in that case you will probably have to set up a different threshold there instead of just setting up a percentage threshold you can also set up an absolute threshold that I am not only saying a 20 percent of it but even when it is going beyond say you know some X amount people I should be able to target them no red color is how many people are changing their behavior compared to their set behavior so so two things one is their recharge pattern which I have broken down in sign and cosine curve and kept it there the other one is their call pattern so how they are calling so suppose you know this is just not 30 days but I have broken it down into 30 cross 24 hours you know that is my time length in that how people are calling when they are calling and what is the call volume so when they are calling is the point where it with the curve will raise the what is the height of the curve that will define by how much they are talking the minute of it right these are the two behavior that we check no it's not the call volume reduce compared to their actual usage if the call volume either goes surprisingly up or surprisingly down so what we are trying to do is because I have based on their normal behavior I have predicted that their normal behavior can be defined by summation of this five sine and cosine curve so every day this people's behavior will be trying to predict based on my set equation okay and I will check compared to what has happened what is the error that I am getting the moment that error increases now either the error can increase because it is high or the error increase because it is low accordingly the color will change so less predictive it is that's how the color will change because I am assuming and we have seen over the time telecom behavior people generally don't change unless there is a reason for them to change okay it's a it's almost like a necessity like we eat food telecom usage is almost like a necessity for that now there are certain situation for example you know geo has come all of a sudden the data usage has spiked up those are the you know very structural change or the structural shift that has happened that we will have to consider separately but in a normal set scenario when nothing is happening people generally don't change their call patterns or the call behavior that's so okay so what we do is we generally try to predict it and reassess our curve every 30 days so it goes by the calendar month and the reason we have to do that particularly for the rural areas their behavior of richer changes during the crop sessions so when the crop cutting happens generally they tend to use way more you know they go and spend way more so therefore we need to change that but it we do that prediction every month but that is done you know through the algorithm that we have written it every month it kind of refreshes it yes but that is for the group not for the individual yes so we tried checking that every three months we found there is hardly any change then later we said once a year it's good enough to change that micro segment it's possible that you know I have suddenly got a hike or you have suddenly got a hike your salary has been doubled but that situation happens little less so once a year is what we said ultimately otherwise it becomes a humongous task yes so the behavior we have checked so when we say that this is my total behavior that is actually one year behavior that has actually been taken but when I check churn I check a 30 days churn I don't check a daily churn I don't check a weekly churn because generally that's how people don't behave okay so generally it's a 30 days calendar month and also we have to see that it's not just how we do the model how even people in the marketing team or in a call center team they get their target everything is you know more set in our world is a more calendar month so it's much better if we do it in a calendar month to also set the process with them yes so there was some basic tagging was there like you know very low spend moderate spend medium spend high medium spend high spend very high spend but to be very honest we didn't you know specifically went ahead and labeled them like that okay because if every you know at the end of one year when I try to do the this segmentation microsegmentation once again people may move from one segment to another segment but in the meanwhile if you know things are changing if even a segment itself is completely moving even there I will find their errors very heavily because that means my set prediction is not working because their whole behavior is changing so much so then we can always go back and either quickly check what is happening there even before attacking them or if we know there is a reason why it is happening then we go back and just attack them so any big changes happening it was very easy to figure out exactly if it's a very slow and you know small changes it's actually difficult to predict but otherwise it was okay to predict any more question yes okay so okay two things one is where you are working where you are staying and what is your spend total spend so now let's take situation where I have come from Bengal to Mumbai to stay and in the same society there is another person who is a Mumbai car bought the flat and staying there with his family two places where you know the biggest change that will you will find is actually the call spend I will be calling my home because I am a migrant person I will be calling my home and therefore my call spend will be way higher particularly at my home location versus the person who is staying with you know his or her family his or her this call spend will be much lower unless the person has you know a spouse or someone which is at a completely different location in either cases both of our you know usage patterns are same I am calling my home he or she is calling his or her spouse so the I ultimately call patterns will be very similar give me an example where you see the difference or a very big difference okay okay we haven't found that we haven't found that happening and the reason is once again we are going back to our human psychology people tend to stay together where they find the similar people are staying so I am not saying that there is no you know difference out of that but what I am saying is in a chunk you know if it's a kind of 50,000 people in a chunk you will hardly find there will be some out layer here and there because the amount spent that is different and yes we are taking a product of all the three yes exactly yeah there will be there will be particularly on the sign the the wavelength and the how many of them you are considering for example the example I gave the person who charges 100 rupees at the starting and then they don't charge versus another person doing five such you know five or 10 such 10 rupee recharges you will always find their curves like the what summation I mean what sine cosine curve you are summing up there is a big difference yes yeah so after this now this is we have just identified who I mean what is happening who are going to turn the most important thing after this to figure out is why they are going to turn if we can't identify them then you know the call center guys will just call what will they speak right and most of the time they will speak okay we will give you this much cash back stay with us now people just shout back why should I get up money repair this right so we have to absolutely understand why people are churning now what is very good to figure out through data science some of the why probably will also be able to analyze within that segment if you analyze that their data usage is falling probably you know that you are there using your network for call and less for the data maybe your data charges are higher always people will have understanding whether they are a you know cheaper data pricing network or a higher pricing network so you can always understand what is happening there if it's a very trivial thing but there are lots of non-trivial reasons why it happens okay for that we had a mandate everyone even the data science people every 15 days one day we will choose a market go and talk to them and analyze the call center calls why the moment a person has challenge there are some people who will find that there is a challenge let's move out but then moving out also has certain price you need to you know then first of all to spread that number to everybody that is the biggest challenge when we speak to people so if I take a new number all of a sudden or if I you know just switch from one network to another even there is a gap of 24 hours or 48 hours when this happens you go and take an MNP and all those stuffs so biggest data was the call center data what we did we listened to so all the call center calls are recorded so there was a big exercise happened and many of this culture also transcribed so we actually had data like all the calls transcribed in English challenge was when it's a Hindi call it was transcribed in English letter and God knows what the transcriber actually transcribed so ideally what we did we picked up lots of sample calls from them and we started analyzing listening to that do some speech analytics trying to understand what is happening many a time you know all those artificial intelligence solution may not give you good solution be human just listen to those calls my suggestion many times that's works wonder compared to you start applying an artificial intelligence to do speech analytics and all those stuffs we are all human the other side people who are human speaking the real challenges just listen to those calls you will actually get the actual reason the heartbeat of the people so that is what we did we actually analyze the call center calls to early identify what is the reason so the moment some segment is changing color that is the early time we will start quickly look at their call center call are they calling for some certain reason are more of them calling for you know one particular reason or not okay and every 15 days we used to pick up one one such segment because we knew where they stay where they work it was easy to actually identify those places go and speak to them just pick up you know people are actually going and recharging just go and ask that I want to talk to you how do you find simple feedback how do you use the mobile we found some wonder thing for example we went to Karnataka I mean around Bangalore some construction site industrial site these are the actually you know we found somehow people are actually spending pretty high there like in telecom someone spending 150 rupees a month is actually considered a pretty high paid person we found at the construction side where most of the migrant laborers are actually staying they're spending so much and we wanted to understand what is happening and their data spend was really really high so when we went and spoke to those laborers they came out during their lunch time we just simply asked how do you actually you know recharge and what do you do with the mobile they actually showed they are from UP they told us that they actually do voice search on google and watch the watch put it dance in the evening now I would have never imagined you know people are at that place so aware about voice search that they go and on where they are doing it viewclip.com which is something even rare but there because those chunk of people they stay together they have figured it out somehow they are doing voice search of Vajapuri dance or Vajapuri Ghana and that is what their evening time spent that is what their entertainment and they are using so much data and how do they recharge they very well know Airtel money they very well know how to actually do this recharge and they told there is actually one auto driver who actually you know help them recharge these things so they're always at the jugars every places in India so it's very good to know you know talk to customer understand you know what they say and the moment you connect this real life examples with your data it becomes very easy to understand what is the reason and how do you change the package or when you are calling them with what you should attack them what is their point what will actually help them to stay back sorry yeah it's a little hidden but it's not that they have stopped so it's generally in the second layer so in the idea second layer the last option is generally you talk to the yes generally it's second layer second or third yes yeah if your questions are solved and that is where also the analytics has been applied we have found what are the people most calling and asking I can tell you if you spend one day at the call center most of the people just call and ask what's my balance now for that there's no point why you should actually pay the call center agent because every call they pick up you pay so then if that is already there in the IVR it's better that you do I can't tell you that you know because most of the call center agents are ladies people actually call just to talk to ladies and I have myself listened to those calls and it's horrible so yes make sense up to some extent you know especially in this domain where multi-lingual people at least in this geography you will be facing but the thing is the sample that you are taking for for example cannot be very diverse because that might be very very common problem and you are wasting your two minutes listening to it don't they just like who's am I calling the guy you called me I listened to your problem solved it or didn't solve it whatever it was I cut the phone I can have a tag right what was the problem not resolved and that makes it much easier for you to target the right calls that you want to listen because they can have other options that it was nothing like you know this and then you can actually go back and target it okay okay so it this options okay so yeah yeah so I don't have much after this I can answer this but probably offline I'll answer I'll quickly rather close this so there is an answer why we could not do that and I'll tell you after this okay and yeah so I just wanted to show you the impact so traditional when we do Charner versus Nonchal and this is not just you know the in-house data science team this is multiple big consulting team came and tried it biggies came and tried it the best one was for every Charner it was predicting eight non-charners we never could apply such model best model was able to predict just before seven to nine days and generally people has already churned out by then and the revenue saved if we apply that was hardly eight percent when we applied that model for every one Charner it was half for every two Charner we predict wrongly predict one non-charner as a Charner and it was at least predicting before 15 to 18 days and revenue saved that way was 60 percent that was a huge difference that was made thanks to Turkish Telecom that I got to know how they have done it and for the Harvard Business School Biostatistics Department one more thing I just want to say at the end industry may not have all the answers please connect with academicians please connect with researchers they might have some very different research which you might be able to apply so in whatever organization you are please try to establish this you know research institute connections it will always help and it you can get wonder solutions from them yeah thank you