 All right. Hi guys. My name is Charu. I work for PTM. I manage all the data science and machine learning teams at PTM and build several products there. Today, I'm going to talk about the personalization story at PTM, right? So I'm going to talk about our journey which we have taken in about last year and a half. I want to tell you state of the union where we are at, where we are going. And we'll also talk about different points in time, how did we pivot and how did we go about building whatever we have today in terms of personalization. So agenda is very simple. So we will talk about why do we care about personalization at PTM? It seems like it's a very large payments company. Then we will talk about the evolution of, you know, how did we just describing you our journey? And then finally, how we engineered a very large scale personalization system, right? Okay. Something about our scale, most of you have heard about us. I'm assuming most people know what PTM is. If you had lunch today, you could have actually paid using the QR code solution. That we built. So something about our scale, we have 100 million plus products and we work in over 2,000 categories. And you know, it's a very wide variety. We go from right from marketplace product like e-commerce products, cell phone covers to fashion accessories. Then we sell travel tickets. We sell movie tickets. We enable a lot of offline payments. We sell deals, name a category. We probably have it, right? So that's why we have very large number of categories and hence very large product set. We have about 100,000 sellers in marketplace alone and about 6 million plus offline merchants. And the reason I'm talking about these numbers is not to advertise but to tell you about the scale, right? So when you have a scale like this, most of the out-of-box recommender systems, whatever people have done in past, kind of starts failing, right? So even your standard databases, standard technologies that you can use to build these, all of that starts failing. And that's when real innovation happens because you actually have to go and build it to match your scale, right? Obviously we have 220 million customers. Most of you have heard this. We have 80 million plus monthly active users, which is a very huge number. That means that people are coming back on the app, back on the website. And then every time they're there, you actually have to serve them some kind of products, right? All right. So personalization at this point, at this scale, think about the categories that I talked about earlier, right? You're selling travel tickets and hotel packages on one side, you know, which can cost you a few thousand rupees. And you're selling a fashion necessity or Bazaar products, if some of you have been shopping at Paytm, which can cost you tens of rupees or a few hundred rupees, right? That's such a huge variety. So how do I actually rationalize what should go on my homepage? How do I rationalize what should go on individual category pages? How do I actually start driving traffic to each one of these pages? It is a very difficult problem to solve. So variety of offering is our number one reason. Number two is when we talk about marketplace itself, we have a very long tail assortment of products, right? So we have roughly 100 million plus products in our marketplace catalog alone coming out of 100,000 merchants, which basically means that not every product is going to sell. Some of the products won't even sell once in a year, right? And this is actually a typical retail problem. It is not unique to us. The uniqueness here is in the scale, right? Then transaction is king, right? So I mean, we are not a content website. So today we are not a content website. So it's not like I want you to come and spend half an hour every day on the website. I want you to come and do a transaction. And I want to make that experience so seamless for you that when you come, I know why you have come here. And I basically show that to you up front so that you can actually just come, do your transaction and get out. I don't want to come in your way. That's the whole idea. And then, like I said earlier, in any digital business, basically you always have, again, a long tail of properties also, right? You have a long tail of products, you have a long tail of properties. What that means is you will have very few properties, which are very impactful. People can come there and people will keep coming there like your homepage, typically. That's a very attractive proposition. But as you go down in the tree, there will be some pages which will get, let's say if your homepage is getting a 10 million views per day. Some of your other pages will get traffic off, let's say a hundred views in that particular day, right? So that's why premium properties are limited. And it is kind of an obligation for us to make sure whatever we are putting on our premium properties is something which is very, very useful for our customers. By the way, putting something on our premium property can make or break a business, right? So we have to be very cautious about that. And this is exactly why personalization was not like a fashion V4 type of a product. It is a necessity for us. All right, let's talk about, so we started this journey somewhere around 2016, early 2016, right? And that's when we were having a lot of discussion. So what do you mean by personalization? What are you going to do? So we said, I come from a background where I have been trained to set metrics against every damn thing, right? So I mean, whatever I do, I try to set a metric against that. So because everything should be measurable, right? So we said, let's set some metrics first and then with the help of some marketing guys, we actually put some nice words also around it. And that's what I have on the slides here. So number one target objective was customer delight, right? So the simple metric against this is basically if the customers who are coming in are actually doing the transaction or not. So if the customer will like it, they will obviously do a transaction. If they know that they don't have to spend a lot of time and once they are on the app or once they are on the website, they can see the product that they need. They will obviously buy it, right? That's the first one. So we want to make sure that we are showing the products preemptively to the customers and not everything needs to go through search because search is again a multi-hop process and you don't want your customers to go through that pain, right? And remember, like I said earlier, transaction is getting my objective is that you come in and I should show you what you are here for. You should do your transaction and you should be able to move on, right? We should show relevant products to a customer based on the prior purchase behavior. This is more like a how. Then the second one is seller delight, right? We have so many sellers. We are a two-sided marketplace. So we have equal amount of respect for the sellers also. Like I said earlier, if I put a product on a home page, it can make or break a business, right? So what that means is that I need to make sure that I am being equally just to my sellers and I'm giving them an equal opportunity to sell on our platform because that's what the purpose of a marketplace is. Marketplaces are not meant for one or two sellers to sell the whole thing. And in that regard, we truly believe in that and the metric that we set around that was the overall spread that a seller gets. So in my top row on my home page, top product row on my home page, I can literally measure at the end of every day, every week, every month how many sellers did I end up promoting? And we will come to how personalization actually made it possible. And then third one is obviously our business objectives, right? So my category managers, my business growth managers, I cannot leave them alone. They are an equally important entity in this trinity. So we actually call this a trinity, a triangle. This has come from our CEO. So there is a Paytm triangle, which is customer seller and then Paytm business objectives, right? So the Paytm business objectives are in a month. It could be I want to drive a lot of transactions. In some other month, I want to drive a lot of GMV. In some other month, I may want to drive a lot of click-throughs and that will be driven by what do I want to achieve in that particular month, right? So I have to, whatever I am building, I have to build it in a way that I can kind of respect that. So remember customer, so I have to make sure that customers are happy and they are getting what they need. I want to make sure that the sellers are happy and they are actually able to sell and it's just marketplace. And Paytm business objectives, which is basically I want to make sure that by making sure that both of them are happy, but we still need to make sure that, you know, whatever we want to drive, we are also able to drive that. Cool. I want to talk about, let's talk about the evolution of personalization now and then maybe I can just go and talk about where we are at. So again, we started this journey somewhere in 2016 and that's when I was given one top carousel on homepage. It's popularly called as Edd. Internally, it's basically the name of that list was Exclusive Discount Deuse. And I will talk about how we used to build assortment for that. But basically, this is a property where we get a lot of clicks and obviously a lot of visibility, right? And the way to build it was people would go and they will find some of the best deals and they will just put it here, right? So we started with that single carousel. Then we said, oh, it's working out. We tested it. We measured it. We'll talk about the measurement journey also. And we said, okay, now why don't I just put it on all the other high traffic properties also? So let me just pick up one high traffic property after other, which is called a category landing page. As an example, you can go ptm.com slash electronics and that will take you on the electronics homepage. And we just started taking one carousel by one carousel on that also. So notice the difference here though, right? If you go to most of the e-commerce companies, they go by a different approach. The general approach is go by the Amazon way. And the Amazon way is that I am going to go in a widget fashion. What that means is that I will build two or three different types of algorithms. So measurements are done and we actually baked all of that into our measurements platform. And now if a customer is getting same type of treatment, then they will continue to get that same treatment across the board so that they are not really polluting the experiment. And initially I was a bit hesitant in doing all that, but now I feel I think that was a great thing that we did and we have seen really good results because of that. So we built experimentation service, then we went on and we did real-time data for evaluating experiments. And we built a really cool reporting tool. I have few folks here sitting from our data lake team who actually helped us build the reporting solution. And now we can literally see all of this data in near real-time. And the next step of this that we took recently is portfolio balancing. So we had another guy who came from financial math background and he was like, you know what, in financial math, basically in people who are doing stocks, the best wish that they have is basically can they actually go and rebalance their portfolio very quickly. So they want to see if some experiment or some algorithm is running and then can they actually go back and they can try and rebalance the portfolio. This experiment is not working. Let me reduce it. Let me increase another experiment which is working. And it is very expensive there because you always have to pay fees to rebalance your portfolio. I told him, why don't you try it here? It's free. And he did it. So what we do, which is also called some kind of multi-arm bandit by the way, is that we run multiple experiments. So we run, I want to say we run about 6,000 experiments in a day across different types of lists. Because we run about 2,000 lists and every list has at least two to three experiments running at any given point. We are comparing them in real time. And at every hour, if I know that this particular experiment is not working on a particular list, then I will start decreasing the exposure of that model on that particular list and I will start increasing some other experiments. And I really like this because when we will talk about, I don't think I talk a lot about adoption challenges here. I will talk about that in the end. But this was one of the pivot points that really helped me get adoption of machine learning. People don't believe in machine learning in any organization. It is a very hard thing and I'm not saying that only for my current company, I've been doing this for 10 years and I have fought this battle multiple times. It is hard to convince growth managers, category managers, business managers that hey, and actually us also when the self-driving cars will come, it is very hard for me to digest the fact that I can go completely hands-off on a self-driving car. Same way they think about it. They are like, I'm not going to go completely hands-off. This is my business and you know what? I am actually going to get bonus done if my category is going to do well or not. So they don't let you go full throttle. This is the same self-driving car problem, right? So this type of thing, portfolio balancing helps you because what you do is you ask them, well, you don't have to be completely off the hook and what you do is why don't you go and curate a list based on whatever assumptions you have, whatever mental model you have and then you will run it in real time and then if you are winning then I should obviously give you more credit and I should increase the exposure and once I do that and I've seen this and I've not just seen this here, I've seen this earlier also in other places I worked that once you have built that kind of a rapport, once you have given them the tools like this then basically the confidence comes and exposure increases. Okay. And the third one is the most interesting one, right? We started with notebooks and that was my foolish idea. So we started writing our entire pipeline. So we were turning about 20 terabytes of data at that point, building the model. Gathering all the data, building the features and then building the model and then saving the model and scoring the model. All of that was done in this very nice notebook environment which is called Databricks. Databricks was a very new product at the time, right? So it has matured a lot since then and then, oh, we have scored everything and now you know what? If you want to put it in production you've got to run it every day, man. So how do I do that? So I wrote a very simple scheduler. It used to come with a cron scheduler. We just utilize that and then we started writing some scripts to basically augment it and then we wrote a scheduler and then I took a flight to India. By the way, I'm based out of Toronto. We have a fairly large data science lab there and that's where we have built all of this. I took a flight to Toronto. I'm going to launch it. And then midway from London, I get a call that, oh, by the way, this is not working because the notebooks are actually not meant for it and it just cannot handle the scale and I return back. But the thing there is that it is an amazing learning from that that never again in my life when I will build something like that, I will underestimate the engineering aspect of it. So I did not underestimate the engineering aspect from the point of view that oh, you should build solid pipelines. You should understand how you're going to put it in production, but when you have to do it, when you have to build these production systems, you have to build it properly. That was the learning. So we built it in notebooks. It failed. In two weeks, we actually converted into Spark jobs and that was an easy thing for us. And then in the first version of it, we basically used Elastic Search to serve this. We had a very tight integration with our catalogue as a serving layer and we were using that. Then we basically started building personalization. We built this personalization service so that there was some more learnings of this tight integration. So we built personalization service, which again, I'll talk about that in a bit. And then finally, we ended up building this real-time scoring thing. So where are we at today? So we are personalization everywhere. I think I have repeated this a few times. So we always reach out to customers with very personalized lists based on some custom assortment. People don't have to time-share lists. Earlier people were time-sharing lists. So fashion accessories and the shoes guy will do this together. Basically they will try to put same products in the same list and so on. I don't think it matters. Okay. This is another learning, right? So anytime we have built models, basically I try to take some... We always try to infer some things from how it is actually done in today's world and then see if you can build intelligent systems that can actually do it. Now, in typical world, for any product list like this exclusive discount deals, anyone who is configuring that, they always start from some catalog. They select some pool of products and they try to push it there. It's a mental model which does not capture a lot of complexity and hence it cannot be generalized, right? And you will always have dependency on humans to reproduce it. So now I want to talk about the postulation framework that we have, right? And this is very standard framework that we have tried to use across different things. And the only time something evolves is basically different types of models and the framework is so general that you can mix and match different types of things. All right. It obviously starts with data lake because that's where you are capturing all the data. The first step that we do is product pool selection, right? This is a very cool forecasting algorithm that we have built. And over the period of time, I would say it has gone through four or five iterations maybe. I have gentlemen here in the crowd who actually built it. So what we do here is we start with the catalog and basically we take... We build a very large time series of every product, right? So for every product, how much it sold in last one month, two months, three months and so on. Actually, we do it on days level, right? And then we try to predict if I were to give X amount of stimulation to this product, how much it will sell, right? So it is based on some time lag variables. It is based on visibility. It is based on how much visibility I'm giving, how much search terms were there for a particular product. What is the current discount running on it today? How much does it sell when there is no discount running on it? And then if I were to give it a 10% discount, how much it will sell? 20% discount, how much? And then based on that, I basically go and look at every discount that is going to run tomorrow. And based on that, I can say that this is my approximation. It is actually going to sell four units. Another added complexity here, remember I told you in the beginning, any item that can go on a homepage or on a prominent page can make or break the business. That's why you need to have the location of the item as a factor in this, right? So we take that into account as well. So based on that, I can basically select some products. The next thing is we define the category affinities. I told you we have 2,000 plus. Precisely we have about 2,500 categories where what we do is we have gone through several iterations of the model. I think six or seven now. The current one being LSTM which is being developed. LSTM is being developed now. I think we just put it in production and before that it was a semi-connected network. Here we are trying to predict in which category customer is going to be most interested. Because remember I have already picked up what are the best products in that category that are going to sell, right? So I have already solved that problem. And when I start doing category predictions which are about 2,500, I'm not talking 10 or 15 categories. I'm talking 2,500 categories. So your shoes and your socks are different categories. In fact your sports socks and your formal socks are actually two different categories. We can do prediction. We do prediction at that level. And once I have all of that, then I use an item recommendation. I don't use item recommendation as my starting point. Item recommendation is my end point, right? This is the last point in the journey. That's the main difference. Most of the time when you talk about a recommender system, that's your starting point. Spark comes with ALS model and all it does is basically takes your view and product signals. Basically calculate the interactions. And then basically you run some kind of alternating least square method which they have implemented. And then you try to find out which product is going to be popular if you have seen this other product. Netflix is doing that, Amazon is doing that. I think that's the most popular thing. And obviously people have used category affinity in past. Not saying we are the first ones, but people do it after the fact. What we are doing is we are doing it before the fact. We have always focused more and more on the category. We are focused on this model. And lately from last six to eight, once we've been focusing a lot on this other model also. And partially because this is actually a very high scaling challenge. Item item collaborative filter, right? 100 million products is a non-trivial size. After this, remember I talked about, I do assembly because remember I talked about my third objective, which was my PTM's business objective, which basically says that in this month I want to increase transactions. In this month I want to increase GMV and so on and so forth. In this month I want to increase profitability. How do I do that? I need to give some kind of levers. And that's where some kind of once I have built my assortment. This is what I'm going to show to a customer. I need to reassort it. I need to re-sort it. And that re-sorting based on the inputs that I get. And this is an optimization problem that we are trying to solve. Right? A regression problem on your left. A category affinity prediction on the right. Collaborative filter, which is a kind of a matrix factorization in your middle. And then an optimization problem in the end. This is our standard framework. And this is what we use in all the places. I'm happy to go into more depth. I see I have only 12 minutes left. Okay. I'm not going to talk about this. Category model is an interesting one. We started with a very simple random forest classifier. It had about 6,500 features. And we were predicting only 200 categories at that time when we started. So we said 2,500 categories is no may when. We cannot predict that. So we said let's go one level up. It's obviously a hierarchy. And we started predicting from that point. It worked out 200 categories. The precisions were decent. The metric that we use to measure our model always is precision at K and NDCG. Then we basically said, you know, I said, we will go on every homepage. So every category page. That's where we started building a hierarchical classifier. By the way, one thing that we added here was in spark ML lab. You could not persist the model, especially the random forest model. We have actually built that. So we could actually persist the model and then we could use the model if it's stable enough. So I'm happy to share that code at some point. We built this hierarchical classifier again. It does not come out of the box. We actually built it. We said let's take all the major categories because there are certain differences in fashion from electronics. So we started building individual models for each one of them. And then obviously it was a joint probability distribution in the end if you want to predict at the leaf level. Or if you just want to predict for that particular category, then you can just use only that model. Then came this new model. By the way, this is a very state-of-the-art mechanism for doing category predictions used by a lot of people. It's based on a paper from Microsoft. What we did was we understand that basically user purchases are a sequence. It is like a sequence of words. So we use word to wek to build a latent factor, a latent space in which we could represent the user purchases. So on the left, you basically see user purchases and views. And then we take some target purchase category and we run it through word to wek. And that gives me that latent space. This guy over here, this is my latent space. And then I take that latent space and then I run a logistic regression on top of that. We like to call it semi-deep learning. This is not truly deep learning because the model is not fully connected. So we build that and then finally we were able to get the category prediction. This actually works on all the 2400 categories. And now we have done another thing on top of this where I don't have it here, but we basically take the next generation of this model, which has been live for about six months now. It takes time buckets. We basically think of a very huge stack matrix, which has this user, this latent space. It is stacked by zero to two days, two to four days, four to six days and so on. And then it is compared against a very large target vector in the end, right? And that's how you run a logistic regression. We try doing basically a fully connected network also using our MLP. But the thing is our data set is so huge that we just could not finish it in time. And I need to finish every model in two hours. If I don't, then basically I'm late for the right prediction. Basically the precision at case starts dropping in collaborative filter. Again, we use ALS, but we initially we use ALS. We give it our flavor by giving it, you know, the, the strength, which is the interactions. We gave it our flavor where we said that not every interaction is equal. So we built a custom function, which exponentially decayed the interaction. And now what we are doing is, and that's why I mentioned the six million plus offline merchants in the beginning is because I'm introducing all the payments data in here as well, where I can literally build a function in ALS. We just say is that I will have X amount of a weight given to interaction from a digital product and a physical purchase. I will have Y amount of weight given to a digital product and a digital purchase. I will have Z amount of weight given to a physical product and a digital purchase and so on. So I can make all those permutations, assign different weights to them. And that's how, and that's basically our variant on how do we define our interactions. This is another use case where we basically had, you know, even after doing all this, you will always have these corner cases keep coming up. This was specifically done for air conditions and TVs, high end electronics type of categories. And what happened here was we were not showing the right products. So basically what we ended up building was another, so similar model, the semi connected network that I showed you. We built a variant of that by using price buckets and brands, right? And this is the TSNE visualization of that. You can see the red ones are the ones which are predicted by this model. And the green ones are the predicted ones predicted by our general model, right? And the blue is the customer's purchase. You can see that the red dots are actually in path of how customers actually purchase it. This is an interesting one because somebody actually challenged us once and this graph is just coming from that email we sent it to them. Saying that by the way you are buying this type of stuff, that's why we are showing you TVs which are very expensive. This is interesting. So now at every point when we start giving out a recommendation, what we do is we're not serving from one model. It really depends on the type of list that is being created. So we start with customer, then at the time of generating the recommendation we look at a few things. As an example, does this list have categories, multiple categories, more than 10 categories? Or is it primarily a list which has only products from electronics? Or is it primarily a list which has products from shoes and fashion necessities? And based on that, we have built a system where we basically invoke different types of models. So it is not one model. Again, that would be another problem with using a vanilla ALS. Where in an ALS you have a very large model and you're trying to treat everyone through that same model. Whereas in this case, you can have multiple different types of models, multiple variations of deal pool selector, multiple variations of category affinity and multiple variations of your collaborative filter. And then combine them based on the type of prediction that you want to make. Okay, I want to share some very quick results. Basically, you can see on the left, you can see some lifts in CTR. But basically 2000 plus lists are personalized today. We serve about a billion plus recommendations every day. We have seen about a 2x increase in GMV per user, similar type of increase in number of transacting users, number of units sold per user. And then we have served a lot more sellers than we would have served by doing manual assortment. This is our overall architecture. I am just going to finish in maybe two more minutes. The overall architecture is basically we have a batch part where we do a lot of feature creation and so on. Then let me go to the next one actually. Then we have these three individual microservices going back to that framework. We had three things. So we had a deal pool selector, which is in the middle here. We had a category classifier and we have, we call it white mage. It's basically the collaborative filter, right? These are three independent microservices completely deployed with their own data sources. I can deploy whatever model I want to deploy there and so on, so forth. And then we have an orchestrator, which is directly talking to something called a serving layer, which is an extension of our catalog. So our catalog, every time it wants to serve anything, it will first make a call to us. Our measurement system is going to determine what type of model needs to be served. Then basically we will go to that particular, we will go to the orchestrator, we will tell orchestrator, pull up that model for this customer for DPS, pull up this other model for CC, pull up this other model for white mage. And then all of them send back the results to orchestrator. Orchestrator then goes to another DB, which tells it in this particular list, my goal is to increase the transactions. So then it says, okay, so let me pick up the optimizer or the racking objective, which I need to solve for is increase transactions. It really assorts everything, resorts everything. And then it basically sends it to, sends the recommended products. And obviously we do a lot of validations like is the inventory there or not, is the merchant good or not, has the merchant been blocked or not and so on. Great. I want to share some learnings. This is my, some of the things that I've learned over the last 10 years. So don't build stuff for yourself because people will not adopt it and that will just lead to more frustration. So always build for your customer, ask them what do they need. This is an interesting one that I've learned. Don't build things to present at conferences. Build to add value to business. Always do that piece. Because when you add value to the business, you will get a lot more ideas and then you'll be actually able to build something which you can present at a conference. So it works other way around. Change is hard and messy. Adoption is very, very hard. Machine learning adoption used to be even more harder five years ago, right? Now because AI is a buzzword, business managers are coming around. There is a top down push, so people are adopting it, but it's hard. Find a business partner. Get executive sponsorship, but more importantly, find a business partner and they will help you get through this, right? Engineering first. So models are absolutely useless. They have zero value. Sorry for interrupting you, but you've overshadowed your time. Okay, I'm almost done. Just start small and keep a trading. Thanks guys.