 I've been working at Walmart labs for the past two years and a few months ago I joined the display advertisements team of your marketing team. What we do is we show ads on the internet, hope that people click on them and then when they come to Walmart.com, they end up buying stuff from here. So one of the challenges we're facing in this, one of the problems that we're solving is this, real-time user scoring, forbidding and display retargeting. I know it sounds quite a mouthful, so let's just break this down into small pieces and understand what exactly we are doing. First of all, what are display advertisements? A display advertisement, as is evident, is simply an ad which conveys its message in the form of images. It can be text or anything. You click on any of the items. So take into Walmart.com the item page over there and then you can shop ahead over there. There's a huge amount of inventory of this kind on the internet. And this is the kind of ads we're trying to show to the users. Okay, we mentioned retargeting. So what is retargeting? In retargeting, you're trying to reach out to customers who have already come to your website. This is in contrast to a prospecting campaign where you try to get new users to the website. We're not doing that. The advantage of a retargeting campaign is essentially that you have information about the user. You know what items he's browsed on your website. You know items what he's bought from Walmart.com, right? So this helps us model his interest. These are the things he's interested in. And we can also model whether he's more likely to buy a game from Walmart or not. This helps us do a highly personalized advertisement. We can decide whether we want to show this user an ad or not. Is it worth it for us? And if we do want to show him an ad, what exactly items do we want to show to him? Okay, what makes all of this possible is this technology called real-time bidding. Okay, here's what happens. Earlier, what would have happened is if you wanted to show ads, you would go to an ad network and say, here's some money, get me, it shows so many impressions or so many, get me so many clicks. What happens now is this. Suppose you have a blog post or something and you want to show an ad. You'll allocate a space on it, give it to an ad network. And now every time your page is loaded by anybody across the world, a call is made to the ad network. Hey, somebody's trying to open my page. Now the real-time bidding part kicks in. The ad network recognizes which user is trying to open your blog post. It communicates this information to every single advertiser it has deals with. So every single advertiser gets to know this user is trying to open this page right now. They're allowed to have roughly 10, 20 milliseconds of time to decide how much money they're willing to spend to show that one impression to the user. So instead of buying bulk of impressions, you're now going to be bid for every single impression. All the advertisers submit the bids. Whoever submits the highest bid gets chosen as a winner and he's allowed to show an ad to the user. So yeah, you have to realize that every single time, every single web page is loaded. There's an auction going on on the internet right now. Okay, so where do we sit? We as a user is browsing on walmart.com. We continuously keep scoring the user. We compute the bid values, how much we're ready to bid. But we do not exactly sit on the bidding on the auction house ourselves. We apply this to a partner and the partner act does the bidding for us. The reason being that it requires a huge amount of engineering effort to actually set up these, you know, you have this quick time response constraints that you have to reply within 10 milliseconds and you have billions of auctions going on, right? So we don't do that yet. We just score the users. We compute the bid values and give this to a partner. Of course, if we do win an ad, then it is up to us to serve, to show the items in the ad. Okay, so it all comes down to this. Computing for every user in real time, a specific dollar value. We do this for every single user who is visiting walmart.com. And real time is important because what we found out in the market was that you have to target the users within minutes. You're gonna do this in one day process or two day process that you submit the bunch of users to the partner every day or something. This has to be in real time. Okay, what's the data that's available for us to do our modeling? We all have the, we have the first party walmart data, which is what is all the site activity that is going on and some form of metadata. Let's look through. Site activity would basically, he's viewed this item. He's added to cart this thing. He's ordered this, all that kind of stuff. We keep track of everything a user does for 30 days. All of this is marked with a timestamp session ID. All of this is stored with us. We also have some form of metadata. For example, we keep track of the user ever logged into walmart.com or not. How frequently he comes to, how frequently does he visit walmart.com? That helps us mark a user's active or passive. The walmart and variety of marketing channels has a user ever click on any of the ads anyway, so that we can mark him as a clicker. And then he tends to click on ads. Yes, okay. One thing we would like to point out is we do not use any personally identifiable information. What this means is that we have a user ID say XYZ. We know what all this person did. But we cannot mark this person as who he is in real life. We're not doing that. We just use a user ID and for user ID, we build our models. Okay. So how do we go ahead to build our user bits? This is done in four stages. Let's go through one by one. First, we compute engagement score, which is essentially a measure of how engaged he is on the website, how much he uses walmart.com, and which converts into how likely he is to convert for walmart.com. Okay. So if you have various types of side activity, like I just said, item pages, add to cards and all that stuff. So for each type of side activity, we convert this. We take the historical data and we try to measure. Hey, if a person does so much of item pages, he's viewed, how likely he is to convert later on. So based on that, we compute a weight for every type of side activity. Then we just multiply this with the number of that type of side activity is done. Weight of item page into the count of item pages of user has viewed. And just sum it over all types of side activity. This gives us a single value for every user. It's essentially how we rank our users. We use this ranking to compute the conversion probability. Okay. Here's how it goes again. A user, you have the user, you have his side activity, and then you have computed the weights. You can use both of them to compute the user to engagement score mapping. Now what we do is we group users for very similar engagement scores into buckets. And then you have this for each group of engagements. I mean for each engagement score group, here are the users in the bucket. What we also take is, okay, now out of these users, how many of them actually ended up converting? Okay. Which gives us basically for each engagement score group, you have a conversion probability. So we can say that if a user, once you compute his engagement score, he falls into this bucket. The probability that he will convert for us within the next few days is so much. This is a graph between the conversion probability and the engagement scores. So essentially, as you can see that as the user browsers more and more on walmart.com, he becomes more and more likely to convert for us. Okay, now up to the next part, which is recommendations. Now, you're not only just bidding for the user, you also have to show him some ad, right? So we need a recommendations engine. We have two models, which are essentially Markovian models, which say if a user has browsed this item, he is most likely to end up buying one of these set of items. Okay. There are two models that we use. One is browse recommendations, which is to say, okay, if he's viewed these items, he ends up buying these. For example, if you're viewing some tablets, then we might end up showing you the most top selling tablet, which is similar to what you were viewing. But also we have another model, which is the next best product. Now, suppose you have bought the tablet, right? Next, you're not going to buy any other similar tablet. You're going to buy some accessories, maybe an external charger or a case or cover for it. So that is the second model, which we use, which is the next best recommendations. Okay. Now we combine both of these and in real time, as a user is browsing on walmart.com, we keep updating his recommendations, using both the models that we just discussed. We also group these by categories. What we do this is because you don't want to show an ad, which would have a pair of jeans, a TV, and a camping tent or something in it, right? You'd want to group them into similar categories so that the ad looks much more coherent. And then we use, and then, so again, these recommendations, which is computed in real time, is used by the ad server to show the ads. Now, the final part of it is the expected revenue. Here, we assume that the items that we've shown in an ad, he's most likely going to buy one of these items. Then we take the average revenue of these items as the expected revenue. So if he ends up buying, this is approximately the amount of dollars walmart.com is going to make. You might first raise eyebrow on this assumption that it seems too aggressive, it seems too confident that he'll buy one of the same items. But our logic here is that if you're showing him ads of TVs, he might not buy one of those three TVs that we just showed him in the ad. He might buy something very similar, in the same cost range. So it gives us a ballpark figure of what we expect, of how many dollars we expect to make out of him. Another thing we noticed, we used is, the metadata we had just mentioned, like whether he's a logged in user, an active user, or whether he's a clicker. We could not use them in combination with our side activity because those were linear counts, like he's in 10 item views, or 20 item views, or 30 item views. These are essentially just boolean values, whether is he a clicker or is he not a clicker. Has he logged in in walmart or has he not logged in in walmart? So what we did was, we took the entire space of walmart.com, all the users, and then divided them into eight groups. One group would basically be all the users who log in, who are active, and who click on ads. And then we computed all our weights, conversion probabilities independently for that group. And similarly for all the eight groups, it is all done mutually exclusively. Now that we have all of this data, computing the bids is simple. Okay, you have a conversion probability, which is, if he clicks, this is a probability that he will convert. You multiply that with the revenue you expect him to generate, and then you simply get the expected revenue per click. Okay, this is, if the user clicks, this is the amount of dollars walmart.com expects to make. ROAS goal. So essentially, if you're a marketing channel, mostly your business goal will be something like, achieve me, get me an ROAS of 10. ROAS stands for revenue over ad spend, which means if I give you $1, you should be able to generate $10 of revenue. So then you can clearly see that ad spend per click, how much money you have to spend per click, is the revenue you're going to generate, divided by the ROAS goal. And now that you have this, you can simply compute the ad spend per impression, just by multiplying by CTR. CTR is click through rate. Essentially, if it takes, if CTR is a 0.1% which means it takes you thousand impressions to get one click, you multiply that with ad spend per click, and you'll be able to get an ad spend per impression. In this whole process, we're able, for every user, we're able to mark that this is the amount of dollars, Walmart.com is ready to spend in a bid. Right? Some of the next features, next steps that we have in plan is essentially to identify more features which would help in identifying, hey, if a user is in this bucket, he tends to click more, or if a user is in this bucket, he tends to convert much more. Right now, we only have a basic statistical model, but we're also looking forward to trying out many machine learning algorithms which might give us, help us improve our performance. Here's just an overview of the systems that we use. So the bottom left you have is the offline weights and conversion probability computations. These are all basically hive jobs which run daily to keep updating the parameters of the model. On the bottom right, we have the recommendations engine, wherein we generate the recommendations and keep them in a MongoDB. At the top left, you have the event stream server which pumps in every event which is happening on Walmart.com. So as events happen on Walmart.com, our real-time system receives them which is the MAPDA at real-time system. Okay, we receive the site activity. We read all of these data or the model parameters and the recommendations for the user, the information about the items. We all read through this into a real-time system. It computes the user bid it computes the recommendations for the user and then puts them into a Cassandra's tool. Since, of course, on the scale at which we are working, we cannot obviously have one machine, so this has to be a distributed machine. We use Flume to keep logging all the bid values into HDFS and then of every few minutes an uploader kicks in, wakes up. It reads from HDFS the bid values that has just been put into HDFS and uploads it to our partner. And then the partner takes care of bidding for us. This is, I guess, the current performance we have. We upload nearly 180k users. This is only a fraction of the total Walmart volume. Since we're in a learning stage right now, we have capped ourselves to spending $500 per day. But even with this, you're able to win up to 250 to 300k impressions every day, generate of almost 270 to 300 clicks at a CTR of 0.107. So yeah, that's our current performance. And yeah, that's about it. Yeah, hi, it's Shakti here. So the basis for all your processing is, you're doing a reactive kind of approach. It's the approach is reactive, right? You're based on the historical data and you're going to give some recommendations. Do you try something that's more proactive? For example, try to incentivize the person who's seeing the ad to click on it. I mean, make it more probable for that person to click through and go. For example, you know this guy cannot say no to a bargain. So there's a one-time code that comes up and you say, okay fine, you use this code and you have $10 off. No, I get your point. As of now, we cannot on the fly create such promotional offers for people. That's something you'll have to work with the business with, right? Because we don't have access to that. So when it comes to clicking, getting the user to click, the best you can do is you... So okay, there are a couple of things. What from our side what we can do is show the best ads to the person. That's about it, what you can do from our side. Yes, if possible, you could add some promotional features to it. What happens also is that our partners to whom we upload these bid values, they take care of on which venues the users most likely to click. Okay, whether on some of the sites that users do not at all click on Walmart ads. Whereas on some of the sites, users do tend to click on the ads. So CTR optimization is something which the partner mostly takes care of for us on that side. As in they optimize on, these are the sites where the ads are doing well. So they'll show more ads over there. But proactively, as in, this is the best, the best you can do is show better and better ads to him. That answers any more questions? Yeah, are you using... Oh yeah, sorry. Are you doing user similarity in your algorithms in any way? If so, how? The only similar, no, we're not doing any similarity. As I said, it's a very simple statistical model. The only similarity we're doing is once you've computed the rank which is basically the engagement score, we just club users together based on the ranking. There's no clustering as of yet we're doing. So as I said, we have to try out many of these machine learning or data mining approaches. It's a very simple statistical model we have as of now. It's only been a few months we started this project. We've been live only for a month or two till now. So that metadata about the user, what was that? Metadata is basically... What we've noticed is these features... You might do a lot of side activity on Walmart.com, but you're not at all a clicker. For example, I'm not a clicker. I do, I buy stuff online. I don't click on ads ever. Clicker is a metadata which you can use to identify such people. You show lots of ads to people and then you see whether that person whether he's clicking or not. If he has shown a tendency to click in the past, he's more likely to click for you again. If he doesn't show any tendency to click for you, it's very less likely he'll show you, he'll click for you again, right? So that is a metadata like clicker we use. Then we've seen that logged in users tend to convert more. If you go to his website and have done the effort of logging in with the email ID, then you tend to convert much more. I don't have the data to show, but yes, I think the clicker was almost 2x better performance. You'd click two times better, two times more frequently. Thank you. How long will you keep me... Keep targeting me with television ads. So how do you know when to stop? Sorry? Oh, okay. So his question was basically if he visits volma.com and he's browsing through televisions, how long will we be showing him ads with regards to televisions, right? If I don't browse anything else in the future. Yes. So what we've noticed is this, I mean, these are real-time systems. You try to target the user as soon as possible. Within the... I think we try for two, three days as of today. And if he isn't... If you do not do anything else on volma.com, then we'll stop showing you ads. You can try and make it a little more, six, seven days, but I don't think you'd go much beyond that. At least we don't as of now, yeah. Hi, this is Vikram here. So how do you make your ads context-sensitive to the publisher's content? For example, the publisher is a real estate site and your ad is something else that you're selling. No. So as I said, we do not do that as of now. You're basically... I think since volma.com is a very generic... It's not like we are into sports business that we'll show only on sports websites or sports... I mean, something like that, right? We're very generic. We show all sorts of... We serve all sorts of items. So it's essentially based on whatever you just viewed. That is a bigger context for us. Whether you're on any site, it's not as important as that you actually added this item to your cart a few hours ago. That is a much stronger signal for us. And wherever you are, we'd go and show you the ad about that. Just the question I had was, when you compared all your click-through rates and the ad penetration, clearly there'll be a lag vis-a-vis, let's say, target and even ad words and people who are, let's say, earlier in the game of the online channel. So when you're looking at your technology, architecture and scalability, what has been your experience in terms of the various components you had that architecture chart in terms of what you have done right and where you probably need to change the system? Right. So we use a MAPDH system. Getting the system, as you said, okay, we target only very few users. When I said the number of users 180k, it's only a fraction of the users on walmart.com. But our systems, even as of today, run for the entire walmart.com traffic. We receive every single site activity that is happening. We use what's a MAPDH system, a real-time processing system. It was built by people who, by a startup which got acquired as walmart labs earlier. So we had tech support for that. Some of the challenges were essentially making sure the system is up all the time. We had to pick which databases to use as in our computation, half of it is offline, which is computing engagement score group. This is a conversion probability that's based on historical data. So all of that is computed offline. So we had a high, we, all of them are high queries. We use HMA to schedule our jobs. It keeps running them. We keep dumping all of this into MySQL. And then again, for recommendations engine, for example, since again, this runs on the scale of, you know, the huge catalog which walmart.com serves. We are initially using HBS. It was being stored in HBS. Then we found that, that we couldn't query it in real time. So we ended up taking a dump of daily dump from HBS into MongoDB, just to be able to serve with, whether it's fast enough for us. About the other parts, I guess, the Cassandra store. So we didn't have much of a choice when it came to where we store all of the mappings. It is the user to site activity, because MAPD8, as of now, is tightly coupled with Cassandra. So we do not have much choice, but then we run into a few problems and then we had to talk to the people maintaining MAPD8 to, I mean, fix those problems for us. I guess, time's up. Thank you very much.