 Hi. Good afternoon, everyone. Yes, this is going to be a lightning session on correlation versus causality. I apologize. The session on measurement is curtsy eforum. And it's in the second half. So i hope you've had enough coffee. You're caffeinated. You'll be able to sit through the next 15 minutes. So i had the india business for publicist epsilon. Epsilon was acquired by publicist about 2019. It was one of the biggest acquisitions after they acquired sapient. So sapient was the services arm that they acquired, and with the Acquisition of epsilon, publicist is getting into the space of Marktech and adtech. So getting into the platform's play. The reason the eforum team requested this session is because Of a case study that we submitted with them on the work That we've done with mama earth. D2C, incrementality, Measurement. And i would like to kick it off today With an avi that showcases these sessions. So if i could have the avi played out, please. Case study, that video talked about a lot. It talked about data. It talked about incrementality. It talked about dynamic creatives. It talked about Personalization, individualization, one-to-one communication. All of these are typically limited to the crm space. Personalization is where first party data comes in. It's normally not talked about when it comes to programmatic Media as a channel, paid media as an option. But what we have been able to do with mama earth is essentially Leverage all of their first party data, going back all the Way three years, and leverage that to drive incremental Return and ad spend. So in the previous session, I have talked about he wrapped up the session with a very, Very important statement. He said it's all about growth. It's all about outcomes for your clients, right? And that's exactly what the focus of our conversation today is. So while we typically talk about three pillars to whatever Epsilon does, doesn't matter if it's on the martec side or On the adtech side. It's about understanding the customer, Identification, therefore. Second is resonating with the customer. So personalization, one-to-one communication with the Customer. And lastly is proving the outcome, Proving the results, and definitively proving the results. Today, instead of covering all of the other pillars and all of The other three-letter acronyms, we're going to focus on Incrementality, we're going to focus on measurement. I decided to start the session with correlation and Causality. These are the two terms that are Fairly misunderstood as far as measurement in programmatic is Concerned. I had a good time researching The content for this slide. So what are we seeing on this slide? Two metrics or two analysis, right? Divorce rate in the State of Maine and per capita consumption of margarine. So if you look at this graph and you just look at the Numbers as is, the trend line as is, what are we saying? Increased per capita consumption in margarine or butter Leads to a higher divorce rate? No, right? And vice versa, what is the relationship between divorce Rates and butter consumption? There is no causal Relationship between the two, right? There is a correlation, There is a positive coefficient of correlation, but there Is no causal relationship between the two. And this is essentially the problem that we face in Marketing analytics today, right? When it comes to Programmatic, we start from the right-hand side, if you look At that slide, right? It's attribution, right? We are going with correlation. I or my channel has been in the Path to conversion. Therefore, i'm attributed or I'm giving credit for that particular conversion. But what we're trying to move to, and this is especially Important in the context of, you know, first party Cookie base, first party solutions, third party cookie Deprecation, device id deprecation potentially is Incrementality. You need more advanced models when it Comes to measurement. So on the right-hand side, You're basically saying, i need to just get in front of as Many conversions that are already going to happen, right? I just need to take credit for those conversions. On the left-hand side, we are saying, let's actually run Experiments, scientific experiments that tell us what Would have happened any which way, what was the baseline going To be any which way without any intervention, without any Personalization, without any usage of first party data, right? What is the incremental revenue, what is the incremental Conversion that a channel is driving for me, right? And so this is the baseline for us. Is the media driving the sale or is that ad basically Taking credit or a sale that was any way going to happen? Now, like i said, interesting time researching for this article, Right? so you see a bunch of images on the slide. So carrying on from the attribution conversation, right? On the right-hand side is what you see as last-touch Attribution, last-click attribution or multi-touch Attribution. So we started out with base Models of last-click and last-touch. And now we're going to a place where we're trying to get more Advanced in terms of how we're measuring performance, right? The reason i picked up the football analogy is because it's The best way of explaining how credit is assigned. So if you have a ronaldo or a macaquire who's scoring the Goal, he's the last person to touch that ball before it Hits the goal. That person gets 100% of The credit, gets 100% of the accord, the recognition, Everything. That is essentially what we're Doing with our current models of measurement. Now this becomes a bit of a problem when you look at the Channel mix. On the one hand, we are Saying that omni-channel is important. Multi-channel is critical to how we're running businesses. We have an influx of d2c businesses, startups, Entrepreneurs who are entering that space, who are present Both offline and online and within online. That mix looks like this. Look at the number of channels That are in that path to conversion. Look at the channels where we are allocating budgets. I'm going to put air bnb here. I think air bnb's cmo Made a very interesting statement. He said the way that we run attribution today is Essentially people are trying to slice that pie and say, Okay, let's attribute credit to these many channels. But the pie remains the same. The objective of Measurement, the objective of optimization is how do we Increase that pie? To increase that pie that you see On the left-hand side, you have to have more robust Methods of experimenting, more robust methods of Optimizing campaigns. Now this is where I have To say, you know, a medium, if you follow a Snap channel, sub-stack, some of these links, if You are interested, I'll send them out to you. But this is a very interesting research done by three Independent freelance marketing analysts and agency guys. And what they've done is they've analyzed the top 40 Consumer brands and how are they running marketing analytics today? So if you see asos, stitch fix, netflix, uber, air bnb, And many more such brands are running at least a Combination of three measurement models. There is marketing mix modeling, there is multi-touch Attribution, they always start with multi-touch Attribution, then move to mmms, and then more Importantly, they're now getting into the space of what You see as cls over there, custom lift studies, Experiments that tell you what is the incremental impact of A particular channel, and then going back and optimizing Their campaigns, optimizing their mmm models, Optimizing their mta models with these. So in our case, for example, with epsilon, urban Outfitters is a massive brand for us. They spend approximately 30 million between publicists in Epsilon. They are leveraging our Insights to optimize their mmm models. I want to specifically read out some of the statements that Are being made. So stitch fix, which is A clothing, a parallel retailer, has clearly said Our north star is incrementality. Everything we do is based off of randomized control Trials. I want you guys to stay in that Randomized control trials in your mind, because that's what We're going to do a little bit of a deep dive in. Interestingly, for netflix, usually we'd like to run a Classic individual level randomized experiment, but Randomizing which individuals see a billboard as an Example is not possible. However, while we cannot randomly assign individuals, we Can randomly choose some series within which to show Billboards and other series to leave out. So you have custom lift studies, and then you have Geo lift studies, and then you have the recognition from Within a behemoth like netflix that we can't do Individual level sort of control test randomized control Trials across the board, so we have to have a mix of Measurement models. Going deeper into randomized control Trials, i'm just Going to give you one reference point. In 2020, cove shield versus covaxin, all of us Overnight became experts at what is the ideal size of control Group? Was the cove shield control group sufficient, Statistically significant to prove efficacy? Go back to that. That's essentially what we're talking about when we say Randomized control Trials. Group, test group, individuals randomly assigned to Either control or test, and then measuring the difference In performance, in conversion range between the two. But the starting point of a randomized control trial is A hypothesis. What am i trying to prove or disprove? So in our case, the hypothesis whenever epsilon works with A client is if we build a campaign, where each ad is Personalized to an individual, keeping that individual in Mind, delivered at a time when they're most likely to respond On a device that they're most likely to engage on, we will See a delta. That is our hypothesis. And sometimes our control test environments prove that there Isn't a significant incremental uplift, the number that you Saw for mama earth of 30.5%. But more often than not, you do see a delta. You see this hypothesis being proven that if you do one To one personalization, you do a segment of one Application through ads, display ads on the open web, you will See a delta in performance. You will see a delta in conversion rate. How does this pan out? This is the most simplistic way of representing a test and Control environment. On the left-hand side, you see this blank image. That's the control group. On the right-hand side is where we're running the test. The test where personalized one-to-one dynamically rendered Ad is being served. The control group has some key important, you know, parameters. Control needs to be of a significant size. There are enough conversions in there that you can say this is Scientifically valid as a result that this is equivalent to What you would have on the test group side. Number one. Number two, you have to have individuals assigned to either Control or test in a random fashion. Why is individual important? It cannot be e-mail ids or i have multiple e-mail ids. I have multiple mobile numbers. Or i have multiple cookies. Average such point that we see in this market is anywhere Between five to six. So would you assign one of my identifiers into control and Another one into test? That would basically contaminate the entire experiment. So that's why you need identity resolution, something that you Saw in the mama earth case study. So identify individual customers, collapse all of their Identifiers into one id, and then randomly assign them into Control or test. For as long as we work with clients, for example, we've Been working with clients for seven to eight years, that Control remains always active. This is not the same as running an ab test. This is not the same as running a hold out group. This is an actual always on control test experiment. The third component to this control and test environment is That both have to be similar to each other and representative Of the overall base of the customer. So you can't pick and choose let me put my best performing Customer, my high value customer base, my high clv Customer base into the test so i can up the number there. I can up the incremental conversion rate over there. That is where the randomization comes. That is where the farm equivalent of this experiment comes Into the picture. What it also does is it takes an offline and online data. So you are able to show causality, cause and effect of This particular channel in isolation both across in Store transactions and digital transactions. So this is where the causation part of our topic today Comes in. If you want to show the impact that a channel has driven in Isolation, you have to get a little more hardcore in terms Of the measurement methodologies, the marketing analytics That you are running within organizations. This is not something that is recommended for businesses That are already starting out. If i were in a d to c conference today, i would say Wait a while. We have that complex channel mix to warrant this kind of Investment, because this needs investment. The reason why most programmatic vendors don't offer Incrementality testing right off the bat is because it needs Data onboarding, it needs identity resolution, it Needs those capabilities that typically a cdp vendor would Talk to you about, and it needs very, very strong Analytics capabilities. And the last and most important piece to all of this Is like any scientific experiment, that's why i've got Scientific advertising the book up on the corner. Claude hopkins ran the first randomized control trial in Marketing in 1928. Is because it needs to be transparent, and a Marketer, an analytics head needs to be able to Replicate this. Every time somebody replicates this, that incremental Conversion lift needs to come out as is the way we Reported it. We're saying you should be able to replicate this just like Any other peer reviewed trial or experiment out there. So if you guys are still with me, this is what Replication looks like. This is what transparency looks like. This is what testing incrementality looks like. On the left-hand side, you see randomization. The fact that we're not gaming the system. The fact that we're not upping the number of high Performing customers in the test group is on the Right-hand side. If you see, this is an actual analysis that our team did For one of our retail or parallel retail clients, you can See that the split of customers across test, which is the Black, grey box, and orange, which is the control, is Pretty similar. That's what we want to do. Control needs to be representative of the Business, so does test need to be representative of the Business as a whole, and the customer base that the Business works with. On the right-hand side is the no bias component. We do not want to manipulate the data. We don't want to manipulate the way that the impressions are Getting served. So both in control and in test, ads are actually getting Served. So in the previous slide, on the left-hand side, you saw This blank space, right? This is an actual ad being served. It's just that there is no personalization in this ad. There is no one-to-one communication dynamic Creativeness. This is a brand neutral ad being served. This is the difference between ab testing and hold out Groups. You're actually serving media both in control and in Test. So now the right-hand side graph that you see, which is Impressions served, right? The orange and black lines need to overlap. This is where we're saying that the frequency of Communication to both the control and test Environments is similar. We're not gaming the system. And these are the kind of tests and health hygiene Checks that our clients actually run, right? Since you need to be a little bit more advanced in where Your business is before we get into this kind of analytics. Going back to identity. I think we covered this briefly. Multiple email ids, mobile numbers, especially in the Bharat side of our ecosystem where there's high turn rate In mobile numbers, right? And the kind of numbers that we are seeing with cookie Churn, five to six average touch points per customer. This is where identity comes in, right? On the left-hand side is the data that the brand or the retailer Knows. So since we're talking about mamma earth, on the left-hand Side is the transactional data, the customer data, the CRM data that they would be working with. On the right-hand side, i'll draw your attention to the Device profile and the content consumption. This is what epsilon adds to the equation. This gives you the single view of customer. So far you have to go into a cdp implementation or Data lake or a data warehouse implementation, all of those Three-letter acronyms to get a single view of customer. We've taken that single view of customer svoc and said 360 degree built into this programmatic solution. Why? Because that's the only way you can do personalization and Prove the hypothesis that personalization in real time Drives incremental conversions. Moving on. This is an interesting one and i picked it up not to Be a tag about the 10-to-1 iROAS that we delivered for Dominos, but mostly to prove that you have a Necessity for identity. The reason Dominos worked with us was because high Frequency, they have a very robust CRM. They had 30 million records in the uk market. But when they started working with epsilon, they Realized that about 30% of those records were duplicate. So how do you run an effective control test Of a house, custom, if you don't have identity baked into the Solution? 30% duplicate records. And with them, i think we've been working with them for about Four odd years, it's been a stable iROAS. All they ask from us is maintain the iROAS at a level where It's profitable for us. That iROAS number would change from business to business Somewhere, it needs to be a two-to-one for that channel To be profitable, for that iROI calculation to pan Out, others it needs to be a little higher. It's a function of average order values, frequency of purchase. And so it changes by vertical to vertical. Some of the other interesting metrics that pan out when You've successfully proven that hypothesis of Personalization. I'll draw your attention to the First column over here. This is an analysis from 2019 to august 2020, so almost, you know, a year plus. Attribution is limited in its window. It's limited to a seven-day, today, with third party Cookie deprecation or 40 days, at best, because it has a High dependency on cookies, right, third party cookies. Now, in the case of an identity-based programmatic Solution, you're able to do analysis like this. May 2019 to august of 2020, you're able to onboard Data and retain that customer base. So 9% of that customer base was still reachable at the end of The analysis period. What does this mean in terms Of conversions, which is the second column? So while your reach is 9%, you've been able to successfully Identify, continue to identify these customers over the open Web, you have successfully impacted the conversion rate. The conversion rate has seen a 5x growth. In the third column, you're able to see new defile customers. Now, because you can identify who your existing customers are, You're able to run more robust new customer acquisition programs. And the third one is the gold star for us, which is what our Clients measure us on, iROAS. Incremental return and ad spend. So while you have a, you Have this decrease in the number, you have a significant Increase in the incremental return and ad spend. So as an overall channel, over a 12 month period, 12 Month period, you have been able to successfully increase that Incremental return and ad spend. This again is an insight that Is important from a testing control perspective. Why for large scale businesses is important to run always on Test and control? Because you can do analysis like this. On the one hand side, you have a new customer base Analysis, which says there is a 20% increase in first Purchase to second purchase between test customers, where You're running personalization, and a 19% increase from first Purchase to second purchase for those who are in the control Group. A 2% at scale would mean Significant monies, especially with the volume we work in Within the indian market. So you have a new customer Analysis, what is the impact of personalization on both Existing customers and on new customers, and within existing Customers, you see a similar result. 33% of existing customers Purchase two plus times. Compared to 26% of control. All of this boils back down to running always on testing Control. So you can do all kinds of Permutation combinations. In the case of certain advanced Scenarios, advanced retailers, they're running mms that are Being optimized on the back of these insights. They're running audience segmentation, audience programs To drive customers with specific creatives, dynamic Treatments, et cetera. So there are various Permutation combinations possible with insights like this. And with that, i come to the end of this measurement session In the second half. Thank you for bearing with me. I'm going to wrap it up with a quote from george pox. So he said all models are wrong, but some are more Useful. So basically, depending upon Where a business is, certain models will work more Efficiently in those scenarios. But it's really what goal Businesses is taking on. If internally, our Businesses are aligned to report click-throughs. Our businesses are aligned to report easy to navigate Dashboards. C-suite is able to Understand clicks better than say incrementality or Scientific experiments. That's what you're going To go with as a model, as a measurement and analytics Exercise. But if you're moving towards Profitability, if you're having a conversation internally Around roi, what is the incremental revenue that Marketing is driving? What is the incremental revenue That a specific channel is driving? Then more advanced methods of measurement need to be Incorporated into the marketing analytics suite. With that, in case you have any questions around the Reports, you need access to the urls. I'll be available for the next two, three hours. If you have any questions on the three letter acronyms that Platforms like ours talk about cdp, dmp, dsp, ssp, mmp, M-m-m's, m-t-a's, please feel free to reach out to me. I will get you the free coffee. Thank you so much.