 So there we have Jacob who's gonna talk about the agile governance data governance a very important topic So thanks again Jacob for taking the time. I know it's a bit of an emergency right now But we appreciate you coming in and presenting this So if you would like to share your some thumbs up, so there are real people in the room All right, so Jacob if you want to share your presentation, please sure I'll get that started I'm also joined here by set them. I don't know if he's sort of connected But but set them has data engineering at Grofer's. I was the CTO there. I had a product engineering. I just left in July and so now I'm independently coaching different organizations And unlicensed corporate psychologist. I like to call myself So this is a talk that set them and I did a year ago And we sort of we're getting ready to do in March We're really excited about and then all this happened and of course like a lot of things is gone stale But I'm going to go back in time So all of this is where I was and what we were doing in March of last year And I hopefully it's still interesting and relevant to all of you So to get started with Can I get a up an up or a heart or whatever a thumbs up you guys do if you're in a committed relationship You're married or you live with somebody or you're sort of have a partner some sort I'm Interesting thing that people Sorry, let me close all those stuff too That people like when you're when you're a married couple you find about a lot of things I was also married at one point one of the things that I often found we fight about is where to get where to get dinner What to eat they would else have that experience of trying to decide what to eat You can up thumb me so we can keep this this thing going yet So I look fairly I looked this up and there's actually a cystic on it The average American couple spends five and a half twenty four hour days a year deciding what to eat It's kind of insane you think about it But it's not always such a simple question and a lot of people have different preferences So if I was to effort and ask these 28 or whatever people we have in the audience right now What your favorite food was I'd probably get 28 different answers and if I was in front of you I would actually ask you this question But when I try to figure out what all we should go out to dinner if we were all going out to dinner It would be pretty difficult to answer that question it would depend sort of who I liked the most in the audience and If you take that and you extend it even further and sort of look at you know what it looks like to Decide for an entire demographic for entire city for entire customer base. It becomes even more challenging and this sort of leads me to the central point of Where where organizations go to be dysfunctional which is that the more people you need to agree with the harder It is to agree. We all agree on that Yeah So when I what I before I worked at grofer's I used to work in B2B So I worked at a software company and we sold software and that is sort of like having a dog When you have a dog Your your customers sort of like that don't mean my customers were dogs or they were like dogs But when you have a dog the dog wants something it's usually wants to eat once you go outside and poop Once lick itself and whatever the dog wants you figure it out and then you get the thing done with the dog needs B2C is a bit more like ant farming. You can't You can't talk to your customers. You don't really know what they want. You can change their environment and see how they behave And oftentimes the ants are maybe more like hornets And your their order is you know two hours late and they hate your guts And you have to sort of figure out what to do this one for instance was a customer of grofer's It says are you dumb and nuggets? I asked you for different issues and you're fucking saying ask on chats. Do you think customer? What's interesting about this customer is he's a really good customer. He's a very loyal He's a very high frequency. He's a very high margin customer. And so it's not always obvious What someone is saying is not always correlated to how they behave And and it becomes very difficult to sort of use these maxims that we use as agile coaches Like why don't you talk to more customers do a story about blah blah blah? If you don't really know who you're talking about and why if and what their behavior actually is and a scale be to be to be B2C setup this becomes a big challenge So I'm going to talk to you a little bit about how those how those decisions get made in a scale B2C organization And one of the things which what makes this really important for grofer's is that grofer's serves a demographic of about 300 million people in India Grofer's at the time was the largest online supermarket in India And that's a giant swath of people of all income levels ethnicities religions Sort of educational backgrounds languages And so figuring out who to build for is a big challenge One of the signals that we use Because we can't talk to all of our customers is NPS I know NPS is somewhat maligned and a lot of people think it's useless and in cases It is but I do find it's a good canary in the coal mine These are the actual NPS course for grofer's about a year and a half ago And I know these aren't like you know Google and Apple numbers, but we deliver potato chips off the back of a motorcycle. So, you know What's interesting though is also how this breaks down and how you segment it this is the data for People who make a household income of under fifty thousand rupees or seven hundred dollars per month and over seven hundred thousand dollars Seven hundred dollars a month and you can sort of see some of the spreads and I'll highlight a few Product quality We have a 21 versus a two So lower income people find it 21 is their NPS score and for higher income into two customer support six minus seventeen Delivery timelines 14 minus 15. That's a 30 point spread. That's a completely different orientation to our product and I think what you can see from this is that there's basically two businesses happening here One is going pretty well and one is a complete shit show and that wouldn't really be a problem Except for the fact that our customer base is roughly split in half So this is the actual Demographics from like from the NPS survey itself and how people identify themselves And so it means that in an organization whenever you do any analysis on customer behavior on average Everything is going to average out You you won't be able to discover the benefit because it'll come out in the wash I need a new product feature you do for the some people it'll go really well for others It will actually degrade the service and so you do your analysis It looks like nothing changed and that's a problem and it's not just rich and poor When we looked at offers and discounts for instance young versus old we had a two and a 13 When you look at customer support an angry, you know an angry man who's frustrated on the phone is a minus seven This is an actual customer and he's he's actually a very sweet guy. He was really he's really nice I hate to use this picture and talk about an angry man But if you talk about a woman who doesn't have to go out and do the shopping there's most women in India Have to do it's a three. She's happy just to talk to somebody get it done. It's not great, but it's acceptable We're talking about delivery timelines in Mumbai where we have, you know, no nation can speak to this He spent half his life in traffic probably Or we have, you know, torrential rain. We have strikes. We have two highways trying to, you know Connect an entire metropolis. It's a negative 34 for delivery timelines in Calcutta Which is a more simple city to get around in and has fewer options. It's 25 The 60 point spread for the same product basically the same demographic And what happens here is not just that you don't know what customer to build for And I'm sure a lot of you can relate to this. It also causes internal dysfunction So if we talk about the teams within grovers, everybody has a different idea of what the problem is So it's sort of the five blind men in the elephant The people who are in the Calcutta team will say that everything is great with with last mile We need to create more regional products. It's a category problem people in Mumbai I will say it's all about last mile. We need to prioritize last mile We need to build a new module for rescheduling because of rain, you know things like that and As a as a tech organization who drives a lot of the improvement in the business We were always conflicted around what we should be investing in from a product standpoint and This becomes a big problem because we also learned the hard way as a company that you cannot serve all masters and I'm gonna show this is a an ad here This is a girl first first big TV ad and I hope this will play There You don't always get what you want, which is why we get what you need All right, is that pretty good thumbs up if you like the video Yeah, I'm just asking for thumbs up. I need validation, please and So, you know who else who else like that video Was our investors so grover's raised about a hundred and sixty million in the first nine months of its existence and Immediately, you know sort of was all over TV was growing like crazy But really it was only a 12 million dollar business and it pretty much went flat revenue-wise And became sort of the laughing stock of the startup community Growers have to lay off a lot of people. They closed a whole bunch of different cities This is just slightly before I joined and Because we basically what we discovered was that there really wasn't much of a market who wanted those things that we were selling in that case And so we And I'm gonna show you sort of what happened after that So what happened after that was we decided to pivot and there was a pivot towards a inventory led model Where grover's sort of ran it's instead of doing hyper local and delivering from anywhere to anywhere We ran our own warehouses. I'm sorry This is Not behaving and so this is kind of the ad that we're running with now and how we how we sort of think about the business now 250 gram Oh peace killer Chupel way Right and so it's a very it's not not as entertaining for sure as a message But let me pull this back up But it was far more effective So we went from a demographic which looked like this demographic which looks more like this we went from You know a lot of good ideas the things that people liked like fruit and veg or express service or gourmet products Into a great idea, which is basically taking Away from the solving the convenience problem mostly for men To solving a savings problem mostly for women who controlled the household budgets and tried to make sure every rupee went as far as it could And you can see the results in terms of our low customer account in terms of our our revenue in terms of our growth in our scale And to give you a sense of that scale really quickly You know two million customers in a month 21 cities. It's actually quite a bit larger than this now One of the fun metrics we came up with is it's called nips. So a nips is neither this nor this but Nips refers to noodle inches per second. So at gruffers clocks about 1700 nips, which is we sell enough instant noodles in one day that if it was one noodle it would extend from Delhi to Mumbai and back. I Think that's pretty fun. So When you're growing this fast everything's breaking. So how do you keep things? How do you figure out, you know, what you should be building for and how you should be Prioritizing your customers because what we don't want to do is go back to the old days of trying to serve everybody and serving nobody So we took a very hard right towards savings and we want to figure out, you know What is the part that we should be looking at what really works? And one of the ways we did this is with a framework called the reforge framework Which I won't go into a ton of detail on but I encourage you to look it up And essentially what it looks at is What early experiences correlate to long-term retention? So if you look at a retention graph which shows month-on-month how many users are left using the platform We see that it flattens out, you know, somewhere over here and this is gruffers sort of actual retention curve Most organizations have a retention curve, which looks something like this There's a huge drop at the beginning and then it sort of Peters and then it flattens And so we call long-term retention around month six The problem with looking at retention is that you never really know what causes retention until much much much later And you that's not any way to run an agile business And so we really need to know what's going to what are early signs of retention in the business And without that we can do a lot of things which pump up our revenue in the short term But don't necessarily lead to long-term benefit And the way we do that is I'm gonna just sort of I'm actually looking at the wrong one here I think we want to be looking at this one I'm sorry. I'm not as well prepared as I should be we Had a bit of a family emergency last night and things got a little delayed There that Here's what I was looking for And so we we we established a few different moments and just very briefly There's habit formation Which is the the point at which somebody has used the service enough times that we consider that they're very highly likely to retain We back correlate that so we say people who make three orders in three months have a You know a point seven five whatever correlation ratio to the long-term retention group So we try to get users to three orders in three months And then we back correlate that to an a-ha experience which is basically in the first order in the first month What are some of the experiences which are highly correlated to establishing a habit and then we focus on those and focus on the Users who need those experiences to occur and that's how we sort of back calculate retention This is Technical I will you can look at the slide. You can take a picture of it I'm not going to explain it in detail odds ratio is essentially a number which shows the strength of the signal So how strongly correlated for instance is an on-time order with habit formation if it's greater than one It's a strong signal if it's much greater than it's someone raised their hand. Is that do they want to say something? No full screen the slides sorry Yes, I will go as full screen as I can Given that I also want to show the I also want to see the chat room and now that I can see the chat room Please please feel free to do that. Let's let's do that much better. Sorry about that. Okay So this is our our our habit formation sort of correlation And so we look at that we sort of a stat we looked at different first-order experiences One of the things that we often hear a lot about is that in the grocery business you you should sell General merchandise products you should sell You know like things like you know Knives and potato peelers you should sell potato chips You should sell pasta sauces package goods the dish of dishwashing detergent all of these things the high-margin products Staples rice lentils wheat, etc. Those are very low margins sometimes negative margin products So we have a lot of push internally that way We should hope we should love we should always be trying to upsell our customers into buying these more expensive things higher margin things But the fact is that customers who don't buy any staples on their first order have a point five to odds ratio So they are negatively correlated with our retention group customers who buy staples much higher core much higher correlation So it's actually interesting to us that in our first order. We want to make sure someone buys staples They only buy general merchandise. They're odds of retaining with us are very low It's a bit counter-intuitive another one that happens all the time is Like Grofer sells a lot of its own brands our own brands that we own You know sort of like a Aldi's or a Trader Joe's in the US And one of the things which is often in conflict is the marketing team wants to push national brands No one wants to run a big campaign on Grofer's catch-up But on Kasan catch-up we say Kasan catch-up two for one you run a big campaign You get a lot of people coming in you'll get more sales that way And so there's always this debate around how much we prioritize our own products versus the national brand ones National brand ones do drive more customers But what's but when you do there when you do actually analyze the data and you figure it out You find that people who buy Grofer's brand products in their first cart are positively correlated with retention Those who don't buy Grofer's brand products have zero are negatively correlated with retention So any customer comes in for that Kasan catch-up deal and doesn't pick up any Grofer's brand products is better than even chance of turning on us and this is a really big revelation because It was very is a very hard pill to swallow because obviously we want to get revenue We want to sell we want to do promotions that work for customers But they aren't the right kind of customers for us and this analysis allowed us to see that We also use this analysis to look at For instance like product development. So these products flour tea salt have positive correlations with retention Whereas these are Grofer's brand products by the way ones we develop on the other hand These products had a lower correlation with retention a part of that could be a selection bias in the case of the toilet cleaner I think it's actually the quality of the product But you know looking at those scores we can sort of back count and figure out alright Is this something that we should be investing in product development? Couple other really interesting ones we did savings So this was quite fascinating if we look at others this charts a little hard to read But this is Grofer's whole thing is about savings. We're cheaper than the competition This actually looks at the price Comparison difference to big basket. So if the price is the same as big basket We have a point seven one correlation to retention if it's zero two percent We have a one point oh nine correlation slightly positive with two to four. It's very positive But then it totally flattens out. It doesn't really matter So cheap is only relevant up to a point and past a point. It actually doesn't help with our retention at all I mean this was this insight, you know, we save us a lot of money eventually Because we were discounting things assuming that the more we discount the better our customer retention is going to be but it's just not true Now what's interesting about this is this is for orders, which are 500 to a thousand rupees If we look at orders, which are sub 500 rupees very small orders. We see it's actually a negative correlation So once we get past four percent savings it actually dips and when we get to ten percent savings or more It's a one point one five. It goes way down and the The reason for this if anyone wants to guess you're welcome to hit the chat room so I can I can see you All right, you've been site solutions welcome to you, too So the reason this occurs is that when you run a huge promotion, which is just for Which is just which is just about one product the super savers people who are really really looking for deals all the time They'll come in they will buy that one product Then they will leave again And so we found a selection bias where we would run these campaigns for some fantastic deal And we get all this traffic and everyone page on the back and say job. Well done But when you look at the data those customers never stay with the platform And so that's what sort of occurs in these small orders with massive savings other things we learned which Weren't that important so Big basket came out with one day delivery guaranteed and everyone bananas they spent all this money advertising it turns out one day Or two day. It's got exactly the same retention Coefficient doesn't matter when you get a three days it starts to drop so customers actually don't really care by and large There's a set of customers who do but by and large our customers didn't seem to care It had no correlation to retention whatsoever So these are a few of the examples of things we found out They allowed us to reprioritize differently. They allowed us to think about our customers differently They allowed us as an organization to stop having some internal debates and start focusing on the problems we wanted to solve I think it's I think it's pretty cool work. It took a very long time. It's a six month project with You know Satyam who is heading data engineering as well as a couple data scientists couple product managers some user researchers Spending all this time, you know going through all this and one of the main reasons that this took so long That to get to this place was that we had really really poor data governance And we have really bad data. So it was hard to read the events we were recording didn't really make sense There was a lot of errors in the translation between sources to analytics tables We really didn't have the right data to start with and so the cleanup effort kind of prevented all this from happening And I want to just take a moment to not really focus just on That's obvious. I think everyone knows that you have to have good data to analyze it but I think the consequence for an agile coach or someone is looking to work with modern scaled organizations that at scale opinions aren't enough and What ends up happening is in less unless the data is clean and unless it's accessible by multiple people Then you will end up Centralizing decision-making whether you want to or not you can decentralize the org all you want You can create cross-functional teams which have autonomy, but if you and you can let them decide their own goals But if you want them to set intelligent goals, which actually makes sense and you want them to measure them intelligently Then they're going to use data to do that and I learned that Despite Organizational changes structural changes changes the way we work We weren't really able to give autonomy to the team unless we're able to give them the kind of analysis of the exposure to data so they can do their own analysis and Where they can make those decisions and find things out And I I have I think we're at like 1130 Satyam is here. I don't want to Like I want to leave time if there's questions, please start rolling them in Satyam did a lot of the technical work in terms of how a technical and organizational work to clean up a lot of our data stores I don't know if it's interesting you guys Maybe you could write in the chat we can go through these slides if you want. This is a lot of the Actual nitty-gritty of how we clean things up If you'd like to see that, please let me know in the chat if you'd like to just ask questions And we can and set them can just answer them or I can answer them. I'm also happy to do that You make it more interactive. So Any any takers? Okay, so like we'll go straight to two questions. I guess unless someone so how is different It's up them. Are you are you on the the Call also, can I hear you? Yeah, I'm there Yeah, all right, cool There's a couple questions here. Let's start with the first one which I'll do and you can take the next one So introduce orange cash around 2018, how is it helping is another significant factor for retention. It's a good question Orange cash was a promotion which was essentially a It's a basically subscription service hidden in a cashback You spend 2,000 bucks during a sale you get 2,000 bucks to spend for free But you can only spend it so much per month and it has a decay on it So you kind of have to keep spending with groper's it definitely influenced retention I Can't speak to the economics of it because I think that's confidential But it definitely increased customer retention among those who had orange cash during the time Yeah The station asking how is data governance different from master data management something do you want to take that? so I Would say that these are very similar things in the sense that at the end of the day Master data management is more about creating centralized data repositories Which can be easily accessed and and you can say that you know some of the portions that we did work upon is Around master data management whereas data governance also talks about not just from how How you are managing data, but how you give accesses to the different people so In a way. Yes, what we did was also, you know getting better at You know getting better at master data management so that people can access the right data at the right place I am not sure if I if I answered your question or not Jacob. Do you think? Yeah, no, I think that's I think what I would say the difference is is that master data management in terms of like schema control and And the definition is extremely important right and that's a fluid ongoing process It's not you don't design your schema once so here it is and here's your data mark now That's that we've modeled the organization is going to stay that way I think data governance is sort of all the things you do to make sure that you are agile enough to adapt To the new and out the new analytical needs that you have You know an operational needs as the business changes and grows But that it doesn't have to flow through one central entity that you give people tools to do that in a decentralized way Besides to clean the data what are the steps that you advise to create data governance in an regulated organization? interesting Yeah, that's a good question Yeah, I'm thinking about Again The idea is that you don't want to stop the access to the data, right? But you also want to ensure that it doesn't get misused so In a in an agile organization. It's more about being flexible in a sense that you understand that this team Requires this kind of data to get their project executed to get this goal To achieve their goal, right? So that kind of understanding and the flexibility in the data team helps different teams achieve their goal, right? While data governance would help you identify. What is the PI data non PI data? It will help you set up all those processes around data, but that flexibility Definitely helps you in achieving those goals for the different teams. So that is what that comes comes to my mind One example, so we talked about like control automation and speed But one of the ways of doing distributed control that's at them and his team worked up was they used a schema repository and Basically, it would do it to get repository where people would they can send pull requests up Etc for different kind of schema changes that they wanted to make so a team could sort of have a process by which they're making Changes and working locally with those changes But not but but not really changing the master the master data sort of interface for everybody else so basically having a difference between gold standard and and sort of prototype Events and event management. I think it's really important So you need to give people playground to work in and it can't pollute the main data source and has to sort of abide by PI I Sort of come security regulations, etc But giving that staging area or people can sort of have a middle ground and that's a complicated technical problem to solve We use segment for a lot of it. So they're a pretty awesome vendor. They got acquired a couple days ago So some of our friends are super rich right now to find out how they're doing what islands they're buying but but yeah Like Jacob said, right? So the idea was that at times in an organization a data team becomes a bottleneck and Everyone is kind of dependent on them for anything to happen with this like this kind of set up that Jacob is showing like this is a positive of the schema registry The idea is that anyone in the team they want to make a certain change They make a PR get it approved one from their team one from our team now It's a shared responsibility. You get it merged and all of the data ops around it, right? Whether the schema it's we have a lot of automation around it to ensure that first of all people are following basic practices They are following the right naming convention They're doing all of that and they don't become a blocker on us, right? So it's a very open repository where anyone can contribute it gets approved and a lot of automation once it gets merged Happens automatically. So that whole process gets moving out You take care of PII you take care of all of that just from this one single source of things and in this way I feel that an organization and different teams can work together on data Which I have usually seen doesn't happen in a lot of organizations and there's always a centralized data team and you have to wait for them for two months and You're not able to get your reports all of that with this It's very easy You want an event to be added create a PR get it approved and it will start flowing into all of your systems Be it your warehouse be it your segment and it gets validated all of that starts happening automatically So that's one of the things how you basically, you know Not just helping cleaning the data, but it also creating that governance Yeah, and you can see here like it actually we sort of in our schema management It's a team that it's tagged with so like there's ways to trace back who created this and why they created We first started cleaning up. That was one of the biggest problems Just like we have so many hundreds and hundreds of events who owns these who's managing them, etc So I think it's not about like gating so much It's about allowing people to do whatever they want but making sure we can trace back who did it when they did it And like is it an automating any kind of QA around it? Also known as internal democratization of data. Yeah, I suppose so yeah, that was a okay name for quite a while. So yeah, I Think that I think one of the challenges though with all of this I will say where I think we kind of I don't know fell short but are still are still working at least when I was there It's that old added you can take a horse to water you can't make him drink So you can you can give people all the data they want the cleanest thing some people don't actually want to look at data So people don't actually want to do analysis Unfortunately, so they think they say they do but really what they want is an analyst to just solve solve answer problem for them And I think that's a cultural change which which is actually harder than the technical stuff to be honest Figuring out which questions to ask. That's why the analysis I showed you I think was was important I think those frameworks for the kind of questions you want to answer are really important because it really defines how you want to think about your data And just just trying to like organize your data without knowing what kind of questions you're going to answer Is a fool's errand and won't accomplish anything I'm curious for anyone else out there You know when you're in your journey with your organizations, particularly as people who are coaches or senior leaders You go to a lot of strategy meetings And there's a lot of data torturing people have an opinion and then they grab a bunch of data support that opinion and sort of beat Everyone up with that data How have you how have you discovered like in your most the most healthy organizations that you've worked with How do you how do you find that people are able to? Sort of go data first as opposed to go opinion first Has anyone been able to crack that nut and found techniques to work on that? I know it's a tough question I'm not expecting to get any good answers Anyway, something something something to think about I think that's the next challenge to sort of for me that I'm sort of pondering Well, cool. Do we have any more questions or should we close up? Just wanted to jump in on the previous question Jacob. I think that's a very powerful question certainly something that many organizations Will themselves saying they are data first not opinion first, but I I think we're all biased And we seem to you know again the whole confirmation bias We seem to pick data that suits our mental model or our hypothesis So I don't know if anyone's really cracked that problem at least not that I'm aware of if you do Next year. We'll get a talk. I Think one of the I think one of the this is the thing to think about actually is how do we How do we go from answer first to question first? So instead of arguing about what's the right answer sort of get everyone to agree on What's the right question? Then it's like it's not about who's got the answer the question. It's about how should we design this experiment? Right, that should be the first step really Sort of changing strategy meetings into here's the plan to here's the question. That's the outcome, right? Maybe that would kind of help. I think you missed the morning keynote from Rajesh But you know having worked a little bit with him and his startups in the past and this current startup What I see him as a leader extremely good at is You know the whole 5-y and keep asking the why till he really gets to what is the question? You know, everyone's coming up with the solution but what's the question and That brings immense simplicity To to the problem that we're trying to solve because often we complicate things because we have a vested Solution within the problem statement And so some of the things he was sharing today in his presentation. We're really interesting around Why they went back to first principles to question things In a very different way and how getting people to solve multiple problems problems simultaneously allowed some of this people not getting to the one problem is people get to Attached to the solution And if you and his his perspective was if you get people to work on multiple problems simultaneously They don't get so attached to the solution They focus more on the problem. I thought that was an interesting way to tackle the you know What's the was the question not this answer or solution first? Yeah, that's cool All right. Well, I think we're we're out of time here and I appreciate you guys hanging out I'm sorry that I kind of rushed into this one I wish I was a little more prepared this time But I hope you still took something away And if there's other things that you would you know like to ask me or talk to me about Whatever, I'm very available and I love having chats about anything that's has to do with people enjoying their work life more and getting more done You can find me on Twitter. It's Jacob Singh. It's my name. It's pretty easy to remember. I think And I hope to see you all sometime whenever we can see people again