 Okay. All right. Let's get started. While we wait for Stojana too. So just a couple of housekeeping notes. Thank you everyone for joining. This is a series of meetups that we are calling making data science work. The idea is to talk to certain serious data practitioners who've been there, done that in terms of actually taking data science initiatives, putting them into operation and being responsible for their outcomes as opposed to just the way that they bring lots of technologies together. But Venkata and I, we are part of a company called Scribble Data and we are an ML engineering company. We're based out of Toronto and Bangalore and we basically help companies with generating data sets to train ML models. Now as far as why we're doing this series of talks, from every conversation that we have with the data science teams of our customers, we find certain persistent themes and questions that they have. And we have our own points of view to help answer those questions. These are questions about putting things into production, into operation. What are the most efficient parts? What are the most robust parts? But we know that there are lots of other people from whom we have learned to have trodden these parts before and we'd love to be able to share their experiences. For the most part, I think that this entire domain is actually fairly young and there's a lot of learning by doing. There's no one rule book by which you are able to put advanced data science or data science in any capacity into production and be very certain about exactly what's going to happen down the road. The format of this entire series of meetups is that it's, we want to get to the authentic voice of all of the panelists that we bring on. So we've kept it fairly conversational and free flowing and so you won't see any slides over here, right? This topic is going to be about setting up data science projects for success. Despite growing teams and budgets, very few machine learning models actually reach the production stage and there are a number of factors contributing to their success or to the failure of these modeling projects and we'll get into some of them. Some of the major factors are about framing and even setting up the context. So during the course of this conversation today, which will be, like I said, free flowing, we hope to touch upon a few of these topics. Like for example, what makes machine learning projects successful and why? How does one go about setting up these projects to be successful? I mean, there's a way you can set up a project to be successful. There's a way in setting up the scaffolding that no matter what you do, it's not going to work out. How does one manage and navigate the surprises and opportunities that occurred during the lifecycle of the machine learning project? So these are some of the topics we'll cover and just by way of housekeeping and some instructions. We have two casts of this going out. The first is a live cast. Some of you that have joined on this cast can see it live. And there's also a YouTube live cast that is going on. You're on mute by default. If you have a question, I encourage you to use the raise hand button and I will get to your mics that you can ask. I would encourage you to actually speak up and ask because this is, we're trying to make it more conversational rather than the usual webinars where we bunch up your questions in the comments and then get to them. This is intended to be a little bit conversational. Of course, there is some decorum there in terms of restricting your questions to just the one question and making it a question and not a comment per se. All of those little bits of etiquette, I'm hoping you'll all follow. And with that, let's talk a little bit about our panelists today. We have Goda and we have Strujana. So Strujana is today, I mean, she's a data science consultant and ML consultant, but she has vast experience working on large-scale learning problems in internet and enterprise education and e-commerce. Strujana, I think you've been at companies like Flipkart, Amazon, IBM, and some startups as well. So you've seen a whole gamut of things around here. Yeah, thanks for having me. Sorry, I had to drop off some stuff. Yeah, internet issues. Go ahead. Yeah. And I think what I'm really looking forward to hearing from you is the different contexts in which you've applied data science, applied machine learning, and how those contexts have shaped your outlook about the space. Goda currently works as the principal data scientist at Swiggy. She's got a number of years of experience building machine learning and optimization-based decision support systems. Before this, she's been a principal data scientist at Ola as well, and also at X to 10X Technologies. And she spent a number of years at Saber, where there was a lot of optimization work at Saber being the airline pricing mammoth, I would say. So that's a little bit about Goda and Strujana, but I would love to get into some of these details. I understand. Let me start with Goda. Goda, I happen to know this, but I'm sure a lot of people that are listening and would be curious. Your undergrad at IIT Madras was in biotech. How on earth are we having this conversation about data here with you today? If you can take us to some of the key points in your decision-making about your journey to get into data? Right. I know that's quite surprising to people who meet me first and get to know this. During my undergrad in biotech, my minor stream was operations research. So back in those times, data science was not even a term. It was more of statistics, operations research, computer science. So at that point, Saber was the only company who was hiring for an operations research profile. And that's how I kind of got into the game. And it slowly evolved into data science. So I spent around 10 years at Saber. And in operations research, mostly focusing on optimization and forecasting. And then crash-landed into the data science world at Ola. And that's how all this term began. Yeah, I mean, I can relate to Goda. Data science was not really used to, like you said, statistics and pattern recognition. It was data mining. Now nobody calls it data mining. This is not so cool anymore. And Srujana, I'd mentioned a little while ago that you've seen many different contexts of data science. And today you work as an independent consultant and you consult with a number of different companies. When you've seen the likes of Amazon and Flipkart and IBM, which are very different piece from the other two, and also smaller startups, which of these roles or cultures at these places have appealed most to you? Where have you found it? Well, maybe you can tell me about where it was difficult and where it was easy. Yeah, I mean, I should like, if I'm commenting on this, I should probably start from the point of journey into ML. So that was, like say, 98 roughly. So when I started doing speech recognition as an undergrad, then I went to grad school because grad school did shape a lot of my thinking around this. So that's where I learned about the rigor and stuff. And then there was Yahoo as well in the mix. So I would say like every place, gave me a little different skill set. So at grad school, you learn being attentive to the math, the modeling, the details, theoretical aspects. At Yahoo, I think the emphasis was more on the research part. And IBM was, I would say like an entry into the applied space. And Amazon was where I started learning about engineering processes and so on. And Flipkart was when I realized that, you know, these things don't come automatically. When you're in a bigger org, you take a lot of things for granted. It's when you go to a somewhat chaotic place, you realize that, you know, these things don't come on their own. And there's a lot of thought and there's a, you know, a lot of care that needs to be put in to make sure things are set up right. And I think, and that is evolved over time. So with each of these startups, when you're doing a zero to one, there's always a new piece to learn. And then I've worked with some startups and nonprofits. So there's little bits to take away. So I can't say there's just one place that I would take a lot from. But if I had to choose, I would probably choose Amazon. So, okay. How would I find it? Yeah. Yeah. That was a bunch of interesting questions. You just talked a little bit about the evolution of the space itself, from being labs, IRL and the TJ Watson's of the world to something that is more on, that is being used every day, right? The culture shift, the problems shift, the kinds of challenges that you have and so on. We'll get into some of those things. Yeah, continue. So I was going to ask, and this is a question for both of you. I'd love both of you to be able to answer one after the other, maybe. At these various organizations, how do they frame the problems that they then deploy the data science resources to us, right? I mean, it's time, it is choice of tools and technologies. And it is when you start to work on one project, effectively there's an opportunity cost of not working on something else. How do they frame these problems? And then I guess, let's lose the data science team on to them. If you can give us something, but how that gets chosen and set up for the data science teams. Krishna, you want to go? Go ahead, go ahead. First you will go in that order. Yeah. Alphabetic. Okay. So I've kind of seen different approaches taken across different companies have been engaged with. So when it comes to more mature and global companies with existing products, usually it's more of a product roadmap view that is taken and every enhancement you put in has a role for data science or operations research to play with. But when it comes to more of a startup ecosystem world, they're kind of the owners to even frame the data science problem lies with the data science function in a way. So while I was working with several startups and with the Ola experience is when we kind of hit upon this approach of what we call drive train model. I can share more about it later, but it's an approach of kind of bringing all the decision makers because usually startups is a small group or small group of leadership trying to solve the core business problem. How to translate it in a very structured manner to data science problems of importance. Because otherwise you usually go the other way around where you have some data and you're trying to find out what problem can be solved with data. But a reverse approach of given a problem where does data science have the best opportunity to play a role in having impact. And that worked very well with a few startups when I was working at X210. Yeah, I mean I agree with Goda largely, right? So in big orgs, there are things already going on. So whatever you're doing is typically additional, not always, there's some orgs that do set aside time and space for people to start from scratch and initiate something completely new. But by and large, there's a lot of problem formulation happening even in smaller organizations, right? And it does start really from the problem. You finally, right, there should be like, it's like the necessity is the mother of invention. There should be a pain point which will answer a lot of questions. That's the right way to start, really. Yeah, yeah. Typically what, in this, I would say there's this, this is something I find personally useful, like a roadmap to ML application activity, like you said, what is the framework or how do you capture this, right? There's sort of a life cycle. The first stage is like really the figuring out the problem. Once you figure out the problem, then there is diligence around data, and there's modeling, there's all of the engineering that comes, putting things together, evaluating deployment and so on. But I think the crux really is getting the problem right. Yeah, and we could talk about each of these in more detail if you wish, but yeah. I would love to learn, yeah, feel free to get into it. No, so I was just thinking, so how is it free? So there's like, okay, let me step back a little bit, and it's not like really particular to data science, but I think in every domain, right, there are these very high level abstractions, like the why, what, who, and when and how, right? So the why is in some sense, captures like why you're doing this, the nature of the problem, you know your goals and so on, right? And what is like the different layers of knowledge that go in and who are the different players and when and how is the process. So what I've been talking when I said life cycle is more of the when and how, but there is a connection to why you're doing something, the problem itself, right? And I think this is pertinent not just to data science, if you go to physics, you go to some entirely different domain, a lot of this will generalize and translate. Now the ML space in particular, right? In ML or data science particular, the key roles are the product engineering, the data owners and the data scientists, right? Product, I'm like, because I don't have so much visibility, all the business and product people and just putting them all together. And typically the formulation of the problem is the critical piece where we all get together just like Goda said, right? We have to figure out what is the problem that really needs to be solved? Why do we want to solve it? What are the metrics around it? How do you represent? How do you make sure there's impact? And you know, how do we set ourselves up for success by making sure data is there, it's all there in the right way and so on. Then comes the next bit about like, actually making sure that data is what it is, right? And then there's automation, the choices of model. I'm not getting into detail because there's probably a lot more to talk about. Before you jump into it. Yeah. I want to zoom in a little bit on this because when we go around talking to people, right? In about three minutes, our, the different companies that we talk to, the executors jump from, I don't have ML2, I need a bandit model, right? Bandit model and it will do this. I see a lot of, most startups are still trying to figure out what problem they're trying to solve, right? And that ambiguity cannot be wished away to be there for a few years in the journey of the startup itself. So even when you have so much ambiguity, do you feel that you can still take constructive steps and identify a reasonable ML problem while they're figuring out the product itself? Yes, I mean, it's largely doable. They're not, their business model itself has not matured. Yeah, yeah, I agree with that. I mean, Godai, please do add to this, right? So what I think I look for, you might have heard of these things called like smart goals, right? Smart goals are like, what is it, specific, measurable, achievable, realistic. These are certain things I've taken from Amazon, right? And tune to whatever I would want to live it. So smart, measurable, achievable, realistic, time up. And I think there's a structure like that that works for problems, how you pick problems, what you can work with, right? Whether it is ripe enough or ready enough for you to take it. Like you said, the business model is not yet ready should you start working on it. But oftentimes, right? As a data scientist, you're also helping tune the business model, especially in an org which is at the stage where the business model is still not ready, you are shaping it. So it makes sense to get in early, but on the specificity, right? We just need to look for some manageable level of ambiguity. It doesn't have to be super specific and clear, but you should at least have manageable level of ambiguity where you kind of, you know, have clarity on the known unknowns at least. So that is what I think that makes it. And then there's also this thing about measurability, right? Even if we don't know right now whether we can measure it, are we, is it possible to do it? Can we define metrics which are measurable? Go the, we do take up a lot of these big problems. So you should, yeah. Yeah, actually I would like to build up on like what Srejana mentioned regarding smart goals, right? One of the important, so even the drivetrain framework starts with the goal. So now one of the important questions to ask at the smart goal level itself is to say what is the objective and what are the constraints, right? Because for any downline solutions that you build this clarity is very important because often all the different goals are not achievable in tandem. It's not feasible. So having one primary goal that we are chasing and having guardrails around the others makes it very clear that every system that I'm building into this whole, what we call the product machine is kind of chasing that goal and keeping the guardrails on the other metrics. And usually from the goal, the question to ask is if we have to achieve this goal, what are really the levers that are under our control as a business or as a company, right? Now these levers are basically the decisions you're making. The decisions could be made by humans. The decisions could be made by machines, right? At the end of the day, all the data science is doing is powering these decisions in some way. It could be analytical approaches, you know which shows up some metrics and helps a human make decision or it could be machine learning models that are enabling machines making decisions. So having this clarity is what helps you identify those low-hanging fruits really that you can, you know you have more confidence in achieving an impact even for a volatile state of a business. Yeah, and I want to add that in lot of the small or like startups, right? The business model is not clear and they're actually chasing multiple opportunities. They don't know which one will pan out. So you want to keep certain doors open. Just like, you know, a youngster at the beginning of a career wants to be trained for multiple potential paths that they could take up. If the same startup is like a baby, it wants to potentially grow in whichever direction, you know, it becomes more possible, right? So we don't want to, I mean, while doing it the guardrails like Goddard said, set you up to explore some door. Maybe that's not the primary, but you know, keeps you ready in case that's the one that catches fire, that actually catches fire. Actually, that reminds me, go up when you were at Ola, I mean, Ola grew so rapidly as did Swiggy. Can you talk a little bit about the culture of good enough to allow experimentation, the culture of tolerance in outcomes, which I'm guessing would be contrasted against your experience at Sabre, for example, just because it's a much more mature business. So if you can tell us a little bit about that, that would be great. Like you mentioned, when I was in Sabre, the approach was usually, you have a timeline of three months or six months to achieve the best or the optimal solution to a problem. Whereas in the world of Ola or even Swiggy to an extent, learning fast and implementing things fast on the ground because by the time you come up with the solution, maybe it's irrelevant, right? Maybe the problem itself goes away. So you say that. So if somebody, you know, making that trade-off between what is that good enough solution you can experiment with and learn quickly and improve, then, you know, trying to hit the North Star right away in your first attempt, right? That's kind of the mindset shift that I saw when I moved from Sabre to Ola. But at the same time, I also realized when you're in that mode, often the pitfall is that the need for speed makes you implement anything and everything fast, experiment and see, but there are a lot of avoidable iterations there where a little bit of upfront thinking of maybe spending a few hours in a structured way and then saying, okay, this is the whole gamut of the problem to be solved, but this is the MVP version of it. Just some upfront thinking will help us avoid a lot of, you know, chaos or reactivity once you go live and then you're trying to, you know, change the wheels while the car is running, kind of thing. Can you zoom into this a little bit? There are upfront thinking. So by avoidable iterations, I think it will be a very concrete takeaway for many people on this call. Right, so for example, when you're thinking about a solution, right? So it's often when you want speed, you might end up thinking only from a backend module perspective. So for example, let's say you're working on a module where you're trying to build a simple matching algorithm where you're trying to match customers to the services. And then in that, you're not really thinking 360 degrees to what are all the pitfalls. For example, what you show on the UI is a pitfall, right? While this whole algorithm is running a fancy thing to come up with the best matches, if you don't have the right messaging on the UI of the app saying, why are you making that customer wait? For example, the customer could just, you know, drop off and you don't see the impact that you saw offline when you did your simulations or some processes, right? So just everything needs a 360 degree upfront view and you consciously decide not to take care of some things in your viewer, right? So that's the approach I'm trying to talk about. Yeah, I mean, I want to mean the great point, Gouda. And I also wanted to add something here. Like typically good enough is also, you know, I mean, it's not like really global optimization that we are doing. We are optimizing among few feasible choice in front of us. There are, you know, maybe two or three directions one could take and then you're trying to do that. And typically it's a question of like figuring out between the resources that you have, between the chances of success you have, which is the one which gives you the biggest delta, like biggest expected delta. I mean, there's a level of risk with everything that you're trying to do. We don't know what the future holds, right? So there's a further time you invest, which are you on, let's say, time horizon of let's say in the next one month, what is the biggest jump we could get? And we would do that. And again, it also depends on the risk taking behavior. So it's like this bigger companies like Amazon or Google have the pockets, deep pockets to take bigger risks. They can, you know, try to do, they can do a big jump, whereas smaller odds will not be able to do that, right? Because, you know, yeah, you're flat till you see something happening. So that's why, you know, that, you know, the level of your risk taking depends also on how you are resourced. And then given that this is the runway you have among the two or three options that you have, you pick whichever makes the most sense, right? So now, if I was a deep... Sorry, and also how reversible the mistakes you made... Exactly, yes, yes. How reversible? Rollback capabilities gives you confidence to experiment. So it sounds like you go ahead. I was going to say, on the rollback capability, I will come to this in a little bit because I also want to talk about what it means to be within thought-through automation so that you actually have this kind of capability. But before we go there, I have a question. If I was a data scientist who had the benefit of having heard all of these things, you know, good enough and framing the problem, understanding constraints and not trying to tackle all goals simultaneously, in my mind, that would be one half of the problem. The second half of the problem might be communicating that properly to all stakeholders so that that culture is an accepted one. Is this something that you guys faced or were you blessed enough that where you got parachuted into everybody already knew and they're like... No, never. Communication is an art. I mean, it's a... Sorry, Godha, you go ahead. You were supposed to alternate. Go ahead, go ahead. All right, like, I'll go ahead. I'm just a half-strong views on that because, I guess, being a parent, you learn this and communication is not trivial. It's a two-way thing, right? It's a two-way street, really. And I feel like, you know, for every communication, it's not just about sharing or gathering information. There's some underlying motive. So there's something that you want to achieve because you're communicating. And the first piece of communication is to know exactly what we want to achieve. And the other piece that helps in doing this effectively is to know whom you're communicating to, right? What drives them? What are their interests? What are their pain points? At least to the extent that is possible. Of course, without talking, you wouldn't know this. But to spend a thought or two, like, to be empathetic about it helps because you can formulate your argument or whatever it is you're trying to pitch in, information to them. And for me, honestly, I have done this badly a lot of times. I still do it badly. I still keep learning. And I've been taught, my friends, to that, yeah, what you said makes sense. If only you had told me like this. So that's there. And yeah, go ahead. Yeah, can you zoom into that? Because this is a real learning. Basically, you've made, you've hit a particular obstacle or you've learned something and then you've amended something. Other people are going to hit that same thing because I'm sure you've put your indents into it. So please zoom in. So let me, so this was, so we were working on a problem. And this was a little bit of a high stakes problem at Amazon and we were missing a potential deadline. So there was, because of that, and because of some kind of miscommunication and things not being handed off, data not being ready and whatever. And we had to communicate to the business unit that our models were actually not progressing because data was changing. There was this expectation that you would always get improvement in performance, right? That it's always a monotonic function, but that's not always the case. And the person we were communicating to was not exactly, he was not well versed with math. He came from a very different background. So to him, it did not make sense that we got good performance in May and we got poor performance in June, right? And that our numbers were going down. So then one of the things we did is we sent him a very detailed report and did not make any sense for it. So because we thought that, yeah, let's over communicate, let's lay it all out. And it did not make sense to him because there was a lot of detail. And then I was coached by my manager and others that this doesn't make sense. This is not what that guy wants. He wants something else. He wants to, so what he wanted in some sense was what to expect. What to expect going forward and why this is happening and whether it will happen again and again, whether we will actually hit the goals on time for the end of the year, what he could do to enable us to work well or unblock us or get us more resources or how he could communicate this to people he was reporting to our upstream. So we had to equip him with that piece of information. So that was the kind of coaching I needed at that point. But in general, I meant like, I had to think from that person's perspective. What were the gaps that he wanted to fill and know about? So, yeah. Also, there is a concept of having a seat at the table. What I mean by that is, so I've seen two kinds of working of data science. One, where every problem comes like an ask. Like this is an ask for data science team from other teams. Whereas there is another part of the thing where data science is also having a seat at the table and defining the askings, right? And the way to get to the table is to speak the language of business and language of product. The moment you only talk about MAs and this is the mean absolute error of the model, but not link it to what it means to either the monetary benefit of some top line and bottom line. You don't get that seat at the table if you're only stopping your language at this level, right? So that's also a part of it. And also, it sounds like we have to be able to communicate the total lifecycle of the work involved. Oftentimes, it is assumed that only modeling is the part, but there is a whole lot of other stuff, including coordination and this failures and learning from it. All of these have to be budgeted for at some level. So I am oftentimes concerned that the expectations from the data science project and the team itself is unrealistic. Have you learned some way of coping with that? I have an example. Yeah, yeah, go ahead, go ahead girls. So often, I come back to that point of primary and secondary objectives. So often you have multiple metrics for a given business function and we were designing an optimization algorithm that was optimizing for one of the things. But every time we went live with some version of it, some other metric got impacted and it was always not the right result, right? So in this part of the communication and how you need to own the entire pipeline, what we had to actually do was simulate different points and explain what the Pareto optimal curve is, where if one objective is on this axis and this you have to choose an operating point towards the message that had to be given because at any point, you cannot say that's the best point because there will always be another solution which could be sweet spot for one pace of your operations and again, you have to choose a different operating point as you reap your business, right? So this whole part of the communication needed what you're saying, like owning the entire life cycle, the downline impact and talking business. Let me also add one more thing to what Goda said, right? So I mean, there's the math and the modeling part is this but there's also a lot of uncertainty and in some sense, right? When you're doing something for the first time around, there is, it's really that you don't have to scramble. This is a system you're building with so many moving parts. So things are bound to fail. It's just like, I mean, if at all you want me to give something, like, you know, a little bit with a little more of certainty it would be that something or the other will break, right? That's way more likely to happen than things would go smoothly. Most people do not have that expectation and they don't have a sense of how this would go. Like one of the, like the most recent thing that I'm working on, you know, some of the product people who haven't developed a modeling application before are very concerned that we are scrambling and they don't know, you know, if this is like this or it is going to be forever. I have to tell them that, you know, yeah, this is a stage, it will pass. We are going to get better because we are going to figure out what pieces have to be fine tuned. So I think what helped and I was, and this is what I did a couple of days back is I just told them that this is a typical cycle and this is the stage we are in right now, right? We have teething problems and we are going to get through these teething problems as long as we get through it properly in an organized way, we'll get to the next stage. So there's like light at the end of the tunnel if you do certain things, right? So not to worry and, you know, that sort of communication is necessary. But I think to even be able to communicate, right? We ourselves need like some kind of like a rough map of what could happen or what are the five, six things that happen, like for example, in a fast moving company or like Ola, right? Where things would change, like you said, requirements change. How do these to have a mental model of how things evolve over time? Like how the first set of iterations look like in a startup? How do those iterations mature later on? That, you know, there's just in some sense like a rough knowledge of that evolution, right? It's helpful because you can tell them that, yeah, this is what's going to happen and you need to convey it. You need to tell them the story that this is how it can evolve. Actually, this, yeah. Sorry, I'll just go ahead. Sorry, let's make one point, sorry about that. The earlier point that Goda made about having a seat at the table and then the joint points that both of you were making about the kind of understanding of business objectives, optimization, and then being able to take that into a business language and being able to communicate that back. If there are any folks in our list of attendees, it sounds to me like the marriage of these two skill sets, one, obviously the fundamentals of data science, but two, the ability to talk about what that means for the business from a cost perspective across various parameters of whatever cost means. That gets you the seat at the table and then if you are interested in management and moving up the corporate ladder, it sounds like this is one of those key skills. Yeah. Okay, I think I'll just take that as a yes from everybody unless my computer's frozen. But on this point of what it means to actually put together a team who leads the team and stuff like that, I would love it if both of you or either of you could speak a little bit about a team structure. Now you've seen so many different team structures and what has operated well. Does any particular team structure seem like very ninja, meaning it was most efficient, it was absolutely perfect, or any learnings about setting up a data science team structure? So it actually depends on the stage of the company in some way, right? So when you start off a data science function, maybe when you're in a series B kind of stage of the startup, it's always good to start with a more central team because your individual functions haven't matured enough and it is important for a function like data science to have that broader big picture view so that you can link different pieces to build the initial Lego block in some way, right? But as companies get more matured and there are different business units and probably at an Amazon scale, then since you have built that core in a strong way, now you can, that hub is built and now the spokes can emerge where they're embedding data scientists into business functions themselves to work closely and solve the problems. That's only because the base architecture and the base core problems have been solved efficiently. So depending on the stage of the company, at least what I have seen working is evolving the structure according to the stage. Yeah, I mean, I think I agree with Godha that it largely depends on the nature of the company the stage and so on, right? But typically even in large companies on a data science project, right? Like, you know, some of you are solving a very specific, we're given the scope. There are broadly speaking, I would say like four types of roles, right? Someone who's responsible for the product, someone who's responsible for the data and someone, and you know, it could be the same person. I'm just saying it's product, there's data and then there is engineering and then there is, you know, all of the modeling. And yeah, again, depending on the stage, you would need someone who's like very hands-on or someone who can like look at things with like a long horizon perspective, right? So I would, yeah, it's tough to answer this, like in a startup, right? So it would be very early startup, you probably want one person just to fill these different roles, right? Like, you know, one or two to do this, all of these roles. But like a good setup in a like reasonably mature company is to have at least one, like at least two people in each role, one person who is, you know, much more experienced, might be spending less time on the project, but much more experienced and one person who's like really hands-on. And- Let me frame this in slightly differently, which is, you know, because you talked about different contexts for different stages of the company, every company obviously will evolve through these stages. So if you imagine you are at the company through its own lifecycle from when it was a fledgling company till it became a mature company, what markers would you look for? Supposing you were responsible for the data science function, what markers would you look for to say that at this point it needs a rethink of the structure of the team? Is there any- Right, I think Goda might be better positioned to answer this, but I will, I'll share what I, yeah, so let me share what I can hear. See, I separate companies more into like the ones where the requirements change a lot and whether they're trying to learn and refine their process or not, rather than just based on their size itself. Am I lost? Am I still there? I would say that a better way to divide a company is probably in terms of like, you know, whether the requirements are, I mean the company or the data science function is in terms of whether the requirements are changing a lot or not, and whether you have, you know, bandwidth and the right DNA and the incentives to get better over time, right? So if you were to split things along these dimensions, right, there would be some large companies who haven't really learned to, you know, learn from their experiences very well. And one of the things that I think separates like companies like Amazon and Google, they've actually gotten very good at like learning from their experience and set up processes to do that. And on the other hand, like requirements changing basically means that, you know, the data science function needs to, like the problem formulation becomes way more important. So the skill set that you need for that is very different. And, you know, the intensity is, so you would invest more in that particular space, right, that sort of skill set and also you would invest more in like Ninja, like people who would quickly, you know, deliver results in that sort of setting. Whereas in a place where requirements don't change and you have some like a longer runway, you might go for a slightly different skill set. Like in a place where you really can, you know, or try to work on research or like very big gems, right? Then you would invest slightly differently. So yeah, I could mean, you know, if we had the bandwidth and want to delve and I'll also take a minute to think through and say something, but you would have a better answer I'm sure for the art of evolution, yeah. Yeah, I was thinking more about it in sequential order in terms of the skill sets that a company would require, right? Because initially when the business is just set up and it's growing fast, just the trackability of how your business is doing and why something is happening becomes important for you, right? Which means that time the skill set required is somebody who can pull the data and convert it into a insight that looks like a fact. More than statistical confidence, what you need is that storytelling capability of this is what is happening, right? So there, you know, people with that analytical background and analysts or analytics team becomes very important. But as you evolve before you hire data scientists, what you need to hire is people with data engineering skills. But what I mean is who can convert your raw data being instrumented into your production DBs to something like a well-designed data warehouse because that's the ratio of data engineering, number of data engineering people to the other, like the data science. The data engineering people come first, even before you hire the data science. Sure, sure. It's, I've seen companies just starting with two people, one who is more at, you know, probably an architect who has the data architecture view of it and one who is very good at, you know, churning out ETL scripts, right? Our two-member team is where people start off with so that if you have that well-designed data warehouse that considers all the metrics that the business is viewing, then getting in the data science team makes it very efficient for them to churn out models quickly. And the skills that is required when you're bringing in the data science team is the translation skill or the formulation skill of converting the business problems in hand to something that can be turned into decision-making systems where you can make machines. So the way I differentiate analytics versus data sciences, you make humans make decisions versus you make systems make decisions in some way, right? And finally, like she pointed out, once you kind of go into that maturity level where you have these systems and you have to churn out models in like, in a month you have to churn out so many models is when ML engineering kind of becomes very important at that point, having the right platforms for your deployment, right feature stores and that team then kind of becomes important as the maturity. Wonderful. One thing I want to add, I'm sorry Indra, I just want to add on this thing about like data collection, right? In a lot of places I've seen that there are data engineers and I've seen this at Eggstep, I've seen this in, you know, at Amazon. They think they're doing good job of data collection but there are some blind spots. Like even now I'm working on a project like this COVID test allocation, right? They're collecting the positives, they're not collecting the negatives. Nobody's consolidating the negatives, right? And then it's just like handicaps as from building a classifier. At Amazon, when they were collecting the clicks, right? Where people were, you know, when you have a product recommended, right? So they were initially collecting the clicks. They were not collecting what else was shown. I mean, they were not logging it. And when it comes to building a model, you can't really do that. Same thing. That's like a mistake I've seen repeated like pretty much in every other recommendation system that I've seen. When we were working with starters, one of the main points we were trying to make was that for data science, you can probably wait for a little, but for data strategy, you have to start thinking about it. Yeah, so the data strategy is not something, typically a data engineer is occupying. You might not hire a person. You still need someone who is enough knowledgeable about data science and what is needed to get there to collect all the pieces. So that is something you can't skip on early days. Because otherwise you spend most of the time in estimating something that could have been instrumented. That could have easily been instrumented, exactly. A very good example, like in Swiggy case itself, right? We have this feature called mark food ready, which is like, you know, when the food is ready for you to mark it ready. Otherwise you would have spent so much time just estimating when the food would have been ready without having ground signals. Like when did the agent reach there? When did he collect it? Was it ready before that? Just little points of instrumenting and logging can make your system so efficient. You can actually focus on problems that you need to predict. It sounds like at least one of the things that I'm taking away is that this whole thing seems to be an exercise in a very disciplined thinking and approach and also systems. I mean, if you have to build a process that is responding very fast to the results at every step, that means both at a intellectual level, at a process level, at a tooling level, all of these things have to be agile in some sense. So how do you, at some level, it seems to ball down to the culture of the team. And at some point, we should talk about that as well. How do you instill that kind of experimentation, iterative observation, thinking through culture? Yeah, I mean, you mentioned a lot of things actually, Venkta, but I want to take one of those pieces which I think is important and often overlooked, but done well in somewhat bigger companies for people who have that sort of experience. That is like early automation. So there are certain things, like problem formulation is one place where things could go wrong, data, everyone knows is messy and comes with its own set of problems. But what people don't realize often is this importance of automating early and not waiting for too long because there's certain things you cannot control. Like data issues, you cannot control. It's going to be messy. So all you can do is set yourself up to iterate faster. If you can iterate fast, it will kind of reduce that impact of all these data quality issues and stuff that are little out of control any which way. And the other thing about when you're trying to do these automation, it helps you get better over time. You get more efficient. You tend to be less error prone. So that's something that's like super important because if you don't do it right, what tends to happen is we spend a lot of effort, ghost waste, the model babysitting or the, essentially like modeling it like manually, right? That takes a lot of time and it becomes throwaway work. And then you haven't really gotten better over time. And you don't really get time to do important things like model diagnostics, thinking through the experimentation framework and so on. So I have a question for both again, both of you. Just before I get into the question, I'm just going to let all the attendees know that if you have any questions, please do use the raise hand function and I will get to your microphones so that you can actually ask those questions like. And while we wait to see if anybody raises their hand. So, you know, we've talked about a few aspects of the life cycle and as we get to the end of our session, I would love to understand a little bit about what's happened, what have you seen that's happened after some of these solutions have gone into production when something's gone wrong, not when it's gone right, when it's gone right, I think it's wonderful. When something's gone wrong, are there any lessons that you'd like to share with the larger audience? We've talked about rollback, quickly being able to roll stuff back if you have a good framework, but anything else? One thing that I've seen is the measurement and the attribution part of, you know, data science models going into production. So often in a very connected environments, whether you take mobility space or logistic space, they are very connected environments. So isolating your control and test is a very difficult thing to do, right? And people often, if you don't think through enough about the experiment design as to how you're going to measure the impact, often you end up doing wrong experiments and wrong conclusions whether something has an impact at all. And especially because of the interference that you see between control and test. And the second thing is, if you have to really isolate and attribute the impact, how do you do that in those connected environments where you might have to, you know, run control on one day, test on another day, more like on-off situations. But the organic on-ground situation is very different between the control and test days, right? So that means somewhere we need to build attribution, like you need to trust a model to see if your model has an impact, where you're building metrics as a function of the organic variables of a day and seeing, you know, between your control and test, where does it lie with respect to your prediction as to how that they should have behaved in a control scenario, right? That's just one example of how experiment design and attribution of impact become very key when you roll these models into production. Yeah, I mean, I think I'll add to what Goda said, like typically when you, you know, release a model in most mature companies, it doesn't happen, you know, like right away. You don't suddenly ramp 0 to 100. You do a careful A-B testing. Of course, there could be issues with the experimentation assumption, the framework assumption, then the design and it might not be quite right. But typically the ramp up is low, but there are certain events that do happen, like when demonetization happened, right? So a lot of assumptions about the data would suddenly be different. And then you don't know what to do, right? Like if it was a rule, okay, you can go and tweak it. But if it's a ML model, you typically don't know what to do with it. So what is a good practice, you know, to either do is to have some kinds of monitoring, some kinds of guardrails around the models themselves and what, how it is affecting decisions. And like, you know, default to the human or generate early alerts whenever something is looking very different. So that's a good practice. And yeah, we've all been there, like something goes wrong and then you pull the model out and put in some just a placeholder rule. So that does happen. Actually, in terms of when something goes wrong, I think this, this time more than any other will throw so many models out of whack in terms of behavior and all of that. And the retraining that needs to happen. Do you find that models in production that organizations have thought through how to be able to monitor and support those models? Or are there things that you could be doing better? So on this, right, see, right now it's a very volatile time. There's going to be a lot of data changes like in a lot of these domains, e-commerce, you know, Swiggy like food ordering and so on. So, and even with the forecasting work that I'm trying to do, right? So what we are trying to move to is more like a scenario modeling rather than just trying to assume that the status quo will hold and that will be the thing to base it on. We want to model not for one setting which is how things have been now, but for a few different possible scenarios. And, you know, that's a slightly different paradigm that you need to think about because of the type of volatility you see. And I think there is a simpler way to do it which is to essentially like, you know, you know, if they're like in India, if there was a way that, you know, there are different scenarios that are holding in different places, right? Certain things have opened up, certain things have not opened up. And it's important to capture those variables, what is, you know, happening on the ground. And then when something, you know, like a red zone turns to orange zone, you know that the behavior will change in a particular way, right? So that's one paradigm to keep in mind. I mean, from a modeling perspective, but like wherever it's possible for a human to intervene, I think, you know, whenever things are not working the way they should or something is like, you know, blowing up some metric is not like in the typical range, you need to step in and look, take a look. But it might be tough to do it on scale right now. Go ahead, you would have more. I just want to highlight the importance of alerting and monitoring frameworks that you would have invested on before, how it's helping during this situation as, you know, we see on-ground behavior changes. It's a, you realize it's an investment well-made before and I'm experiencing that at Swiggy, right? So that is just a fine number. Yeah, Indira, we are about a minute away. Yeah, so yeah, we are. So I guess one of the things I would say is, thank you first of all very much for doing this. I am going to ask all members in the audience if there's any one urgent question. Otherwise, I am so grateful for you making the time because so many of these things that you've talked about, I know that you could spend other hours going into much more detail about those things. So, you know, this has been very enlightening. We've almost covered the entirety of the life cycle and at certain points, we've gone a little bit deeper. Oh, it looks like on the chat, we say that we can go a little bit longer if both Boda and Shujana are okay with that. Yeah, I'm fine. Okay, wonderful. Venkata, is there something that comes to mind while I look through the chat to see if there's any other question? So, we will continue this conversation for sure. We have an upcoming meetup to discuss the ML operations, including the alerting and so on, a concept drift, some of those things also, it will not end. On the has deep side, there is a comments page. So, if you have further questions after this session, if you want to go back and think about it, feel free to post it there. We'll definitely reach back to both Shujana and Gouda to address some of those concerns. Beyond that, I don't have, as far as announcements or the next one is, we don't have anything as of now. All right, then I think we're on the clock. Oh, there is one question, give me a second. Manu, you have a question, I'm just, I'm asking you to unmute your mic, give me a second. Yeah, go ahead. Can you hear me? Yes, we can, please go ahead. Yeah, so I come from a software engineering background. So, I wanted to know what is the difference between the life cycle of a machine learning model deployment and the general software deployment? You want to go first, Gouda? So, well, yeah. So, the deployment of a machine learning model, right? Typically, there are, unlike in software, there are roughly, I would say, three moving parts, right? Then there is data, then there is the models themselves, and then there are the engineering components, right? And in case of software engineering, usually it is the engineering components and there's like lots of tests and unit tests and things that are already in place, right? There's a lot of thinking that has gone into refining how those things should be done. In case of model deployment, right? Because data changes, when data changes, it's not deterministic anywhere. The model behavior is not very deterministic. And because of that, things could go wrong. So, there's a lot of need for doing pre-deployment testing and like monitoring how data distribution itself changes over time. So, these are the things that kind of are different. And again, the other challenge that we typically tend to face is that it requires a handshake between two sets of people talking different languages, right? So, the engineering folks might not, like when there's a problem, they might not realize that the problem is coming, not because of an error in a model, but because of certain assumption that doesn't hold true, right? It's not like really a bug, but something inherent in the model itself, or it could be because of a distributional change. So, these are things that are like, it's a little trickier to separate. That's one place where things change. You want to add Koda? No, I think you put it very well. So, basically, I think it's also, many times the ML model deployment, it's relying on existing, you know, libraries, open source libraries, which are there either from, you know, Escalar or Emly for, it's using existing open source frameworks a lot of times. And there could be inherent issues within that, which sometimes becomes difficult to isolate to whether it's a problem with the applying feature stream that's coming in or something inside the model or distribution. So, debugging becomes more difficult in this case from what I've seen compared to. Yeah, and I actually think like what you said, like it's important to have some things in place which can point out where the errors are coming. We do not, we are not yet there. Our systems are not yet there. We don't clearly point out to the source of error and then, you know, make it easier to solve it yet. There's a question from YouTube, I think, that Zainab has posted. This is from Harshit Gupta. He says that he's an internet credit lending company and they're working on new fraud and underwriting models as COVID has highly impacted the distribution of the data that their previous models were trained on. So he says, are all before COVID models junk now? How can we create new models considering COVID-19 factors? Are they all junk? No, no, no, no, no, no. No, I see it's basically about, you know, figuring out what part will transfer and what will not transfer, right? That is the trick in any place, right? There will be, you know, say if you've learned, like, as we learn, human beings are learning machines, right? So we learn something and then we move to a different setting. A part of will hold and a part doesn't. So the trick is to figure out what part holds and what doesn't, right? And that also requires a bunch of experimentation. You might not be able to use the model as it is, but, you know, you will be able to correct for certain pieces, right? So, like, let me give an example, right? So there is a, again, I'm taking a code, this is top of my mind, so I can tell you about a forecasting model. Let's say forecasting, you're trying to forecast how many infections are going to be, you know, there in a particular city. So that depends on how things spread. That also depends on certain disease parameters, like how long it takes for someone to recover and so on. Now, when you take this from one city to another city, the parts about the disease progression, how long it takes someone to recover, how long, you know, it takes for the onset of symptoms, those still translate. The spreading behavior does not. So same thing with your credit model as well, you would have to see what parts of the signal still remain so, right? Like, still have the same type of effect and which ones do not. And for the ones that are changing, you would want to, you know, adjust in a, you know, in a meaningful way, right? Like that it would still make sense. You might even guess estimate some of those effects. Wonderful. I think with that, we will bring this to a close just because you want to give people to more. Is that okay? Did I miss something? Yes. There was one question that has been pending for a few minutes comes from the Zyna herself. When we talk about all of these complex processes, there are evolving roles of the individual, data engineer, ML engineer, data scientist and so on. Do you have a point of view on these roles and what they're supposed to do and how they will all take about 30 seconds or one minute? Yeah. So like I mentioned. A long topic, but... So you have to put it in three lines. Data engineer role is more about taking the raw data signals that are getting instrumented and converting it into a process data store that can be used by analytics and data science. Data scientists are formulating a business problem into a mathematical formulation and building a model. And ML engineers are helping productionize that model in a very reliable and reproducible way. This is the way I would put it. Right. Yeah. And I would add to be a complete data scientist, you should be able to do all three. All three. Full stack. Yeah. Full stack. Wonderful. As good a note as any, wrap this up. Sounds like deja vu to me, but yes, that is. So thank you so much, Srijana and Goda for joining us. But what was the first meetup in this series of conversations called making data science work? You guys have given us so much that we might impose on you and have you come back once again because there's a lot to dig into what you've already spoken about. It was a pleasure, yeah. We had fun too, yeah. Wonderful. All right. Well, thank you so much, everyone. And Hazgeek will put up an update about our next conversation, which will be around ML ops. So please stay, look out for that and stay tuned. And thank you so much for having joined us today. Thank you, Srijana Goda. Yeah. Thank you everyone. Thank you. Thanks everyone. Bye. Bye.