 Good afternoon everyone. It's really nice to see you all today. So when I started working in this space around 20 years back, as an undergrad, it was very different. And honestly, I would say it's a huge understatement to say that the field has come a really long way. And one thing that does strike me that even though so many years have gone past, there are certain core elements to building ML systems that remain as important as ever. And these are things that we keep getting wrong even now. So in today's talk, I'm going to touch on some of those core elements and hopefully leave you with some stories and questions that could potentially help you improve your process of building ML systems. So I'm going to begin with a little story. So my son asked me the other day a few weeks back. He asked me, what is the difference between an awesome chess player and a solid good player? And he asked me this because he's a chess player himself. And I was just thinking, is it talent? Is it hard work or it's something else? And we had this little discussion. We had a back and forth talking about different players. And then we figured, of course, there are going to be lots of different factors. But the one that really stood out for us was deliberate learning and abstraction of knowledge into mental models. And you could say that the same ideas spills over into other areas. One could ask, what is the difference between these so-called great tech companies like Google, Amazon, whatever is your favorite? I shouldn't be biased. And the rest. And again, there are bound to be a lot of differences. But one key differentiator is how they abstract their learnings and use that to come up with models and processes that generalize. So in today's talk, I'm going to talk about one such model, an application lifecycle. As you all know, to build good software systems, software engineers need to not only master data structures, algos, programming languages, design aspects. But they also have to be very familiar with the process of software development itself. Similarly, to build good ML systems, data scientists and ML engineers need to not only learn about ML concepts, algorithms, and tools, but they have to be very conversant with the process of building ML systems all the way from problem formulation to deployment and maintenance. Unfortunately, what happens in academic programs is that there's very little emphasis on this model building in system building process. So most of us, including those who have PhDs, end up learning about these aspects while building these real systems on the job. Clearly, we should be able to do better. And the way to go is to actually figure out an abstraction that represents this process and to master it. So let's take a look. Similar to software development lifecycle, there is a typical ML application lifecycle which looks somewhat like this. So here the color coding corresponds to the skill sets needed in the different stages. So the journey begins with problem formulation, which is the collective responsibility of a business expert who brings in the application requirements, an engineering expert who brings in the engineering constraints, and an ML expert who translates these application requirements into one or more ML or optimization problems. Now, once that stage is done, we move on to data definitions. And this is not data cleaning, this is the place where you do a precise recording of all the sources and definitions of the data elements in different stages across the cycle. So you'll have three copies, one for training, one for offline evaluation and for deployment. For each data element. And when I say a definition, you're pinning down which database and what SQL query you're working with. And this is also the place where you review metadata associated with the actual data. I mean, so this is before you touch the data so that you can review it and prevent issues later on. Now, once you have a clear handshake on the data and there's a decoupling of these modeling and engineering responsibilities, we have a fork. And the ML scientists go on to build their models, which means training and offline evaluation and that's the blue box here. And this process itself is a complex multi-step iterative process, something that looks like this. I'm not going to get into details here, but this is what happens in the modeling stage. At the same time, the engineering folks go on to design and build the production systems, which are these orange boxes here. Now, once the models and the production system is ready, you would want to deploy. But what happens typically is that they're going to be discrepancies between the offline modeling environment and the deployment environment, especially when you're working with these models for the first time. So it is really important to fix these issues and establish equivalence. And that's where pre-deployment testing comes in. Once we have passed that phase, we move on to deployment and maintenance where we automate the predictions, data monitoring, data logging, and even periodic retraining of models with minimal human involvement. Now, is this the end of the story? No, because rarely can we build something that will work perfectly well the very first time. So there are bound to be mistakes and there are bound to be some issues. So that brings us to the last box, which is online evaluation and evolution. So what we do here is do an actual assessment of the system's performance on a live load. And we do split test, we check if the system is indeed effective on the desired business metrics. And this is also a stage where you do a proper diagnosis of your system to find the areas of improvement for the next iteration. Now, this figure here is not universal. If you go and talk to somebody at an ML savvy company like Google, they have their own version with their detailed project templates. But the problem is that most other organizations have a very incomplete view of this process. Even though they have been building ML models, they do not cover all the stages. For instance, data definitions and pre-deployment testing is rarely done in a systematic way. And even when it comes to something like problem formulation, this often missing participation from the data scientists and engineers in pinning down some essential elements. And even folks who have some course knowledge of this process don't exactly cross all the T's and dot the I's and invariably things go wrong. In fact, I should say, you know, my strong feelings about this aspect emerged from an experience with a high-profile project and we literally had to do 17 rounds of modeling before we released the first version of, you know, a project, a very important project at that time for us. And the reason had nothing to do with the ML aspects. It had just to do with the diligence across the cycle. Right? Now, if you step back and think, what is that we really want from a process? We want to be able to build systems and models which have some very desirable characteristics. Like, we might want effectiveness with respect to business metrics. We want them to either to ethics. We might care about reusability, security, a whole lot of things. But the key point I want to convey here is that all of these can be achieved only if we have a formal approach to building systems with checklists, templates, and automated tests, just like how you do in the software world. This aspect is still missing in the ML world, right? Now, some of you might be thinking, this sounds all great for a big organization that's building a lot of models. But is this applicable for a startup which is working on a single application? Well, having worked with startups, I would say it's actually the answer is yes. And even more so because you have limited resources, you have tight deadlines. So you really want to get the prioritization right, right? So if you have knowledge of the process, you'll be able to make those decisions in an informed way. To illustrate that idea, let's annotate the different stages with how specific the associated work is for a particular application. So what you see here, H means it's highly application specific. So what you see here is that problem formulation and data definitions are very application dependent. They involve a lot of uncertainty, whereas the engineering effort is typically application agnostic and reusable across applications and offline modeling is somewhere in between. So depending on the focus that you care about, if you really care about getting an application right or whether you care about a lot of reuse, you could prioritize accordingly. Now, this is one type of annotation, but it's possible to annotate the stages and the substages with other on other dimensions, like uncertainty, the effort involved, skill sets and so on. And it's particularly useful when you're trying to manage an ML project. Now this is all I'm going to see about the ML lifecycle. Now, before we delve into specific stages, I want you to pause and think about how we view this ML space. The data science space. Typically it is something like this, especially when we are learning about it, right? A lot of emphasis on learning algorithms, a little bit on modeling and the ML tools. And arguably this is justified because algorithms involve a lot of mathematical complexity. But having been around doing this for a while, I would say that the success of ML systems actually requires a very different distribution of focus and effort. As anybody who has worked on a real project would know, data and data processing becomes super important. But what most people often overlook is that problem formulation is also extremely critical. It's in fact more critical than modeling process and learning algorithms. Because if you get it wrong, it's almost like you've lost your keys in the bedroom and you're looking for it in the kitchen. And it's literally so. So in this talk, because we might not have a lot of time, I'm going to primarily focus on the problem formulation stage and breeze through the others or, you know, let's see how we do on time, right? So that's where we should be starting. So now on the problem formulation. So this is the stage where we are trying to convert a business problem into one or more ML or optimization problems. So typically a business goal is centered around optimizing some decision process, like you see in the yellow box, so that you improve certain business metrics. And when the decisions are suboptimal because of missing pieces of information, then you want to estimate them somehow. And when there are no easy alternatives, that's where ML methods come in. And that's how you have these ML problems. But one thing to remember here, it's super critical to not jump to the ML problems. Let's say there is a business expert who comes to you and says, you know, build me this model to predict X, whatever be the X. The very first question one needs to ask them is why? Right? And this is like really, really important because, you know, I have learned this lesson the hard way, it's cost me, my teammate, a lot of time and effort. Just because we didn't question the problem premise early on. In fact, you know, if I were to ask my son also, right, you know, do something, he's going to ask why because he wants to make sure he agrees with the goal and, you know, can do it effectively. The reason I'm emphasizing it is because folks finally care about the larger goal. No one cares about the beautiful models you build or like near perfect predictions on some target variables. But what often tends to happen is that ML experts are handed over poorly formulated problems with very little visibility to the larger goals. And as a result, the solutions don't work as expected. Let me give you an example so that this becomes clear. So let me talk about less resell of fraud. You might have heard of this. It's very common in e-commerce. So what happens is whenever Flipkart or Amazon has this big sale, lots of small businessmen buy items in bulk like 100 phones and then after the sale is over, they sell it at a higher price. It's actually fraud according to the terms but very difficult to come. So naturally the e-commerce companies would want to figure out whether an order is going to result in fraud and block it or do something about it. Now what could happen here? A product manager could come to an ML expert and say, look, I have this resell of problem and I have this historical data that human auditors have labeled as fraud or not fraud. And why don't you build this binary classifier for me which can automate this process? Fine, it looks very straightforward. The ML expert can get to work and quickly build a model. But do you all see what the problem here is? Let's see. The issue with this model is that it essentially reverse engineers, the human auditors decisions along with their biases and their shortcomings. And this model won't be able to adapt to changes in the fraudsters' tactics or drifts in the data distribution. And most importantly of all, it doesn't have a direct connection to the business metrics that we really want to optimize with the fraud detection. So this might not succeed like how we want it to. On the other hand, if we had just asked the product manager, why? Then he would tell us that, yeah, I want to, then I want to do this fraud detection because we want to reduce the return shipping cost from the resellers. We want to increase the number of users served, especially during the sale time. And now the problem space starts looking very different. In fact, not all resellers will be bad because if there is a reseller in a tier four town who's selling phone just because nobody else has internet and that guy rarely returns the products, then there's no harm at all. In fact, you would want to partner with somebody like that. But at the same time, we do want to block the orders which result in higher return costs or impact high demand products. So now in this case, to optimize this decision, we need multiple pieces of missing information. We want to know about the likelihood of resale. We want to know about the likely return costs, what the unserved demand for a product is and whether that particular reseller is serving some untapped customer base. This is a very different picture than what we had earlier. So we have multiple ML problems here and we would have to solve all of them and combine the predictions to come up with an optimal decision. Fine? So that was the motivation for asking why, to get things right. Now let's see how we would solve one particular problem here like a likelihood of resale. Say we want to flesh it out. So to keep the discussion simple, I just want to limit ourselves to a supervised learning setting, though a lot of this applies in other scenarios too. Now for a particular problem, there are multiple key elements like what you see here. The ones on the left correspond to the representation of the problem elements, instances, targets and input features. These are the definitions, not the actual data. And the one at the bottom is data sources, which is where we get our observations from and the ones on the right are our objectives, metrics and constraints. So we have to make all these choices to specify this problem. Now let's see how we make these choices. Let's start with the instance. So this is a simple one. So to figure out the definition of an instance, we need to ask, is this the right granularity from the decision-making perspective and is it feasible from the data collection perspective? And let's just going back to the reseller fraud example, there could be multiple options. The instance could be a customer, it could be a complete order with multiple products or it could be a customer product pair, right? Here we might decide that if the decision-making involves blocking or not, we might not want to block a customer or an entire order. So the last option makes more sense. Now let's look at the target variable which was the second element we had to pin down. So with respect to target variable, the most important thing to ask is can we express the business metrics in terms of the prediction quality of the target variable or variables, right? The final business metrics. Now this connection might be approximate but you should be able to make it because if you cannot make the connection, you cannot assess the likely impact of your modeling effort and it is crucial to do that early on. Let me tell you a story. So we were working on this problem on sorting packages to minimize delivery delays. And we worked on this project for a couple of months before we realized that even if we had solved this problem perfectly well, we would make only a minor difference to the overall business metric because there were bottlenecks elsewhere. So it was like wasted effort. So this particular issue is important. So before jumping to modeling, before you touch the data, you need to ask will accurate predictions actually improve those business metrics? And in fact, we need to construct rough estimates of the likely business metrics for the ideal case, for the current baseline and whatever is likely to happen. And this will help you even assess the relative efforts and benefits when especially when you have multiple problems to choose from and your resource constraint. Now apart from this first set of questions, there's a second set of questions that you need to worry about again related to the collection effort. We want to figure out, can we really get observations for this target variable? Is it going to involve manual labeling? Is it going to cost some money? And more importantly, can we get high quality observations? And for this, you need to look at the uncertainty in the definition of the targets and the noise or bias in the labeling process. Another example here, so what I've seen in a lot of projects, for example, like product classification e-commerce, the target definition is very fine grade. So much so that even human experts don't agree and there's not much benefit even to the business process because of that fine granularity. So in such cases, it's actually a good idea to step back and have a clearer, useful definition because it will save you modeling effort. So let me now go to the third element, which is the raw input signals of features. So the first question is something most folks are familiar with, which is whether a feature is predictive of a target or not. And there are a lot of ways to evaluate the features on that. The second question is what I think is even more important, which is whether a feature is going to be available in the production setting with the same semantics. And this is where a lot of errors creep in because this is really taught at school. In fact, when I was at Amazon, we were wired to think that any time our models do really good, it's either because of a target leakage or a feature semantics mismatch. So let me tell you what happens. So essentially training data is created in retrospect. What I mean is that you can use any data till the current time instant to create your features. Let me take the reseller fraud example. Let's say we have labeled orders from January to September. Now it is number. So I can actually accidentally create a feature, for example, average monthly spend corresponding to an order on September 15th, right? So this particular feature not only looks at the first half of the September, but also the later half. It's doable because I'm doing down in number. And because this feature has peaked into future, it might do very well in offline modeling, but it will fail really badly when it comes to production because the semantics are not the same. Now it's not you are with me on this. This is an important thing. You should read up more on these data creeps and leakages if you're building system because this happens all the time. And now I wanted to say that it's not just enough to just look at the data just before September 15th because there are lags in the real pipelines, right? So even if you look at only the data just before 15th, it might be possible that there are data lags and it's not possible to create an equivalent feature in the deployment setting. So to create any of these features, you need to understand the data flows, the lags, and figure out what is a safe buffer and design the features and the time windows appropriately. And especially when we get to aggregates which involve the targets, it's even more risky because you are taking a direct look at the targets. Now apart from this, the third question you need to think about again, which comes back to the data collection just like we had with the target and the incident is to think about the cost of computing or acquiring the feature. Sometimes it might be monetary, sometimes it might be a computational cost. And the computational cost is particularly important if you are operating in a low resource deployment scenario, say a client-side computation, you need to really think about that. So so far I talked about elements of the representation. Now let's look at the data sources. Now with respect to the data sources, the most critical thing to worry about is whether the distribution of the training data or the offline modeling setting is similar to the production data. And here it's okay if the joint distribution of target and inputs don't exactly match, but you at least need the conditional distribution of the target given inputs to be more or less the same. Now apart from this, we need to worry about fairness issues and Radhika talked about it, you know in the context of gender bias, but this is a real important problem. So if only we do a little bit of resampling or we do a little additional data collection, these are issues that can be addressed and it's good to be mindful of those. Now the other thing to think through is whether the production data changes with time. If that is the case, we need to incorporate a mechanism that keeps collecting the data and we need to retrain our models on that data. Now so far I talked about questions on distributional value, but sometimes we know that there are systemic biases because of how we are collecting. So typically this happens when we collect supervision on a subset of instances through some filtering process. Let me give you an example. So for reseller fraud, they didn't have a lot of auditors and they just got people to label only the orders which had five or more repeat items. Now a model built on that data cannot be used on all orders, but it is good enough for that segment. So in this case, you could adjust the prediction scope to match with this filter and you can do that when the filtering is very explicit and of this fixed nature. Sometimes the filtering is lot more certain. Let's say we have a system in place with this blocking mechanism, again for the reseller fraud, right? So what would happen in that case is that we know the outcome of the orders only for the orders that went through. We would know if the reseller not only for the orders that went through. The orders that got blocked, we actually don't have any information about that. We don't know the counterfactures. What would have happened if it was not blocked? And because of that, the data gets biased over time. And I've seen this happen in a lot of systems which are operational over years, the metrics start moving. And to correct this, what you need to do is early on adopt an explore, exploit strategy where you keep exploring even the so-called blocked segment to a small scale and then you use statistical correction methods to adjust for the bias, right? So there are a lot of things to be talked about on data sources, but this is all I'm going to say. Now let's discuss the objectives, metrics and constraints. So before we go there, let me just talk about two broad classification of metrics. The first is online versus offline. Online metrics are computed on a live system and typically they directly relate to your business metrics and we measure these with split tests. You know, and because these incorporate a lot of interacting factors, it gives us a very grounded sense of what is happening with the system. Is it really doing well or not? On the other hand, offline metrics are meant to be computed on retrospective label data. And these are related to the prediction quality of the models. And the primary purpose here is to use them in offline experimentation so that we get a good model that we can test online. The second classification of metrics is primary and secondary. When I say primary, that's what we are actively trying to optimize. And secondary constraints are more like guardrails just to make sure the system doesn't go haywire. Now to pin down metrics, what we really need to ask first is what are the key metrics? And these include both the primary ones and the secondary ones. Often people only mention the primary one and forget to leave the list out the secondary one till you build a system and things go bad and then the second iteration it comes back. So that is really important. Now this question on online metrics is usually answered by the product or the business person but there are some important components that everyone needs to pitch in. And let's for now, assume that we do have these metrics in place. Now the second question to think through is whether your offline modeling metrics are aligned with the online ones. Again, let me tell you a little story. So I was at some point working on product recommendations and we had to build models for predicting the click-through rate of a customer for products to be used in recommendations. So we had this nice model and then we pushed it out to an AB test and nicely the CTR went up. Unfortunately, that model was recommending lots of free videos and it was recommending cheap gadgets and the revenue took a big hit and we had to obviously go and redo it but the learning here was we should have aligned our offline metrics to online metrics and we should have known the online metrics in the first place. So essentially the point here is don't go with some default offline metrics make sure you actually work backwards from the online metrics in deployment scenario and arrive at these. Now in the previous slide, I said that the metrics are primarily chosen by the product or the business person but there is one aspect where everyone needs to pitch in and that is ethics and fairness and Radhika spoke on that a little bit and I think it's really important for us to be mindful of this. Just because we are data scientists or engineers we cannot walk away from being responsible for what we create. And so it's important for us to think about the long-term secondary effects of the systems we create so that we minimize the negative unintended consequences. We need to try to ensure that as far as possible the systems are fair to all the user segments. For example, when we are working with content recommendations it's important to not personalize objective news like we do with entertainment. Even when we are personalizing entertainment we should not overdo it so that it becomes addictive. There are a lot of other areas like data abuse and privacy. So there's regulation coming up, there are a lot of frameworks but this is something that all of us need to think through early on in the problem formulation state and make sure it goes into the modeling metrics itself at least as a secondary guard break. So now the last thing I want to talk about with respect to problem formulation is deployment constraint. This is again something that the product expert might not be able to come up with but the engineering folks can pitch in and here we need to worry about the application constraints stemming from user interface restrictions or connectivity issues, the hardware constraints to worry about where the computation is happening, what are going to be the memory and the compute power resources when you are trying to work especially when you are doing prediction with low latency. Same thing, it's scalability, the size of the data, how frequently you have to do that processing and in case of online predictions you need to worry about the arrival of prediction instances and latency bounds. You have five more minutes? Ah, okay, great, thank you. So again let me tell you a quick story here. This again from product recommendation. So we build these nice models with lots of features and I put it out to test and it turned out that product recommendations requires very low latency because you don't want people to click away. So we worked for a long time build these models and then we had to junk all of those because the only ones that could meet those latency restrictions were features with four or less, models with four or less features. So this is again, you know, it's not orthogonal to modeling. You might think this doesn't matter but it does matter to your modeling choices and nowadays it's particularly important because there are these packages and tools like TensorFlow Lite, Proto NN so which operate better in low resource environment. So I have some key tenets. So I do have five minutes, right? So let me take the time. So I spent so much time on the problem formulation. I'll just breeze through the other ones a little bit. So the first state after problem formulation is data definitions. So in this stage, I mean, what we do is we actually record all the definitions. Remember, this is not about data cleaning. This is not about, you know, actually touching the data. This is about looking at the metadata. So we record the definitions exactly which database it's coming from, what query you're dealing with and you make copies of this for training evaluation and production and the main focus of this stage is to establish parity that the semantics are actually the same across these stages and there are in fact a whole lot of things. I had, you know, almost eight slides that I compressed into one because I knew I wouldn't have time but I'll probably add it later on. So there are a lot of things that one needs to look at whether the definitions vary with time. So if you have something like spend till date and that might look very different when you're doing offline modeling and in production. So you need to think through all of those when you're establishing parity and you also have to review for leakage. Just from the definition you would know whether the semantics are going to be the same or not whether this is looking at the target or creeping into future and so on. And the other thing which came up when I was talking to one of the attendees today is about handling missing values. So the tough job about handling missing values is because we do not know the reason why they are missing. So if you proactively talk to folks about the data quality issues you might be able to, you know, devise a way to solve these up thread. So there might be reasons why data is corrupted because there was a sale going on or there's a server down, there's some join that has broken up. If you know about this you will know the appropriate response whether to throw out the data or fill it with some default values, compute it with conditional expectations. To be able to do that you need to be very proactive about these data issues. And let me now say a little bit about offline modeling and I'm going to skim through it a bit. The very first thing is ensure that data is of high quality because garbage goes in and garbage comes out otherwise. And the other things to think about which are points to entry is that you have to be a little smart about how you do modeling. Right now there's a lot of algorithms and the only way you can get around this is to understand characteristics of your data, assumptions of the different algorithms where they work well, where they do not write the limitations and choose those and choose the right options. And we need to be very smart about the trade off between feature engineering and model complexity. If it's a domain that you know a lot about then fine go ahead with feature engineering. But if you do not know much and you do have a lot of data then you should throw a very complex model at it. So this is something that you need to be smart about. And another very important issue is to make sure that your offline evaluation is a very good proxy for the unseen data. This again speaks back to the point I made on you know the equivalence between offline modeling and deployment, right? So to the extent possible, we need to create our split so that they're reflective of what you're going to see in production. Yeah, so now on the engineering again the first point is work backwards. This applies everywhere not just in engineering scenario. So you need to consider your data choices, your compute choices, cloud, or your own in-house servers, the ML framework choice, all these based on what your deployment requirements are going to be. Is it going to be stream, batch, you know what is the scale? Depending on that you need to think through. And another important thing to be mindful of when you're designing this engineering system is to make sure that there's a clear decoupling of the modeling effort and the production system. This is important because it makes sure that you can push your models to production much faster if it is agnostic to the modeling process. And maintaining version repositories of pretty much everything that is a no-brainer because data keeps changing models, then you're going to have lots of models not just one model, they're going to be experiments you need to keep track of that. And rather than thinking about a single model plan for a connected system of models which you can put together and form flows over. And lastly coming to deployment, I mean when I said deployment, this actually includes, you know, pre-deployment testing, deployment and online evaluation. The first point is to ensure parity. And when parity, parity of the pipelines, parity of the data distribution, every moving part that you can think of make sure they're comparable before you go out and deploy. And before you scale up your deployment, start your split test, on a small fraction of data and then scale it up as you see improvement in businessmen. And when you have models out there, trust them but always audit. So you need to have some monitors, some safeguards and some dashboards. Otherwise, if there is data that you have never seen, if there's a segment that you've never seen, this model might not work as you expect it to be. And lastly, think of this model building not as a one-time effort but something that you do continuously. So you design for that need and retrain period. So these are some things I thought would be useful to keep in mind when you design these systems and work through all of these operations. But I haven't done justice to these all because it takes a lot more time. The one thing I do want you to take away in regardless of what I said in the last couple of slides is this bit, right? This bit on map your application life cycle. See, when you go back in whatever we've discussed so far, right? It's not just for me to share something with you but I would actually be very happy if you go back on Monday and think through how you're doing your ML projects. So plan out an application life cycle based on what you're doing with your ML projects. Introduce checklists, templates, and automated tests wherever you can. You might be able to create some in-house or borrow what is out there. I also probably will put out some whatever information templates I have. And again, very importantly, invest effort in getting the problem formulation, right? So ask why early on, why are we doing the different things and make those choices carefully because they have repercussions later on. And be very proactive about data issues. What this means is examine your metadata, examine the definitions before you go to clean up the data and do things around the actual data. So this is pretty much all I wanted to say. I'm happy to take questions. Thank you so much for listening. When am I writing a book? Oh, no, so this is, I just touched the iceberg. So I actually have lots of slides. So, you know, I've been doing this workshop for a while. So I have like 200 screen slide deck, which I will probably put out once I clean it up so I can do that. It's great. But I'm happy to share in case you want to look at more material. Great talk, and happy to see if you can share the slides and maybe... Definitely. And I actually have a longer slide deck. I didn't do justice to the other stages and I have some templates too that I use personally and in fact, went through a lot of hardship before we came up with those templates. Happy to share it online after I do some cleaning. So one question I had is to manage the life cycle. Can you give some pointers towards, let's say, open source tools? So I have looked into this ML flow, but not in detail. So I would like to know open source tools where, in which aspects can they help? Right. What I find is that, you know, the best resources I've found online on how to go about this is Google's AI education course. So they do have a lot of material which is very actionable. Now, apart from that, there are specific resources which, you know, help you with data checks and so on. But I honestly don't see a lot of, you know, automated testing code, right? Like our tests for doing the pre-deployment checks, I haven't seen a lot of that. So there is material, isn't like material you can look through, but not automated tests. There are checklists, there are some templates, and I'll point you to, I'll actually put this thing along with some references so you'll be able to look at it. But do check out the Google's course on, you know, problem formulation, the different stages of, you know, the ML application life cycle.