 And now we are going to the next speaker. It's Jay Stevens, I see is in the room. Please start sharing your slides. Jay's will talk about machine learning again. His talk is painless machine learning in production. Hi, thank you very much. I'm in Green's room in Boston. So as Martin just said in the introduction, my name is Chase. I am a engineer at a company called Take a Metrics. I've been working there for about the last three years or so as the tech lead for the data science team. I'm very excited to be talking about this topic. So I'll just get started. First of all, I just wanted to help clarify, well, what is the point of this talk? What am I going to be introducing you to? If you read the description, you know that the talk is going to be essentially about the internal machine learning platform that we use at Take a Metrics. But I just wanted to clarify like, what specifically about that am I going to be talking about? And this is not a machine learning talk, right? This is not about painless machine learning in production. It's about painless machine learning in production. It's going to be much more about sort of taking models which are sort of our bread and butter as data scientists and machine learning practitioners. We have that pretty down pat, but taking the models and bringing them into this sort of productionized environment where internal stakeholders can use them or internal consumers, or we can expose them to the rest of the world is something that we maybe need to mature a little in. So that's what this talk is going to focus on. Also to Martin's point, I say this is painless machine learning in production. That's maybe a lofty goal. When I say painless, don't think of that as like expressionless or colorless. Think more like stainless steel, right? Where it's not going to be a description of totally pain-free utopia that we built at my company, but it's rather more of a pain-resistant sort of platform that we've built. So given these sort of caveats, I did consider other names for the talk, but somehow I thought this wouldn't detract as many attendees. So painless machine learning in production it is. I also want to talk a little bit about the company and this will hopefully help sort of contextualize why we have this machine learning platform, what we needed it for, all that sort of stuff. And this is not, hopefully not going to be your typical like shill for the company, but really it's more about some of the intrinsic problems that we face in our space. So take a metrics goal is essentially to help sellers on all sorts of e-commerce platforms, manage their businesses and sort of optimize for profitability, I guess you would say. If you're someone who has a product, either a very small business or perhaps a single entrepreneur all the way up to very sort of large businesses, if you have a product, the things that you want to think about are like improving that product, how are sales doing, how am I doing against my competition, what are like new opportunities in the market? These are the sort of things that you're probably really good at and really would like to focus on. Unfortunately, the reality of e-commerce today is that there's a bunch of sort of extrinsic things that you also have to worry about. And that's sort of where we come in, right? We're trying to sort of smooth over that process and give you as much insight and as much empowerment as you possibly can have as a seller, right? And so the sort of things I'm talking about are things like managing your advertising spend, right? It's just not going to be an interesting problem for most people or maybe not something that people are going to be that proficient at, right? Things like predicting what the demand for your product is going to be over the next six months, right? Managing the inventory and warehousing. These are the sorts of things where you really do need a machine learning solution and the average seller is just not going to be either have access to the data to produce that sort of solution or frankly, probably not going to be sophisticated enough or have the right background to sort of tackle those sorts of problems as effectively as they might like. The other thing that I guess will help sort of situate this talk is we're a fairly small Boston based startup, although we have offices all over the world now. And like I said, I joined three years ago. When I joined, it was about three or so people on the data science team. So a very, very small team. I think two years ago when we started sort of building this machine learning platform, it was around four or five people. Now we're at about 11 people and hoping to double the size of the team by the end of the year. So if you find this talk very fascinating and want to get in on this stuff, there are openings available. But I guess central conceit here is we've developed this to sort of help our relatively small team be much more effective than we might otherwise be. And it's really over the past couple of years accelerated the amount of work we've been able to do and the sort of solutions we've been able to develop. But in addition to that, like I said, a pretty small team, and we haven't been working on it continuously for the past two years. We've been working sort of piecemeal here and there. We've probably put in a collective, I don't know, let's say nine to 12 developer months into developing this platform. So the point being that regardless of the size of your organization, whether you also are working at a startup or whether you're working at like a Fortune 500 company, some of the principles that I'm going to be talking about and the technologies that we use are something that you can employ yourselves, right? This is not like a solution that's been developed by one of your big tech giants. So in terms of what I'm actually going to be talking about in the talk, first I'm going to try and convince everyone to actually continue listening and motivate the rest of what I'm going to talk about. I'm then going to talk about, well, if you are a data scientist at Takenmetrics, what is your experience when you start developing a new model? And I should say that for the purposes of this talk, essentially all the models we develop, we expose as internal services. So everything is an endpoint that you can hit and query and get a response back from. Then I'm going to talk about sort of the stack that helps power that, some of the tools and technologies we're using. I'm not going to go too in depth about that. This talk is really more focused on, I guess, the mechanics, the developer ergonomics and sort of the general guiding principles rather than the nitty gritty. So I'm not going to be showing you any code, but hopefully it should be an interesting reference if you would like to develop a similar solution at your organization. And then the last thing I want to cover is basically what are the lessons that we've learned over the past two years or what are things that might have done differently or I guess how did we evolve the platform to the States in today? So without any further ado, first there are a couple of, I guess, premises to this talk that will help motivate why I'm giving it and why we sort of need this solution. So I'll help justify these later on, but for now let's just assume these things are true. First of all, ops is intrinsic to ML. And what I mean by ops is I mean sort of the process of bringing your machine learning model into production and monitoring, maintaining that model. It's really direly important for having an effective machine learning solution. In addition to that, there's this concept called ML ops, which is unsustainable. And when I say ML ops, I mean sort of this concept of well, you have the data scientists or machine learning practitioners or what have you, they know how to do data science stuff or develop the algorithms and whatever those data scientists do. And when they're done with that, you take whatever solution they develop and you hand it off to someone else and they are the ones who maybe have a little more engineering background or a little more experience that take it and expose it to the rest of the world or integrate with other services or what have you. So assuming these two premises are true, the sort of inevitable conclusion is that, well, if the ML ops team isn't a sustainable practice, but operations are important to machine learning intrinsically, then data scientists need to be able to productionize their own models, right? Data scientists need to have the tooling and be empowered to take things from conceptualization all the way through to production. There's a catch though. Data scientists want to do data science and I won't help motivate this one, but this hopefully should be pretty uncontroversial, right? As data scientists or machine learning practitioners, the thing that we are best at and the thing that we find most interesting is sort of getting down to the nitty gritty and developing new innovative interesting solutions, right? People come into data science from all sorts of different backgrounds with all sorts of different levels of I guess engineering expertise or experience. And we can't uniformly assume that every data scientist is going to know the best route forward to bring a model into production. So hence, we need to develop these sort of tooling and services to really minimize the overhead, minimize the pain and make it so that that whole process is easy. So now I'd like to start justifying those first two premises. I'm sure people have seen diagrams like this a million times, but I'll just go through it. This is sort of the platonic ideal of machine learning life cycle, right? And so the idea roughly is you have some data that lives somewhere in like a data warehouse or accessible through some APIs or something like this. And the first thing you need to do is take that, do some preprocessing query. Those APIs are query that data warehouse and turn it into a training set, a test set of validations that all that sort of good stuff. Maybe do some feature engineering, maybe some data cleaning, things like that. Then you have some model that either is off the shelf or you developed yourself that you would like to train using the training data, right? After that's done, you have some set of parameters you've learned for that model, which you can then evaluate. And the evaluation might be some sort of manual evaluation like, hey, does the accuracy of this model look good? Or it might be a comparison against an established baseline or something like this, right? And then you deploy it somehow, right? And this is sort of the step that I think as a discipline we haven't focused enough on perhaps because there's a lot of complexity in here, right? If you think about like normal software engineering, this is actually the majority of the complexity is figuring out how to look at things like logs, monitoring, even auto-scaling, all these sorts of considerations that again, as data scientists, we don't wanna have to worry about or think about or really aren't our wheelhouse. And in addition to that, you have this sort of intrinsic orchestration component where you'd like all of this cycle to just be automated for you basically. There's also this other strange aspect where, okay, you have a deployed model. Why do you need to do pre-processing again? Like why is there this little arrow from deployment to pre-processing? The answer is, unfortunately, the world is sort of pulling the rug out from under us at all times, right? There are always new observations that you're making. Hopefully whatever models or solutions you've actually deployed in production are having some effect on that world that you'd like to monitor. You might just be learning new things, right? And that, I guess, environment is constantly changing and you need to be able to adapt your model to survive. So two years ago, we had this sort of concept, which I think is fairly standard, that we want to find a solution for and we searched high and low for some service that lives out there in the big wide world that will just sell us this platform. And to be frank, we couldn't find it. You could find places maybe that will do the offer solutions for pre-processing and training or might offer all of these, but don't have that orchestration compiler, right? Instead, want you to manually run all the cells in some Jupyter notebook you, because data scientists love Jupyter notebooks, you know, some Jupyter notebook that you have, which is just not a sustainable practice and to sort of help evidence that. So this is a very interesting chart. This is basically a few different classifiers, a few different models that were learned on data at the beginning of January of 2017, right? And what they're trying to do is determine or I guess classify whether a particular URL is malicious or not. And you can see that at the beginning of January, they all do really well. I guess this is their area under the curve over time in using real world URLs, right? And then you see this sort of precipitous drop all of a sudden in the middle of February. If you're in an organization or if you're a part of a data science team that is not constantly retraining the models on the newest, freshest data, if you aren't monitoring the accuracy of your models or some of these operational metrics, then it's gonna come as a big surprise to you and all of a sudden things might start failing that you didn't expect to fail. And to show this is not, I guess, a phenomenon isolated just to this one domain. This is some of our data, right? So the blue line here is showing you the daily number of orders for a household cleaning product that one of our sellers sells. And the orange dot line is showing you Oxford data set, a measurement of the stringency of the coronavirus countermeasures in the United States, right? Which is sort of an aggregate of all sorts of different individual measures that have been implemented. And you can see this is sort of a correlation made in heaven, I guess. But the general gist of this is that if we, we're in a space where if we had learned this model, predicting what the daily order volume, for instance, is at the tail end of 2019, we wouldn't be doing very well come the middle of March, right? And this sort of pervades into a whole different, a whole different set of domains. All sorts of use cases have the same sort of patterns, not just the sort of time series data or not just necessarily because the world is sort of stochastic and changing out from underneath you. But also, anytime you have a new client or a new user or a new customer or a new use case for the model, you might have the same problem. Again, in our domain, we have some models that we learn on a marketplace-specific basis. We learn parameters on a marketplace-specific basis where a marketplace might be Great Britain or Germany or the US or wherever. And for instance, suppose it's the case that we only have one seller in that marketplace and they sell, I don't know, pencils for 10 cents a piece, something like this, and then tomorrow, we have a new seller that's selling luxury watches for $500 up, right? The parameters that you've learned for that first seller aren't necessarily going to apply to the second seller. So we constantly need to be retraining, reevaluating these models and measuring the performance of them. And that's why it's so, so important to have a good grasp and a good platform for easily enabling sort of these whole operational world that you might like to have some system to orchestrate for you. I also wanted to talk about, why do I think machine learning operations is an unsustainable practice? And let's go a little bit back in time. Let's say you're a hotshot developer in the 1970s, right? What does your day-to-day look like? Well, you write your code onto punch cards and when you want to have that code run or what do you want to compile it, what do you do? Well, you take the stack of punch cards that you've written and you go down to the computer lab and you hand it over to some computer operator because in the 1970s, well, programmers can't be trusted with computers, right? That's a ridiculous notion. Why, you know, the programmer doesn't know how to run the punch cards through the main frame. They don't know, you know, how to figure out, oh, this transistor's blown out or replace it. None of those things. You would never have a programmer touching the computer. That's a very odd notion. And so what was, what did people at the time think about this? And this next slide, these are all real quotes that I've pulled from different accounts of people that are working in the 1970s using this sort of punch card flow, I guess. And they're pretty amusing, but it was not a great experience. So for instance, this middle quote here, I think helps sort of justify what I was just saying, only a select few programmers were allowed in the computer lab. Again, there was this very odd division. But as you sort of read through some of these, they're pretty insidious accounts or very, I guess, poor development experiences where the latency of you submitting that program and getting back to the output or the compiled program or whatever is on the order of hours, days. There's one account here where someone's doing something for a final exam and they have four days before the deadline and they can only compile it four times. And if it happens to be the case that you forgot a semicolon or something like that, well, try again next time, right? So how did we end up moving past this into a, I guess, more humane ergonomic environment for development? And the answer was the advent of this sort of dumb terminal, right? So I can directly connect to the mainframe. I have some time on that mainframe. And I can just run the programs myself. I don't have to have this intermediary that's helping me compile my programs. So there was an advancement in technology but that was also met with, I guess, change in the skill set that was expected of programmers. So it wasn't sufficient anymore to just know how to write the program. You also had to get familiar with the command line and all these sorts of things, figure out how to compile your program. So it was sort of this measure. We met in the middle a little bit. And of course we learned our lesson. So we never had anything like this happen again. So now let's say that you're a programmer in the heady early web days, right? Your life is pretty sweet. What do you do? You code? Then do you write unit tests or anything like that? No, of course not. You have a QA team and you give your solution to the QA team. They run in some sort of environment and they go, okay, it looks good for release. And then they pass it along to, you know, you're like sysadmin or IT team or whomever that's going to release it. Easy peasy, right? You're living in the lap of luxury except that sometimes there are problems, right? So you maybe were using a functionality in PHP that is not actually installed on the server or you know, whatever. And you know, fundamentally this was an insurmountable obstacle. Obviously you don't have the people doing the coding also doing the releasing because coders don't know how to, I don't know, install RPM packages or update OpenSSL or configure Apache or anything like that, right? So you just don't have the same skillset and you wouldn't want those same people doing it. But again, over time we sort of remedied this situation through the advent of services like AWS where all of a sudden, hey, at least I don't have to actually physically put a server in some like co-located data center. I can just spin up a box, I guess and start installing things on that box, which is great. And then we got a little further along and you have services like Heroku that it's sort of one level of abstraction higher where now I don't even have to know, how to install things on my box or anything like that. I don't have to worry about security patches or any of those sort of operational concerns. This service will take care of it for me. And then of course, there are things like Squarespace or Wix or whatever where you don't even have to know how to code, right? You can make a web app or what have you pretty easily. But again, this wasn't just technological solutions helping us gloss over this problem. You also had a commensurate increase in the, I guess, set of responsibilities or the remit of the average developer. Developer software engineers, I guess today have a lot more skills about things like dockerization or managing web server configurations than they used to. And so again, it was sort of this, let's meet in the middle, scale up a little bit, but also significantly reduce the amount of work that you have to do or the amount of pain that you have to encounter when trying to deploy things to production or when trying to release things. So obviously, fully one shame on you, fully twice shame on me, as an industry we've gone past this and we'll never make the same mistake again. Or so I thought, imagine my surprise when just last year I was at a big data conference and I heard a lot of, I guess industry leaders from very large tech companies talking about ML Ops and the idea behind ML Ops is of course, well, you have your data scientist but they don't know anything about deploying things into production. So you need to have this other team to help them do that. And, you know, we had, take a metrics our own dalliance with this and I just want to, I guess, recount how that went. So we only ever tried it once with one model but we thought at the beginning of some quarter, well, maybe things will work a little bit better, maybe things will go smoother if we sort of split the team up and have a research division and engineering division. And the researchers can focus on the, you know, real meat of the modeling and then they can hand the solution off to the sort of engineering team who will productionize it. All right, so we tried this, how'd it go? Well, the initial model was developed and unfortunately immediately there was a problem where, oh, they were using a data set but actually that data is not available in production. So, you know, there's an obvious problem there. We have to account for that somehow. Okay, went back to the researchers. Try this different approach instead. I fixed the code. We tried running it, we're encountering errors. Turns out wrong version of NumPy, right? The researcher was using a particular version in their sort of notebook environment which wasn't the one we were using in production boxes. Okay, okay, okay. So now it's pretty easy to correct. It should be all set now. Ah, well, it turns out in the training set or development set that was being used to create the model, this one column was entirely populated but in production it wasn't always the case. Okay, so back to the drawing board. Try again, blah, blah, blah. Well, there were a bunch of sort of diagnostic graphs that really were going to help us figure out why the model was performing in the way it was or help us identify ahead of time issues with the model that might be arising. And for whatever reason, those graphs just weren't displaying when running production. Okay, so maybe we can admit that part and get rid of it and we'll just learn to live without those. Ah, well, it turns out that our hosting provider has a particular timeout on the model and the processing was taking longer than that timeout. And by this time, and I don't joke when I say this, there was already a new version of the model that the researcher had developed. And so this particular version never made its way into production. And again, I wanna emphasize that just like the punch card example, each of these steps was on the order of hours or days, right? Because maybe the researcher gets told there's an issue with the model, but they just started working on something else so they have to revisit it a few days later, et cetera, et cetera, et cetera. And so this is just sort of like operating in this way, at least in my experience, is going to make you want to tear your hair out. So it's simply not a viable way forward for us at Taken Metrics, or I think across the industry. So what's the alternative? Well, let me tell you about what it now looks like two years past this point to develop a model at our company. So don't worry too much about reading through all of this command line output, but the basic idea is you have a cookie cutter, which is sort of like a template repo. That template repo is using a framework that we have developed internally. And basically when you start out, you're asked a few questions about the model you're developing, right? So you're asked what your GitHub username is and what the name should be for the model. And you can see that things start pre-populating based off of the answers you've given before. You're asked, how are you going to evaluate this model? What's an appropriate validation metric? You're asked, well, for that validation metric, is this something that you want to maximize or minimize? Or under what circumstances should I be promoting this model in comparison to a model that lives on production, let's say? How many CPUs do you need for the model? How much memory do you need during pre-processing, training, serving? What's the appropriate test set proportion? What's the maximum runtime of the model? You answer these sort of pretty easy, pretty baseline questions. And what you get out of this is you get basically your entire repo made for you, right? You have a lot of very useful things you wouldn't have to want to write over and over and over again, like a good standardized Python package structure, you have your CI integration, you have all these sorts of things. And really, of this generated code, there are only four or so files that you need to actually edit to implement your model. So what does the implementation then look like? Well, if you think back to that initial, I guess model lifecycle that I showed you, it more or less maps directly onto that, right? So you have to define some function that does your pre-processing, which normally in our case looks like, oh, we're making some SQL queries, we're taking that, we're loading into a pandas data frame, we're doing manipulations on that, et cetera, et cetera. You then have to say, well, the training set and validation set that are produced by that pre-processing step, how do I actually train the model using that data and how do I extract out the parameters that I'm going to want to use when serving it, right? So this step runs, you get out these sort of artifacts, which are arbitrary binaries, that you can then are asked to describe how to load into a function or a callable that can be invoked at serving time or on a per-request basis. So three steps, obviously I'm glossing over a lot of the code here because it varies on a model by model basis, but the general gist is that all these three steps are things that as machine learning practitioners or data scientists, you should feel fairly comfortable implementing. You do have to specify a few more things. So there's a config file that's generated for you. Most of the config file is pre-populated, but we also use JSON schema just to define, I guess, the request and response schemas for the service, all of our machine learning models are sort of the services that can be invoked internally at the company. And this is actually beyond giving you some really nice properties, like being able to ensure that your model conforms to these schemas and that requests always conform to these schemas, that it has some really nice properties, like when you see a PR that is implementing these schemas, we can have a discussion as a team about, well, is this actually how this model should be shaped? Do you not need this feature to be specified? Is there not this setting that we want to expose? Things like that. So as a cultural practice, this is very interesting, but also it gives you a lot of nice benefits as both the developer and as a putative integrator with this service. So I guess what sort of things are provided for you out of all that generated code that I showed you? Well, for one, you get a test suite that's integrated in CI that is testing things that are common across all of our models, like does the pre-processing step output work with the training step? Does the training step output work with loading the model? Does the loaded model conform to the schemas that I just mentioned? You get this sort of standard linting suite. You get dockerization, you get CI CD through Circle. We use Airflow to orchestrate that machine learning lifecycle. You don't want to have to write the DAG yourself, so the DAG is generated for you. By virtue of that, you get this training orchestration, you get automated model evaluation and promotion, you get gradual roll-out, you get automated roll-out, monitoring, alerting diagnostics, auto-scaling, schema validation, data capture, health-chase cost monitoring, and all this other stuff that as a data scientist, you might not be good at, you probably don't want to do, and is ideally best handled in a uniform fashion that you don't have to think about. You just really want to distill your job, your role at the company, down to the things that you're good at, ideally. So basically you get all this stuff for free. Obviously all this didn't exist at the beginning of the project's inception, but over time we've added more and more features to it, incrementally, little by little, essentially basically anytime you have something you run into and you go, I don't want to have to do this for every model going forward, there you go, just add to our framework. So what is the stack that powers us actually look like? Well, this is the general architecture. Again, I'm not going to go into too much detail on this, but essentially the framework that we've built, like I said, is all orchestrated by Apache Airflow. We use Snowflake as our sort of data lake, and most of it is just sort of glue between various services. We happen to use primarily AWS services, but in principle there's no reason why that platform especially should be favored. But essentially this is a diagram showing that exact same sort of life cycle that I talked about before, right? You have some data that's fetched from Snowflake, you pre-process it, the results of that are admitted to S3, you take those files from S3 and you do training on them that emits some sort of model parameters or model artifact, you then can evaluate the efficacy of that model artifact against whichever validation metric you've chosen and compare that against the current models in production. The evaluation there happens in ECS. If the promotion criterion says that based off of the comparison between production and your new model, you should promote it, then it gets promoted to an endpoint. Very simple stuff. And again, all of this is sort of just taken care of for you. In terms of what the rollout looks like, this is again the same sort of very standard experience that you'd have if you were doing software development that as I guess data science practitioners, we haven't necessarily matured into yet, right? But the general idea being that you have your two models, your monitoring error rates, specifically four and 500s in H&P, and you just want to gradually start shifting traffic over from one mall to another. So this is the FSM, but at every step here, you have one of three options. You can either say, well, based off the comparison, it looks like the new model's statistically indistinguishable from the old model in terms of error rate or better, in which case you'd like to start shifting more traffic to it. Or maybe it looks worse in case you'd like to roll back just to the old production model variant. Or maybe it's the case, and all of our services have a uniform traffic that's amortized over the entire day. So they have sort of spiky access patterns that can change depending on what services integrate with them over time. So it might be the case that you actually don't have enough observations to determine whether it's safe to continue to promote. And so you can just sort of defer and wait for more data to come in, basically. And I should say that all of these steps when they are run, they're emitting things to Slack channel that we've set up so that we can monitor this. And in the case of a rollback, someone is alerted so that we can investigate and figure out what the problem was. Like I said, this is all orchestrated through Airflow. If you're familiar with Airflow, writing the DAGs can be sort of a drag, right? So if you look at this diagram, there's about, I don't know, 50, 60 nodes here. This ends up being 1500 lines of Airflow DAG declaration code, which you don't have to write, right? That is provided for you by virtue of our DAG generation. This is the stack we're using or some of the technologies we're using. This really isn't anything apart from bog standard. Again, the key thing that we've done is just sort of glue these various components together. And again, anytime that we've run to something where we go, that's gonna be painful later on down the line. Well, we embrace it and we try and add it as a feature to our existing platform. So now I wanna go over some of the key, I guess, principles and things we've learned over the past two years, right? So one is, probably three months into developing this, this is what the state of the platform look like, right? So in Slack, you'd see stuff like, oh, this model's been created, we're running the validation, here are the results. And now the onus was on the developer to look at those results and say, okay, yes, this looks good, let's promote it to the endpoint, right? And so the issue there is we started talking about like, okay, operationally, logistically, or sort of culturally as a team, who's gonna be on the line for pressing that button? Who's gonna be on the line for making sure that when it's promoted that it's working well? Who's gonna be looking at these metrics and making sure that there aren't alerts going off? And we talked about that as a team, transpires maybe unsurprisingly, that no one was really gung-ho about taking on that responsibility. And so again, especially since we were planning on scaling this to many, many models, we thought this is unsustainable, this is gonna be painful for us, let's just have the framework take care of this for us. So it was a little bit of effort to implement this sort of automated promotion and rollout, but at the end of the day, it's paid huge dividends. Also, I don't know if you've ever used AWS before, but these are some of the instances you can choose from and they say how much memory and how much CPU they have. This is the full list and actually I've cut off a little tail end of it. This all lives on one webpage and a completely different webpage tells you what the prices are. So if you're trying to figure out which instance size is going to match the needs of my model and also be cheap, very painful process. Don't do that. Instead, we have implemented basically a little tool that you just define what your memory and CPU requirements are and it finds the cheapest instance size for you. So what I guess I'm in to say here is that this notion of identifying painful parts of the development cycle isn't just for these really big architectural concepts, but also needs to pervade into these sort of smaller rocks in the shoe, if you will. And then about Airflow specifically, I just wanted to highlight, we started out hosting our own stack using CloudFormation and we ran into these same sort of operational issues where, hey, we didn't know how many workers provision or all this sort of stuff. We ended up just having someone hosted for us. So don't be precious about the things you've built. If something is really not within your wheelhouse, give it away. There are also issues with deployment interruptions and sort of these issues with different contributed operators not always being exactly what you want. So I guess just do your due diligence if you're gonna use Airflow and make sure that things will integrate the way you're expecting. But the bottom line here essentially is if you want to achieve the same sort of thing that we have where you minimize the painfulness, you have to start solving the things that are painful and as a team, dedicate yourselves to making the developer experience in ergonomics as simple and enjoyable as possible. And that's all I wanted to leave you with. So thank you very much. Thank you very much, Chase. Very interesting talk. The questions are coming in. The burning question everyone is asking probably is your Python logo in the background. Ah! So this is actually from a local artist in Massachusetts, but if you go to bluefrogcollective.com, those are the people that I had commissioned this for me. So they do the great job. So one comment is awesome talk. It's not for you to question. Let me answer that. According to your architecture, most ML is happening on Amazon Cloud where did you deploy Airflow? Yeah, so like I alluded to, we initially were also deploying Airflow on AWS through CloudFormation. The hosted service we're using is astronomer.io. They've been really great so far and it's really reassuring to have a support team that you can sort of call up and ask questions for when you run into sort of these painful operational issues. And then the next question, it's a little bit cryptic but I think the question is cookie cutter better than ML hops? Well, I guess it's not better but it helps you avoid having to have a dedicated ML Ops team, right? The idea behind all of this is basically anything that you run into where it's painful or you think, oh my goodness, I don't want to have to deal with this when I'm developing my models. This is not why I'm interested in. Bite the bullet, do it, have some sort of set of libraries or tools that you can just contribute that to. And then you've solved that problem in perpetuity for your entire team. So it's not about this is better or worth an ML Ops, it's about just making the productionization process as painless as possible. Okay, thank you very much again. Thank you.